Command-line interface

1.0

The emberbench command exposes the library contracts as a deterministic, provider-free command line. Every command calls the existing library API (the scenario loader, the suite loader, the generated-suite loader, the manifest builder, the episode runner, the replay check, the evaluation runner, the result loader, the aggregate statistics, the paired comparison). The CLI adds argument parsing, fail-closed input loading, no-overwrite atomic output and stable exit codes. It does not add a second simulator, a second validator or a second artifact format.

The CLI uses only the standard library (argparse).

Entry points

emberbench version --json

python -m emberbench version --json

Both run the same function, the emberbench command. The console script is installed by the package ([project.scripts] in pyproject.toml).

python -m normally puts the current directory first on Python import path. The module entry point removes it, so the current directory never decides which agent code is imported (see Agent factories).

Command tree

emberbench
  version                     contract versions
  scenario validate FILE      one scenario file
  suite validate DIR          a suite directory (manifest.json + scenario files)
  generated validate DIR      a frozen generated suite (generation.json + suite/)
  evaluation manifest         build an evaluation manifest from a suite and ordered seeds
  evaluation run              run every manifest episode with a local agent (resumable)
  evaluation verify           authenticate an evaluation result
  episode run                 run one scenario with one seed and a local agent
  replay verify               authenticate one run record by replay
  statistics aggregate        aggregate statistics of one result
  statistics compare          paired statistics of two results (A - B)

Every command accepts:

• --json: print only the result, as one canonical JSON line on stdout.
• --debug: print a Python traceback to stderr on failure.
• --help: print the usage of the command.

Options go after the command words (emberbench suite validate DIR --json). Option names are never abbreviated.

Commands that take a suite accept exactly one of:

• --suite DIR: a suite directory, loaded with the suite loader.
• --generated-suite DIR: a frozen generated suite, loaded with the generated-suite loader. The freeze record is checked as well.

Commands

version

emberbench version --json

• benchmark_revision and every benchmark-defining version, exactly the entries of the benchmark version list that evaluation provenance pins: simulator_version, observation_version, action_schema_version, schema_version (the scenario schema), metrics_version, crowd_diagnostics_version, statistics_version, generator_version, generator_config_version, generation_manifest_version, evaluation_manifest_version, evaluation_result_version, scoring_version, environment_pack_version, environment_world_version, family_rules_version, calibration_panel_version, difficulty_rules_version, admission_rules_version, composition_version, selection_rules_version, recipe_catalog_version and suite_freeze_record_version;
• the tooling contracts that do not change what a score means: cli_version, baselines_version, integrity_protocol_version, replay_audit_version and animation_format_version.

The benchmark-defining part is built from the benchmark version list, so a version that provenance pins is always reported (see versioning.md, section 3, and docs/integrity.md, section 12). No field depends on the environment (no paths, platform, interpreter or install location). These keys were added before any release, so the CLI version stays 1.0.

scenario validate FILE

emberbench scenario validate scenarios/public/v1/v1-pub-01-single-exit-sparse.json --json

Loads the file with the strict scenario parser. Reports scenario_id, scenario_hash (the content hash) and schema_version.

suite validate DIR

emberbench suite validate scenarios/public/v1 --json

Loads the suite with the suite loader: every listed file must exist, no file may be unlisted, and every pinned hash and derived field must match. Reports suite_id, suite_version, suite_hash, visibility, seed_role, schema_version, scenario_count and scenarios (each scenario_id and the scenario content hash, in manifest order).

generated validate DIR

emberbench generated validate path/to/frozen-batch --json

Loads a frozen generated suite with the generated-suite loader and checks the freeze record against every scenario. Reports the suite fields above and generation_manifest_version, generator_version, batch_id and the configuration hash.

evaluation manifest

emberbench evaluation manifest --suite scenarios/public/v1 --seed 1 --seed 7 --seed 42 --out manifest.json

Builds the manifest with the manifest builder and writes its canonical file text (the manifest file text, the bytes that manifest writer writes).

• --seed N is repeatable and required. The order is kept exactly. The CLI never generates, sorts or removes seeds.
• A seed is a decimal integer in [0, 2**64 - 1]. A duplicate seed is a usage error.
• --out FILE is required. --force replaces an existing file.

Reports the manifest hash, manifest_version, the suite fields, episode_count and seed_count. The seed values are never printed: they stay in the file the operator asked for.

episode run

emberbench episode run --scenario scenarios/public/v1/v1-pub-01-single-exit-sparse.json --seed 3 --agent my_agents:make_agent --out record.json

Runs one episode through the single simulator path: The episode runner with the agent and a seed. The agent is built fresh with the factory, receives only observation values and its output is validated by the action check. The record is then checked with the replay check and written as canonical JSON plus one LF (the run record JSON text with a final newline).

• --scenario FILE, --seed N, --agent MODULE:FACTORY and --out FILE are required. The scenario file's development seed is not used.
• --agent-path DIR (repeatable) and --force are optional.

Reports scenario_id, scenario_hash, simulator_version, observation_version, record_digest, steps and termination_reason.

replay verify

emberbench replay verify --scenario scenarios/public/v1/v1-pub-01-single-exit-sparse.json --record record.json --json

1. The record file must be JSON that run record loader accepts and that round-trips exactly (no unknown or changed fields). Otherwise: exit 3.
2. The replay check replays the action log. Any version, scenario-hash, initial-state or trajectory difference: exit 5.
3. The run record check requires a complete, well-formed episode. A replayable prefix that stops before termination: exit 3.

Reports verified: true, scenario_id, scenario_hash, record_digest, simulator_version, observation_version, steps and termination_reason. There is no best-effort recovery.

evaluation run

emberbench evaluation run --suite scenarios/public/v1 --manifest manifest.json --model-id my-model --agent my_agents:make_agent --out result.json

Calls the evaluation runner unchanged. Every episode gets a new agent.

• --manifest FILE, --model-id ID, --agent MODULE:FACTORY and --out FILE are required. --model-id must pass the library model ID check (non-empty, printable, at most 256 characters, no outer whitespace).
• An existing --out file is never overwritten. Without --resume it is an output conflict (exit 6). With --resume it must load and authenticate completely and carry the same manifest hash and model ID (otherwise exit 3 or 6). Completed episodes are not run again. This is the library resume behaviour; the CLI does not change it.
• The evaluation runner saves the artifact atomically after every episode. If the agent fails, the last checkpoint stays on disk so that --resume can continue it. This is the only command that leaves an artifact behind after a failure, and that artifact is always a complete, valid checkpoint.

Reports model_id, result_version, the manifest hash, the suite fields, result_digest, scheduled_episodes, completed_episodes and complete.

evaluation verify

emberbench evaluation verify --suite scenarios/public/v1 --result result.json --manifest manifest.json --require-complete --json

Calls the result loader: the embedded manifest must match the suite and the running versions, and every episode's run record is authenticated by replay and its metrics are recomputed. A malformed, duplicate, unknown, non-canonical or tampered episode fails the whole artifact (exit 3, or exit 5 when the cause is a replay mismatch). No episode is dropped or repaired.

• --manifest FILE also requires this exact evaluation manifest (exit 5 if the embedded manifest differs).
• --require-complete also requires every scheduled episode (exit 5 if any is missing). Without it, a partial result that authenticates is valid.

Reports verified: true and the same fields as evaluation run.

statistics aggregate

emberbench statistics aggregate --suite scenarios/public/v1 --result result.json --out stats.json

Loads and authenticates the result, then writes the aggregate statistics JSON text with a final newline.

statistics compare

emberbench statistics compare --suite scenarios/public/v1 --a a.json --b b.json --out paired.json

Loads and authenticates both results, then writes the paired comparison JSON text with a final newline. Deltas are A - B. Results of different manifests are rejected.

Both statistics commands accept --resamples N. Without it the library default (default bootstrap resample count, 2000) is used: the CLI does not pass the argument at all. Out-of-range values are a usage error. They report statistics_version, digest (the artifact's the digest), the model IDs, the manifest hash and resamples.

Exit codes

The codes are the exit code list and are pinned by tests.

A Python traceback is never printed unless --debug is given.

Output

JSON mode (--json)

• On success, stdout holds exactly one line: the result object as canonical JSON (sorted keys, no whitespace, ASCII). Nothing else is printed to stdout.
• Every result has a command field. Results never include input or output paths, timestamps, hostnames or environment details.
• On failure, stdout is empty and stderr holds exactly one line: {"error":{"exit_code":N,"kind":"...","message":"..."}}. With --debug the traceback follows on stderr.
• There are no ANSI escape codes or colours.

Usage errors found while parsing the command line, before the command is known, are also reported in this JSON form when --json is present.

Human mode

A short key: value list on stdout. Errors go to stderr as emberbench: error: MESSAGE.

Error messages are deterministic for the same inputs. Library messages can contain the input paths as given on the command line.

Files

Inputs are never written

Validation, verification and statistics commands only read their inputs. An output path that is the same file as an input is refused, even with --force.

Output rules

• An existing output file is never replaced unless --force is given. evaluation run has no --force: an existing result can only be resumed.
• Parent directories are never created. A missing parent is exit 6.
• The output is written to a temporary file in the same directory, flushed and fsynced. With --force the temporary file is then renamed over the output (atomic file replace). Without --force it is hard-linked to the output path. The link fails if the file exists, so a file that another process creates in the meantime is never replaced either.
• If the filesystem has no hard links, the CLI creates the output with O_CREAT | O_EXCL (this also fails if the file exists), writes the bytes, flushes and fsyncs. On such a filesystem the output is visible while it is written. If that write fails, the CLI removes the file, but only if it is still the file that the CLI created. There is never a check-then-replace step.
• On any failure, no output file and no temporary file are left behind (evaluation run checkpoints are the documented exception above).
• Inputs are validated and the output path is checked before an agent is imported or an episode runs.

Artifact bytes

All artifacts are ASCII with LF line endings on every platform. They hold no absolute paths and no timestamps. A record or statistics digest is the SHA-256 of the file without its final LF.

Agent factories

episode run and evaluation run import the operator's agent code. This is the only place where the CLI imports code that is not part of EMBERBench, and it happens only when --agent is given.

--agent MODULE:FACTORY

• MODULE is a dotted Python module path (my_pkg.agents).
• FACTORY is an attribute path in that module (make_agent or the agent factory).
• Both parts use ASCII identifiers only. Anything else is a usage error (exit 2).

The CLI imports exactly MODULE with Python module import and reads FACTORY from it. There is no entry-point discovery, plugin registry, fallback module, network location or search of the current directory. The module must be importable from Python import path as the interpreter sets it, or from a directory given explicitly with --agent-path DIR (repeatable; the directories are put first, in the order given, for this command only).

The factory must satisfy the agent factory contract (docs/model-interface.md):

• It is a zero-argument callable.
• Every call returns a new object with an act(observation) method. A returned agent that was already used for an earlier episode is rejected.
• act receives only the observation. It never receives the scenario, seed, manifest, episode index or run record.

Example agent module:

# my_agents.py
from emberbench import Action, PolicyAgent

def make_agent():
    return PolicyAgent(lambda obs: {p.id: Action.STAY for p in obs.active})

emberbench episode run --scenario scenarios/public/v1/v1-pub-01-single-exit-sparse.json --seed 3 --agent my_agents:make_agent --agent-path . --out record.json

Importing a module runs its code. Use --agent only with code that you trust.

Configuration, determinism and network policy

• There is no configuration file, no home-directory configuration, no .env file and no environment variable that changes a result. Every value that affects a benchmark result comes from an explicit argument, an explicit input file or a versioned library default.
• The current directory does not change results. Relative paths are resolved against it, as usual, and nothing else is taken from it.
• The same command with the same input files gives byte-identical output files and stdout, under any PYTHONHASHSEED, from any current directory and on Linux, Windows and macOS. The order of options does not matter. The order of --seed options does: it is the manifest order.
• The CLI core never opens a network connection: it does not download models, contact providers, check for updates or fetch URLs. A test runs every command under an audit hook that fails on any socket or URL event. An agent that the operator loads with --agent may use the network itself. That is outside the CLI core and is the operator's explicit choice.

Examples (public development assets)

emberbench suite validate scenarios/public/v1

emberbench evaluation manifest --suite scenarios/public/v1 --seed 0 --seed 1 --seed 2 --out manifest.json --json

emberbench evaluation run --suite scenarios/public/v1 --manifest manifest.json --model-id stay --agent my_agents:make_agent --agent-path . --out stay.json

emberbench evaluation verify --suite scenarios/public/v1 --result stay.json --manifest manifest.json --require-complete

emberbench statistics aggregate --suite scenarios/public/v1 --result stay.json --out stay-stats.json

The public seeds 0, 1, 2 give the evaluation manifest hash 7b9aae45e5e2828f881b2f14d8e5585627e5f8bdbb137a1ed2331bf10ce8c876. Development seeds are for development only; scored seed schedules belong to the evaluator (docs/evaluation.md).