Paired identical evaluation

This document specifies how EMBERBench runs several models on exactly the same episodes and how their results are matched.

Out of scope here: model-provider adapters, win rates, confidence intervals and other aggregate statistics (see statistics.md), procedural scenario generation, integrity and provenance (see integrity.md), and the EMBER Score, leaderboard order and frozen environment packs (see scoring.md and environment-pack.md).

1. Common random numbers

An episode is fully defined by a scenario and a seed. All randomness comes from simulator-owned streams derived from the seed (see simulation-contract.md). In V1 the fire does not depend on the actions, so two models that play the same scenario with the same seed face the same fire, cell for cell and step for step.

Paired evaluation uses this. Every model plays the same ordered list of (scenario, seed) episodes. A difference between two models on one episode then comes from their decisions only, not from different fire. This is the common-random-numbers method. It removes the between-seed noise from each paired difference, so fewer episodes are necessary to compare two models.

The test test_two_policies_see_identical_stochastic_fire checks this on the stochastic scenario v1-pub-10-route-invalidation. A greedy policy and an always-stay policy see identical burning sets at every step that both runs reach, and two different seeds give different fire.

2. The evaluation manifest

An evaluation manifest is an immutable, versioned episode schedule.

{
  "manifest_version": "1.0",
  "suite_id": "emberbench-v1-public",
  "suite_version": "1.0.0",
  "suite_hash": "<sha256 of the canonical suite manifest>",
  "simulator_version": "0.2.0",
  "observation_version": "1.0",
  "metrics_version": "1.0",
  "episodes": [
    {"index": 0, "scenario_id": "v1-pub-01-single-exit-sparse", "scenario_hash": "<sha256>", "seed": 3},
    {"index": 1, "scenario_id": "v1-pub-01-single-exit-sparse", "scenario_hash": "<sha256>", "seed": 1}
  ]
}

Rules (all enforced on construction and on parsing):

• manifest_version must be a supported version (1.0).
• suite_id, suite_version and suite_hash must be well formed.
• episodes is non-empty. index equals the position in the list (0, 1, 2, ...).
• Each seed is an integer in [0, 2**64 - 1]. Booleans and floats are rejected.
• A seed occurs at most once per scenario. The same seed may be used for different scenarios.
• One scenario ID is pinned to exactly one content hash.
• Unknown and missing keys are rejected.

The manifest hash is the SHA-256 of the canonical JSON of the whole manifest. A change to any field, any seed, any scenario hash or to the episode order gives a different hash.

Building a manifest

The manifest builder schedules every suite scenario under every seed: the outer order is the suite manifest order, the inner order is the order of seeds as given. The function does not sort, remove or generate seeds. An empty list, a duplicate or an out-of-range seed is an error. A set, a str or a generator is a type error, because it has no reliable order.

The manifest check requires that the manifest targets exactly this suite (ID, version, hash) and the running code's simulator, observation and metrics versions, and that every pinned scenario hash matches both the suite entry and the scenario file. The evaluation runner and the result loader always call it.

The manifest writer and the manifest loader store a manifest as JSON. The loaded manifest is checked with the same rules.

3. Public development manifests and scored manifests

The seeds decide the fire. If a model developer knows the scored seeds, they can tune a model to that exact fire. Therefore:

• Development manifests use small, fixed, public seeds (for example [0, 1, 2]). They are fine in tests, examples and documentation. The test suite pins one: the public suite under seeds (0, 1, 2) has manifest hash 7b9aae45e5e2828f881b2f14d8e5585627e5f8bdbb137a1ed2331bf10ce8c876.
• Scored manifests use seeds chosen by the evaluator. They are not published here.

This repository never generates scored seeds. The seed in a scenario file is a development default only. The evaluation runner always overrides it with the manifest seed.

A published manifest hash lets anyone check later that a published schedule is the one that was used, without revealing the seeds before the evaluation.

4. Seed privacy and episode isolation

The evaluator knows the manifest and the seeds. The decision policy does not.

The evaluation runner does this for each episode:

1. New policy from the factory. The factory receives no arguments: no scenario, seed, episode index or other metadata.
2. The episode runner with the episode seed. The policy is called with exactly one positional argument, an observation, once per step.

The policy never receives the evaluation manifest, an episode entry or index, the seed, the scenario, the simulator, the run record or any evaluator metadata. The observation contains no seed and no RNG state (see v1-fairness-spec.md).

A new policy is built for every episode, so state that a policy keeps cannot carry over into the next episode. This also makes resume correct: an episode gives the same result whether it runs in a fresh process or after other episodes. The factory must return a new policy each time. A plain stateless function is also acceptable.

The tests prove this:

• test_policy_receives_only_observations uses a policy that records every argument it receives and a factory that records its own arguments. The factory is called with () once per episode. Each policy call has exactly one argument of type observation and no keyword arguments. Every object inside the observation is one of the known observation types. The 64-bit seed (in decimal and hex), the manifest hash, the suite hash and the scenario hashes do not occur in its JSON form.
• test_fresh_policy_per_episode checks that there is one distinct policy instance per episode and that each instance sees one episode, from step 0.

Limit: this is an interface contract for in-process Python policies, not a sandbox. Python code can inspect its caller's stack. A real model adapter must send only the serialized observation to the model.

For per-episode agents, action validation, optional execution accounting and the canonical serialized observation, see model-interface.md.

5. The per-model result artifact

The evaluation runner writes one JSON artifact per model run. It is canonical JSON (sorted keys, no extra whitespace) followed by a newline.

{
  "result_version": "1.0",
  "model_id": "model-a",
  "evaluation_manifest_hash": "<sha256>",
  "suite_id": "emberbench-v1-public",
  "suite_version": "1.0.0",
  "suite_hash": "<sha256>",
  "simulator_version": "0.2.0",
  "observation_version": "1.0",
  "metrics_version": "1.0",
  "manifest": { "...": "the complete evaluation manifest" },
  "episodes": [
    {
      "index": 0,
      "scenario_id": "v1-pub-01-single-exit-sparse",
      "scenario_hash": "<sha256>",
      "seed": 3,
      "record_digest": "<sha256 of the canonical run record>",
      "record": { "...": "the complete run record" },
      "metrics": { "...": "episode metrics" }
    }
  ]
}

• The header fields repeat the embedded manifest so a reader can see them at once. They must equal the manifest.
• episodes holds completed episodes only, in manifest order. Each one names its exact manifest entry and contains the full run record, so anyone can replay and audit it, plus the raw metrics computed from it.
• model_id is a label chosen by the evaluator (1 to 256 printable characters, no leading or trailing whitespace). It is not proof of which model produced the actions, and it is not a unique key.

The result loader accepts an artifact only if all of these hold:

• the JSON has exactly the expected keys at every level;
• the embedded manifest is valid, its hash equals evaluation_manifest_hash, and every header field equals the manifest;
• the manifest passes the manifest check against suite and the running code;
• every episode index is in the manifest, occurs once, and episodes are in manifest order;
• every episode's scenario_id, scenario_hash and seed equal its manifest entry, and the record's header agrees with them;
• every record is in canonical form and matches record_digest;
• every record is authenticated: Episode metrics checks its structure and replays its action log in a fresh simulator, which must reproduce it byte for byte;
• every stored metrics object equals the metrics recomputed from the authenticated record;
• the artifact as a whole re-serializes to exactly the loaded JSON.

Any failure raises an evaluation error. Nothing is repaired, skipped or ignored.

6. Resume guarantees

The evaluation runner saves the artifact after every completed episode. Each save writes a temporary file in the same directory, flushes and fsyncs it, and renames it over the artifact with atomic file replace. A crash therefore leaves either the previous artifact or the new one, never a partial file. After a hard crash (for example a power loss) a stray .<name>.*.tmp file can remain next to the artifact. It is never read and can be deleted. If the output path does not exist, an artifact with zero episodes is written before the first episode runs.

If output_path exists:

• With resume=False it is an error. An existing artifact is never overwritten.
• With resume=True (the default) the artifact is loaded and authenticated completely, as in section 5. Then it must have the same evaluation manifest hash and the same model ID as the call. A changed schedule, seed order, scenario, suite or version, or a different model ID, is rejected.
• Only the episodes that are not in the artifact run, in manifest order. A completed episode is never run again.

Because each episode uses a fresh policy and its own seed, an interrupted and resumed run gives the same artifact, byte for byte, as an uninterrupted run with a deterministic policy. test_interrupted_run_resumes_without_rerunning checks this.

A policy exception or an invalid action stops the run with that exception. The artifact keeps the episodes completed before it.

Cost note: the whole artifact is rewritten after each episode, so total bytes written grow with the square of the number of episodes. This is acceptable for V1 schedules. A sharded layout can replace it later without changing the pairing rules.

7. Exact pairing

The result pairing compares two loaded artifacts. It first requires that both use the same evaluation manifest (equal hash and equal content). Otherwise it raises an evaluation error: results from different schedules are never compared.

Each scheduled episode has the key

(evaluation manifest hash, episode index, scenario ID, scenario hash, seed)

Definition. An episode is a comparable pair exactly when both artifacts contain a result whose key equals that scheduled key. Nothing else (model ID, file name, run order or time) is part of the key.

The function returns a pairing report:

An episode that neither artifact has is in both missing lists. The report holds no scores, win rates or statistics. statistics.md defines the paired statistics; they use only paired episodes.