Frozen environment packs (environment pack version 1.0)

An environment pack freezes the complete evaluation environment once, before any model runs. Every model is then evaluated on the same bytes, the same seed schedule and the same fire worlds. The score that uses a pack is in scoring.md.

1. What a pack freezes

pack.json lists the SHA-256 of every other file, so the pack digest pins every byte of the pack.

2. Layout

<pack>/
  pack.json                    the record; its canonical digest is the pack digest
  evaluation-manifest.json     the ordered seed schedule
  suite/                       a plain suite: manifest.json + scenario files
  generated-suite/             or a generated suite:
    generation.json
    suite/

pack.json has exactly these keys: environment_pack_version, pack_id, source (suite or generated_suite), versions, suite (ID, version, suite hash, and per scenario its ID, content hash and primary family), evaluation_manifest_hash, episodes (per episode: index, scenario ID, scenario hash, seed, the fire-world digest) and files (sorted [path, sha256] pairs).

3. Verification fails closed

The pack loader raises an environment pack error when any of these is true:

• pack.json is missing, is not JSON, or has a missing or extra key;
• the pack version or any entry of versions differs from the running code;
• a listed file is missing, a file is not listed, a path is unsafe, or a symbolic link is present;
• a file's SHA-256 differs from its listed value;
• the suite does not load (the suite loader) or the generated suite does not verify (the generated-suite loader);
• the evaluation manifest does not target the suite (the manifest check) or its hash differs;
• the record differs from the record rebuilt from the files, including any fire world recomputed from the scenario and the seed;
• expected_digest is given and differs.

The scoring step with the pack also refuses results whose manifest is not the pack's manifest, and checks every episode's fire trajectory against the pack.

4. Identical environments for every model

• Never regenerated per model. A pack is written once. Models only read it. A model's score names the pack digest; two scores are comparable only with the same digest.
• No model-dependent state. In V1 the fire ignores people and actions. The burning set of every snapshot is a function of the scenario and the seed (the oracle fire trajectory). The pack stores its digest, and scoring checks every recorded snapshot against it. So every model meets the same fire at the same step: common random numbers by construction. Nothing in the pack depends on a model or on any mid-episode state that a model can change.
• People are the model's. Positions, blocks and outcomes depend on the model's actions. They are in the run record and are authenticated by replay; they are not in the pack.

5. API and external models

An external model (an LLM API or any remote service) is a policy behind the adapter of model-interface.md:

1. The evaluator holds the pack. The model never receives the pack, the seed, the fire trajectory or any privileged reference.
2. At each step the model receives one observation and returns one action output. Invalid output raises a model output error; the episode is declared invalid_output and scores 0.
3. The evaluator runs the simulator locally and records the run. Scoring is local, from replayed records only.
4. Provider-reported success, accuracy, safety or cost is never trusted and never used. Cost for the EMBER-Efficiency view is measured by the evaluator.
5. A timeout is the adapter's own deadline. A timed-out episode is declared timeout and scores 0. Provider retries are an adapter policy, recorded in the run's configuration, not hidden.
6. A model error (the provider or the adapter raised) is declared model_error and scores 0. A failure can never disappear: an episode without a result and without a declaration is missing and scores 0.
7. Operational data is recorded separately in an operational record (scoring.md, section 11): model ID, provider ID, the inference configuration digest, latency, input tokens, output tokens, cost and its unit, retries and timeouts. It is never a score input.

Stochasticity protocol.

• Pin the inference configuration (model name and version or snapshot, temperature, top-p, max tokens, system prompt, adapter version) and record it with the submission as the inference configuration digest. All trials of one submission use the same digest.
• temperature = 0 is not a determinism guarantee. Batching, numerics and provider-side changes can change outputs.
• Run K >= 3 trials of the full pack for any sampled model (K = 1 only for a policy shown to be deterministic). Score them together with the scoring step over all trials and the pack. Each episode is the mean of its trials. Report K.
• Never select the best trial. Every trial that started is scored; a trial that failed contributes zeros.

6. Privacy and publication

A scored pack stays private until its evaluation ends. Publish the pack digest first, as a commitment. Publish the pack after the evaluation; anyone can then run the pack check with the published digest and re-score every submitted result.

7. Versioning

1.0. It changes when the layout, the record or the fire-world digest changes. The scoring correction pass (person-level attribution, severe quality gates, the exit-capacity time bound) changed none of these, so the version stays "1.0": the privileged references are recomputed by the scorer from the pack, and are not stored in it. A pack is valid only under the exact versions in its versions entry; a simulator or schema change invalidates every pack, by design.