EMBERBench V1 fairness and semantics specification

This document is normative. It freezes the V1 fairness contract, the model-visible observation and the V1 simulator rules. The words MUST, MUST NOT and SHOULD have their RFC 2119 meaning.

If the code and this document disagree, it is a bug.

V1 in one sentence. A model sees the full current state and knows the episode dynamics (the rules). The evaluator's randomness (the seed, the RNG state and every future draw) stays hidden. V1 evaluates decisions under known rules, not inference of hidden rules.

1. Fairness contract

1.1 Matched evaluation

Every compared model MUST run on exactly the same:

• scenario (the same scenario content hash),
• simulator version,
• initial population, with the same person IDs and positions,
• hazard configuration (ignitions, fire model and its parameters),
• evaluator seed schedule (the same list of seeds for each scenario),
• exogenous stochastic process (the fire RNG stream derived from each seed).

The only input that differs between two matched runs is the sequence of actions that the models submit. A run record header (simulator_version, observation_version, scenario_hash, seed) identifies the matched episode and the observation contract the model received.

1.2 Evaluator-private and model-visible state

The simulator state has two separate parts.

Evaluator-private state. Only the evaluator and the harness may read it. A decision model MUST NOT receive any of it, in any form, before or during a run:

Model-visible state. A decision model receives only the observation returned by the simulator observation step. Section 4 defines its fields. It has the full current state (map, people, fire) and the episode rules (rules view): every action-relevant rule that a scenario can vary. The rules are copied from the scenario, so the model never holds a scenario reference.

The harness boundary is the simulator observation step and the simulator step. A harness MUST give a model only observation objects (or their dictionary form). It MUST NOT give a model the simulator, the scenario, a simulation state, a step record or a run record until the evaluation is complete. The episode runner obeys this rule: the policy receives only observation objects.

1.3 The seed

The seed exists for reproducibility and for matched evaluation only. It is not a model input.

• The evaluator owns the seed schedule. The schedule MUST NOT be disclosed to a model before the evaluation is complete.
• Scored evaluations SHOULD draw seeds uniformly from [0, 2**64) with a cryptographic generator. A model that knows the simulator code could test small seeds (0, 1, 2, ...) against the fire it observes and then predict future fire. A 64-bit seed space, with stream seeds derived by SHA-256, makes that search impractical.
• A model sees the realised fire (the burning cells). That is the intended information. It does not see the draws that produced the fire, or any future draw.

1.4 Person and exit IDs

Person IDs and exit IDs are model-visible, because actions are keyed by person ID. Scenario authors MUST NOT encode hidden test-set metadata in them. IDs SHOULD carry no information about the solution (for example, IDs SHOULD NOT be ordered by distance to an exit), because the collision rule (R1) uses the ID order.

2. Stochastic-world fairness

Fire is exogenous in V1. People and actions do not affect fire propagation:

• The fire model receives only the static terrain, the burning cells, the step number and the fire RNG stream (the fire spread step). It does not receive people or actions.
• People do not block fire, and fire spread does not depend on where people are.
• The stochastic model takes one draw per (candidate cell, burning neighbour) edge on every spread step. The number of draws depends only on the geometry and the burning set.

Therefore, for the same scenario and the same evaluator seed, two runs with different actions MUST produce identical ignitions, burning sets and fire RNG positions on every step that both runs complete. The fire trajectory is equal to the trajectory the fire process produces with no people at all. This is the "common random numbers" property: a difference in outcome between two matched runs comes from the decisions only, not from luck.

A run can end earlier than another run (for example when everyone evacuates). The invariant holds for the overlapping steps.

The exact probability and draw-count rules are in simulation-contract.md.

It uses several development seeds on multiple ignitions, flammable and non-flammable exits, spread_interval 2, p = 0, p = 1 and the public route-invalidation scenario. It compares ignitions, burning sets and the fire RNG state on every step.

A future subsystem that needs randomness MUST use a new stream label (see simulation-contract.md). If its draws depend on actions, it MUST NOT share a stream with the fire.

3. Frozen V1 rules

Each rule has a short rationale and the tests that pin it. V1 keeps the Phase 1 behaviour for every rule. No rule changed in Phase 2: the simulator dynamics are the same as in 0.1.0. The simulator version is 0.2.0 only because the run record header gained observation_version (see section 6).

Every rule that a scenario can vary is in the observation rules. The model knows the rules it plays under.

R1. Collision tie-break: lowest person ID wins

If two or more movers target the same cell, the mover with the lowest person ID wins. IDs compare as plain strings by code point ("P10" < "P2"). Every other mover gets blocked_conflict. The decision uses intent only. A loser is not re-awarded the cell if the winner fails later.

Rationale. The rule is deterministic, simple and visible to the model (IDs are in the observation, and rules.conflict_resolution names the rule). It is identical for every model on a scenario. A random tie-break would need RNG draws whose count depends on the actions. That would either couple the fire stream to the actions (section 2) or add a second stream with no benefit for fairness. The known bias (low IDs win) is the same for every compared model.

R2. Swaps are blocked

Two people who try to exchange cells (A <-> B) both get blocked_cycle and stay in place.

Rationale. A cell holds one person. A swap needs two people to pass through each other on a shared edge. This is not possible in a one-cell-wide corridor, and the grid cannot tell a wide passage from a narrow one. Blocking is the conservative, simple rule.

R3. Rotations are blocked

Any closed chain of movers (a rotation of length 3 or more) is blocked. Every member gets blocked_cycle. An outside mover with a lower ID than the cycle member it meets wins the conflict (R1). That cycle member then becomes a non-mover, and the chain resolves as blocked_occupied.

Rationale. Same rule as R2, with no special case. Open chains (people who follow each other in one step) are allowed, so a queue still moves at full speed.

R4. Exit throughput: one person per exit cell per step

An exit is a normal cell for movement, with capacity one. A person who moves onto a usable exit is evacuated in the same step (phase 6). The exit is therefore empty at the start of the next step. Each exit cell lets at most one person through per step. A queue flows with no gaps: the next person can move into the vacated cell in the same step. Two adjacent exit cells let two people through per step.

Rationale. Throughput follows from the one-person-per-cell rule. It needs no extra parameter. Congestion at exits is part of what V1 measures.

R5. Burning exits are unusable

An exit evacuates a person only if the exit was not burning at the start of the step. A burning exit never evacuates anyone. Moving onto a burning exit follows R6: it is refused (blocked) or the person becomes a casualty (casualty). An exit can burn only if fire.exits_flammable is true. A person who reaches an exit in the same step that the fire reaches it is evacuated, because evacuation (phase 6) comes before fire spread (phase 7).

Rationale. A burning exit is the main "exit becomes unusable" event that V1 measures. Evacuation before spread gives the model the benefit of the doubt for a move that was valid when it was chosen: the model saw a usable exit.

R6. enter_fire: explicit in every scenario, default casualty

movement.enter_fire is blocked or casualty. With blocked, a move into a cell that was burning at the start of the step gives blocked_fire, and the person stays in place. With casualty, the person moves and becomes a casualty with cause entered_fire. Scenario files MUST state the value (the key is required). The Python default (default movement settings) is casualty. Scored V1 scenarios SHOULD use casualty.

Rationale. casualty makes a bad decision have its real consequence, so it measures decision quality. blocked hides some bad decisions behind a safety net. Both values stay available for tests and ablations, and the value is part of the scenario hash, so two runs can never silently use different policies.

R7. Ignition timing

• The scenario ignitions burn at step 0. They are in the first observation. No person can start on an ignition.
• Spread happens in phase 7 of step t (1-based), only when t % fire.spread_interval == 0. The first possible spread is in step spread_interval.
• Movement (phase 5) happens before spread (phase 7) in the same step. A person who moves away from a cell that ignites in this step survives. A person who stays on it becomes a casualty with cause fire_spread.
• Cells that ignite in step t are in the observation before step t + 1.

Rationale. The model always acts on the fire it can see. No cell burns "between" the observation and the move.

R8. No burn-out

A burning cell burns until the end of the run. Burn-out does not exist in V1, for any cell type (floor or exit). The burning set never shrinks.

Rationale. Simple and monotone. A burned-out cell that becomes passable again would add a timing rule and new routing behaviour that V1 does not need. A later version can add burn-out with a simulator version bump.

R9. Trapped definition

trapped is the set of active people who have no path to an exit that is not burning. A path uses 4-neighbour steps and may cross any cell that is not a wall and not burning. Other people are ignored. The set is computed at step 0 and after phase 8 of every step.

trapped is an evaluator diagnostic. It is recorded in simulation state and run record. It is not model-visible in V1. It does not change a person's status, and it does not end the run.

Rationale. Walls and fire are permanent in V1 (R8). People are not, so congestion does not count as trapped. The diagnostic is for metrics (for example "trapped from the start" against "trapped by a bad decision"). It is kept out of the observation because it is a pre-computed planning aid. A model can compute the same set from the observation (the map plus the burning cells), so hiding it removes no information.

R10. Termination precedence

After phase 8 of step t:

1. If no person is active, the run ends with all_resolved.
2. Else, if t >= termination.max_steps, the run ends with max_steps.

all_resolved wins when both hold. Inside a step, evacuation (phase 6) comes before casualties (phase 8). A run does not end early because every active person is trapped. A terminated simulator raises a simulation error on the simulator step.

Rationale. A run that resolves everyone on its last step is a full success, so all_resolved is the more informative reason. Early stop on "all trapped" would make the run length depend on a diagnostic. It would not change any outcome, because a trapped person cannot evacuate.

4. Observation contract (version 1.0)

The simulator observation step returns an observation. In V1 the current state is fully observable (the observation shows every burning cell) and the episode dynamics are known (rules). The observation holds no reference to the scenario, the simulator or any RNG. Its object graph contains only these types: Observation, map view, rules view, exit spec, person state, person status, action, position, fire model name, enter fire policy, conflict resolution, tuples, frozensets, integers, floats, booleans, strings and None.

Rules view holds every action-relevant rule that a scenario can vary. Each value is copied from the scenario when the simulator is built:

The spread probability is a parameter of the hazard distribution. It is not evaluator randomness: it tells the model how likely spread is, but not which draws will succeed.

The observation dictionary form gives the serialised form, with the extra key observation_version. The action contract is unchanged: The simulator step needs exactly one action from allowed_actions for each active person. A move that cannot happen is legal and resolves to a blocked_* outcome.

The observation does not contain: the seed, RNG state, stream seeds, future draws, scenario ID, description, content hash, schema version, or trapped.

A change to the fields or the meaning of the observation MUST bump the observation version.

The tests check:

• the exact field lists of observation, map view and rules view, and of the serialised form,
• that the object graph (walked through every instance attribute, not only the declared fields) holds only the public types above, and never a scenario, simulator, simulation state, step record, run record, RNG or configuration object,
• that the seed, stream seeds, RNG positions and hidden metadata are absent from the observation and its JSON form,
• that observations are identical for different seeds and different hidden metadata,
• that two scenarios that differ only in one public rule give different observations, and that the rules are the only difference,
• that every rules view value equals the scenario configuration.

5. Metrics input

A metric MAY need both the scenario and the run record. The run record has the actions, movements, ignitions, evacuations, casualties, trapped sets and states of each step. The static geometry (walls, exit positions, grid size) and the configuration (for example max_steps) are in the scenario. The scenario_hash in the record binds the two together. Not every metric can be computed from a run record alone.

6. Provenance and change control

Every run record header stores simulator_version, observation_version, schema_version, scenario_id, scenario_hash and seed. A new record always stores the active the observation version. The replay check rejects a record with a different simulator_version or observation_version, or a different scenario hash, with an explicit replay mismatch error. The run record loader rejects a record that has no observation_version.

• A change that changes simulator output (any run record byte) for the same scenario, seed and actions MUST bump the simulator version. The golden digest test fails until the bump is made.
• A change to the observation fields or their meaning MUST bump the observation version.
• A change to the scenario file format MUST bump the scenario schema version.
• A documentation-only clarification of visibility does not need a bump.

History.