Per-episode metrics (metrics version 1.0)

This document defines every field returned by episode metrics for a scenario and run record.

The metrics describe one episode: one scenario, one seed, one action log. Aggregation across seeds or scenarios, robustness statistics and leaderboards are out of scope here (see Issue #12).

1. Principles

1. Raw metrics are canonical. Every field is a count, a step number, a cell distance or a ratio of named counts. Each ratio states its numerator and denominator.
2. No composite score in the metrics. The metrics have no single score and the metrics version does not change. The EMBER Score is a separate, separately versioned layer on top of these metrics: see scoring.md and section 11.
3. Safety first. Speed fields that only cover evacuees always come with the counts of people who did not evacuate. The person-time-to-safety metric counts every person.
4. Proxies are named as proxies. The fire-exposure and replanning fields describe observable events. They do not measure risk or model intent.
5. Fail closed. Inputs that do not belong together, a malformed record, or a record that the simulator could not have produced raise a metrics input error. No partial result is returned.
6. Exact arithmetic. Every ratio is computed as an exact fractions.Fraction and converted to float once. A value is the correctly rounded float of an exact rational, on every platform.
7. Pure function. The scenario and the record are not mutated. The metrics version changes whenever a definition in this document changes.

2. Inputs and validation

Episode metrics validates its inputs in three stages, in this order. All stages always run; there is no option to skip one. It raises a metrics input error when any check fails.

Stage 1 and 2: compatibility and structural validation.

• Compatibility. simulator_version equals the simulator version, observation_version equals the observation version, and schema_version is supported and equals the scenario's. scenario_id equals the scenario's. scenario_hash equals the scenario content hash.
• Initial state. Step 0, every scenario person active at the scenario position, burning cells equal to the ignitions, not terminated, and a trapped set equal to a recomputation.
• Each step t. Step numbers are 1, 2, ..., T. Actions and movements cover exactly the people active in s_{t-1}. Each movement agrees with its action, origin and outcome; a moved target is in bounds and not a wall; no two people end in one cell. Ignitions are new, non-wall cells, and burning(s_t) = burning(s_{t-1}) ∪ ignited_t. Every evacuation and casualty event matches one status change (with the correct exit ID and casualty cause), resolved people never change, no active person stands on a burning cell or a usable exit, and the trapped set equals a recomputation.
• Termination. Only the last snapshot is terminated. Its reason is all_resolved when nobody is active, otherwise max_steps when T = max_steps. A record with no steps, or one that does not reach termination, is rejected: metrics need a complete episode.

These checks give precise diagnostics and confirm every assumption the metrics rely on. On their own they only prove internal consistency.

Stage 3: simulator replay authentication. Episode metrics then calls the replay check. It replays the recorded action log from the recorded seed and requires a byte-identical record. A replay mismatch error becomes a metrics input error whose message starts with record failed simulator replay verification:; the original error is kept as __cause__. This stage rejects records that pass every structural check but are impossible under the simulator, for example a blocked_wall outcome for a move into an open cell, or an ignition that the configured fire model and RNG stream would not produce. Callers do not need to call the replay check themselves.

3. Notation

• N = population = number of people in the scenario (always >= 1).
• T = episode_steps = number of recorded steps.
• H = scenario.termination.max_steps (the horizon).
• s_0 = initial snapshot; s_t = snapshot after step t, for t = 1..T.
• A(s) = people with status active in snapshot s.
• A decision is one movement record: one active person at one step. The number of decisions is D = Σ_{t=1..T} |A(s_{t-1})|.
• ratio(a, b) = a / b, or null (None) when b = 0.

4. Safety (safety)

Invariants: evacuated + casualties + stranded = N exactly (checked; a mismatch raises), and terminal_trapped_count <= stranded. When the episode ends with all_resolved, stranded = 0 and the trapped set is empty.

5. Time (time)

Person-time-to-safety. For each person i:

τ_i = evacuation step of i     if i evacuated
τ_i = H                        otherwise (casualty or stranded)

Non-evacuees are censored at the horizon H. Every evacuation step is <= H, so a casualty or a stranded person can never look faster than any evacuee. The sum covers all N people, so this metric cannot hide a casualty. The censoring value is the scenario horizon, not T: an episode that ends early because the last people died does not get a smaller value.

mean_evacuation_step and median_evacuation_step are evacuee-only statistics. They always come with evacuees and non_evacuees, and a run must never be compared on them alone. A run that loses its slowest people gets a lower evacuee mean.

6. Movement and congestion (movement)

Each decision has exactly one move outcome: stayed, moved, or one of the six blocked_* outcomes.

Invariant: successful_moves + blocked_moves = movement_attempts (checked). Under enter_fire: casualty, a move into fire is moved; the person then becomes a casualty in the same step.

7. Path efficiency (path_efficiency and per person)

For each person i:

• cells_moved_i = number of the person's decisions with outcome moved. Each such move changes the cell by exactly one 4-neighbour step.
• static_exit_distance_i = L_i = length of the shortest 4-neighbour path from the initial cell to any exit, through any non-wall cell in the grid. It ignores all fire (including the initial ignitions) and all people. It is null when no exit can be reached around the walls.

L_i uses only the scenario geometry. It is a static lower bound: it is not a dynamically feasible path and not an oracle-optimal dynamic path. Under spreading fire and congestion, the best achievable path can be longer.

For each evacuee (L_i is always defined and >= 1, and cells_moved_i >= L_i; a violation raises):

path_efficiency_i = L_i / cells_moved_i        in (0, 1]
detour_ratio_i    = cells_moved_i / L_i        in [1, ∞)

Both are null for casualties and stranded people. Episode aggregates, over the M evacuees:

Path efficiency measures distance, not time. A blocked or stay step costs time but no distance.

8. Fire-exposure proxy (fire_exposure and per person)

This is a deterministic proximity proxy, not a physical model of heat or smoke. It has no tuned constants.

The person-states are all pairs (t, i) with t = 0..T and i ∈ A(s_t): every active person in every recorded snapshot, including s_0 and the terminal snapshot. Let B_t = burning(s_t). For each person-state with B_t ≠ ∅:

d(t, i) = min_{f ∈ B_t} ( |row_i - row_f| + |col_i - col_f| )

This is the Manhattan distance on the grid. It ignores walls: a wall does not shield a person in this proxy. An active person is never on a burning cell in a snapshot, so d >= 1. The exposure is:

exposure_total = Σ_{(t, i) : B_t ≠ ∅} 1 / d(t, i)
mean_exposure  = exposure_total / active_person_states

Per person: min_fire_distance and fire_exposure (the same sum, limited to that person's states).

Recomputing alternatives. distance_histogram and person_states_without_fire are the raw components. Any proxy of the form Σ g(d) can be recomputed from them, for example a threshold count (Σ_{d <= k} count) or another decay. exposure_total equals Σ count / d over the histogram.

Limitation. People who leave the grid stop accumulating exposure. A casualty therefore can have a low exposure. Always read the proxy together with the safety counts.

9. Exit utilization (exits)

Let n_e be the number of evacuations through exit e, E = Σ n_e, and K the number of exits in the scenario.

p_e          = n_e / E
exit_balance = (1 - Σ_e p_e²) / (1 - 1/K)        in [0, 1]

exit_balance = 1 when every exit has the same share, and 0 when one exit takes every evacuee. This is descriptive only. Equal use is often not optimal: one exit can be much closer, or threatened by fire. Do not reward or penalize a run for its balance.

10. Replanning and recovery proxies (replanning)

These fields count observable reactions to a blocked move. They are proxies. They do not observe model intent, and a model may change its action for reasons that have nothing to do with the block.

A blocked event is a decision with a blocked_* outcome. Its next decision is the same person's decision at the next step. It exists only when the person is still active and the episode has not terminated.

Invariant: moved_after_block + blocked_after_block + stayed_after_block = B'. Per person: blocked_moves and max_consecutive_blocks.

11. Rules for any future composite or ranking

The metrics deliberately have no single score. Any ranking or composite score built on them MUST meet the rules below. The EMBER Score (scoring.md, scoring version 1.0) states how it meets each rule in its section 13.

Any ranking or composite score MUST:

1. be versioned, separately from the metrics version, and state the exact raw fields and formula it uses;
2. be decomposable: every published composite value must ship with the raw metrics it was computed from;
3. never let speed buy safety: a faster run that is materially less safe (more casualties or stranded people) must never outrank a safer run because of speed alone. Safety outcomes take precedence, for example through a lexicographic order with safety first, not a weighted sum in which time can offset lives.

Cross-seed robustness and statistical aggregation are in statistics.md.

12. Per-person fields (people)

One person metrics per scenario person, sorted by person ID:

13. Edge cases

14. Serialisation

Episode metrics dictionary form returns plain JSON types. Pairs become objects keyed by ID, outcome, cause or distance (as strings). null marks an undefined ratio. No value is NaN or infinite, so the result encodes with the canonical JSON encoding.