Aggregate and paired statistics (statistics version 1.0)

This document defines the statistical summaries that EMBERBench computes over evaluation results.

There is no EMBER score or global winner in this module. (The EMBER Score is a separate module: see scoring.md.) This module does not rank models, combine metrics or produce a leaderboard. Every number belongs to one named metric. Win/loss/tie counts exist per metric only.

Out of scope here: leaderboards, a composite score, model rankings, provider adapters, procedural generation and a CLI.

1. Fail-closed input

An evaluation result in memory is not trusted as given. Before any episode is used, both functions:

1. rebuild the evaluation manifest from its dictionary and require that it is equal to the original;
2. call the manifest check: the suite ID, version and hash and the simulator, observation and metrics versions must match the running code;
3. rebuild the evaluation result, which repeats its structural checks (manifest order, no duplicates, every episode equal to its manifest entry, record headers equal to the manifest);
4. for every episode: resolve its scenario from the suite by ID and content hash, recompute episode metrics from the run record (this authenticates the record by simulator replay), and require that the stored metrics equal the recomputed metrics exactly (canonical JSON).

Any failure raises a statistics input error (a subclass of evaluation error). Nothing is repaired, skipped or partly summarized.

The paired comparison first calls the result pairing. Results from different evaluation manifests raise an evaluation error before any replay.

2. Supported metrics and directions

Each metric is a per-episode scalar from episode metrics. The module rebuilds it as an exact Fraction from the integer counts in the metrics (see §10) and requires that the float of that fraction equals the stored value.

The per-episode definitions are in metrics.md. Notes:

• Kinds. Outcome metrics are terminal safety and time results. Behaviour metrics describe observed movement. Proxy metrics approximate a quantity that is not measured directly (path optimality, fire danger). Descriptive metrics describe the run and are not objectives.
• Directions are used only to read the sign of a paired difference (§8). They do not make a metric an objective by itself.
• mean_person_time_to_safety counts every non-evacuee at max_steps (censoring), so it mixes speed and safety. Read it with the safety rates.
• blocked_rate and congestion_block_rate are lower-is-better per metric, but a run can lower them by not moving. Never read them alone.
• Exit metrics have no direction. Equal exit use is often not optimal (one exit can be much closer, or threatened by fire). exit_balance is never an objective, and no win/loss/tie count exists for any exit metric.

3. Groups: overall, per scenario, per family

Each artifact holds the same summaries for three kinds of group:

A scenario with several families contributes all of its episodes to every one of those families. Family views can therefore overlap, and the family counts do not add up to the overall counts. A family with no scheduled scenario is not listed.

Overall statistics use the episode observations directly. They are never an average of scenario or family means. With unequal group sizes (for example after missing episodes) the two are different, and only the episode-level value is reported. Each episode has equal weight; a scenario with more scheduled seeds has more weight in overall.

4. Descriptive statistics

For one metric in one group, let x_1, …, x_n be the defined values of the observed episodes, and x_(1) <= … <= x_(n) the same values in ascending order.

An undefined statistic is null (None). It is never 0.

Quantile rule (Hyndman-Fan type 7)

With 0-based sorted values y[0] <= … <= y[n-1] and a level p in [0, 1]:

h = (n - 1) · p
j = floor(h)
g = h - j
Q(p) = y[j] + g · (y[j+1] - y[j])        (Q(p) = y[j] when g = 0)

This is the default of R (type = 7) and of NumPy (method = "linear"). The levels are fixed: p25 = 1/4, p50 = 1/2, p75 = 3/4, p90 = 9/10, p95 = 19/20. With n = 1 every quantile is the single value.

Example: for 1, 2, 3, 4 the quantiles are p25 = 1.75, p50 = 2.5, p75 = 3.25, p90 = 3.7, p95 = 3.85, and the sample variance is 5/3.

5. Robustness and completeness counts

Each aggregate group has a robustness object of raw episode counts. There are no rates here, so nothing hides the denominators.

all_resolved + max_steps = observed. An episode can count in several of the last four rows.

6. Missing and undefined data

There are two different kinds of absent value. Both are reported; neither is imputed.

• Missing episode. The manifest schedules it but the result has no entry for it (for example, an interrupted run). It is missing, not a model failure: it is not counted as a casualty, a timeout or zero evacuation. The artifact lists every missing episode key (missing_episodes), and every group gives missing_indices and robustness.missing.
• Undefined metric. The episode has a result, but the metric is None for it (for example blocked_rate when nobody tried to move). The per-metric summary counts it in undefined.

For each metric summary:

scheduled = observed + missing
n         = observed - undefined

The statistics use the n defined values only. No value is filled in with zero, a mean, a worst case or a value from another episode.

7. Paired comparison

The paired comparison uses the result pairing for the exact episode join. An episode is paired only when both results contain the same key (manifest hash, index, scenario ID, scenario hash, seed). Unpaired episodes are never compared; they are listed in missing_from_a and missing_from_b.

For one metric in one group, with paired episodes k where the metric is defined for both A and B:

delta_k = value_A(k) - value_B(k)        (always A - B)

Each paired metric comparison reports:

Win/loss/tie, per metric:

There is no count across metrics, no majority vote and no composite. Swapping A and B negates every delta and swaps a_better and b_better.

8. Confidence intervals

Each metric summary has mean_ci (aggregate) or mean_delta_ci (paired): a 95 % percentile bootstrap interval for the mean.

1. Let x_1, …, x_n be the defined values (aggregate) or the paired deltas (paired). If n < 5 the interval is null.
2. For each of R resamples, draw n indices uniformly with replacement from 0 … n-1 and take the mean of the selected values. In a paired comparison the resampling unit is the paired episode, so A and B stay together in the delta.
3. Sort the R resample means. The bounds are their type-7 quantiles at 0.025 and 0.975.

R is resamples: the default is 2000; any integer in [100, 1 000 000] is accepted. All means are exact fractions; only the bounds become floats.

Determinism. The bootstrap never uses random, the time or OS entropy. Its stream seed is

seed = SHA-256( canonical JSON of {
    statistics_version, purpose ("aggregate_mean" or "paired_mean_delta"),
    evaluation_manifest_hash, result_digests, group [kind, name], metric, resamples } )

where each result digest is the SHA-256 of the canonical evaluation result (the artifact as saved). The 64-bit words come from SHAKE-256 in counter mode: block k is SHAKE-256("emberbench/statistics/bootstrap/v1\0" || seed || k) (k as 8 big-endian bytes), read as 512 big-endian unsigned 64-bit words. An index below n is drawn by rejection sampling (word < 2^64 - 2^64 mod n, then word mod n), so there is no modulo bias. The seed is in the output for audit. Identical inputs give identical intervals on every platform. Different data, labels or manifests give different streams. This stream shares nothing with the simulator's SplitMix64 streams.

Interpretation. An interval describes the uncertainty of the mean over this evaluation sample of scenarios and seeds, resampled at the episode level. It does not prove that one model is better in general, on other scenarios or on other seeds. Episodes of one scenario are not independent of each other, and the public suite is small, so treat intervals as descriptive.

9. Exact arithmetic and serialization

• Per-episode values are exact Fractions rebuilt from integer counts (for mean_exposure, from the fire-distance histogram: Σ count_d / d).
• Means, medians, variances, quantiles, min/max, deltas and bootstrap means are exact Fractions in the dataclasses.
• The output boundary is the dictionary: each exact value becomes its correctly rounded float. The standard deviation is the one irrational value: it is math.sqrt(float(variance)), which IEEE 754 rounds correctly.
• Every list is in manifest, suite or scenario family order. No result depends on dict or set iteration order or on PYTHONHASHSEED.
• The JSON text is canonical JSON (sorted keys, no whitespace, ASCII, no NaN). The digest is its SHA-256. The model ID label, the input result digests and resamples are part of the artifact, so a change to any of them changes the digest.

10. Artifact layout

The aggregate statistics dictionary form:

statistics_version, artifact = "aggregate", model_id,
evaluation_manifest_hash, suite_id, suite_version, suite_hash,
simulator_version, observation_version, metrics_version,
result_digest, scheduled_episodes, observed_episodes, missing_episodes[],
metric_definitions[], methods{quantiles, variance, std, bootstrap},
overall{kind, name, episode_indices, missing_indices, robustness, metrics[]},
scenarios[], families[]

Each metrics[] entry: metric, scheduled, observed, missing, undefined, statistics{n, mean, median, variance, std, min, max, quantiles{p25…p95}}, mean_ci{method, statistic, level, resamples, seed, lower, upper} | null.

The paired statistics dictionary form has the same header with model_a, model_b, result_digest_a, result_digest_b, delta = "A - B", paired_count, missing_from_a[], missing_from_b[], and groups with paired_indices, missing_from_a, missing_from_b and the §8 fields per metric.

11. Summary of what each section is

There is no EMBER score or global winner in this module.