EMBERBench is live
Today we are releasing EMBERBench, the first project from Anantam Labs. It asks one question: can a decision model guide people to safety as a fire spreads?
What it tests
Each scenario is a building with people, exits and a fire that grows step by step. At every step, the model sees the current situation and decides how each person moves. Routes close, queues form, and the model never knows where the fire will go next.
The first release has 10 public scenarios and 19 frozen worlds. Every model faces exactly the same worlds and the same fires.
How we score it
Every recorded run is replayed and checked before it is scored. A run that does not match its replay is rejected.
The EMBER Score runs from 0 to 100. It combines two things: Safety, which asks whether people who could be saved were saved, and Quality, which asks how well they were moved (time, danger, path and replanning). It is a benchmark score, not the share of people who got out.
What we found
We tested 21 decision models and 8 hand-coded agents.
- The best decision model scored 51.70. The best hand-coded agent scored 88.59. Even the best model is far behind the strongest hand-written planner.
- Most decision models scored between 20 and 33. Two scored close to zero.
- Cost and speed do not change the score, but we show them beside it, so you can see what each result cost to run.
Each model ran once on the public set, so small gaps between models are not proof that one is better.
On the private set of 300 hidden episodes, scores are lower: the best decision model scored 20.97. See the private set results.
What it is not
These results come from a public development set that models can train on. The official EMBERBench test is private and separate; we publish only its totals, on the private set page. EMBERBench is simulated research. It is not guidance for a real evacuation.
Explore it
- Results: public and private set results, sorted by score, safety, cost or speed.
- Methodology: how the score works, in plain words.
- Documentation: how to run EMBERBench on your own model.
- Source code: the public benchmark repository.
If you find a mistake, write to us.