How a forecast is scored
A forecast says when fire will reach each patch of ground. We check that against where the fire really went, counting only ground that burned after the forecast was made. Here is one made-up fire, step by step, with a Pyrecast forecast. FARSITE forecasts are scored the same way; section 5 explains how we make them.
1The timeline
- Monday 15:40: firefighters map the fire's perimeter. It's the latest one before the forecast, so it's our before picture.
- Tuesday 07:04: a satellite passes over and detects heat. Pyrecast starts its forecast right then, from the perimeter plus those satellite detections, which reach beyond the perimeter where the fire has grown since Monday.
- Tuesday 18:10: a new perimeter is mapped, 11.1 hours after the forecast started. It's our after picture, and the forecast is judged by what it expected to have burned by 18:10: the window runs from 07:04 to 18:10.
- Every perimeter in the forecast's first 7 days gets its own score like this, each at its own time, so a run is scored many times, never at made-up times between perimeters.
- Satellite passes at 13:25 and 14:05 detect heat during the window: a second, separate check.
2Four maps of the same ground
The forecast works on a grid of small squares (30 m across in reality). To keep it readable, each square here stands for 10 acres.
ABefore the forecast
Gray is inside Monday's perimeter. Hatched is where the forecast started burning beyond it, from the 07:04 satellite detections (dots). Together they're burned before: this ground can't count for or against the forecast.
BThe forecast
When the forecast expects fire to arrive. Squares it expects to have burned by 18:10 (dark purple) are its prediction for this perimeter. Later arrivals don't count against this one.
CWhat happened
Tuesday 18:10's perimeter (red line) and the ground that burned beyond what had burned before (pink). Dots are the satellite detections during the window.
DThe score
Lay B over C. Each square of new ground is a hit (predicted and burned), a false alarm (predicted, didn't burn) or a miss (burned, not predicted).
3The calculation
Count the squares in map D. Burned-before squares are left out entirely.
Why the forecast's own start counts as burned
Pyrecast treats the hatched squares as already burning when it starts, so it never predicts fire arriving there. If we only removed Monday's perimeter, those squares would count as fire the forecast missed, though it was never trying to predict them.
Recall would fall from to ( ÷ ), and IoU from to . Precision wouldn't change. In real runs this start reaches a median of about 2,400 acres beyond the perimeter.
4The rest of the rules
- Five versions per run. Each forecast comes in five versions, the 10th to 90th percentile: Pyrecast's five weather scenarios, and five of FARSITE's 25 simulations (section 5). Each is scored on its own.
- Which perimeters. Each run is scored against every perimeter with a mapped time in its first 7 days (a day is 24 hours from the run's start). On a day with none, the latest perimeter published that day is used instead, timed by its last edit; a copy of an earlier shape is skipped. The perimeters scored before November 2026 are kept too, rescored at their own time. FARSITE forecasts 4 days, so it's scored against the perimeters in its first 4 days.
- Perimeter times. A perimeter is timed by when it was mapped. When that time is missing or doesn't look right, we use when it was last edited instead, and record which (the map shows those in italics).
- Everything is shown. There's no minimum growth and no off-grid rule: growth beyond the edge of the forecast's map counts as a miss. When the fire didn't grow, recall is blank, and precision and IoU are 0 if the forecast predicted growth.
- Every model the same way. Pyrecast and FARSITE are scored by the same code, against the same perimeters, with one common "burned before": the perimeter at run time plus each model's own start (Pyrecast's satellite-seeded start, and the edge FARSITE was lit from).
- When the forecast's start wasn't saved. For about 8% of runs, our archive missed the forecast's first hour. Then every satellite detection up to the run stands in for its start.
- Why hotspots are only a side check. Satellites pass a few times a day and can't see through thick smoke or cloud, so missing detections don't mean the ground didn't burn. Hotspots feed the capture check only, never precision.
- A detection reported twice counts once. Our feed often lists the same detection twice: the same satellite pass, a minute or two apart, with the same brightness and fire power, at spots up to about 300 m apart where the products it came through place it differently (most over steep ground). Two detections from one satellite within 3 minutes count as one when they have the same reading within 300 m, or are closer than a sixth of the sensor's pixel (62 m for VIIRS), which two different pixels can't be. On four test fires that took out 35–50% of the VIIRS listings (none from MODIS or Landsat), but only about 2% of the ground they cover, since the copies overlap. The map shows each detection once too.
- Detections that don't look like the fire are left out. The capture check skips a detection that's more than 1 km from the fire and that no later perimeter ever reaches, and one that no satellite's nearest pass, before or after, also saw within 1 km. These count neither for nor against the forecast. On four test fires that was about 2 in 100 detections.
- Same grid throughout. The perimeters and detections are laid onto the forecast's own grid, and every area above is a count of its squares.
5How the FARSITE forecasts are made
FARSITE is a second fire-spread model, which we run ourselves as a hindcast. For each archived Pyrecast run of the pilot fires (Dome and Plaskett), FARSITE starts at the same moment from the same perimeter, using only data that was public at that moment. Every setting below is fixed in code, and the random parts are seeded by the fire and run, so any run can be reproduced exactly.
What goes in
The baseline perimeter: the latest official perimeter mapped by the run's start, the same one the scoring calls "before".
Only its active edge is lit: the ground it gained since the last perimeter mapped at least 18 hours before the run, plus a 150 m strip inside its edge wherever satellites detected heat in the 24 hours before the run (each detection padded by 300 m). Pieces under an acre are dropped.
The rest of the burn scar is set to non-burnable, as an analyst would, except for a 30 m strip around the lit edge that FARSITE needs to get going.
LANDFIRE 2024 fuels (the 40 fire behavior fuel models) and canopy (cover, height, base height, bulk density), and LANDFIRE 2020 elevation, slope and aspect, in 30 m cells, out to 15 km beyond the fire's perimeters.
NOAA's National Blend of Models (NBM): its latest 00 or 12 UTC run that had been out for at least 2 hours at the forecast's start. Values every 3 hours, interpolated to hourly: temperature, humidity, precipitation, and sustained 10 m wind speed and direction.
The weather is taken at one point, the middle of the baseline perimeter, for the whole fire.
Initial dead fuel moistures of 5% (1-hour fuels), 6% (10-hour) and 8% (100-hour); live herbaceous 30%, live woody 90%, foliage 100%.
Crown fire by Finney's method, a 30-minute time step, and no spotting.
The fire spreads only during a daily burn period centered on 16:00 local time, and holds overnight. Its length is the simulation's own (see below) times the calibration factor.
96 hours (4 days) from the run's start.
30 m distance and 60 m perimeter resolution. Runs whose lit edge covers more than 3,000 acres start at 60 m and 120 m, since FARSITE slows down as the edge gets longer.
A simulation that runs past its time limit (25 minutes; 10 minutes for 21 early Plaskett forecasts that we re-ran) is retried once at double those. One that fails twice is left out of the ensemble.
An ensemble of 25 simulations
Each run is 25 FARSITE simulations. The first uses the weather as forecast; the other 24 each vary it.
| Simulation 1 | Each of the other 24 | |
|---|---|---|
| Wind speed | as forecast | × 0.6 to 1.6 (typically within ±25%) |
| Wind direction | as forecast | turned by a random angle (typically within ±15°) |
| Dead fuel moisture | as above | up to 2 points wetter or drier |
| Burn period | 8 hours | 6, 7, 8, 9 or 10 hours |
Calibrating to the day before
Before forecasting, FARSITE replays the day before the run. Nine of the simulations start from the previous official perimeter (the latest one mapped 20 to 54 hours before the baseline perimeter), lit and set up the same way, with the weather forecast published back then, and run until the baseline perimeter's time. Their median growth is compared with the growth that really happened between the two perimeters. Satellite detections aren't used for this: along a slow edge they read as several times the perimeter growth.
The ratio scales the burn period, the hours of active spread per day, for all 25 simulations of the forecast. A second replay with the scaled burn period refines it when the first was more than 25% off. The factor is kept between 0.05 and 2.5, and burn periods between 15 minutes and 16 hours. If the perimeter didn't grow, the factor is 0.05; if there's no earlier perimeter in that window, or the replay can't run, the run isn't calibrated (factor 1).
Example: for Dome's run at 19:56 UTC on 20 September, the two perimeters were 23.4 hours apart and the fire grew 337 acres between them. The replay grew 1,772 acres. The refined factor is 0.20, so the first simulation's 8-hour burn period becomes about 1.6 hours a day.
From 25 simulations to five maps
The 25 simulations (those that finished) are ranked by how many new acres each burns in its 4 days. The 10th, 30th, 50th, 70th and 90th percentile maps are the 3rd, 8th, 13th, 18th and 23rd smallest of the 25: each map is one whole simulation, not a blend of cells, so it's always one consistent fire.
Each map is moved onto the same run's Pyrecast grid, taking the nearest cell so arrival times don't change, and both models are scored on the same squares. What FARSITE started from is left blank, as Pyrecast's start is: the lit edge, which joins the common "burned before" just like Pyrecast's start in the callout above, and anything inside the baseline perimeter except its unburned islands.
What it leaves out
- Firefighting. The version on the map has no control lines or other suppression.
- Spotting. Embers that start fires ahead of the front aren't simulated.
- Weather across the fire. The weather changes hour by hour but is the same everywhere on the fire.
- Changes to fuels since LANDFIRE 2024, other than the fire's own burn scar.
The map shows the calibrated version with a constant factor. We keep three more for the pilot's write-up: uncalibrated, a factor that eases off over the forecast, and calibrated with control lines read from the incident action plans' operations maps (Plaskett only; Dome's maps have no vector layers). Each run's settings and calibration are recorded beside its maps, for example Dome's 20 September run, and the code is farsite/hindcast.py.
