CorneaProjector

How a forecast is scored

A forecast says when fire will reach each patch of ground. We check that against where the fire really went, counting only ground that burned after the forecast was made. Here is one made-up fire, step by step, with a Pyrecast forecast. FARSITE forecasts are scored the same way; section 5 explains how we make them.

1The timeline

  1. Monday 15:40: firefighters map the fire's perimeter. It's the latest one before the forecast, so it's our before picture.
  2. Tuesday 07:04: a satellite passes over and detects heat. Pyrecast starts its forecast right then, from the perimeter plus those satellite detections, which reach beyond the perimeter where the fire has grown since Monday.
  3. Tuesday 18:10: a new perimeter is mapped, 11.1 hours after the forecast started. It's our after picture, and the forecast is judged by what it expected to have burned by 18:10: the window runs from 07:04 to 18:10.
  4. Every perimeter in the forecast's first 7 days gets its own score like this, each at its own time, so a run is scored many times, never at made-up times between perimeters.
  5. Satellite passes at 13:25 and 14:05 detect heat during the window: a second, separate check.

2Four maps of the same ground

The forecast works on a grid of small squares (30 m across in reality). To keep it readable, each square here stands for 10 acres.

ABefore the forecast

Gray is inside Monday's perimeter. Hatched is where the forecast started burning beyond it, from the 07:04 satellite detections (dots). Together they're burned before: this ground can't count for or against the forecast.

BThe forecast

When the forecast expects fire to arrive. Squares it expects to have burned by 18:10 (dark purple) are its prediction for this perimeter. Later arrivals don't count against this one.

CWhat happened

Tuesday 18:10's perimeter (red line) and the ground that burned beyond what had burned before (pink). Dots are the satellite detections during the window.

DThe score

Lay B over C. Each square of new ground is a hit (predicted and burned), a false alarm (predicted, didn't burn) or a miss (burned, not predicted).

3The calculation

Count the squares in map D. Burned-before squares are left out entirely.

Hits
squares
acres · predicted and burned
False alarms
squares
acres · predicted, didn't burn
Misses
squares
acres · burned, not predicted
Precision
hits ÷ (hits + false alarms) = ÷
Of the ground the forecast said would burn, how much did.
Recall
hits ÷ (hits + misses) = ÷
Of the ground that burned, how much the forecast called.
IoU
hits ÷ (hits + false alarms + misses) = ÷
How much the predicted and the burned areas overlap: 100% only if they match exactly.
Hotspot capture
detections inside the prediction ÷ detections on new ground = ÷
The separate satellite check. The detection on burned-before ground is left out, and a detection our feed lists twice counts once (section 4). (The pipeline weighs each detection by a disk of ground around it: 375 m across for VIIRS, wide enough that neighbouring detections touch; 500 m for MODIS, whose 1 km pixels are usually much bigger than the fire they caught; and 30 m for Landsat.)

Why the forecast's own start counts as burned

Pyrecast treats the hatched squares as already burning when it starts, so it never predicts fire arriving there. If we only removed Monday's perimeter, those squares would count as fire the forecast missed, though it was never trying to predict them.

Recall would fall from to ( ÷ ), and IoU from to . Precision wouldn't change. In real runs this start reaches a median of about 2,400 acres beyond the perimeter.

4The rest of the rules

5How the FARSITE forecasts are made

FARSITE is a second fire-spread model, which we run ourselves as a hindcast. For each archived Pyrecast run of the pilot fires (Dome and Plaskett), FARSITE starts at the same moment from the same perimeter, using only data that was public at that moment. Every setting below is fixed in code, and the random parts are seeded by the fire and run, so any run can be reproduced exactly.

What goes in

Start

The baseline perimeter: the latest official perimeter mapped by the run's start, the same one the scoring calls "before".

Only its active edge is lit: the ground it gained since the last perimeter mapped at least 18 hours before the run, plus a 150 m strip inside its edge wherever satellites detected heat in the 24 hours before the run (each detection padded by 300 m). Pieces under an acre are dropped.

The rest of the burn scar is set to non-burnable, as an analyst would, except for a 30 m strip around the lit edge that FARSITE needs to get going.

Landscape

LANDFIRE 2024 fuels (the 40 fire behavior fuel models) and canopy (cover, height, base height, bulk density), and LANDFIRE 2020 elevation, slope and aspect, in 30 m cells, out to 15 km beyond the fire's perimeters.

Weather

NOAA's National Blend of Models (NBM): its latest 00 or 12 UTC run that had been out for at least 2 hours at the forecast's start. Values every 3 hours, interpolated to hourly: temperature, humidity, precipitation, and sustained 10 m wind speed and direction.

The weather is taken at one point, the middle of the baseline perimeter, for the whole fire.

Fuel moisture

Initial dead fuel moistures of 5% (1-hour fuels), 6% (10-hour) and 8% (100-hour); live herbaceous 30%, live woody 90%, foliage 100%.

Fire behavior

Crown fire by Finney's method, a 30-minute time step, and no spotting.

Burn periods

The fire spreads only during a daily burn period centered on 16:00 local time, and holds overnight. Its length is the simulation's own (see below) times the calibration factor.

Length

96 hours (4 days) from the run's start.

Resolution

30 m distance and 60 m perimeter resolution. Runs whose lit edge covers more than 3,000 acres start at 60 m and 120 m, since FARSITE slows down as the edge gets longer.

A simulation that runs past its time limit (25 minutes; 10 minutes for 21 early Plaskett forecasts that we re-ran) is retried once at double those. One that fails twice is left out of the ensemble.

An ensemble of 25 simulations

Each run is 25 FARSITE simulations. The first uses the weather as forecast; the other 24 each vary it.

Simulation 1Each of the other 24
Wind speedas forecast× 0.6 to 1.6 (typically within ±25%)
Wind directionas forecastturned by a random angle (typically within ±15°)
Dead fuel moistureas aboveup to 2 points wetter or drier
Burn period8 hours6, 7, 8, 9 or 10 hours

Calibrating to the day before

Before forecasting, FARSITE replays the day before the run. Nine of the simulations start from the previous official perimeter (the latest one mapped 20 to 54 hours before the baseline perimeter), lit and set up the same way, with the weather forecast published back then, and run until the baseline perimeter's time. Their median growth is compared with the growth that really happened between the two perimeters. Satellite detections aren't used for this: along a slow edge they read as several times the perimeter growth.

The ratio scales the burn period, the hours of active spread per day, for all 25 simulations of the forecast. A second replay with the scaled burn period refines it when the first was more than 25% off. The factor is kept between 0.05 and 2.5, and burn periods between 15 minutes and 16 hours. If the perimeter didn't grow, the factor is 0.05; if there's no earlier perimeter in that window, or the replay can't run, the run isn't calibrated (factor 1).

Example: for Dome's run at 19:56 UTC on 20 September, the two perimeters were 23.4 hours apart and the fire grew 337 acres between them. The replay grew 1,772 acres. The refined factor is 0.20, so the first simulation's 8-hour burn period becomes about 1.6 hours a day.

From 25 simulations to five maps

The 25 simulations (those that finished) are ranked by how many new acres each burns in its 4 days. The 10th, 30th, 50th, 70th and 90th percentile maps are the 3rd, 8th, 13th, 18th and 23rd smallest of the 25: each map is one whole simulation, not a blend of cells, so it's always one consistent fire.

Each map is moved onto the same run's Pyrecast grid, taking the nearest cell so arrival times don't change, and both models are scored on the same squares. What FARSITE started from is left blank, as Pyrecast's start is: the lit edge, which joins the common "burned before" just like Pyrecast's start in the callout above, and anything inside the baseline perimeter except its unburned islands.

What it leaves out

The map shows the calibrated version with a constant factor. We keep three more for the pilot's write-up: uncalibrated, a factor that eases off over the forecast, and calibrated with control lines read from the incident action plans' operations maps (Plaskett only; Dome's maps have no vector layers). Each run's settings and calibration are recorded beside its maps, for example Dome's 20 September run, and the code is farsite/hindcast.py.