Model Methodology

1. Overview

Flood Sentinel uses an ensemble machine learning approach to forecast river levels at configured gauge stations. It combines multiple model types to produce a consensus forecast with confidence intervals.

The system is designed as a decision-support tool for flood operations centres, council emergency managers, and infrastructure operators. It does not replace official Bureau of Meteorology flood warnings.

2. Ensemble Architecture

For each forecast station and time horizon, Flood Sentinel trains and maintains multiple models:

These four model types — plus a meta-learner that combines them — are what Flood Sentinel actually serves. They run on ordinary CPUs. Deep-learning sequence models (LSTM variants) and conceptual rainfall-runoff heads exist in the codebase as research scaffolding; none of them is served to customers, and we do not claim them as part of the product.

Members are selected per forecast horizon, not averaged by a fixed weighting. For each station and each lead time, every candidate member is scored out-of-fold on CRPS (Continuous Ranked Probability Score — the standard proper scoring rule for probabilistic forecasts), and a member is admitted to that horizon's ensemble only if it holds or improves the out-of-fold CRPS. A horizon can therefore be served by a different subset of members than its neighbours.

Because skill is measured per station and per horizon, coverage is uneven and we publish it that way: short lead times (1–6 hours) are model-backed at every gated station, and model-backed forecasts extend to 72 hours only at the stations that have passed the gate at that lead. Beyond the gated horizon the system falls back to a climatology or persistence outlook — useful guidance, but not a skill-bearing forecast, and labelled as such in the API response (num_models: 0).

3. Input Features

Models are trained on the following feature categories:

• Current conditions: Latest river level and rate-of-change at the forecast station and upstream gauges
• Upstream propagation: Time-lagged readings from upstream stations (lag hours configured per catchment)
• Rainfall: Observed and forecast rainfall at rain gauge stations within the catchment
• Temporal features: Hour of day, day of year (seasonal patterns), day of week
• Antecedent conditions: Rolling averages of upstream levels and rainfall over 6h, 12h, 24h, 48h, 72h windows
• Tidal influence: Tidal predictions for tidally-affected stations
• Rate features: Rate of rise/fall at upstream stations

4. Training Process

• Data split: Chronological split (no random shuffling) to prevent data leakage. The most recent 20% of data is used for validation.
• Horizon-specific models: Separate models are trained for each forecast horizon (e.g., 6h, 12h, 24h, 48h) because error characteristics change with lead time.
• Cross-validation: Time-series aware cross-validation with expanding windows.
• Retraining: Models can be retrained through the Training UI as new data accumulates. The system tracks model versions in its registry.
• Cutover gate: a newly trained model never replaces the one in production on training metrics alone. It must beat the incumbent on CRPS at a two-day lead and on threshold-conditional CRPS during major-flood conditions, measured on held-out historical flood events it was not trained on. A candidate that fails either half is not promoted, however good its headline R² or RMSE looks. Several candidates have failed this gate and been discarded.

5. Safety Clamps & Guardrails

To prevent physically impossible or dangerous forecasts, the system applies several safety mechanisms:

• Physical bounds: Forecasts are clamped to station-specific min/max values (e.g., a river cannot have a negative level)
• Rate-of-change limits: Forecasts that imply physically impossible rates of rise or fall are flagged and dampened
• Confidence degradation: Confidence intervals widen with forecast horizon, reflecting increasing uncertainty
• Honest fallback: where no gated model exists for a station and lead time — or where model confidence is very low — the system serves a persistence forecast (last observed level) or, at station/lead combinations where the evidence supports it, a climatology blend. These responses are labelled: the API reports num_models: 0 and the fallback method used, so an outlook is never presented as a model forecast
• Ensemble disagreement: When individual models disagree significantly, the system flags this and widens confidence bounds

6. Confidence & Uncertainty

Forecast confidence is expressed as a percentage (0–100%) and as confidence bands on the hydrograph:

• High confidence (80–100%): Models agree and the observations feeding them are fresh
• Moderate confidence (50–79%): Some model disagreement or data quality issues
• Low confidence (<50%): Significant model disagreement, stale data, or conditions outside training range

Confidence describes the internal state of the ensemble at run time — member agreement (spread), data freshness, and whether current conditions sit inside the training distribution. It is not a measured hit rate, and a high-confidence reading is not a claim about how often we are right. The only evidence we treat as skill is the CRPS gate described in §4, evaluated on held-out flood events.

7. Limitations

Flood Sentinel forecasts are advisory only. Users should be aware of these limitations:

• Peak magnitude at major flood is undemonstrated. In hindcast testing across four model generations, including with radar QPE rainfall, peak heights at major flood have been under-predicted by 17–88% at every gate station. Treat a forecast rise as early signal, not as the expected peak. Skill in ordinary conditions and on event rises is measured and good; peak magnitude is not.
• Training data range: Models perform best within the range of conditions seen during training.
• Infrastructure changes: Dam operations, new levees, channel modifications, or urban development can change catchment behaviour. Models need retraining after significant infrastructure changes.
• Rainfall forecast dependency: Longer-range forecasts depend on rainfall predictions, which have their own uncertainty. Forecast accuracy degrades beyond 24–48 hours.
• Flash floods: Very rapid (sub-hourly) flood events in small catchments may develop faster than the system's data refresh cycle.
• Tidal interactions: Complex tide-flood interactions at estuary stations are approximated and may not capture all dynamics.
• Not a replacement for BoM warnings: The Bureau of Meteorology is the authoritative source for official flood warnings in Australia. Flood Sentinel should complement, not replace, official warnings.
• Model degradation: All ML models can degrade over time as catchment conditions evolve. Regular retraining and performance monitoring are essential.

8. Validation & Performance Metrics

Model performance is continuously monitored using:

• CRPS (Continuous Ranked Probability Score) — the primary metric, and the only one that governs whether a model reaches production. It scores the whole predicted distribution, not just the central estimate, so a forecast that is confidently wrong is penalised properly. We also compute it threshold-conditionally at major-flood levels, because average-case skill over a quiet year says nothing about the events that matter.
• RMSE (Root Mean Square Error) — deterministic error, in metres
• MAE (Mean Absolute Error) — average forecast error magnitude
• NSE (Nash-Sutcliffe Efficiency) — standard hydrological performance metric (1.0 = perfect, <0 = worse than mean)
• Bias — systematic over/under-prediction tendency
• Threshold hit rates — accuracy of predicting when levels cross alert/warning thresholds

Performance reports are accessible via the Model Health dashboard.