Definition
Error Budget Forecast Fidelity (EBFF) measures the calibration of AI-driven error budget forecasts against actual error budget consumption at the decision-relevant horizon.
The Measurement Gap
A SaaS company's reliability team uses an AI-driven error budget forecaster to inform deployment freeze decisions. The forecaster ingests the last 30 days of SLO performance and projects error budget consumption for the rest of the quarter. When the forecast predicts the budget will be exhausted before quarter end, the platform recommends a deployment freeze. The team has been operating this way for nine months. Freeze recommendations are issued roughly every 4-6 weeks, accepted by engineering leadership, and lifted when the forecast improves.
In Q3, a senior engineering leader pushes back. "We froze deployments for two weeks in July based on the forecast. By the end of July, the budget was actually fine: we ended the quarter at 35% remaining. Did the freeze actually save us? Or did we delay deployments for no reason?" The reliability team digs into the historical forecast versus actual data. Across the past nine months: 8 freeze recommendations. Of those 8, 3 were followed by actual budget exhaustion within the next two weeks (forecast was correct). 3 were followed by budget recovery (forecast was overly conservative; freeze was probably unnecessary). 2 were ambiguous.
The leader's question lands: if 5 of 8 freeze recommendations were either overly cautious or ambiguous, the team has been imposing real engineering cost (delayed releases, accumulated change backlog, customer feature delays) based on forecasts that may be no better than gut-feel.
The legacy metrics on the team's dashboard report a healthy error budget program.
Error budget remaining: tracked per SLO, updated continuously.
Burn rate: tracked at 1-hour, 6-hour, and 24-hour windows.
SLO attainment: tracked weekly and quarterly.
Freeze decisions made: 8 in the past 9 months.
Time spent frozen: roughly 14 weeks of the trailing 39, or 36% of the time.
Deployment frequency during frozen vs unfrozen windows: significantly lower during freezes (as expected).
None of these report on whether the forecasts driving freeze decisions are calibrated. The forecasts produce a number; the freeze decision is made; the deployment pipeline responds. Whether the forecast was actually accurate is rarely measured after the fact. The team trusts the forecaster because it is the only forecaster they have.
These metrics were designed for an error budget practice that did not include AI forecasting. The freeze decision used to be made by a human looking at current burn rate and using judgment. The judgment was sometimes wrong, but it was bounded by what the human could see; nobody mistook a hunch for a prediction. With AI-driven forecasting, the prediction is treated as a reliable signal because it is produced by a machine. The accuracy of that prediction is rarely audited.
There is no metric that tells you whether error budget forecasts correlate with actual reliability outcomes, which means deployment freeze decisions are based on predictions that may not reflect current system risk.
Formula
Error Budget Forecast Fidelity (EBFF) measures the calibration of AI-driven error budget forecasts against actual error budget consumption at the decision-relevant horizon. In plain language: every time the forecaster makes a prediction about error budget consumption at a future horizon (typically 7 or 14 days out), wait for that horizon to arrive, then compare the prediction against actual consumption. EBFF is one minus the mean absolute prediction error, normalized by total error budget. A value near 1 means the forecasts are well-calibrated. A value near 0 means the forecasts are random with respect to actual outcomes.
The formula:
EBFF = 1 − (1/N) × Σ |EB_predicted_i − EB_actual_i| / EB_total
Where:
- EB_predicted_i = the forecasted error budget remaining at the end of the forecast window (i = each comparison point)
- EB_actual_i = the actual error budget remaining at the end of the same forecast window
- EB_total = the total error budget for the window (used to normalize the error)
- N = the count of forecast-to-actual comparison pointsUnits are a ratio from 0 to 1. A value of 1.0 means perfect forecasting. A value of 0 means the forecast is no better than chance. A worked example. The SaaS company computes EBFF for their authorization service over a 12-week measurement period. They generate a forecast each Monday for the upcoming 7-day window, then compare against actual consumption the following Monday. Each weekly comparison produces a prediction error, normalized by the weekly error budget. Sample data across 12 weeks:| Week | EB predicted | EB actual | EB total | Absolute error | Normalized ||------|--------------|-----------|----------|----------------|------------|| 1 | 87% | 91% | 100% | 4% | 0.04 || 2 | 82% | 88% | 100% | 6% | 0.06 || 3 | 76% | 79% | 100% | 3% | 0.03 || 4 | 70% | 65% | 100% | 5% | 0.05 || 5 | 60% | 71% | 100% | 11% | 0.11 || 6 | 53% | 62% | 100% | 9% | 0.09 || 7 | 48% | 56% | 100% | 8% | 0.08 || 8 | 41% | 49% | 100% | 8% | 0.08 || 9 | 35% | 42% | 100% | 7% | 0.07 || 10 | 27% | 38% | 100% | 11% | 0.11 || 11 | 19% | 33% | 100% | 14% | 0.14 || 12 | 10% | 28% | 100% | 18% | 0.18 |Mean normalized error = (0.04 + 0.06 + ... + 0.18) / 12 ≈ 0.085
EBFF = 1 − 0.085 = 0.915
Per the catalog guidance (above 0.80 means operationally useful, below 0.60 means freeze decisions are essentially random) this EBFF of 0.915 sits well above the operational threshold. The forecaster is calibrated. The example also reveals a directional bias: the forecaster systematically underestimates remaining error budget (predictions are consistently lower than actuals). This is worth noting separately. Even with high EBFF, a consistent directional bias means the freeze decisions may be triggering earlier than necessary. Track the signed error alongside the absolute error. Three edge cases need explicit treatment. First, exclude all-green windows from the calculation: periods with no error budget consumption are trivially correct to predict and inflate EBFF without testing the forecaster's actual capability. Second, standardize on the horizon where freeze decisions are actually made. A forecast made 30 days out has different error characteristics than a forecast made 7 days out; comparing them in the same EBFF calculation is incoherent. Third, the minimum sample is 12 comparison points per service per quarter; below that the mean is noise. Cross-references. Upstream of EBFF is
Adversarial Confidence Audit (ACA, GMI-IT-004): forecaster calibration is a specific case of AI confidence calibration. Downstream is SLO-Experience Alignment Score (SEAS). If the underlying SLO does not align with user experience, the forecasts on that SLO are not actionable regardless of fidelity. The misinterpretation to watch for: a team reports EBFF = 0.95 (excellent calibration) and concludes the forecaster is reliable. They miss that the high EBFF is driven by long stretches of trivially-correct predictions during low-incident periods, while the actual decision-relevant predictions (during periods of elevated risk) have much worse error characteristics. EBFF should be computed both overall and during periods of active error budget consumption; the latter is the operationally relevant fidelity for freeze decisions.
Proposed formula, pending validation pilots.