Definition
Degree to which the AI's stated confidence in regulatory penalty threshold forecasts corresponds to the forecast's actual accuracy at each confidence band.
The Measurement Gap
There is no metric that measures whether AI regulatory penalty threshold forecasts are calibrated to their stated confidence, which means organizations defer capital investments based on forecast assurances whose accuracy has never been verified.
Building performance standards (NYC's Local Law 97, Washington DC's BEPS, Boston's BERDO) impose escalating penalties for buildings exceeding carbon thresholds. AI sustainability platforms forecast whether each building will clear or exceed these thresholds, often with confidence scores. Capital planners use these forecasts to time retrofit investments: if the AI says a building will clear the threshold with 90% confidence, the capital plan defers the retrofit. But confidence calibration is rarely checked. A platform that says 90% confidence but is actually right only 60% of the time is producing forecasts that systematically encourage deferral when intervention is needed.
The consequences of overconfident forecasts compound. Deferred retrofits that should have proceeded mean the building exceeds the threshold, incurs penalties, and then requires emergency capital investment at higher cost and shorter timeline than planned investment would have required.
Decision Supported
Should the organization defer capital investments based on the AI's forecast that a building will clear its penalty threshold, or does the forecast's historical calibration warrant more conservative planning? PFCI provides the calibration data needed to adjust planning confidence.
Formula
Plain Language
Group past penalty-threshold forecasts by confidence band. Within each band, compare how often the forecast correctly predicted whether the building would be above or below the penalty threshold. Perfect calibration means 85%-confidence forecasts are right 85% of the time.
Notation
PFCI(b) = Actual_accuracy(b) / Stated_confidence(b)
Variables
| Variable | Definition | Unit |
|---|---|---|
| b | Confidence band (e.g., 80-90%) | Band |
| Actual_accuracy(b) | Proportion of forecasts in band b that correctly predicted above/below penalty threshold | Ratio (0-1) |
| Stated_confidence(b) | Midpoint of the confidence band | Ratio (0-1) |
Units: Ratio. 1.0 equals perfect calibration. Greater than 1.0 means underconfident (conservative). Less than 1.0 means overconfident (dangerous for deferral decisions).
Proposed formula, pending validation pilots.
At a Glance
Decision Supported
Should the organization defer capital investments based on the AI's "safe" forecast, or does the forecast's track record warrant more conservative planning?