Penalty Forecast Calibration Index (PFCI)

GMI-FAC-011Facilities ManagementIntegration-RequiredPattern O-1: Confidence Calibration

Definition

Degree to which the AI's stated confidence in regulatory penalty threshold forecasts corresponds to the forecast's actual accuracy at each confidence band.

The Measurement Gap

There is no metric that measures whether AI regulatory penalty threshold forecasts are calibrated to their stated confidence, which means organizations defer capital investments based on forecast assurances whose accuracy has never been verified.

Building performance standards (NYC's Local Law 97, Washington DC's BEPS, Boston's BERDO) impose escalating penalties for buildings exceeding carbon thresholds. AI sustainability platforms forecast whether each building will clear or exceed these thresholds, often with confidence scores. Capital planners use these forecasts to time retrofit investments: if the AI says a building will clear the threshold with 90% confidence, the capital plan defers the retrofit. But confidence calibration is rarely checked. A platform that says 90% confidence but is actually right only 60% of the time is producing forecasts that systematically encourage deferral when intervention is needed.

The consequences of overconfident forecasts compound. Deferred retrofits that should have proceeded mean the building exceeds the threshold, incurs penalties, and then requires emergency capital investment at higher cost and shorter timeline than planned investment would have required.

Decision Supported

Should the organization defer capital investments based on the AI's forecast that a building will clear its penalty threshold, or does the forecast's historical calibration warrant more conservative planning? PFCI provides the calibration data needed to adjust planning confidence.

Formula

Plain Language

Group past penalty-threshold forecasts by confidence band. Within each band, compare how often the forecast correctly predicted whether the building would be above or below the penalty threshold. Perfect calibration means 85%-confidence forecasts are right 85% of the time.

Notation

PFCI(b) = Actual_accuracy(b) / Stated_confidence(b)

Variables

Variable Definition Unit
b Confidence band (e.g., 80-90%) Band
Actual_accuracy(b) Proportion of forecasts in band b that correctly predicted above/below penalty threshold Ratio (0-1)
Stated_confidence(b) Midpoint of the confidence band Ratio (0-1)

Units: Ratio. 1.0 equals perfect calibration. Greater than 1.0 means underconfident (conservative). Less than 1.0 means overconfident (dangerous for deferral decisions).

Proposed formula, pending validation pilots.

At a Glance

Category
Sustainability Forecast & Attribution Accuracy

Decision Supported

Should the organization defer capital investments based on the AI's "safe" forecast, or does the forecast's track record warrant more conservative planning?

← Carbon Attribution Accuracy Index (CAAI)Retrofit Optimization Horizon Alignment (ROHA) →

Assess Your Organization

The AI-GPS Assessment evaluates your AI governance maturity and recommends KPIs like this one for your implementation roadmap.

Take the Free AI-GPS Assessment