Prediction Confidence Calibration - AI Infrastructure (PCCAI)

GMI-DC-004Data CentersIntegration-Required

Definition

Measures whether the predictive failure model's stated confidence levels for critical infrastructure components correspond to actual failure and health outcomes at each confidence band, quantifying the degree to which operations can trust the model's assessments.

The Measurement Gap

Predictive maintenance systems report component health as a confidence score. "95% probability this component will not fail within the next 30 days." Operations teams use these scores to authorize maintenance deferrals. What no existing metric measures is whether 95% confidence actually delivers 95% reliability in practice, or whether the model is systematically overconfident, producing 95% confidence assessments for components that fail at 15% or 25% rates. An overconfident model converts normal-looking health scores into hidden failure risk, and every deferral authorized on the basis of an overconfident score is a deferral made on false information. PM completion rate measures whether deferred work gets done; it cannot tell you whether the deferrals were safe in the first place.

Decision Supported:

Can operations trust the predictive model's health assessments enough to authorize maintenance deferrals, or is the model systematically overconfident in ways that convert normal-looking scores into unrevealed failure risk?

Formula

Plain language: Group components by the model's stated confidence band. For each band, compare the actual percentage that did not fail against what the confidence level claimed. Compute the aggregate calibration score as 1 minus the average absolute error across bands.

Notation:

PCCAI(b) = Actual_healthy_rate(b) / Stated_confidence(b)

PCCAI_agg = 1 − (1/B) × Σ |Actual_healthy_rate(b) − Stated_confidence(b)| for all bands b

Variables:

Variable Definition Unit
b Confidence band (e.g., 90-100%, 80-90%, 70-80%) Band label
Actual_healthy_rate(b) Percentage of components in band b that did not experience failure requiring unplanned maintenance within the prediction horizon %
Stated_confidence(b) Midpoint of the confidence band (e.g., 95% for 90-100% band) %
B Number of confidence bands Count
PCCAI_agg Aggregate calibration score across all bands 0 to 1

A per-band PCCAI of 1.0 indicates perfect calibration. Above 1.0 indicates the model is underconfident (conservative). Below 1.0 indicates overconfidence: the dangerous direction.

Proposed formula, pending validation pilots.

At a Glance

Unit
Score
Trend Direction
Toward 1.0
Reporting Frequency
Quarterly
Category
Predictive Maintenance Calibration
← Deferral Risk Correlation Index (DRCI)Compliance Margin Remaining (CMR) →

Assess Your Organization

The AI-GPS Assessment evaluates your AI governance maturity and recommends KPIs like this one for your implementation roadmap.

Take the Free AI-GPS Assessment