Prediction Confidence Calibration Index (PCCI-RE)

GMI-RE-001Renewable EnergyIntegration-RequiredPattern O-1: Confidence Calibration

Definition

Ratio of a predictive maintenance model's actual prediction accuracy to its stated confidence level within each confidence band, measuring whether the model's certainty claims correspond to reality.

The Measurement Gap

Measurement Gap

Every predictive maintenance platform in renewable energy reports an aggregate accuracy score for its failure predictions. The number usually looks good, often north of 85%, because it blends low-confidence predictions (where being wrong is expected) with high-confidence predictions (where accuracy should be near-certain). The aggregate hides the pattern that matters most: how calibrated is the model at each confidence level?

Legacy maintenance metrics are structurally blind to this problem. Capacity factor measures production output, not prediction quality. Availability tracks uptime, not whether the alerts that triggered maintenance interventions were trustworthy. MTTR measures repair speed, not whether the repair was warranted by a reliable prediction. PM compliance counts whether scheduled work happened on time, not whether the prediction that generated the work order was calibrated. None of these metrics distinguish between a model that is 90% confident and 90% accurate versus a model that is 90% confident and 65% accurate. The first model deserves trust. The second model is systematically overconfident, and every high-confidence alert it produces should be treated with skepticism rather than triggering immediate maintenance mobilization.

The gap is not whether the model predicts failures. The gap is whether the model knows how well it predicts failures, and whether the organization can verify that self-assessment.

Decision Supported

Should the asset manager act on a predictive alert at a given confidence level, or is the model systematically overconfident, meaning high-confidence alerts don't deserve high-confidence responses? PCCI answers this directly: if the 90-95% confidence band shows a PCCI of 0.72, the model is claiming near-certainty but delivering only moderate accuracy. The response protocol for that band should be downgraded until calibration improves.

Formula

Plain Language

Group all predictive alerts by their stated confidence level (e.g., 80-85%, 85-90%, 90-95%, 95-100%). Within each band, count how many predictions actually came true. Divide actual accuracy by stated confidence. A result of 1.0 means the model is perfectly calibrated at that band. Below 1.0 means the model is overconfident. Above 1.0 means the model is underconfident (conservative).

Notation

PCCI(b) = Actual_Accuracy(b) / Stated_Confidence_Midpoint(b)

Variables

Variable Definition Units Source
b Confidence band (e.g., 5% wide bins from 50% to 100%) Categorical Predictive analytics platform
Actual_Accuracy(b) Predictions in band b that matched actual outcome divided by total predictions in band b Dimensionless ratio Derived from analytics platform alerts cross-referenced with CMMS outcomes, per rolling 6-month window
Stated_Confidence_Midpoint(b) Midpoint of the confidence band (e.g., 0.925 for the 90-95% band) Dimensionless ratio Predictive analytics platform

All thresholds in this entry are proposed starting points for pilot validation, not proven benchmarks.

Proposed formula, pending validation pilots.

At a Glance

Unit
Ratio
Trend Direction
Toward 1.0
Reporting Frequency
Rolling 6-month window
Category
Model Calibration and Prediction Trustworthiness
← Forecast Regime Bias Index (FRBI)RUL Estimation Accuracy Index (REAI) →

Assess Your Organization

The AI-GPS Assessment evaluates your AI governance maturity and recommends KPIs like this one for your implementation roadmap.

Take the Free AI-GPS Assessment