Model Confidence Calibration Error (MCCR)

GMI-MRO-004MROIntegration-Required

Definition

The weighted average gap between a predictive model's stated failure probabilities and the actual observed failure rates within matched probability bins, measuring how far the model's confidence scores deviate from reality.

The Measurement Gap

Confidence scores drive every triage decision in predictive maintenance, but legacy metrics (precision, recall, F1, AUC) evaluate only the binary correctness of a prediction. They do not interrogate whether the probability attached to each prediction means what it says. A model can have strong F1 while wildly misstating its own confidence: when it says "90% likely," the actual observed failure rate in that bin may be 55%. The failure mode is the inflated-confidence alert that pulls resources from genuinely higher-risk assets. Miscalibrated confidence also corrupts economic thresholds (expected-value math breaks when probability is wrong) and erodes technician trust when high-confidence alerts repeatedly fail to pan out. Research on modern neural networks shows that calibration degrades as accuracy improves: the best-performing models are statistically the most miscalibrated.

Decision Supported:

When the model says "X% likely," can I trust that number for threshold-based prioritization and economic decisioning?

Formula

Plain language: Bin the model's predictions by confidence score. For each bin, compare the average predicted probability against the actual failure rate observed in that bin. Weight each bin's gap by the share of predictions it contains, then sum. The result is the model's expected calibration error.

Notation:

MCCR = Σ_b (n_b / N) × | observed_rate_b − predicted_prob_b |

Variables:

Variable Definition Unit
b Confidence bin index (typically 10 equal-width bins from 0% to 100%) index
n_b Number of predictions falling in bin b count
N Total number of predictions across all bins count
observed_rate_b Actual failure rate among assets in bin b within the prediction window Ratio
predicted_prob_b Average predicted failure probability for assets in bin b Ratio
MCCR Expected calibration error: weighted mean absolute gap across bins Ratio

Proposed formula, pending validation pilots.

At a Glance

Unit
Ratio (0 to 1)
Trend Direction
Lower; approaching 0
Reporting Frequency
Monthly
Category
Algorithmic Efficacy & Model Health
← Feature Importance Stability (FIS)Sensor-vs-Asset Degradation Disambiguation (SADD) →

Assess Your Organization

The AI-GPS Assessment evaluates your AI governance maturity and recommends KPIs like this one for your implementation roadmap.

Take the Free AI-GPS Assessment