Definition
The weighted average gap between a predictive model's stated failure probabilities and the actual observed failure rates within matched probability bins, measuring how far the model's confidence scores deviate from reality.
The Measurement Gap
Confidence scores drive every triage decision in predictive maintenance, but legacy metrics (precision, recall, F1, AUC) evaluate only the binary correctness of a prediction. They do not interrogate whether the probability attached to each prediction means what it says. A model can have strong F1 while wildly misstating its own confidence: when it says "90% likely," the actual observed failure rate in that bin may be 55%. The failure mode is the inflated-confidence alert that pulls resources from genuinely higher-risk assets. Miscalibrated confidence also corrupts economic thresholds (expected-value math breaks when probability is wrong) and erodes technician trust when high-confidence alerts repeatedly fail to pan out. Research on modern neural networks shows that calibration degrades as accuracy improves: the best-performing models are statistically the most miscalibrated.
Decision Supported:
When the model says "X% likely," can I trust that number for threshold-based prioritization and economic decisioning?
Formula
Plain language: Bin the model's predictions by confidence score. For each bin, compare the average predicted probability against the actual failure rate observed in that bin. Weight each bin's gap by the share of predictions it contains, then sum. The result is the model's expected calibration error.
Notation:
MCCR = Σ_b (n_b / N) × | observed_rate_b − predicted_prob_b |
Variables:
| Variable | Definition | Unit |
|---|---|---|
| b | Confidence bin index (typically 10 equal-width bins from 0% to 100%) | index |
| n_b | Number of predictions falling in bin b | count |
| N | Total number of predictions across all bins | count |
| observed_rate_b | Actual failure rate among assets in bin b within the prediction window | Ratio |
| predicted_prob_b | Average predicted failure probability for assets in bin b | Ratio |
| MCCR | Expected calibration error: weighted mean absolute gap across bins | Ratio |
Proposed formula, pending validation pilots.