Definition
Prediction Confidence Calibration Index (PCCI) measures the degree to which a prediction model's stated confidence levels match its actual accuracy, segmented by failure mode and asset class. A perfectly calibrated model that says "85% confident" should be right 85% of the time for that specific prediction context.
The Measurement Gap
A reliability engineer at a refinery reviewed her dashboard. The predictive system was showing three open alerts: a cooling water pump flagged at 91% failure probability within 30 days, a heat exchanger fan at 78%, and a compressor seal at 85%. She had resources to address two of the three this week. Standard triage: take the two highest-probability items. She scheduled the pump and the compressor seal.
The pump intervention found nothing. Bearing condition was good, vibration within normal range, no degradation visible. The compressor seal work found a genuine early-stage leak that would have progressed to failure within the predicted window. The heat exchanger fan she deferred? It seized two weeks later during a production run, causing a $340,000 unplanned outage.
The post-incident review asked the obvious question: how did a 91% confidence prediction turn out to be wrong while a 78% prediction was deferred and proved correct? The answer wasn't that the model was "bad". Across all its predictions, it was right about 75% of the time. The problem was that its 91% didn't mean 91%. For cooling water pumps specifically, predictions above 85% confidence were only correct 55% of the time. For compressor seals, predictions above 80% confidence were correct 90% of the time. The model's confidence was well-calibrated for some asset classes and catastrophically miscalibrated for others.
No one knew this because no one was measuring it.
MTBF captured the fan failure as a data point in the running average, one more failure in the denominator. It couldn't tell you that the failure was predicted, that the prediction was deferred because a higher-confidence prediction took priority, or that the confidence ranking was unreliable. PdM Hit Rate showed the system's overall accuracy at 75%, which looked acceptable. But aggregate accuracy masked the per-asset-class variation that actually drove the wrong decision. PM Compliance scored the pump intervention as compliant: the scheduled work was completed. The fact that it was unnecessary never showed up in any metric.
There is no metric that tells you whether a prediction model's stated confidence matches its actual accuracy for each failure mode and asset class, which means organizations prioritize their responses using numbers that may bear no relationship to actual risk.
Formula
Prediction Confidence Calibration Index (PCCI) measures the degree to which a prediction model's stated confidence levels match its actual accuracy, segmented by failure mode and asset class.
In plain language:
When the model says it's 90% sure, is it actually right 90% of the time, or is it bluffing?
The Formula
PCCI = 1 - (1/B) x Sum_b |C_b - A_b|
Where B is the number of confidence bins, C_b is the mean stated confidence for predictions in bin b, and A_b is the actual accuracy for predictions in that bin.
Think of it this way: you take every prediction the model made and sort them into bins by their stated confidence. All the predictions the model said were 80 to 90% likely go in one bin. All the 90 to 100% predictions go in another. Then, for each bin, you check: what percentage of those predictions actually came true? If the model said 85% and was right 85% of the time, the bin error is zero. If it said 85% and was only right 55% of the time, the bin error is 0.30. Average the bin errors and subtract from 1. A PCCI of 1.0 is perfect calibration. A PCCI of 0.50 means the model's confidence scores are almost meaningless.
Worked Example
A mining operation's prediction model generated 840 predictions over 6 months for haul truck powertrain failures. Binned by confidence:
| Confidence Bin | # Predictions | Stated Confidence (mean) | Actual Accuracy | Bin Error |
| 50-60% | 210 | 0.55 | 0.48 | 0.07 |
| 60-70% | 195 | 0.65 | 0.59 | 0.06 |
| 70-80% | 178 | 0.74 | 0.63 | 0.11 |
| 80-90% | 152 | 0.84 | 0.71 | 0.13 |
| 90-100% | 105 | 0.93 | 0.76 | 0.17 |
Average bin error = (0.07 + 0.06 + 0.11 + 0.13 + 0.17) / 5 = 0.108
PCCI = 1 - 0.108 = 0.892
That looks decent overall, but the pattern is concerning: calibration degrades systematically as confidence increases. The model is worst at exactly the predictions it's most confident about. Predictions marked 90%+ are only right 76% of the time. The reliability engineer who prioritizes the 93% prediction over the 74% prediction is being misled. The actual accuracy difference between those two is only 13 percentage points (76% vs. 63%), not the 19 points the confidence scores suggest.
Now segment by asset class. The same model on haul truck engines shows PCCI of 0.93. On haul truck transmissions, PCCI drops to 0.71. On hydraulic systems, PCCI is 0.58. The aggregate PCCI of 0.892 hid the fact that the model is essentially unreliable for hydraulic system predictions.
Sign Convention
Higher is better. PCCI of 1.0 means perfect calibration. PCCI below 0.70 should trigger model review. Below 0.50, the confidence scores are providing essentially no useful prioritization information. These thresholds are proposed, not proven.
Cross-References
PCCI is the most foundational metric in this catalog. Almost every other KPI either depends on model calibration or is affected by it.
Directly downstream: Prediction Threshold Value Ratio cannot be optimized if the underlying confidence scores are miscalibrated. You're optimizing a threshold on a broken ruler. Override Outcome Index must be interpreted in context of PCCI: if PCCI is low, overrides may be rational, and an Override Outcome Index above 1.0 is expected. Autonomy Boundary Calibration Score depends on prediction reliability. You shouldn't give autonomous authority to a system whose confidence you can't trust.
Upstream of PCCI, there's nothing in this catalog. It's foundational. PCCI depends on the ML model's training data, architecture, and calibration approach, which are engineering inputs, not KPI-derived.
If you ignore PCCI and only track Override Outcome Index, you might see operators consistently overriding the system and conclude they're not trusting the AI. The real problem might be that the operators have implicitly learned what PCCI would tell them explicitly: the model's confidence scores are unreliable for certain asset classes, and the overrides are a rational adaptation to uncalibrated predictions.
Proposed formula, pending validation pilots.
At a Glance
Decision Supported
Should I act on this prediction at face value, or does this model systematically overstate or understate its certainty for this type of failure on this type of asset?