Definition
The ratio of a predictive maintenance model's actual accuracy to its stated confidence, measured per failure mode and per confidence band, indicating whether the organization can trust the model's confidence scores at face value.
The Measurement Gap
The predictive model reports 90% confidence in a wheel bearing failure prediction. Is the actual hit rate at that confidence level really 90%, or has the model drifted to where 90% confidence means 72% accuracy? No legacy metric answers this question. Locomotive availability measures whether the asset is running, not whether the predictions keeping it running are trustworthy. Mean distance between failures counts the failures that happen, not the near-misses where the model said "fine" and luck held. Bad-order ratio does not attribute cause to model error.
Decision Supported:
Should we act on this model's high-confidence predictions with the same urgency as its confidence scores imply, or has the model become overconfident?
Formula
Plain language: For each confidence band (e.g., 80-90%, 90-95%, 95-100%), count how many predictions in that band turned out to be correct. Divide the actual hit rate by the midpoint confidence. A perfectly calibrated model scores 1.0 across all bands. Above 1.0 means underconfident (conservative). Below 1.0 means overconfident (dangerous).
Notation:
PCI_b = (TP_b / N_b) / C_b
Overall PCI (weighted): PCI = SUM(PCI_b x N_b) / SUM(N_b)
Variables:
| Variable | Description | Unit |
|---|---|---|
| TP_b | True positive predictions in confidence band b (predicted failure occurred within horizon) | Count |
| N_b | Total predictions issued in confidence band b during evaluation window | Count |
| C_b | Midpoint confidence value for band b (e.g., 0.85 for the 80-90% band) | Dimensionless |
Sign Convention: 1.0 = perfect calibration. < 1.0 = overconfident (bad). > 1.0 = underconfident (less dangerous but suboptimal). Distance from 1.0 in either direction is the concern.
Proposed formula, pending validation pilots.