Prediction Calibration Index (PCI)

GMI-RAIL-001Rail TransportationIntegration-Required

Definition

The ratio of a predictive maintenance model's actual accuracy to its stated confidence, measured per failure mode and per confidence band, indicating whether the organization can trust the model's confidence scores at face value.

The Measurement Gap

The predictive model reports 90% confidence in a wheel bearing failure prediction. Is the actual hit rate at that confidence level really 90%, or has the model drifted to where 90% confidence means 72% accuracy? No legacy metric answers this question. Locomotive availability measures whether the asset is running, not whether the predictions keeping it running are trustworthy. Mean distance between failures counts the failures that happen, not the near-misses where the model said "fine" and luck held. Bad-order ratio does not attribute cause to model error.

Decision Supported:

Should we act on this model's high-confidence predictions with the same urgency as its confidence scores imply, or has the model become overconfident?

Formula

Plain language: For each confidence band (e.g., 80-90%, 90-95%, 95-100%), count how many predictions in that band turned out to be correct. Divide the actual hit rate by the midpoint confidence. A perfectly calibrated model scores 1.0 across all bands. Above 1.0 means underconfident (conservative). Below 1.0 means overconfident (dangerous).

Notation:

PCI_b = (TP_b / N_b) / C_b

Overall PCI (weighted): PCI = SUM(PCI_b x N_b) / SUM(N_b)

Variables:

Variable Description Unit
TP_b True positive predictions in confidence band b (predicted failure occurred within horizon) Count
N_b Total predictions issued in confidence band b during evaluation window Count
C_b Midpoint confidence value for band b (e.g., 0.85 for the 80-90% band) Dimensionless

Sign Convention: 1.0 = perfect calibration. < 1.0 = overconfident (bad). > 1.0 = underconfident (less dangerous but suboptimal). Distance from 1.0 in either direction is the concern.

Proposed formula, pending validation pilots.

At a Glance

Unit
Ratio
Category
AI/Model Trustworthiness
← Model Currency Index (MCI)Defect Detection Latency (DDL) →

Assess Your Organization

The AI-GPS Assessment evaluates your AI governance maturity and recommends KPIs like this one for your implementation roadmap.

Take the Free AI-GPS Assessment