Definition
The agreement between the AI-generated critical-spare risk score and the actual realized production-impact distribution (probability times dollars), measured against the tail (P95, P99) rather than the mean.
The Measurement Gap
The AI gives you a risk score of 7.2 out of 10 for a critical spare. That single number drives safety stock level, reorder priority, and sourcing strategy. But the reality is not a single number: it is a probability distribution of dollar impact. A P50 outcome might be $40,000 in production loss. A P95 outcome might be $2.4 million. A P99 outcome (a simultaneous failure during a planned outage) might be $8 million. The risk score of 7.2 captures none of this distribution. If the score is calibrated to the mean, it systematically under-weights the tail events that drive the most catastrophic outcomes. If it is calibrated to the P95, it over-stocks average-risk parts. The legacy metrics are no help. Carrying cost percentage penalizes high stock regardless of risk. Service level rewards hits regardless of consequence. Stockout rate counts events but not dollars. ABC classification sorts by movement, not by impact. The score-driven stocking decision ignores the shape of the production-impact distribution, and the shape is where the $8 million surprise lives. A critical spare risk score that matches P50 reality but misses P95 reality is a tool that works until it doesn't.
Decision Supported:
Whether the critical-spare risk score can be trusted to set stocking policy for high-impact, low-frequency stockouts, and whether tail-risk-aware stocking rules are needed for parts where the P95 impact exceeds a defined threshold.
Formula
Plain language: The AI gives you one risk score. Reality is a probability distribution of dollar impact. Measure how well the score matches the distribution, especially the tail. If the P95 predicted impact matches P95 realized impact, the calibration is good where it matters most.
Notation:
CSTRC = 1 - |realized_P95_impact - predicted_P95_impact| / realized_P95_impact
Computed across critical SKUs. Report with a confidence grade.
Variables:
| Variable | Definition | Unit |
|---|---|---|
| realized_P95_impact | 95th percentile production-impact dollars from realized stockout events or simulation | $ |
| predicted_P95_impact | 95th percentile production-impact dollars from the AI risk model | $ |
Confidence Grading (required alongside every CSTRC value):
| Grade | Basis | Interpretation |
|---|---|---|
| A: Empirical | 5+ critical-spare stockout events in the measurement window provide direct tail observation | Measurement is grounded in observed outcomes; highest confidence |
| B: Hybrid | 1-4 observed events plus Monte Carlo simulation with Bayesian updating from those events | Measurement blends observation and simulation; moderate confidence |
| C: Simulated | Zero observed events; CSTRC derived entirely from Monte Carlo simulation of impact distributions | Measurement is modeled, not observed; treat as indicative, not validated |
Organizations must report the confidence grade alongside the CSTRC score. A CSTRC of 0.85 Grade A is a validated risk model. A CSTRC of 0.85 Grade C is a simulation that agrees with itself: it tells you the model is internally consistent but not that it matches reality. Grade C scores should trigger proactive stress-testing against historical near-miss events and cross-asset analogues rather than acceptance as calibrated.
Proposed formula, pending validation pilots.