Safety-Critical Prediction Calibration Index (SCPCI)

GMI-NUC-001Nuclear OperationsIntegration-Required

Definition

Degree to which a predictive model's confidence scores at each confidence band correspond to actual prediction outcomes, weighted by the nuclear safety significance of the component being predicted.

The Measurement Gap

Nuclear plants making component deferral decisions based on predictive model output face a calibration problem that no legacy metric can detect. When a predictive model reports 90% confidence that a safety-related pump will survive to the next refueling outage, the plant needs to know whether that confidence score is trustworthy. Capacity factor, scram rate, Maintenance Rule (a)(2) criteria, and PM completion rate all measure outcomes of maintenance decisions, but none of them measure whether the predictive model's confidence corresponds to its actual accuracy. A model can report high confidence while being systematically overconfident, and the plant discovers the miscalibration only when a component fails that the model said would survive. The invisible failure mode is this: the model's stated confidence and its actual prediction accuracy diverge without any existing metric detecting the gap, and the plant makes deferral decisions based on confidence scores that do not mean what they appear to mean.

Decision Supported

Should the plant defer replacement or surveillance of a safety-related component based on a predictive model's confidence that the component will survive until the next outage?

Formula

Plain language: For each confidence band, compare how often the model's predictions at that confidence level were actually correct. Weight the calibration error by the safety significance of the predicted components, a 10% calibration gap on a reactor coolant pump matters more than a 10% gap on a condensate polisher.

Notation:
SCPCI(b) = Σ(w_s × actual_accuracy_s(b)) / Σ(w_s × stated_confidence(b))

Composite (plant-level):
SCPCI_plant = Σ_b (n_b × SCPCI(b)) / Σ_b n_b

Variables:

  • b = confidence band (e.g., 70-80%, 80-90%, 90-95%, 95-100%)
  • w_s = safety significance weight for component s (based on Maintenance Rule safety significance: 1.0 for safety-related, 0.7 for risk-significant non-safety, 0.4 for balance of plant)
  • actual_accuracy_s(b) = fraction of predictions in band b for components of significance s where the predicted outcome matched actual outcome within the prediction window (dimensionless, 0 to 1)
  • stated_confidence(b) = midpoint of confidence band b (dimensionless)
  • n_b = number of predictions in band b (count)

Sign convention: SCPCI = 1.0 is perfect calibration. Values above 1.0 indicate underconfidence (model is better than it claims: conservative, acceptable in nuclear). Values below 1.0 indicate overconfidence (model claims more accuracy than it delivers: dangerous in nuclear). Lower is worse.

Aggregation period: Rolling 24 months (nuclear operating cycles require longer measurement windows than industrial verticals due to lower decision frequency per component).

Edge Cases

  • Fewer than 10 predictions in a confidence band: exclude band from composite, report as insufficient sample
  • No safety-related component predictions: report SCPCI for non-safety population only, flag that the most important calibration question remains unanswered
  • Model predicts "no failure" and component doesn't fail: this is a correct prediction, but only if the component was genuinely at risk. Survival of a healthy component isn't calibration evidence. Require that predictions be limited to components where the model detected degradation signals.
  • Zero stated confidence (model declines to predict): this is actually good behavior. Exclude from SCPCI, track as "model humility rate" separately

Proposed formula, pending validation pilots.

At a Glance

Unit
Ratio
Trend Direction
Toward 1.0
Reporting Frequency
Rolling 24-month
Category
AI Model Trustworthiness in Safety-Critical Decisions
← PRA Input Currency Index (PRICI)Cross-Boundary Data Fidelity Index (CBDFI) →

Assess Your Organization

The AI-GPS Assessment evaluates your AI governance maturity and recommends KPIs like this one for your implementation roadmap.

Take the Free AI-GPS Assessment