Stability Prediction Calibration Score (SPCS)

GMI-PH-006Pharmaceutical OperationsInfrastructure-Dependent

Definition

Percentage of stability results falling within the model's predicted confidence interval, compared against the interval's stated confidence level. A well-calibrated model lands near its nominal level; scores below it mean the model is more confident than it has earned.

The Measurement Gap

The Gap: No current metric catches the difference between stated model confidence and actual model accuracy. Stability failure rate only counts OOS results. Shelf-life adherence metrics track whether product is sold before expiry, not whether the expiry was correctly set. OOS rate at stability pull is an individual-measurement metric, not a model-calibration metric. The problem is not that the model is wrong (most predictions are directionally correct) the problem is the model does not know how uncertain it is.

The Decision:

SPCS stratified by formulation type becomes a decision input for which products can rely on model-predicted shelf lives, which need more conservative dating, and where the model needs retraining. Products with SPCS below 85% should not rely on model-predicted shelf lives without additional real-time confirmation. SPCS above 98% indicates underconfidence: intervals too wide, reducing the model's practical utility.

Formula

Notation:

SPCS = (Σ Results_Within_CI / Σ Predictions_With_Actuals) × 100

Variables:

  • Results_Within_CI = stability results at a given time point that fell within the model's predicted 95% confidence interval
  • Predictions_With_Actuals = total model predictions for which real-time stability data is now available for comparison

Worked Example: Over two years, 43 prediction-actual pairs available. 34 actual results fell within the model's 95% CI. SPCS = 34/43 × 100 = 79.1%. Of 9 outside the CI, 7 were below the lower bound (model over-predicted stability) and 2 above. The asymmetry reveals the model is biased toward over-prediction: the dangerous direction for shelf-life decisions. Stratified: immediate-release tablets 83.3%, extended-release 91.7%, oral solutions 61.5%. The model is most overconfident for oral solutions.

Edge Cases:

  • SPCS above 95% (e.g., 99%) means underconfident intervals: too wide. Not dangerous but reduces decision utility.
  • SPCS requires patience: 12–18 months of retrospective data before sufficient prediction-actual pairs accumulate
  • Stratify by product, formulation type, and storage condition to avoid masking formulation-specific problems

Sign convention: SPCS should approximate the stated confidence level. Both too high and too low are informative, but too low is dangerous.

Confidence Grading (required alongside every SPCS value):

Grade Basis Interpretation
A: Empirical 5+ observed events in measurement window provide direct tail observation Measurement grounded in observed outcomes; highest confidence
B: Hybrid 1-4 observed events plus Monte Carlo simulation with Bayesian updating Blends observation and simulation; moderate confidence
C: Simulated Zero observed events; derived entirely from Monte Carlo simulation Modeled, not observed; treat as indicative, not validated

A SPCS of any value at Grade C is a simulation that agrees with itself. It tells you the model is internally consistent but not that it matches reality. Grade C scores should trigger proactive stress-testing against historical near-miss events and cross-asset analogues rather than acceptance as calibrated.

Proposed formula, pending validation pilots.

At a Glance

Unit
Percentage
Trend Direction
Higher
Reporting Frequency
Quarterly (requires accumulated prediction-actual pairs)
Category
AI Model Fidelity & Drift
← Regulatory AI Currency Index (RACI)CAPA Scope Adequacy Trend (CSAT) →

Assess Your Organization

The AI-GPS Assessment evaluates your AI governance maturity and recommends KPIs like this one for your implementation roadmap.

Take the Free AI-GPS Assessment