Definition
Percentage of stability results falling within the model's predicted confidence interval, compared against the interval's stated confidence level. A well-calibrated model lands near its nominal level; scores below it mean the model is more confident than it has earned.
The Measurement Gap
The Gap: No current metric catches the difference between stated model confidence and actual model accuracy. Stability failure rate only counts OOS results. Shelf-life adherence metrics track whether product is sold before expiry, not whether the expiry was correctly set. OOS rate at stability pull is an individual-measurement metric, not a model-calibration metric. The problem is not that the model is wrong (most predictions are directionally correct) the problem is the model does not know how uncertain it is.
The Decision:
SPCS stratified by formulation type becomes a decision input for which products can rely on model-predicted shelf lives, which need more conservative dating, and where the model needs retraining. Products with SPCS below 85% should not rely on model-predicted shelf lives without additional real-time confirmation. SPCS above 98% indicates underconfidence: intervals too wide, reducing the model's practical utility.
Formula
Notation:
SPCS = (Σ Results_Within_CI / Σ Predictions_With_Actuals) × 100
Variables:
- Results_Within_CI = stability results at a given time point that fell within the model's predicted 95% confidence interval
- Predictions_With_Actuals = total model predictions for which real-time stability data is now available for comparison
Worked Example: Over two years, 43 prediction-actual pairs available. 34 actual results fell within the model's 95% CI. SPCS = 34/43 × 100 = 79.1%. Of 9 outside the CI, 7 were below the lower bound (model over-predicted stability) and 2 above. The asymmetry reveals the model is biased toward over-prediction: the dangerous direction for shelf-life decisions. Stratified: immediate-release tablets 83.3%, extended-release 91.7%, oral solutions 61.5%. The model is most overconfident for oral solutions.
Edge Cases:
- SPCS above 95% (e.g., 99%) means underconfident intervals: too wide. Not dangerous but reduces decision utility.
- SPCS requires patience: 12–18 months of retrospective data before sufficient prediction-actual pairs accumulate
- Stratify by product, formulation type, and storage condition to avoid masking formulation-specific problems
Sign convention: SPCS should approximate the stated confidence level. Both too high and too low are informative, but too low is dangerous.
Confidence Grading (required alongside every SPCS value):
| Grade | Basis | Interpretation |
|---|---|---|
| A: Empirical | 5+ observed events in measurement window provide direct tail observation | Measurement grounded in observed outcomes; highest confidence |
| B: Hybrid | 1-4 observed events plus Monte Carlo simulation with Bayesian updating | Blends observation and simulation; moderate confidence |
| C: Simulated | Zero observed events; derived entirely from Monte Carlo simulation | Modeled, not observed; treat as indicative, not validated |
A SPCS of any value at Grade C is a simulation that agrees with itself. It tells you the model is internally consistent but not that it matches reality. Grade C scores should trigger proactive stress-testing against historical near-miss events and cross-asset analogues rather than acceptance as calibrated.
Proposed formula, pending validation pilots.