Stability Prediction Retrospective Accuracy (SPRA)

GMI-PH-034Pharmaceutical OperationsInfrastructure-Dependent

Definition

Proportion of AI stability predictions confirmed accurate against real-time data, reported alongside validation coverage: how many predictions were ever checked at all.

The Measurement Gap

An AI stability platform justified deferring 1,847 individual stability tests across a 67-product portfolio, saving $4.2M in direct testing costs. The stability budget was cut 18%. But of those 1,847 deferred tests, how many predictions were actually checked against reality when the physical test eventually happened?

In most implementations, the prediction justified the deferral, the deferral was approved, and the prediction was never validated. The model could be systematically overconfident (predicting "within specification" for products trending toward OOS) and nobody would know until a physical test revealed an unexpected failure. Stability failures often occur at inflection points (packaging changes, excipient substitutions) that break the model's assumptions.

Decision enabled:

When SPRA drops below 90%, the model is too inaccurate to justify broad deferrals without additional controls. When validation_coverage (the companion metric) is below 30%, the organization is accumulating unvalidated risk. Both trigger increased physical testing requirements.

Formula

SPRA = predictions_confirmed_accurate / predictions_with_retrospective_data

Companion: validation_coverage = predictions_with_retrospective_data / total_predictions_made

A prediction is confirmed accurate if the actual result falls within the AI's stated prediction interval.

Worked Example:

Twenty-month retrospective analysis:

Total predictions used to justify deferrals: 1,847.
Predictions where subsequent physical testing occurred: 683.
Validation_coverage = 683 / 1,847 = 37.0%.

Predictions confirmed accurate: 614.
SPRA = 614 / 683 = 89.9%.

Two layers deeper: 1,164 predictions (63%) remain unvalidated. $2.4M in deferred testing justified by unchecked predictions. Among the 69 inaccurate predictions, 47 were minor (within spec but outside the prediction interval). 22 were material: actual result exceeded the specification or was trending faster than predicted. In 8 of those 22, the eventual test triggered an OOS investigation that would have started 6–12 months earlier.

By confidence band: 95%+ predictions had 94.1% SPRA, 85–95% had 82.7% SPRA, 75–85% had 71.3% SPRA. Slightly overconfident at every level.

Sign Convention:

Higher is better. Above 90% means predictions are reliable. Below 80% means the model is too inaccurate to justify deferrals. Validation_coverage below 30% signals unvalidated risk accumulation.

Proposed formula, pending validation pilots.

At a Glance

Unit
Percentage
Trend Direction
Higher
Reporting Frequency
Annually; per deferred-test retrospective
Category
Quality System & Investigation Effectiveness
← Root Cause Investigation Independence Score (RCIIS)Regulatory Change Classification Accuracy (RCCA) →

Assess Your Organization

The AI-GPS Assessment evaluates your AI governance maturity and recommends KPIs like this one for your implementation roadmap.

Take the Free AI-GPS Assessment