Definition
Percentage of AI-eliminated stability time points that would have detected a trajectory deviation earlier, among eliminations where subsequent data makes that judgment possible. Puts a value on the information the model chose not to collect.
The Measurement Gap
When the stability prediction model says "I am confident enough that you can skip the 15-month time point," the organization saves testing cost. But nobody tracks whether the skipped time point would have revealed a degradation pathway the model didn't predict. For one product, the eliminated 15-month point turns out to be exactly where a secondary degradation impurity becomes detectable, an impurity the model's training data never included. The $340K in testing savings is quantified and celebrated. The information lost from the eliminated time points is assumed to be zero based on model confidence. No metric measures the gap between what the model thinks it knows and what the eliminated data would have revealed. The cost savings are visible; the information loss is invisible until a supplementary regulatory filing or a product quality event forces the organization to confront what it chose not to measure.
Decision Supported:
Did eliminating those stability testing time points actually save money net of information value, or did we trade measurable cost savings for unmeasured information loss?
Formula
Notation:
SIVI = (Σ Informative_Eliminations / Σ Total_Assessable_Eliminations) × 100
Variables:
- Informative_Eliminations = eliminated time points where subsequent data showed a trajectory deviation that the eliminated time point would have detected earlier
- Total_Assessable_Eliminations = eliminated time points for which sufficient subsequent data now exists for retrospective assessment
Worked Example: 23 time points eliminated over 18 months. Retrospective analysis: 18 confirmed the model's predicted trajectory (correctly eliminated). 3 showed trajectory deviations the eliminated time point would have caught earlier. 2 have insufficient subsequent data (too recent). SIVI = 3/21 = 14.3%. Above the 10% action threshold: one in seven eliminated time points would have provided useful information.
Edge Cases:
- Exclude eliminations where insufficient subsequent data exists (too recent) from both numerator and denominator
- "Informative" means the eliminated time point would have detected a deviation or trend at least one time point earlier than actual detection
- SIVI inherently requires patience: 12–18 months before assessable data accumulates
Sign convention: Lower SIVI is better. Zero means all eliminations were correctly made.
Proposed formula, pending validation pilots.