Reject Attribution Accuracy (RAA)

GMI-PH-010Pharmaceutical OperationsIntegration-Required

Definition

Percentage of reject rate anomaly events where the initial attribution was confirmed correct by subsequent investigation. Reject rate tells you how many units failed; this tells you whether the stated reason was right.

The Measurement Gap

The Gap: Reject Rate tells you how many units failed, not why. Batch Success Rate is often unaffected because the underlying quality shift does not cause OOS results, just marginal quality. No standard metric tracks the accuracy of the attribution decision. The dangerous pattern: a product quality issue causes the reject rate to rise, the team attributes it to model drift, recalibrates the model to be less sensitive, and the underlying quality issue goes uninvestigated for weeks until a downstream test catches it.

The Decision:

RAA below 60% triggers mandatory dual-investigation protocol. Always check both model AND product for every anomaly, regardless of initial attribution. Persistently low RAA for specific attribution directions (e.g., frequently blaming the model when it is the product) drives targeted investigation training and root-cause-specific response protocols.

Formula

Notation:

RAA = (Σ Correct_Attributions / Σ Total_Anomaly_Events) × 100

Variables:

  • Correct_Attributions = anomaly events where the initial attribution was confirmed by subsequent investigation
  • Total_Anomaly_Events = all reject rate anomaly events that triggered investigation

Worked Example: Over 6 months, 17 reject rate anomaly events across 4 inspection lines. 11 attributed to model drift: 8 confirmed correct (model diagnostics showed sensitivity shift, product testing showed no change), 3 later found to be product-related. 6 attributed to product quality: 5 confirmed correct, 1 turned out to be a model issue. RAA = (8 + 5)/17 = 76.5%. By direction: correct model attribution 72.7%, correct product attribution 83.3%. The team is better at identifying product issues than model drift: the dangerous errors are the 3 events attributed to model drift that were actually product quality.

Edge Cases:

  • Attribution to "model drift" is confirmed correct if model diagnostics show sensitivity change AND product testing shows no quality change
  • Attribution to "product quality" is confirmed correct if product testing reveals a genuine issue
  • Inconclusive investigations should be tracked separately, a high inconclusive rate means the investigation process needs improvement
  • Minimum 10 events per measurement period

Sign convention: Higher RAA is better.

Proposed formula, pending validation pilots.

At a Glance

Unit
Percentage
Trend Direction
Higher
Reporting Frequency
Monthly; per reject investigation
Category
Human-AI Trust & Override Governance
← Environmental Alert Action Rate (EAAR)Case Seriousness Classification Accuracy (CSCA) →

Assess Your Organization

The AI-GPS Assessment evaluates your AI governance maturity and recommends KPIs like this one for your implementation roadmap.

Take the Free AI-GPS Assessment