CAPA Scope Adequacy Trend (CSAT)

GMI-PH-016Pharmaceutical OperationsPilot-Ready

Definition

Ratio of the 12-month repeat deviation rate for AI-scoped CAPAs to the rate for human-scoped CAPAs. A ratio above 1.0 means AI scoping is producing narrower corrective actions that miss the systemic cause.

The Measurement Gap

The Gap: CAPA Closure Rate does not detect this. CAPAs are closed on time. Deviation Rate tracks count, not the relationship between CAPAs and subsequent deviations. Repeat deviation rate captures it eventually but with a 6–12 month lag and no attribution to AI scoping. The efficiency metrics look positive: cycle time is down, format consistency is up, less time debating scope in review meetings. But the AI learned the organization's historical under-scoping habit and replicates it.

The Decision:

CSAT above 1.50 triggers a review of AI training data and scoping algorithm. CSAT trending upward over 3 consecutive quarters triggers mandatory human review of all AI-scoped CAPAs before closure. The 6-month lag inherent in repeat deviation measurement means CSAT is a medium-horizon indicator: it does not catch the problem immediately but catches it before the regulatory audit does.

Formula

Notation:

CSAT = Repeat_Rate_AI / Repeat_Rate_Human

Variables:

  • Repeat_Rate_AI = percentage of AI-scoped CAPAs where a repeat deviation (same root cause category) occurred within 12 months
  • Repeat_Rate_Human = percentage of human-scoped CAPAs where a repeat deviation occurred within 12 months

Worked Example: Over 18 months (12-month lookback + 6-month observation): AI-scoped CAPAs: 94 total, 17 resulted in repeats = 18.1%. Human-scoped CAPAs: 62 total, 7 resulted in repeats = 11.3%. CSAT = 18.1/11.3 = 1.60. AI-scoped CAPAs produce 60% more repeat deviations than human-scoped ones. The AI is systematically under-scoping.

Edge Cases:

  • CSAT below 1.0 means AI outperforms human scoping: the AI is catching systemic scope that humans miss
  • CSAT = 1.0 means equivalent performance
  • Minimum 12-month observation window for repeat deviation detection: shorter windows undercount repeats
  • Control for deviation severity: AI-scoped CAPAs may be assigned to different severity categories than human-scoped ones

Sign convention: CSAT at or below 1.0 is ideal. Higher means AI-scoped CAPAs produce more repeats.

Proposed formula, pending validation pilots.

At a Glance

Unit
Percentage
Trend Direction
Higher
Reporting Frequency
Quarterly (6-12 month repeat-deviation lag)
Category
Autonomous Decision Amplification
← Stability Prediction Calibration Score (SPCS)Digital Twin Variance Amplification Ratio (DTVAR) →

Assess Your Organization

The AI-GPS Assessment evaluates your AI governance maturity and recommends KPIs like this one for your implementation roadmap.

Take the Free AI-GPS Assessment