Batch Record AI Reliance Gap (BRARG)

GMI-PH-011Pharmaceutical OperationsIntegration-Required

Definition

Percentage of sampled AI-certified batch records in which a human auditor found completeness issues the AI did not flag. Quantifies whether human review reduction has outpaced the AI's actual capability.

The Measurement Gap

The Gap: The QA director progressively reduced human review depth based on AI performance: full review to 20% spot-check to 10% spot-check. Efficiency gains are real and quantified: fewer FTEs, faster batch release, labor savings presented in quarterly reviews. But nobody is measuring whether the review reduction outpaces the AI's capability. At 20% spot-check, humans still caught issues the AI missed. At 10%, the sample is too small to catch them reliably. The issues did not go away; the detection net got too sparse.

The Decision:

BRARG trends directly inform whether to increase, maintain, or decrease human review depth. Rising BRARG at any reduction level triggers reversal of the most recent depth reduction. BRARG above 5% at critical gap level requires increasing human review depth. BRARG above 10% at any level requires suspending AI-driven review depth reduction and conducting a retrospective audit.

Formula

Notation:

BRARG = (Σ Records_With_AI_Missed_Gaps / Σ Sampled_AI_Certified_Records) × 100

Variables:

  • Records_With_AI_Missed_Gaps = sampled batch records where the human auditor found completeness issues the AI did not flag
  • Sampled_AI_Certified_Records = batch records the AI certified as complete that were randomly selected for deep human audit

Edge Cases:

  • Gap severity must be classified: Critical (patient safety or data integrity risk), Major (regulatory compliance gap), Minor (documentation quality below standard)
  • Human auditor standards must be defined by SOP to avoid measuring auditor variation instead of AI gaps
  • If BRARG is consistently 0%, either the AI is perfect (unlikely) or the audit standard is too lenient

Sign convention:

Lower BRARG is better. 0% means the AI catches everything a deep human audit would.

Proposed formula, pending validation pilots.

At a Glance

Unit
Percentage
Trend Direction
Higher
Reporting Frequency
Quarterly
Category
Human-AI Trust & Override Governance
← AI-Human Discordance Resolution Consistency (AHDRC)Deviation Root Cause Verification Rate (DRCVR) →

Assess Your Organization

The AI-GPS Assessment evaluates your AI governance maturity and recommends KPIs like this one for your implementation roadmap.

Take the Free AI-GPS Assessment