Definition
Percentage of sampled AI-certified batch records in which a human auditor found completeness issues the AI did not flag. Quantifies whether human review reduction has outpaced the AI's actual capability.
The Measurement Gap
The Gap: The QA director progressively reduced human review depth based on AI performance: full review to 20% spot-check to 10% spot-check. Efficiency gains are real and quantified: fewer FTEs, faster batch release, labor savings presented in quarterly reviews. But nobody is measuring whether the review reduction outpaces the AI's capability. At 20% spot-check, humans still caught issues the AI missed. At 10%, the sample is too small to catch them reliably. The issues did not go away; the detection net got too sparse.
The Decision:
BRARG trends directly inform whether to increase, maintain, or decrease human review depth. Rising BRARG at any reduction level triggers reversal of the most recent depth reduction. BRARG above 5% at critical gap level requires increasing human review depth. BRARG above 10% at any level requires suspending AI-driven review depth reduction and conducting a retrospective audit.
Formula
Notation:
BRARG = (Σ Records_With_AI_Missed_Gaps / Σ Sampled_AI_Certified_Records) × 100
Variables:
- Records_With_AI_Missed_Gaps = sampled batch records where the human auditor found completeness issues the AI did not flag
- Sampled_AI_Certified_Records = batch records the AI certified as complete that were randomly selected for deep human audit
Edge Cases:
- Gap severity must be classified: Critical (patient safety or data integrity risk), Major (regulatory compliance gap), Minor (documentation quality below standard)
- Human auditor standards must be defined by SOP to avoid measuring auditor variation instead of AI gaps
- If BRARG is consistently 0%, either the AI is perfect (unlikely) or the audit standard is too lenient
Sign convention:
Lower BRARG is better. 0% means the AI catches everything a deep human audit would.
Proposed formula, pending validation pilots.