Definition
Percentage of NLP auto-cleared batch record exceptions that an independent qualified reviewer confirms were correctly cleared against current SOP and data integrity requirements.
The Measurement Gap
Batch Record Review Time (the metric everyone celebrates) improves as the NLP auto-clear rate climbs, because each auto-cleared exception is one fewer exception a human sees. But Batch Record Review Time cannot distinguish between auto-clears that happened because the AI learned the right pattern and auto-clears that happened because the AI learned from reviewers who were rushing through exception queues under time pressure. Deviation Rate does not catch it because auto-cleared exceptions do not become deviations. Data integrity metrics, where tracked, measure whether records exist and are attributable, not whether the AI's clearing decisions are correct. The risk accumulates silently: each auto-cleared borderline exception is a small data integrity gap that, over thousands of batches, creates the systematic documentation weakness an FDA investigator will find during a pre-approval inspection.
Decision Supported:
Is our batch record NLP model's auto-clear function maintaining accuracy, or is it learning bad habits from rushed reviewers and accumulating data integrity risk?
Formula
Plain language: Take a random sample of exceptions the NLP model auto-cleared during the measurement period. Have a qualified human reviewer (one who was not part of the original training data set) independently assess each one against the current SOP and data integrity requirements. Count how many the reviewer confirms were correctly cleared. Divide by the total sampled.
Notation:
BRACA = (Σ Confirmed_Correct_Auto-Clears / Σ Sampled_Auto-Clears) × 100
Variables:
| Variable | Definition | Unit |
|---|---|---|
| Confirmed_Correct_Auto-Clears | Auto-cleared exceptions the independent reviewer confirms as correctly dispositioned | Count |
| Sampled_Auto-Clears | Total auto-cleared exceptions in the random sample (use √N or risk-based sampling plan, minimum 30 per period) | Count |
Proposed formula, pending validation pilots.