Definition
Measures the accuracy rate of AI or NLP classification of quality events into regulatory versus non-regulatory categories.
The Measurement Gap
As manufacturing organizations deploy AI and NLP systems to process quality events, nonconformance reports, and incident notifications, these systems increasingly make the initial classification decision that determines whether an event enters the regulatory reporting pathway or is processed as a routine quality issue. The accuracy of this classification has direct regulatory consequences. A false negative (reportable event classified as routine) means the organization fails to report an event that regulations require, which can result in enforcement actions, fines, and loss of certifications. A false positive (routine event classified as reportable) wastes compliance resources but creates no regulatory risk.
No existing quality metric specifically measures the accuracy of this AI classification step. Event management systems track event counts and resolution times, but nobody systematically compares AI classifications to ground truth regulatory criteria. RECA fills this gap by sampling AI-classified events, having the compliance team independently classify them against regulatory criteria, and computing the agreement rate. This creates a validation feedback loop that detects classification drift before it results in missed regulatory reports.
Formula
Plain language: Of the events that compliance reviewers confirm are regulatory-reportable, what fraction did the AI miss? That regulatory false-negative rate is the headline metric. Overall classification accuracy is reported alongside it for model health, but the false-negative rate on reportable events is the number that matters for compliance risk.
Primary metric: Regulatory False-Negative Rate:
RECA_FNR = Missed_regulatory / Total_regulatory_events_sampled
Secondary metric: Overall Classification Accuracy:
RECA_OA = Correctly_classified_regulatory / Total_regulatory_events_sampled × 100
Variables:
| Variable | Definition | Unit |
|---|---|---|
| Missed_regulatory | Events confirmed as regulatory-reportable that the AI classified as non-reportable (false negatives) | Count |
| Total_regulatory_events_sampled | Total confirmed regulatory events in the verification sample | Count |
| Correctly_classified_regulatory | Events the AI correctly classified as reportable, confirmed by compliance review | Count |
The manufacturing variant applies to quality events with FDA MDR, OSHA recordable, and EPA exceedance significance. Focus verification on events near the classification boundary. Minimum 50 events per verification period, stratified by category, severity, and time period.
Why false-negative rate is primary:
In manufacturing, false positives (flagging routine events as reportable) create investigation labor. False negatives (missing actual reportable events) create unreported FDA MDR filings, OSHA recordable omissions, and EPA exceedance violations with personal liability for EHS managers. The asymmetry is a consequence structure, not a preference. A 96% overall accuracy that misses 10% of OSHA-recordable incidents is a regulatory liability, not a success.
Proposed formula, pending validation pilots.