Definition
RECA measures the accuracy of AI-assisted regulatory event classification by comparing automated classifications against expert human review, separately for high-consequence Tier 1 events and routine events.
The Measurement Gap
Measurement Gap and Decision Supported
The Invisible Failure Mode: A combined water and wastewater utility deploys an AI-assisted event management system classifying operational incidents for regulatory reporting. Over 18 months, it classifies 1,247 events. Internal audit of 150 finds routine classification at 89% accuracy. But among 23 Tier 1 events (immediate notification required), 3 were misclassified: two Tier 1 events incorrectly classified as routine (delayed notification by 14 and 22 hours), and one routine event escalated to Tier 1. The 87% Tier 1 accuracy represents a 13% misclassification rate on the events where errors have the most severe consequences.
What Existing Metrics Miss:
Event management systems report total event counts and resolution times but not classification accuracy by consequence tier. A system correctly classifying 95% of all events may still misclassify 15% of Tier 1 events, invisible in aggregate accuracy.
The Decision RECA Enables:
Is AI-assisted event classification accurate enough for highest-consequence events without mandatory human review, or do Tier 1 classifications require human confirmation before notification timers begin?
Cross-References:
Misclassified events may produce incorrect DMR entries flagged by DMRPS (GMI-WT-028). Incomplete permit condition inventory (PCIC, GMI-WT-030) means the classifier may lack rules for certain reportable conditions. Poor alert quality (ASNR, GMI-WT-010) makes classification harder by increasing upstream noise.
Formula
Primary metric (false-negative rate on reportable events):
FNR_reportable = Reportable_classified_as_non_reportable / Total_actually_reportable
Secondary metric (overall accuracy):
RECA = Correctly_classified / Total_sampled
Why false-negative rate on reportable events is primary: Classifying a non-reportable event as reportable triggers an unnecessary notification but causes no regulatory harm. Failing to classify a reportable event means a missed SDWA or state notification, creating enforcement exposure, public health risk from unreported violations, and potential consent decree liability. Set FNR_reportable below 1%; overall RECA above 95%.
Plain language: Sample regulatory events classified by the AI. Have a compliance officer independently classify against current SDWA and state reporting requirements. Compare AI classification to expert classification.
Decomposition:
False_positive_rate = Non_reportable_classified_as_reportable / Total_actually_non_reportable
Where:
- Correctly_classified = AI classification matches expert compliance officer classification
- Total_sampled = audited events in the measurement period
- Total_actually_reportable = events confirmed reportable by expert review
Units: Percentage.
Proposed formula, pending validation pilots.