Regulatory Event Classification Accuracy (RECA-WT)

GMI-WT-029Water TreatmentIntegration-RequiredPattern R-2: NLP Classification

Definition

RECA measures the accuracy of AI-assisted regulatory event classification by comparing automated classifications against expert human review, separately for high-consequence Tier 1 events and routine events.

The Measurement Gap

Measurement Gap and Decision Supported

The Invisible Failure Mode: A combined water and wastewater utility deploys an AI-assisted event management system classifying operational incidents for regulatory reporting. Over 18 months, it classifies 1,247 events. Internal audit of 150 finds routine classification at 89% accuracy. But among 23 Tier 1 events (immediate notification required), 3 were misclassified: two Tier 1 events incorrectly classified as routine (delayed notification by 14 and 22 hours), and one routine event escalated to Tier 1. The 87% Tier 1 accuracy represents a 13% misclassification rate on the events where errors have the most severe consequences.

What Existing Metrics Miss:

Event management systems report total event counts and resolution times but not classification accuracy by consequence tier. A system correctly classifying 95% of all events may still misclassify 15% of Tier 1 events, invisible in aggregate accuracy.

The Decision RECA Enables:

Is AI-assisted event classification accurate enough for highest-consequence events without mandatory human review, or do Tier 1 classifications require human confirmation before notification timers begin?

Cross-References:

Misclassified events may produce incorrect DMR entries flagged by DMRPS (GMI-WT-028). Incomplete permit condition inventory (PCIC, GMI-WT-030) means the classifier may lack rules for certain reportable conditions. Poor alert quality (ASNR, GMI-WT-010) makes classification harder by increasing upstream noise.

Formula

Primary metric (false-negative rate on reportable events):

FNR_reportable = Reportable_classified_as_non_reportable / Total_actually_reportable

Secondary metric (overall accuracy):

RECA = Correctly_classified / Total_sampled

Why false-negative rate on reportable events is primary: Classifying a non-reportable event as reportable triggers an unnecessary notification but causes no regulatory harm. Failing to classify a reportable event means a missed SDWA or state notification, creating enforcement exposure, public health risk from unreported violations, and potential consent decree liability. Set FNR_reportable below 1%; overall RECA above 95%.

Plain language: Sample regulatory events classified by the AI. Have a compliance officer independently classify against current SDWA and state reporting requirements. Compare AI classification to expert classification.

Decomposition:
False_positive_rate = Non_reportable_classified_as_reportable / Total_actually_non_reportable

Where:

  • Correctly_classified = AI classification matches expert compliance officer classification
  • Total_sampled = audited events in the measurement period
  • Total_actually_reportable = events confirmed reportable by expert review

Units: Percentage.

Proposed formula, pending validation pilots.

At a Glance

Category
Regulatory Compliance AI Integrity
← Permit Condition Inventory Completeness (PCIC)Disinfection Byproduct Precursor Model Currency Index (DPMCI) →

Assess Your Organization

The AI-GPS Assessment evaluates your AI governance maturity and recommends KPIs like this one for your implementation roadmap.

Take the Free AI-GPS Assessment