Definition
RDCA measures the percentage of AI-classified regulatory deadlines where the assigned timeline category (routine, expedited, or conditional) matches the regulatory body's actual requirement for that specific event type. This metric targets a critical vulnerability in AI-assisted compliance management: NLP systems that default to "routine" classification for rare or unfamiliar event types, creating invisible compliance violations that surface only when deadlines pass.
The Measurement Gap
The Measurement Gap
Renewable energy compliance involves multiple regulatory bodies with overlapping and sometimes conflicting deadline structures. NERC reliability standards, FERC interconnection requirements, state public utility commission filings, and environmental permit renewals each carry their own timeline categories and urgency classifications. When AI scheduling systems process these deadlines, they rely on NLP classification to assign the correct timeline category. The problem is that NLP models trained on historical data perform well on frequently occurring event types but struggle with rare or novel regulatory requirements.
Without RDCA, organizations discover classification errors only after a deadline has been missed. A NERC standard that requires expedited response gets classified as routine because the NLP model has seen few examples of that specific event type. The filing goes out on a routine timeline, arrives late, and triggers a compliance investigation. The NLP system showed no warning because it was confident in its classification. RDCA provides the feedback loop that catches these errors before they become violations.
The Decision This Metric Supports
RDCA answers a specific operational question: is the NLP system correctly identifying which regulatory filings require expedited timelines, or is it defaulting to "routine" for rare event types and creating compliance violations? When RDCA is high and stable across all timeline categories, the compliance team can trust the AI classification for scheduling purposes. When RDCA drops for any specific category, particularly expedited or conditional timelines, the metric signals that human review must be reinstated for those categories until the model is retrained. This is not an efficiency metric. It is a compliance risk metric where the cost of misclassification is measured in regulatory penalties, audit findings, and potential operating restrictions.
Formula
Primary metric (misclassification rate on expedited/conditional filings):
MCR_expedited = Misclassified_expedited / Total_actually_expedited
Secondary metric (overall accuracy):
RDCA = Correctly_Classified / Total_Sampled x 100
Why misclassification rate on expedited filings is primary: Classifying a routine filing as expedited triggers premature preparation effort but ensures the filing receives attention. Classifying an expedited filing as routine means the organization misses a time-sensitive regulatory deadline, creating NERC/FERC enforcement exposure and potential penalty assessment. Set MCR_expedited below 3%; overall RDCA above 95% for routine, above 85% for expedited/conditional.
| Variable | Definition |
|---|---|
| Correctly_Classified | Regulatory events where NLP-assigned timeline category matched actual requirement per compliance specialist |
| Total_Sampled | All regulatory events reviewed in the measurement period |
| Misclassified_expedited | Expedited/conditional filings NLP classified as routine |
| Total_actually_expedited | All filings confirmed as expedited/conditional by compliance specialist |
Measurement frequency: Quarterly, minimum 30 classified events per quarter.
Segmentation requirements: Track RDCA separately by timeline category (routine, expedited, conditional) and by regulatory body (NERC, FERC, state PUC, environmental). Aggregate RDCA masks the most dangerous errors because high accuracy on routine filings hides poor accuracy on expedited filings.
Edge cases:
- Conditional timelines where the deadline depends on triggering event characteristics: track separately, expect lower accuracy during initial deployment
- New regulatory requirements not in training data: track "novel regulation" rate as a leading indicator
- Regulatory bodies that change timeline requirements without updating formal guidance: verification must reference actual current practice
Proposed formula, pending validation pilots.