Definition
Accuracy of NLP regulatory change classification, with the false-informational rate on action-required changes as the primary measure and overall accuracy secondary. Missing an action-required change is the failure that carries consequences.
The Measurement Gap
The NLP regulatory intelligence system ingests 1,200 regulatory changes per quarter and reports 94% classification accuracy. But that 94% includes a massive category of easy-to-classify informational items (meeting announcements, workshop schedules) that represent 60% of volume. Strip those out, and accuracy on action-required items drops to 83%. Strip out the straightforward action-required items, and accuracy on ambiguous high-impact changes drops to 71%.
A 29% error rate on high-impact changes means the team misses roughly one in three regulatory changes requiring a substantive response. Some are false negatives: action-required changes classified as informational that nobody reads. One such miss was an EMA guideline revision on nitrosamine control that affected 6 products across 3 sites, discovered 7 weeks late.
Decision enabled:
When the false-informational rate exceeds 5%, manual review of all NLP "informational" classifications is required. Above 10%, the NLP system is unreliable for triage and must be supplemented with expert review of all high-impact regulatory changes.
Formula
Primary metric (false-informational rate):
FIR = action_required_classified_as_informational / total_action_required_in_sample
Secondary metric (overall accuracy):
RCCA = correctly_classified_changes / sampled_changes
Why false-informational rate is primary: Classifying an informational change as action-required wastes reviewer time but ensures the change gets evaluated. Classifying an action-required regulatory change as informational means nobody reads a document that required a response, creating compliance gaps, missed filing deadlines, and potential enforcement action. Set FIR below 5%; overall RCCA above 90%.
Error direction decomposition:
- false_action_rate = informational_classified_as_action_required / total_informational_in_sample
Worked Example:
Q2 validation sample: 83 regulatory changes reviewed by senior regulatory affairs specialists.
Correctly classified: 72 of 83. RCCA = 72/83 = 86.7%.
Of 11 misclassifications: 7 were false-action (wasteful but safe). 4 were false-informational (dangerous).
Total action-required in sample: 31. FIR = 4/31 = 12.9%.
One in eight action-required changes classified as informational and potentially ignored. For a company processing 300+ action-required changes per year, that projects to 38 missed changes annually.
Sign Convention:
Higher RCCA is better. FIR above 10% means the NLP is unreliable for triage.
Proposed formula, pending validation pilots.