Regulatory Change Classification Accuracy (RCCA)

GMI-PH-038Pharmaceutical OperationsIntegration-Required

Definition

Accuracy of NLP regulatory change classification, with the false-informational rate on action-required changes as the primary measure and overall accuracy secondary. Missing an action-required change is the failure that carries consequences.

The Measurement Gap

The NLP regulatory intelligence system ingests 1,200 regulatory changes per quarter and reports 94% classification accuracy. But that 94% includes a massive category of easy-to-classify informational items (meeting announcements, workshop schedules) that represent 60% of volume. Strip those out, and accuracy on action-required items drops to 83%. Strip out the straightforward action-required items, and accuracy on ambiguous high-impact changes drops to 71%.

A 29% error rate on high-impact changes means the team misses roughly one in three regulatory changes requiring a substantive response. Some are false negatives: action-required changes classified as informational that nobody reads. One such miss was an EMA guideline revision on nitrosamine control that affected 6 products across 3 sites, discovered 7 weeks late.

Decision enabled:

When the false-informational rate exceeds 5%, manual review of all NLP "informational" classifications is required. Above 10%, the NLP system is unreliable for triage and must be supplemented with expert review of all high-impact regulatory changes.

Formula

Primary metric (false-informational rate):

FIR = action_required_classified_as_informational / total_action_required_in_sample

Secondary metric (overall accuracy):

RCCA = correctly_classified_changes / sampled_changes

Why false-informational rate is primary: Classifying an informational change as action-required wastes reviewer time but ensures the change gets evaluated. Classifying an action-required regulatory change as informational means nobody reads a document that required a response, creating compliance gaps, missed filing deadlines, and potential enforcement action. Set FIR below 5%; overall RCCA above 90%.

Error direction decomposition:

  • false_action_rate = informational_classified_as_action_required / total_informational_in_sample

Worked Example:

Q2 validation sample: 83 regulatory changes reviewed by senior regulatory affairs specialists.

Correctly classified: 72 of 83. RCCA = 72/83 = 86.7%.

Of 11 misclassifications: 7 were false-action (wasteful but safe). 4 were false-informational (dangerous).

Total action-required in sample: 31. FIR = 4/31 = 12.9%.

One in eight action-required changes classified as informational and potentially ignored. For a company processing 300+ action-required changes per year, that projects to 38 missed changes annually.

Sign Convention:

Higher RCCA is better. FIR above 10% means the NLP is unreliable for triage.

Proposed formula, pending validation pilots.

At a Glance

Unit
Percentage
Trend Direction
Higher
Reporting Frequency
Quarterly; stratified by change criticality
Category
Regulatory Intelligence & Compliance
← Stability Prediction Retrospective Accuracy (SPRA)Regulatory Deadline Margin Index (RDMI) →

Assess Your Organization

The AI-GPS Assessment evaluates your AI governance maturity and recommends KPIs like this one for your implementation roadmap.

Take the Free AI-GPS Assessment