Definition
The regulatory false-negative rate of NLP event classification (the fraction of genuinely regulatory-significant events that NLP misclassifies as routine) reported as the primary compliance risk metric, with overall classification accuracy as a secondary diagnostic.
The Measurement Gap
Legacy MRO supply chain metrics do not measure whether NLP event classification is accurately distinguishing regulatory-significant events from routine ones. Event processing volume measures throughput. Average response time measures speed. Neither measures classification accuracy: whether the NLP correctly identified a vendor quality notification as a reportable event versus a routine quality note, or whether a receipt discrepancy with regulatory implications was classified as a compliance event versus a routine variance. False-negatives on regulatory events are far more costly than false-positives: a missed reportable event creates regulatory exposure; a false alarm on a routine event creates only investigation labor. Rare regulatory events are the highest-consequence classification target and the hardest for NLP to learn: standard random sampling will not capture enough of them for stable accuracy measurement. The decision RENCA supports: whether NLP classifications can route events automatically or must be human-reviewed by regulatory class, and whether NLP needs targeted training on rare regulatory event types.
Formula
Plain language: Of the events that a qualified human reviewer determines are regulatory-significant, what fraction did the NLP miss? That is the number that matters for compliance risk. Overall accuracy is reported alongside it for model health, but the false-negative rate on regulatory events is the headline metric.
Primary metric: Regulatory False-Negative Rate:
RENCA_FNR = FN_regulatory / (TP_regulatory + FN_regulatory)
Secondary metric: Overall Classification Accuracy:
RENCA_OA = correctly_classified / sampled_events
Variables:
| Variable | Definition | Unit |
|---|---|---|
| FN_regulatory | Regulatory-significant events the NLP classified as routine (false negatives) | Count |
| TP_regulatory | Regulatory-significant events the NLP correctly classified (true positives) | Count |
| correctly_classified | All events (regulatory + routine) where NLP matches ground truth | Count |
| sampled_events | Audit sample, stratified to oversample rare regulatory event types | Count |
Worked example: Audit sample of 400 events, stratified to include 80 known regulatory events and 320 routine events. NLP correctly classified 374 overall. RENCA_OA = 374/400 = 93.5%. Of the 80 regulatory events, NLP missed 4 (classified as routine). RENCA_FNR = 4/80 = 5.0%. The headline: 1 in 20 reportable events will be missed. Overall accuracy looks acceptable; the false-negative rate is the operational concern.
Why false-negative rate is primary:
In regulated environments, false positives (overcounting regulatory events) create investigation labor. False negatives (missing regulatory events) create unreported violations, audit findings, fines, and personal liability. The asymmetry is not a preference: it is a consequence structure. A 96% overall accuracy rate that misses 15% of environmental release events is a regulatory liability, not a success.
Proposed formula, pending validation pilots.