Definition
Agreement rate between NLP-assigned near-miss severity classifications and expert safety officer classifications of the same events, measuring whether NLP is correctly identifying precursor events that warrant investigation.
The Measurement Gap
Why This Metric Didn't Exist Before
Before NLP classification, near-miss reports were read and classified by safety officers. The classification quality depended on the individual officer's experience and judgment, but every report got human eyes. When NLP took over initial classification, it introduced speed and consistency, and also introduced a new failure mode that nobody was measuring.
NLP classifiers are trained on historical data. If the training data contains systematic underclassification (safety officers in a previous era who minimized certain event types), the NLP learns that bias and applies it at scale. Worse, NLP struggles with the specialized vocabulary of railroad safety reporting: experienced engineers use shorthand, euphemisms, and technical jargon that general NLP models interpret poorly.
Here's the decision NMSCA supports: Is the NLP severity classifier identifying the near-miss events that predict future accidents, or is it systematically underclassifying events and burying precursors? The directional error matters enormously: underclassification (NLP says low severity, expert says high) is far more dangerous than overclassification (NLP says high, expert says low). Overclassification wastes investigation resources. Underclassification buries the warnings that precede catastrophic events.
Without NMSCA, you trust the NLP's classifications uniformly. You don't know which event categories it handles well and which it misclassifies. The buried precursors stay buried until the accident they predicted actually happens.
Formula
Plain language: Of the near-miss reports that a safety officer classifies as high-severity, what fraction did the NLP classify at a lower severity? That underclassification rate is the primary metric: it represents buried precursors. Overall classification accuracy is secondary.
Primary metric: High-Severity Underclassification Rate (False-Negative Rate):
NMSCA_FNR = Expert_high_NLP_low / (Correctly_classified_high + Expert_high_NLP_low)
Secondary metric: Overall Classification Accuracy:
NMSCA_OA = Correctly_classified / Total_sampled
Variables:
| Variable | Definition | Unit |
|---|---|---|
| Expert_high_NLP_low | Near-miss events the safety officer rated high-severity but NLP rated low (false negatives: buried precursors) | Count |
| Correctly_classified_high | Near-miss events the safety officer rated high-severity and NLP agreed (true positives) | Count |
| Correctly_classified | All near-miss reports where NLP severity matched expert severity | Count |
| Total_sampled | Total near-miss reports in the audit sample | Count |
| Expert_low_NLP_high | Events the expert rated low-severity but NLP rated high (overclassification: conservative direction) | Count |
Expert disagreement: use a panel of 2+ safety officers for ambiguous cases; count NLP agreement with either as correct. Track accuracy by reporter since experienced versus new employees use different terminology. Minimum 50 near-miss reports per audit, quarterly.
Why underclassification rate is primary:
In railroad safety, overclassification creates additional investigation labor but ensures precursors receive attention. Underclassification means high-severity near-miss events are buried in the low-severity queue, making accident precursor signals invisible to FRA safety trend analysis. The asymmetry is a consequence structure, not a preference. A 90% overall accuracy that underclassifies 12% of high-severity near-misses is burying the signals that prevent derailments and collisions.
Proposed formula, pending validation pilots.