Near-Miss Severity Classification Accuracy (NMSCA)

GMI-RAIL-045Rail TransportationIntegration-RequiredPattern R-2: NLP Classification

Definition

Agreement rate between NLP-assigned near-miss severity classifications and expert safety officer classifications of the same events, measuring whether NLP is correctly identifying precursor events that warrant investigation.

The Measurement Gap

Why This Metric Didn't Exist Before

Before NLP classification, near-miss reports were read and classified by safety officers. The classification quality depended on the individual officer's experience and judgment, but every report got human eyes. When NLP took over initial classification, it introduced speed and consistency, and also introduced a new failure mode that nobody was measuring.

NLP classifiers are trained on historical data. If the training data contains systematic underclassification (safety officers in a previous era who minimized certain event types), the NLP learns that bias and applies it at scale. Worse, NLP struggles with the specialized vocabulary of railroad safety reporting: experienced engineers use shorthand, euphemisms, and technical jargon that general NLP models interpret poorly.

Here's the decision NMSCA supports: Is the NLP severity classifier identifying the near-miss events that predict future accidents, or is it systematically underclassifying events and burying precursors? The directional error matters enormously: underclassification (NLP says low severity, expert says high) is far more dangerous than overclassification (NLP says high, expert says low). Overclassification wastes investigation resources. Underclassification buries the warnings that precede catastrophic events.

Without NMSCA, you trust the NLP's classifications uniformly. You don't know which event categories it handles well and which it misclassifies. The buried precursors stay buried until the accident they predicted actually happens.

Formula

Plain language: Of the near-miss reports that a safety officer classifies as high-severity, what fraction did the NLP classify at a lower severity? That underclassification rate is the primary metric: it represents buried precursors. Overall classification accuracy is secondary.

Primary metric: High-Severity Underclassification Rate (False-Negative Rate):

NMSCA_FNR = Expert_high_NLP_low / (Correctly_classified_high + Expert_high_NLP_low)

Secondary metric: Overall Classification Accuracy:

NMSCA_OA = Correctly_classified / Total_sampled

Variables:

Variable Definition Unit
Expert_high_NLP_low Near-miss events the safety officer rated high-severity but NLP rated low (false negatives: buried precursors) Count
Correctly_classified_high Near-miss events the safety officer rated high-severity and NLP agreed (true positives) Count
Correctly_classified All near-miss reports where NLP severity matched expert severity Count
Total_sampled Total near-miss reports in the audit sample Count
Expert_low_NLP_high Events the expert rated low-severity but NLP rated high (overclassification: conservative direction) Count

Expert disagreement: use a panel of 2+ safety officers for ambiguous cases; count NLP agreement with either as correct. Track accuracy by reporter since experienced versus new employees use different terminology. Minimum 50 near-miss reports per audit, quarterly.

Why underclassification rate is primary:

In railroad safety, overclassification creates additional investigation labor but ensures precursors receive attention. Underclassification means high-severity near-miss events are buried in the low-severity queue, making accident precursor signals invisible to FRA safety trend analysis. The asymmetry is a consequence structure, not a preference. A 90% overall accuracy that underclassifies 12% of high-severity near-misses is burying the signals that prevent derailments and collisions.

Proposed formula, pending validation pilots.

At a Glance

Category
Passenger Experience and Safety Analytics Integrity
← Grade Crossing Risk Currency Index (GCRCI)Schedule Resilience Under Disruption Index (SRUDI) →

Assess Your Organization

The AI-GPS Assessment evaluates your AI governance maturity and recommends KPIs like this one for your implementation roadmap.

Take the Free AI-GPS Assessment