Definition
Percentage of NLP-classified Aviation Safety Action Program (ASAP) reports where the AI-assigned severity level and hazard category match the assessment of a qualified aviation safety officer upon independent expert review: measuring whether the NLP model is producing an accurate risk picture from safety narrative data.
The Measurement Gap
Safety management systems now process hundreds or thousands of ASAP voluntary safety reports per month. Manual review of every report by a qualified safety officer is impractical at scale. NLP models that automatically classify reports by severity and hazard category enable safety analytics at volume: identifying emerging trends, prioritizing investigation resources, and feeding regulatory reports. The accuracy of these classifications is the foundation on which all downstream safety analytics rest.
Traditional safety metrics measure report volume, investigation completion rates, and corrective action closure rates: process metrics that don't reveal whether the NLP model's risk classifications are reliable. An airline can have a fully functioning safety reporting program and still have a safety trend picture that is distorted by systematic NLP misclassification: classifying a flight control anomaly as minor because the crew used casual language in the narrative, or misrouting an airworthiness-relevant event into a routine operations category.
Decision supported:
Is the NLP safety classification model producing an accurate risk picture, or is it systematically misclassifying events in ways that distort safety trend analysis and suppress high-priority items from investigation queues? SCCR provides the human-validation feedback loop that the NLP model needs to detect when its accuracy has degraded: whether due to model drift, changes in reporting culture, or evolution of safety taxonomy. Without SCCR, safety leadership has no mechanism to know whether the AI's risk picture is reliable.
Formula
Plain language: Of the ASAP reports that a qualified aviation safety officer classifies as high-severity, what fraction did the AI classify at a lower severity? That is the number that matters for safety risk. Overall concordance is reported alongside it for model health, but the underclassification rate on safety-significant events is the headline metric.
Primary metric: Safety Event Underclassification Rate:
SCCR_FNR = Underclassified_safety_events / (Correctly_classified_safety_events + Underclassified_safety_events)
Secondary metric: Overall Safety Classification Concordance:
SCCR_OA = Concordant_classifications / Sampled_classifications
Variables:
| Variable | Definition | Unit |
|---|---|---|
| Underclassified_safety_events | Reports expert-assessed as high-severity that AI classified at a lower level (false negatives) | Count |
| Correctly_classified_safety_events | Reports expert-assessed as high-severity that AI correctly classified (true positives) | Count |
| Concordant_classifications | Reports where AI severity AND hazard category both match expert assessment | Count |
| Sampled_classifications | Total reports in the stratified sample | Count |
Minimum sample size: 50 reports per quarter, stratified by hazard category to ensure coverage of rare high-consequence categories. Use a panel of two expert reviewers for borderline cases; count as concordant if the AI matches either reviewer.
Why underclassification rate is primary:
In aviation safety reporting, overclassification creates review burden but ensures events receive attention. Underclassification means safety-critical ASAP reports bypass the scrutiny they require, risking missed airworthiness findings, unaddressed hazard trends, and FAA enforcement actions. The asymmetry is a consequence structure, not a preference. A 90% overall concordance that underclassifies 8% of safety-critical events is a safety liability, not acceptable performance.
Proposed formula, pending validation pilots.