Definition
Percentage of NLP-determined AD/SB applicability classifications that match expert engineering assessment: specifically measuring the rate at which the model correctly classifies airworthiness directives and service bulletins as applicable or not applicable to each aircraft in the fleet.
The Measurement Gap
The measurement gap ADCR addresses: airlines using NLP to automate AD/SB applicability determination track overall system throughput (how many directives processed per day, how many flagged for engineering review) but not whether the "not applicable" determinations are correct. The determination that an AD does not apply to a specific aircraft is the one that ends scrutiny. If that determination is wrong, the error passes undetected until an audit, an accident investigation, or an FAA surveillance finding brings it to light.
NLP applicability determination fails at scale because AD/SB applicability logic is conditional and context-dependent. A directive may be "applicable if aircraft is equipped with modification X AND serial number is in range Y AND engine variant is Z." Getting one condition wrong produces a wrong answer. Complex conditional applicability is where NLP most frequently fails, and it's exactly where the regulatory stakes are highest.
The decision ADCR supports: can the engineering team rely on NLP "not applicable" determinations, or must every determination be verified by a human engineer? At what concordance rate does the NLP provide productivity value versus liability exposure? If ADCR is 94%, the team is comfortable with the system's applicability work on simple directives but knows it needs human review for complex conditional applicability. If ADCR is 87%, the NLP is making too many errors to be trusted for unsupervised applicability determination. ADCR makes this decision calculable rather than intuitive.
Formula
Plain language: Sample AD/SB applicability determinations made by the NLP model. Have a qualified engineer independently assess applicability for the same aircraft-directive pairs. Compare AI to expert on two outcomes: applicable or not applicable.
Primary notation:
ADCR = Concordant_determinations / Sampled_determinations
Directional error rates (computed alongside ADCR):
False_not_applicable_rate = AI_says_not_applicable_but_is / Sampled_determinations
False_applicable_rate = AI_says_applicable_but_isnt / Sampled_determinations
Variables:
- Concordant_determinations = sampled AD/SB-aircraft pairs where AI applicability matches expert assessment (count)
- Sampled_determinations = total sampled AD/SB-aircraft pairs (count)
- False_not_applicable_rate = the safety-critical direction: AI says "not applicable" but the directive IS applicable
Units: Percentage. Track overall concordance and both directional error rates separately. The False_not_applicable_rate is the regulatory compliance metric; False_applicable_rate is the efficiency metric.
Stratification requirements:
By applicability complexity: simple applicability (model applies to all aircraft of this type) versus complex conditional applicability (multiple conditions required). ADCR will be naturally higher for simple applicability and lower for complex conditional. This stratification reveals where the NLP fails and what the acceptable use boundary is.
Minimum sample: 50 AD/SB-aircraft pairs per quarter, stratified to include both simple and complex applicability logic. If complex-conditional applicability represents 20% of the directive workload, it should represent at least 20% of the sample.
Proposed formula, pending validation pilots.