Definition
Percentage of AI-classified condition reports where the significance level and regulatory reportability determination matched the conclusion reached in the subsequent engineering evaluation.
The Measurement Gap
The corrective action program is the backbone of nuclear safety culture, and condition report screening determines which issues get engineering evaluation and which are handled as routine. When AI screening assigns significance levels and reportability flags to incoming CRs, the plant needs to know whether those classifications are accurate. CAP trending metrics track CR volume, closure rates, and aging. NRC performance indicators measure broad safety outcomes. INPO metrics evaluate program health at the process level. None of them measure whether the AI screening function is correctly sorting CRs by significance and reportability. The invisible failure mode is that a safety-significant condition gets classified as routine and sits in the general queue while expedited resources focus on benign issues the AI flagged as significant, or worse, a reportable condition gets flagged as non-reportable and the plant misses a 10 CFR 50.72/50.73 reporting obligation. CR volume metrics look healthy while the classification quality degrades underneath.
Decision Supported
Is the AI condition report screening process routing CRs to the right queue, or are safety-significant conditions sitting in the routine pile while benign conditions consume expedited review resources?
Formula
Primary metric (under-classification rate on significance):
UCR_significance = Significance_underclassified / Total_high_significance
Where Significance_underclassified = CRs where AI assigned lower significance than engineering determined, and Total_high_significance = CRs confirmed as high-significance by engineering evaluation.
Secondary metric (composite accuracy):
CRSCA_significance = Correct_significance / N_evaluated
CRSCA_reportability = Correct_reportability / N_reportability_relevant
CRSCA = 0.6 x CRSCA_significance + 0.4 x CRSCA_reportability
Why under-classification rate on significance is primary: Over-classifying a condition report triggers unnecessary engineering resources but ensures the condition receives evaluation. Under-classifying a significant condition means degraded equipment or a reportable condition is screened to a lower review tier or screened out entirely. In a nuclear facility, the consequence is unreported conditions that may violate 10 CFR 50.72/50.73 reporting requirements. Set UCR_significance below 1%; CRSCA above 90%.
Variables:
- Correct_significance = CRs where AI-assigned significance matched engineering-determined significance (count)
- N_evaluated = CRs that completed full engineering evaluation in the period (count)
- Correct_reportability = CRs where AI reportability flag matched licensing engineer's determination (count)
- N_reportability_relevant = CRs where reportability was evaluated by licensing (count)
- Composite weights: 0.6 significance / 0.4 reportability
Sign convention: 1.0 = perfect classification. Track under-classification and over-classification separately.
Aggregation period: Monthly rolling, with quarterly deep-dive by condition type.
Edge Cases
- CRs where significance was upgraded due to information not available at screening: exclude if information genuinely was not available to AI; include if information was available but missed
- CRs screened out at intake: highest-risk gap. Run a separate random audit of screened-out CRs
- Minimum sample: 50 evaluated CRs per month
Proposed formula, pending validation pilots.