Condition Report Significance Classification Accuracy (CRSCA)

GMI-NUC-005Nuclear OperationsPilot-Ready

Definition

Percentage of AI-classified condition reports where the significance level and regulatory reportability determination matched the conclusion reached in the subsequent engineering evaluation.

The Measurement Gap

The corrective action program is the backbone of nuclear safety culture, and condition report screening determines which issues get engineering evaluation and which are handled as routine. When AI screening assigns significance levels and reportability flags to incoming CRs, the plant needs to know whether those classifications are accurate. CAP trending metrics track CR volume, closure rates, and aging. NRC performance indicators measure broad safety outcomes. INPO metrics evaluate program health at the process level. None of them measure whether the AI screening function is correctly sorting CRs by significance and reportability. The invisible failure mode is that a safety-significant condition gets classified as routine and sits in the general queue while expedited resources focus on benign issues the AI flagged as significant, or worse, a reportable condition gets flagged as non-reportable and the plant misses a 10 CFR 50.72/50.73 reporting obligation. CR volume metrics look healthy while the classification quality degrades underneath.

Decision Supported

Is the AI condition report screening process routing CRs to the right queue, or are safety-significant conditions sitting in the routine pile while benign conditions consume expedited review resources?

Formula

Primary metric (under-classification rate on significance):

UCR_significance = Significance_underclassified / Total_high_significance

Where Significance_underclassified = CRs where AI assigned lower significance than engineering determined, and Total_high_significance = CRs confirmed as high-significance by engineering evaluation.

Secondary metric (composite accuracy):

CRSCA_significance = Correct_significance / N_evaluated
CRSCA_reportability = Correct_reportability / N_reportability_relevant
CRSCA = 0.6 x CRSCA_significance + 0.4 x CRSCA_reportability

Why under-classification rate on significance is primary: Over-classifying a condition report triggers unnecessary engineering resources but ensures the condition receives evaluation. Under-classifying a significant condition means degraded equipment or a reportable condition is screened to a lower review tier or screened out entirely. In a nuclear facility, the consequence is unreported conditions that may violate 10 CFR 50.72/50.73 reporting requirements. Set UCR_significance below 1%; CRSCA above 90%.

Variables:

  • Correct_significance = CRs where AI-assigned significance matched engineering-determined significance (count)
  • N_evaluated = CRs that completed full engineering evaluation in the period (count)
  • Correct_reportability = CRs where AI reportability flag matched licensing engineer's determination (count)
  • N_reportability_relevant = CRs where reportability was evaluated by licensing (count)
  • Composite weights: 0.6 significance / 0.4 reportability

Sign convention: 1.0 = perfect classification. Track under-classification and over-classification separately.

Aggregation period: Monthly rolling, with quarterly deep-dive by condition type.

Edge Cases

  • CRs where significance was upgraded due to information not available at screening: exclude if information genuinely was not available to AI; include if information was available but missed
  • CRs screened out at intake: highest-risk gap. Run a separate random audit of screened-out CRs
  • Minimum sample: 50 evaluated CRs per month

Proposed formula, pending validation pilots.

At a Glance

Unit
Percentage
Trend Direction
Toward 1.0
Reporting Frequency
Quarterly
Category
Regulatory Interpretation and Classification Accuracy
← Regulatory Fluency Atrophy Index (RFAI)NLP Regulatory Interpretation Accuracy Rate (NRIAR) →

Assess Your Organization

The AI-GPS Assessment evaluates your AI governance maturity and recommends KPIs like this one for your implementation roadmap.

Take the Free AI-GPS Assessment