Definition
Percentage of NLP-generated regulatory interpretations that match the conclusion an independent qualified licensing engineer would reach when evaluating the same regulatory text against the same plant conditions.
The Measurement Gap
Nuclear plants adopting NLP tools for regulatory interpretation assistance face an accuracy question that no existing metric addresses. When an NLP system analyzes Technical Specification language against plant conditions and concludes that an operability determination or 50.59 screening yields a specific result, the plant needs to know whether that conclusion is reliable. Human performance error rate tracks human mistakes in procedure execution. Condition report volume tracks the corrective action program throughput. NRC inspection findings measure regulatory compliance outcomes. None of these metrics measure whether an NLP tool's regulatory interpretation matches what a qualified licensing engineer would conclude from the same text and conditions. The invisible failure mode is that the NLP produces confident regulatory interpretations that are subtly wrong, particularly on ambiguous text, novel plant configurations, or recently amended licensing basis sections, and no metric distinguishes NLP-assisted interpretations that happen to be correct from those that are confidently incorrect.
Decision Supported
Can the plant trust NLP-assisted Technical Specification interpretation for operability determinations, LCO entry/exit decisions, and 50.59 screening, or does every NLP output require full independent verification?
Formula
Plain language: Sample NLP interpretations from recent regulatory analysis work. Have a qualified licensing engineer independently evaluate the same regulatory text against the same plant conditions without seeing the NLP output. Compare conclusions. Score each NLP interpretation as Correct (same conclusion and reasoning), Directionally Correct (same conclusion but reasoning gaps), or Incorrect (different conclusion).
Notation:
NRIAR = (Correct + 0.5 × Directionally_Correct) / N_sampled
Variables:
- Correct = NLP interpretations where the conclusion and reasoning both match the independent human evaluation (count)
- Directionally_Correct = NLP interpretations where the conclusion matches but the reasoning is incomplete or cites the wrong regulatory basis (count)
- N_sampled = total NLP interpretations sampled in the measurement period (count)
- The 0.5 weight on Directionally_Correct reflects that right-answer-wrong-reason is dangerous in nuclear licensing, a correct conclusion reached through flawed reasoning will produce incorrect conclusions when applied to a slightly different scenario
Sign convention: 1.0 = perfect accuracy. Lower is worse. The threshold for acceptable accuracy in nuclear regulatory interpretation is much higher than in non-regulated domains.
Aggregation period: Quarterly, with stratification by interpretation type (operability, reportability, 50.59, Tech Spec applicability).
Edge Cases
- Ambiguous regulatory text where two qualified engineers might reasonably disagree: use a three-engineer panel for the benchmark. Score as "ambiguous" if the panel itself splits, and exclude from NRIAR denominator
- NLP flags a potential issue that the human evaluator dismisses as clearly inapplicable: this is a false positive, scored as Incorrect, but track false-positive vs. false-negative rates separately: in nuclear, false negatives (NLP misses a real issue) are far more dangerous than false positives (NLP flags something unnecessarily)
- Minimum sample: 30 interpretations per quarter per interpretation type
Proposed formula, pending validation pilots.