Definition
Percentage of AI-surfaced operating experience events that the plant's OE review board determined were genuinely consequence-relevant to the plant's specific configuration, systems, and operating conditions: as opposed to keyword-similar but configuration-irrelevant events.
The Measurement Gap
Nuclear plant operating experience (OE) programs screen thousands of industry events per year to identify those relevant to the plant's specific configuration, systems, and operating conditions. When AI systems take over this screening function, the plant needs to know whether the AI surfaces the right events. No legacy metric measures OE screening quality at the event level. OE program metrics track volume (events reviewed, actions generated) and timeliness (response time to OE notifications). INPO evaluation criteria assess the OE program's effectiveness in broad terms. None of them measure whether the specific events the AI surfaces are genuinely consequence-relevant to the plant or whether the AI misses events that use different technical vocabulary to describe the same failure mechanism. The invisible failure mode is that the AI's keyword-based relevance scoring surfaces events that look similar (same component names, same system designations) while missing events that describe the same failure mechanism using different vocabulary, and the OE review board reviews an increasingly irrelevant event stream while truly relevant events go unexamined.
Decision Supported
Is the AI OE screening system surfacing the events that matter for this plant's specific vulnerability profile, or is it generating review workload based on superficial keyword matching while missing configuration-relevant events with different vocabulary?
Formula
Plain language: Of all OE events the AI flagged for plant-specific review, count how many the OE review board determined were genuinely applicable to this plant's configuration. Also audit a sample of events the AI did NOT flag to check for misses (events that were relevant but weren't surfaced).
Notation:
OERA_precision = Relevant_surfaced / Total_surfaced
OERA_recall = Relevant_surfaced / (Relevant_surfaced + Missed_relevant)
Composite:
OERA = 2 × (OERA_precision × OERA_recall) / (OERA_precision + OERA_recall)
(F1 harmonic mean: balances precision and recall equally)
Variables:
- Relevant_surfaced = AI-surfaced OE events confirmed as genuinely applicable to this plant by the OE review board (count)
- Total_surfaced = all OE events AI flagged for plant review (count)
- Missed_relevant = OE events NOT surfaced by AI that the audit found to be genuinely applicable (count: measured by auditing a random sample of non-surfaced events)
- OERA is the F1 score, balancing the "too much noise" problem (low precision) with the "missed relevant events" problem (low recall)
Sign convention: OERA = 1.0 means perfect relevance accuracy. Lower is worse. Precision and recall should be tracked separately because they have different operational consequences.
Aggregation period: Quarterly (aligned with OE review board cycles).
Edge Cases
- Events with different technical vocabulary describing the same failure mechanism: these are the AI's hardest challenge. Track as a sub-population ("vocabulary-variant misses") to specifically measure the failure mode identified in Phase 1
- Events from different reactor types (PWR event flagged for BWR plant, or vice versa): tag as a specific irrelevance category. Configuration mismatch is the most common false-positive pattern
- Minimum sample for recall audit: 50 non-surfaced events per quarter reviewed by the OE coordinator
Proposed formula, pending validation pilots.