Operating Experience Relevance Accuracy (OERA)

GMI-NUC-014Nuclear OperationsIntegration-Required

Definition

Percentage of AI-surfaced operating experience events that the plant's OE review board determined were genuinely consequence-relevant to the plant's specific configuration, systems, and operating conditions: as opposed to keyword-similar but configuration-irrelevant events.

The Measurement Gap

Nuclear plant operating experience (OE) programs screen thousands of industry events per year to identify those relevant to the plant's specific configuration, systems, and operating conditions. When AI systems take over this screening function, the plant needs to know whether the AI surfaces the right events. No legacy metric measures OE screening quality at the event level. OE program metrics track volume (events reviewed, actions generated) and timeliness (response time to OE notifications). INPO evaluation criteria assess the OE program's effectiveness in broad terms. None of them measure whether the specific events the AI surfaces are genuinely consequence-relevant to the plant or whether the AI misses events that use different technical vocabulary to describe the same failure mechanism. The invisible failure mode is that the AI's keyword-based relevance scoring surfaces events that look similar (same component names, same system designations) while missing events that describe the same failure mechanism using different vocabulary, and the OE review board reviews an increasingly irrelevant event stream while truly relevant events go unexamined.

Decision Supported

Is the AI OE screening system surfacing the events that matter for this plant's specific vulnerability profile, or is it generating review workload based on superficial keyword matching while missing configuration-relevant events with different vocabulary?

Formula

Plain language: Of all OE events the AI flagged for plant-specific review, count how many the OE review board determined were genuinely applicable to this plant's configuration. Also audit a sample of events the AI did NOT flag to check for misses (events that were relevant but weren't surfaced).

Notation:
OERA_precision = Relevant_surfaced / Total_surfaced
OERA_recall = Relevant_surfaced / (Relevant_surfaced + Missed_relevant)

Composite:
OERA = 2 × (OERA_precision × OERA_recall) / (OERA_precision + OERA_recall)
(F1 harmonic mean: balances precision and recall equally)

Variables:

  • Relevant_surfaced = AI-surfaced OE events confirmed as genuinely applicable to this plant by the OE review board (count)
  • Total_surfaced = all OE events AI flagged for plant review (count)
  • Missed_relevant = OE events NOT surfaced by AI that the audit found to be genuinely applicable (count: measured by auditing a random sample of non-surfaced events)
  • OERA is the F1 score, balancing the "too much noise" problem (low precision) with the "missed relevant events" problem (low recall)

Sign convention: OERA = 1.0 means perfect relevance accuracy. Lower is worse. Precision and recall should be tracked separately because they have different operational consequences.

Aggregation period: Quarterly (aligned with OE review board cycles).

Edge Cases

  • Events with different technical vocabulary describing the same failure mechanism: these are the AI's hardest challenge. Track as a sub-population ("vocabulary-variant misses") to specifically measure the failure mode identified in Phase 1
  • Events from different reactor types (PWR event flagged for BWR plant, or vice versa): tag as a specific irrelevance category. Configuration mismatch is the most common false-positive pattern
  • Minimum sample for recall audit: 50 non-surfaced events per quarter reviewed by the OE coordinator

Proposed formula, pending validation pilots.

At a Glance

Unit
Percentage
Trend Direction
Toward 1.0
Reporting Frequency
Quarterly
Category
Data Fidelity and Information Quality
← Cross-Boundary Data Fidelity Index (CBDFI)AI Verification Thoroughness Ratio (AVTR) →

Assess Your Organization

The AI-GPS Assessment evaluates your AI governance maturity and recommends KPIs like this one for your implementation roadmap.

Take the Free AI-GPS Assessment