Execution Alert Signal-to-Noise Ratio (EASNR)

GMI-NUC-026Nuclear OperationsIntegration-Required

Definition

Ratio of execution monitoring alerts that identified a genuine deviation requiring intervention to total alerts generated by the digital work execution system, measuring the actionable signal content of the alert stream.

The Measurement Gap

Digital work execution monitoring systems generate alerts when field execution deviates from the planned work package: steps performed out of sequence, measurement values outside expected ranges, time durations exceeding estimates, or tool usage inconsistencies. But many of these alerts are triggered by legitimate field variations (experienced workers performing steps in a different but equally valid order, environmental conditions affecting measurement readings, or the work package itself being overly prescriptive). No legacy metric measures the actionable signal content of the alert stream. Alert volume is tracked as an activity metric. Alert acknowledgment rate is tracked as a compliance metric. Neither measures whether the alerts are worth acknowledging. The invisible failure mode is that the alert stream becomes noise-dominated, the maintenance foreman learns to dismiss alerts without investigation, and a genuine execution deviation (wrong torque value, skipped safety step, incorrect material) gets the same dismissive treatment as the dozens of false positives that preceded it.

Decision Supported

Is the digital execution monitoring system producing alerts that help the maintenance foreman catch real problems, or has the alert volume degraded into noise that conditions the foreman to dismiss all alerts?

Formula

Plain language: Count all alerts generated by the digital work execution system during maintenance activities. For each, determine whether the alert identified a genuine deviation that required (or should have required) corrective action. Divide genuine by total.

Notation:
EASNR = Alerts_genuine / Alerts_total

Variables:

  • Alerts_genuine = execution monitoring alerts where post-review determined a genuine deviation existed that required or would have required intervention (count)
  • Alerts_total = all execution monitoring alerts generated in the measurement period (count)
  • Genuine = the alert identified a real step sequence error, wrong tool/torque/material, missed hold point, or other deviation from the approved work package. Includes alerts that were dismissed by the foreman but post-review determined should have been acted on.

Sign convention: EASNR = 1.0 means every alert was a genuine deviation (perfect signal). Lower is worse. EASNR of 0.10 means 90% of alerts are false positives.

Aggregation period: Monthly rolling, stratified by alert type (step sequence, torque, tool, hold point, timing).

Edge Cases

  • Alert was technically a deviation but the deviation was acceptable per field conditions (e.g., experienced mechanic performing steps in a different order due to physical access): classify as "justified deviation" rather than false positive or genuine. Track separately. The alert was factually correct, but the alert wasn't a problem. High rates of "justified deviation" alerts indicate the AI model doesn't understand legitimate field practice variations.
  • Alert acknowledged and dismissed by foreman with appropriate justification: still count as a non-genuine alert if post-review agrees the foreman's dismissal was correct. The alert consumed attention without adding value
  • Minimum sample: 100 alerts per month (most active digital execution programs will exceed this easily)

Proposed formula, pending validation pilots.

At a Glance

Unit
Ratio
Trend Direction
Toward 1.0
Reporting Frequency
Monthly
Category
Execution Monitoring Effectiveness
← AI Verification Thoroughness Ratio (AVTR)FME Alert Response Quality Index (FARQI) →

Assess Your Organization

The AI-GPS Assessment evaluates your AI governance maturity and recommends KPIs like this one for your implementation roadmap.

Take the Free AI-GPS Assessment