Definition
The fraction of AI-triggered work orders where the model's top contributing factors map to a defensible physical relationship with the verified root cause recorded in the failure report.
The Measurement Gap
Dashboards now show factor importance (the "why" behind every AI alert) and technicians use those explanations to scope work. But no metric checks whether the explanations are correct. Precision, recall, F1, and AUC evaluate whether predictions were right; they say nothing about whether the model's stated reason was right. The failure mode is the plausible-but-wrong explanation: a model correctly predicts a pump failure within 14 days while attributing it to discharge pressure when the real root cause is coupling misalignment. The prediction is a true positive; the explanation sent the crew to the wrong place inside the asset. Feature Importance Stability measures whether explanations are consistent, but consistency does not imply correctness, an explanation can be perfectly stable and perfectly wrong.
Decision Supported:
When the model tells me why an asset is anomalous, can I trust that "why" to scope investigation and intervention?
Formula
Plain language: For every AI-triggered work order in the period, check whether at least one of the model's top-k contributing factors maps (via a feature-to-cause lookup) to the verified root cause recorded in the closure failure report. EAV is the ratio of matches to total validated work orders.
Notation:
EAV = N_match / N_validated
Variables:
| Variable | Definition | Unit |
|---|---|---|
| N_match | Number of AI-triggered work orders where at least one top-k factor maps to the failure-report Cause | count |
| N_validated | Total AI-triggered work orders with completed failure reports | count |
| top-k | Number of contributing factors evaluated per alert (recommended k=3) | count |
| Match | Defensible physical relationship between a model feature label and a failure-cause code (defined by a reference mapping table) | binary |
Proposed formula, pending validation pilots.