Explainability-to-Action Validation (EAV)

GMI-MRO-008MROResearch-Forward

Definition

The fraction of AI-triggered work orders where the model's top contributing factors map to a defensible physical relationship with the verified root cause recorded in the failure report.

The Measurement Gap

Dashboards now show factor importance (the "why" behind every AI alert) and technicians use those explanations to scope work. But no metric checks whether the explanations are correct. Precision, recall, F1, and AUC evaluate whether predictions were right; they say nothing about whether the model's stated reason was right. The failure mode is the plausible-but-wrong explanation: a model correctly predicts a pump failure within 14 days while attributing it to discharge pressure when the real root cause is coupling misalignment. The prediction is a true positive; the explanation sent the crew to the wrong place inside the asset. Feature Importance Stability measures whether explanations are consistent, but consistency does not imply correctness, an explanation can be perfectly stable and perfectly wrong.

Decision Supported:

When the model tells me why an asset is anomalous, can I trust that "why" to scope investigation and intervention?

Formula

Plain language: For every AI-triggered work order in the period, check whether at least one of the model's top-k contributing factors maps (via a feature-to-cause lookup) to the verified root cause recorded in the closure failure report. EAV is the ratio of matches to total validated work orders.

Notation:

EAV = N_match / N_validated

Variables:

Variable Definition Unit
N_match Number of AI-triggered work orders where at least one top-k factor maps to the failure-report Cause count
N_validated Total AI-triggered work orders with completed failure reports count
top-k Number of contributing factors evaluated per alert (recommended k=3) count
Match Defensible physical relationship between a model feature label and a failure-cause code (defined by a reference mapping table) binary

Proposed formula, pending validation pilots.

At a Glance

Unit
Ratio (0 to 1)
Trend Direction
Higher; approaching 1.0
Reporting Frequency
Quarterly
Category
Algorithmic Efficacy & Model Health
← Edge Deployment Fidelity (EDF)Feature Importance Stability (FIS) →

Assess Your Organization

The AI-GPS Assessment evaluates your AI governance maturity and recommends KPIs like this one for your implementation roadmap.

Take the Free AI-GPS Assessment