Counterfactual Intervention Accuracy (CIA)

GMI-MRO-021MROResearch-Forward

Definition

CIA measures the fraction of digital twin what-if predictions whose actual post-intervention outcomes fell within engineering-defined tolerance bounds of the predicted outcomes.

The Measurement Gap

Twins are used to evaluate interventions ("what happens if we replace this impeller?") but the loop is rarely closed. The prediction lives in a simulation log or PowerPoint; the actual outcome lives in work-order actuals and post-intervention sensor data. The two are never formally compared, so the twin's track record on the predictions that drive capital decisions remains unknown. MTBF, OEE, RMSE, and R² cannot see this: they measure historical fit or running-state behavior, not the interventional accuracy that Pearl distinguished from observational accuracy. Commissioning validation tests the twin under known conditions, not under decision conditions.

Decision Supported:

How much margin to apply to a twin's next intervention recommendation, and which prediction classes are weak enough to warrant recalibration before further capital allocation.

Formula

Plain language: For every twin prediction that led to an actual intervention with measured outcome, check whether the prediction fell inside the engineering tolerance band. Divide hits by total scored predictions.

Notation:

CIA = (1/N) * SUM[ I( |Y_predicted(i) − Y_actual(i)| <= B(i) ) ] for i = 1 to N

Variables:

Variable Definition Unit
N Total twin predictions that were followed by an actual intervention with post-intervention outcome data Count
Y_predicted(i) Twin-predicted outcome for intervention i (throughput, vibration, RUL, energy, etc.) Variable-specific
Y_actual(i) Measured post-intervention outcome for the same variable over the same window Variable-specific
B(i) Engineering-defined tolerance bound for prediction i, reflecting decision sensitivity (not arbitrary statistics) Variable-specific
I(·) Indicator function: 1 if within tolerance, 0 otherwise Binary

Proposed formula, pending validation pilots.

At a Glance

Unit
Ratio (0 to 1)
Trend Direction
Higher
Reporting Frequency
Rolling 12-month
Category
Digital Twin & System Resilience
← Cascading Failure Containment (CFC)Critical Dependency Visibility (CDVIS) →

Assess Your Organization

The AI-GPS Assessment evaluates your AI governance maturity and recommends KPIs like this one for your implementation roadmap.

Take the Free AI-GPS Assessment