Definition
Override Outcome Index (OOI) is the ratio of outcome quality when humans override AI recommendations compared to outcome quality when they follow them. An OOI of 1.0 means overrides produce the same results as compliance. Above 1.0, the humans are adding value. Below 1.0, the overrides are destroying value.
The Measurement Gap
A water treatment utility deployed an AI-based system to predict membrane fouling in its reverse osmosis units. The system would flag membranes likely to require cleaning within the next 14 days, allowing the operations team to schedule cleaning during low-demand periods rather than reacting to throughput loss.
Six months in, the reliability manager noticed something troubling in the maintenance records. The AI system was making an average of 22 predictions per week. Operators were following about 13 of them and overriding 9, a 41% override rate. But when the manager asked whether those overrides were good decisions, nobody could answer. There was no tracking mechanism that connected override decisions to outcomes.
The manager commissioned a manual review. What she found was messy: of the 9 weekly overrides, roughly 4 were cases where the operator correctly recognized a false alarm (the membrane was fine, and no cleaning was needed within the window). Three were cases where the operator deferred action to a more convenient time, which sometimes worked out and sometimes didn't. And two were cases where the operator overrode a correct prediction, and the membrane fouled during a peak demand period, exactly the outcome the AI was trying to flag.
So the operators were improving on the AI about 44% of the time, matching it about 33% of the time, and making things worse about 22% of the time. That's a meaningful split, but without a metric tracking it, nobody knew the ratio. The organization had no visibility into whether its human-AI decision system was functioning, and no way to identify which operators were well-calibrated and which were systematically over- or under-riding the system.
PM Compliance captured all of this as "work done or not done". It couldn't distinguish between work correctly deferred by a skilled operator and work incorrectly skipped by an operator who ignored a valid prediction. Asset Availability would eventually capture the downstream failure when a fouled membrane reduced throughput, but by then the causal chain of AI predicted, human overrode, failure occurred was invisible. PdM Hit Rate captured aggregate prediction accuracy but said nothing about what happened after the prediction was delivered to the human.
There is no metric that tells you how often operators override AI recommendations and whether those overrides improve or degrade outcomes, which means the human-AI trust relationship is invisible and unmanaged.
Formula
Override Outcome Index (OOI) is the ratio of outcome quality when humans override AI recommendations compared to outcome quality when they follow them.
In plain language:
When our people ignore the AI, does that make things better or worse?
The Formula
OOI = S_o / S_f
Where S_o is the success rate in the override group (successful outcomes among overridden recommendations) and S_f is the success rate in the followed group (successful outcomes among followed recommendations).
"Successful outcome" must be defined per recommendation type. For failure predictions: success means no unplanned failure within the P-F interval. For intervention recommendations: success means the intervention resolved the detected condition. For "no action needed" recommendations: success means no failure within the review period.
Worked Example
A food processing plant's AI system made 480 recommendations in Q1. Of those, 312 were followed and 168 were overridden.
In the followed group: 271 of 312 had successful outcomes. S_f = 271/312 = 0.869.
In the override group: 128 of 168 had successful outcomes. S_o = 128/168 = 0.762.
OOI = 0.762 / 0.869 = 0.877
The overrides are producing worse outcomes than following the AI, about 12% worse. But that's the aggregate. Segmented by confidence band:
Overrides of high-confidence predictions (above 85%): S_o = 0.64. The operators are overriding correct predictions.
Overrides of medium-confidence predictions (60 to 85%): S_o = 0.81. Closer to parity with the followed group.
Overrides of low-confidence predictions (below 60%): S_o = 0.91. The operators are outperforming the AI on its weakest predictions.
This is the pattern that makes OOI actionable. The operators have developed reasonable intuition about when the AI is unreliable (low-confidence predictions) but are also overriding high-confidence predictions that are usually right. Targeted training on when to trust high-confidence outputs could shift OOI above 1.0, and the human-AI team would then be outperforming either one alone.
Sign Convention
OOI of 1.0 is break-even. Above 1.0 means overrides add value. Below 1.0 means overrides destroy value. An OOI of 1.20 means overrides produce 20% better outcomes. An OOI of 0.80 means overrides produce 20% worse outcomes. Thresholds are proposed, not proven.
Cross-References
OOI sits at the center of the human-AI interaction layer. It connects upward to model quality and downward to trust and feedback integrity.
Upstream: Prediction Confidence Calibration Index directly determines whether overrides are rational. If that index is low for a particular asset class, an OOI above 1.0 for that class is expected and healthy: the operators have learned what the uncalibrated model hasn't told them. Knowledge Retrieval Accuracy Score plays the same role for knowledge-based recommendations.
Downstream: Operator Trust Calibration Score uses OOI as ground truth. If OOI shows that overrides of high-confidence predictions are destroying value, but the trust score shows operators are frequently overriding high-confidence predictions, you have a specific, actionable trust calibration problem. Override Feedback Degradation Rate tracks whether the override pattern is causing the AI to get worse over time. OOI tells you if overrides are good or bad in the moment; the degradation rate tells you if they're causing long-term system damage.
If you track OOI without the Prediction Confidence Calibration Index, you'll know the overrides are bad but not why. The operators might be overriding because the model's confidence is misleading them. Fixing operator behavior without fixing model calibration treats the symptom, not the cause.
Proposed formula, pending validation pilots.
At a Glance
Decision Supported
Are our operators improving on the AI's recommendations, or are they overriding correct predictions and causing the failures the AI tried to prevent?