Definition
Operator Trust Calibration Score (OTCS) measures whether operators respond appropriately to AI alerts, acting on true positives and investigating or dismissing false positives, without over-trusting (blindly acting on false alarms) or under-trusting (ignoring valid alerts).
The Measurement Gap
A semiconductor fab deployed an AI-based contamination detection system on its chemical vapor deposition tools. The system monitored 47 process parameters in real time and generated alerts when it detected conditions likely to produce particle contamination events, events that could scrap $200,000 worth of wafers in a single run.
In its first year, the system's false positive rate was about 15%, which was manageable. Operators treated most alerts seriously, pausing production to investigate. The system caught 34 genuine contamination precursors that traditional monitoring did not surface.
In year two, the system was updated with a new model. The new model's sensitivity increased. It caught more genuine events but also increased the false positive rate to 38%. Operators received more alerts but found fewer of them justified. Over six months, operator behavior shifted. Response time to alerts increased from an average of 4 minutes to 22 minutes. Some operators started categorically ignoring certain alert types. One shift team developed an informal rule: if the tool's been running clean for more than 8 hours, ignore the first alert and wait for a second.
Then Tool 14 generated a contamination alert at hour 11 of a clean run. The operator on shift, following the informal rule, waited for confirmation. No second alert came. The contamination event proceeded. By the time the 25-wafer lot completed processing, 22 wafers were scrapped. $176,000 lost because the operator had been trained by experience to distrust the system.
The organization had no visibility into this behavioral adaptation. No metric tracked whether operators were responding appropriately to alerts. The reliability and process engineering teams saw alert counts and response counts but couldn't distinguish between operators who were well-calibrated (trusting good alerts, dismissing bad ones) and operators who had either checked out entirely or were wasting time investigating every false alarm.
Alert acknowledgment rate showed operators were acknowledging alerts, but acknowledging and acting are different things. Defect rate captured the scrap event but couldn't connect it to the operator's behavioral adaptation. System uptime showed the tool was running, which was exactly the problem.
There is no metric that tells you whether operator trust in AI alerts is calibrated.
Formula
Operator Trust Calibration Score (OTCS) measures whether operators respond appropriately to AI alerts, acting on true positives and investigating or dismissing false positives.
In plain language:
Are our operators ignoring real warnings or chasing false ones?
The Formula
OTCS = 1 - (W_o x R_ov + W_u x R_un)
R_ov = FP_acted / FP_total (over-trust rate)
R_un = TP_ignored / TP_total (under-trust rate)
Default weights: W_o = 0.4, W_u = 0.6. Under-trust is weighted higher because ignoring real alerts has worse consequences.
Worked Example
A chemical plant evaluates operator responses to AI process alerts over Q3:
Total alerts: 480
True positives (actual conditions): 168
False positives: 312
True positives acted on promptly: 131
True positives ignored or significantly delayed: 37
False positives acted on without investigation: 94
False positives correctly investigated and dismissed: 218
R_un = 37 / 168 = 0.220 (22% under-trust)
R_ov = 94 / 312 = 0.301 (30% over-trust)
OTCS = 1 - (0.4 x 0.301 + 0.6 x 0.220) = 1 - (0.120 + 0.132) = 0.748
Operators are under-trusting 22% of the time (ignoring valid alerts) and over-trusting 30% of the time (acting on false alarms without investigation). The OTCS of 0.748 is in the range where trust calibration issues are worth addressing.
Segmented by shift team reveals the variation:
Day shift A: OTCS = 0.84 (well-calibrated)
Day shift B: OTCS = 0.71 (moderate under-trust)
Night shift: OTCS = 0.62 (significant under-trust, ignoring 34% of valid alerts)
Sign Convention
Higher is better. 0.80 and above is well-calibrated. 0.60 to 0.80 needs attention. Below 0.60 is dysfunctional. These thresholds are proposed, not proven.
Cross-References
Upstream: Override Outcome Index provides ground truth on whether override behavior is improving or degrading outcomes. Prediction Confidence Calibration Index matters because if model confidence is miscalibrated, operator trust responses may be rational adaptations.
Downstream: Override Feedback Degradation Rate, because trust-driven override patterns determine whether the system's learning loop is intact. Autonomous WO Signal-to-Noise Ratio, because operator trust levels affect how many machine-generated work orders get executed vs. cancelled.
If you fix the model (improve the calibration index) but don't address operator trust calibration, the behavioral patterns from the low-accuracy period persist. Operators who learned to distrust the system don't automatically recalibrate when the system improves.
Proposed formula, pending validation pilots.
At a Glance
Decision Supported
Are our operators appropriately calibrated in their trust of AI alerts, or have they developed learned helplessness (ignore everything) or blind trust (act on everything)?