Definition
The longitudinal change in a technician's unaided diagnostic performance (current unaided diagnostic accuracy divided by baseline unaided diagnostic accuracy) measured against a standardized scenario set without AI assistance.
The Measurement Gap
Every metric on the maintenance dashboard measures performance while the AI is running. PM Compliance, MTTR, OEE, and even the proposed human-AI metrics (TTCT, OA) measure collaboration quality during normal operations. None of them capture what happens when the AI is unavailable: during planned outages, server migrations, or model failures. Certification is a one-time event; training completion measures inputs, not retained diagnostic capability. Aviation solved this problem at policy level by mandating periodic manual flight to prevent skill decay; maintenance has no equivalent. The risk is a brittle workforce that performs well with AI and collapses without it: invisible until the system goes down.
Decision Supported:
Whether AI deployment is building diagnostic capability or eroding it, and which individuals need scheduled AI-off practice cycles to maintain unaided skill.
Formula
Plain language: Score each technician on a structured AI-off diagnostic assessment at baseline (when they first start working with the AI) and again every quarter. Divide current score by baseline score. Above 1.0 means the AI is teaching, exactly 1.0 means stable, below 1.0 means atrophy.
Notation:
SAT = P_manual(t) / P_manual(t₀)
Variables:
| Variable | Definition | Unit |
|---|---|---|
| P_manual(t) | Current unaided diagnostic performance on standardized scenarios at time t | Ratio (0–1) |
| P_manual(t₀) | Baseline unaided diagnostic performance at time zero | Ratio (0–1) |
| Scenario set | 15–20 cases covering common failure modes for AI-monitored asset classes, scored on failure-mode ID (40%), root cause (40%), first action (20%) | Rubric |
| ρ(AI-assist%, ΔSAT) | Correlation between AI exposure duration and SAT change: diagnostic complement | Correlation |
A SAT of 0.82 means the technician has lost 18% of baseline diagnostic capability since deployment.
Confidence Grading (required alongside every SAT value):
| Grade | Basis | Interpretation |
|---|---|---|
| A: Empirical | 5+ observed events in measurement window provide direct tail observation | Measurement grounded in observed outcomes; highest confidence |
| B: Hybrid | 1-4 observed events plus Monte Carlo simulation with Bayesian updating | Blends observation and simulation; moderate confidence |
| C: Simulated | Zero observed events; derived entirely from Monte Carlo simulation | Modeled, not observed; treat as indicative, not validated |
A SAT of any value at Grade C is a simulation that agrees with itself. It tells you the model is internally consistent but not that it matches reality. Grade C scores should trigger proactive stress-testing against historical near-miss events and cross-asset analogues rather than acceptance as calibrated.
Proposed formula, pending validation pilots.