Definition
Operator Skill Atrophy Index (OSAI) measures the change in mean resolution time for novel incidents (those that no AIOps runbook or pattern-matching model can handle) compared to a baseline established early in the AIOps deployment.
The Measurement Gap
Twelve months after deploying AIOps, the operations team was handling fewer than 10 incidents per month that required human investigation. The rest (90% of incident volume) was handled entirely by automated detection and remediation. MTTR for AIOps-handled incidents averaged 6 minutes. The business case for the investment looked airtight.
Then, on a Friday afternoon, a novel configuration error propagated across the load balancer fleet. No AIOps runbook matched the failure pattern. The correlation engine grouped a collection of HTTP 502 alerts, connection timeout events, and certificate validation failures into one major incident and escalated to the on-call SRE. It had done exactly what it was supposed to do: surface the incident and hand it off.
Resolution took 7.4 hours. In the pre-AIOps era, based on historical incidents of comparable complexity, the same failure category averaged 2.1 hours.
Nobody had hidden any data. Nobody had made a mistake in the post-AIOps period. The SREs were technically skilled and motivated. The problem was that they hadn't done real load-balancer troubleshooting in eight months. The pattern recognition that makes experienced SREs fast at novel incidents (the instinct that says "this timeout pattern combined with certificate errors usually means X") doesn't stay sharp when you're not using it. The team had gotten better at managing the AIOps platform and worse at debugging the infrastructure it managed.
MTTR as a metric never surfaced this. The 6-minute average for AIOps-handled incidents dominated the monthly report, masking the trend in human-resolved incidents. Because novel incidents were rare (the automation was handling almost everything) the human MTTR data was too sparse to trend without deliberate tracking. No one was monitoring whether the team's capability on hard incidents was changing direction.
This is the failure mode that sits behind successful AIOps deployments: as the system handles more incidents automatically, human operators get fewer opportunities to practice the diagnostic skills that make them effective when automation can't help. Troubleshooting capability is a perishable skill. Atrophy is gradual, invisible on any standard operations metric, and fully visible only during the incidents you least want to discover it on.
The DORA MTTR metric averages across all incidents. It doesn't distinguish between AI-handled incidents and human-handled novel incidents, which is precisely the distinction that matters for measuring skill degradation. SLA compliance doesn't measure capability at all. Incident volume measures events, not the quality of the human response to events that do require humans.
The measurement gap: there is no metric that tells you whether SRE troubleshooting capability is degrading as AIOps handles more incidents automatically, which means novel-incident response time silently increases while the overall MTTR dashboard stays green.
Formula
Operator Skill Atrophy Index (OSAI) measures the change in mean resolution time for novel incidents (those that no AIOps runbook or pattern-matching model can handle) compared to a baseline established early in the AIOps deployment. A rising OSAI means the team is taking longer to resolve the incidents the automation can't, which is the leading indicator of a workforce capability risk. OSAI is Infrastructure-Dependent. Computing it requires that your AIOps platform consistently classifies incidents as pattern-matched (handled by automation) vs. novel (escalated to humans), and that your ITSM system maintains incident resolution timestamps going back to a meaningful baseline period. Both should be true in a mature AIOps deployment, but they require infrastructure decisions that most organizations don't make explicitly in year one.
In Plain Language
collect every incident escalated to human engineers because AIOps had no matching pattern. Compute the average resolution time for those incidents in the current period. Compare to the average from your baseline period: ideally the first 6 months of AIOps deployment, before atrophy has had time to develop. The percentage change is OSAI.
Formula
OSAI = (MTTR_novel_current − MTTR_novel_baseline) / MTTR_novel_baseline × 100
Where:
- MTTR_novel_current = mean time to resolve novel (non-pattern-matched) incidents in the current measurement period
- MTTR_novel_baseline = mean time to resolve novel incidents in the baseline period (first 6 months post-AIOps deployment, or pre-AIOps historical data if available)
Worked Example
An SRE team establishes a baseline in the first 6 months after AIOps deployment. During that period, they resolved 47 novel incidents with a mean resolution time of 2.8 hours.18 months later, in the current measurement period, they resolved 12 novel incidents (volume has dropped significantly as AIOps handles more) with a mean resolution time of 4.6 hours. OSAI = (4.6 − 2.8) / 2.8 × 100 = 64.3%
The team is taking 64% longer to resolve novel incidents compared to their baseline. The operations dashboard doesn't show this because AIOps-handled incidents dominate the MTTR calculation, and the novel-incident sample is too small to surface as a trend without deliberate tracking.
Sign Convention
Positive OSAI is bad: it means resolution time for novel incidents is increasing relative to baseline, indicating skill degradation. Negative OSAI is improvement: the team is getting faster on novel incidents. Zero means no change. Track direction over time; the trend matters more than any single measurement.
Edge Cases
Low novel incident volume: small samples create noise. In a mature AIOps environment, novel incidents may be infrequent, which is what makes them high-stakes when they do occur. Use a 6-month rolling window rather than monthly measurement to accumulate enough incidents for statistical meaning. Below 10 incidents in the measurement window, treat OSAI as directional rather than precise. System complexity genuinely increasing: rising MTTR for novel incidents might reflect harder problems, not weaker skills. If the environment has grown significantly more complex (more services, more dependencies, new technology stacks) control for incident severity where possible to reduce this confound. Team turnover: engineers hired post-AIOps deployment never had the baseline troubleshooting exposure. If turnover is significant, segment OSAI by team tenure: the atrophy pattern will differ for engineers who were there before AIOps vs. those who joined after. Cross-references: Upstream, OSAI is fed by RMR (Remediation Masking Rate, GMI-IT-005) and ACA (AIOps Correlation Accuracy, GMI-IT-004). High RMR means the team gets fewer reps at real troubleshooting: every masked remediation is a missed practice opportunity. Poor ACA means more engineer time goes to untangling correlation errors rather than building genuine diagnostic skill. Downstream, OSAI directly predicts extended MTTR on the incidents that matter most: the novel failures during which your automation provides no assistance. Maturity: Infrastructure-Dependent. The incident classification data (pattern-matched vs. novel) must exist in your AIOps escalation logs and must be consistent enough to join with ITSM resolution timestamps across a multi-year period. This is achievable but requires that the AIOps platform was configured from deployment to log escalation reasons. Organizations that didn't capture this data retroactively face a gap.
Proposed formula, pending validation pilots.