Definition
Drift Classification Accuracy (DCA) measures the share of AI drift classifications that match ground-truth assessment by domain experts.
In plain language: take a sample of AI-classified drift events.
The Measurement Gap
A platform engineering team operates infrastructure-as-code at scale. They have an AI-driven drift detection and classification platform that watches the actual state of production infrastructure against the IaC-declared state, classifies any drift as either intentional (someone made the change for a legitimate operational reason) or accidental (the drift is from configuration error, manual fixes that bypassed IaC, or system mutations that should be corrected). The platform auto-remediates the accidental cases by reverting them to IaC-declared state. The team has been running this for nine months. The dashboard reports auto-remediation success rate at 97%.
Then a production incident occurs on a Friday afternoon. A database cluster's connection pool size was tuned manually two weeks ago by an on-call engineer addressing a performance issue. The change was deliberate, well-reasoned, documented in the incident channel, but never made it into the IaC. The AI classifier observed the drift, classified it as accidental, and auto-remediated by reverting the pool size to the IaC value. The reversion happened at 2:47 PM on Friday. By 3:15 PM, the database was hitting connection exhaustion under normal afternoon load. Customer-facing services degraded. The on-call team had to scramble to manually re-apply the change while figuring out what the AI had done and why.
Investigation reveals that the AI classifier was correct about the drift (the live state did not match IaC) but wrong about its classification (intentional, not accidental). The classifier had no way to know the change had been made deliberately; the team had not encoded the reasoning in any system the classifier could read. The classification was based on heuristics (recent change patterns, who typically makes changes to this resource, whether the change matches a known operational pattern) none of which captured the actual intent.
The legacy change metrics report a healthy platform.
Drift detection rate: 100% of detected drift was classified.
Auto-remediation success rate: 97% (the remediation completed cleanly).
Time from drift to remediation: under 5 minutes for accidental cases.
Manual remediation rate: low.
IaC adherence: 99% by configuration count.
None of these report on whether the classifications were correct. The remediation success rate measures whether the technical remediation executed; the classification accuracy (was this drift really accidental) is not measured. The team trusts the classification because the system is automated, and the failure surfaces only when the auto-remediation produces a customer-visible problem.
These metrics were designed when drift was investigated by humans before remediation. The human read the change context, understood whether it was intentional, and made the call. With AI classification and auto-remediation, the human is out of the loop on most decisions, and the classification accuracy becomes a measurement question that nobody has been asking.
There is no metric that tells you the accuracy of AI drift classification: whether "intentional" drift is actually intentional and "accidental" is actually accidental, which means misclassified drift persists as risk or triggers unnecessary remediation.
Formula
Drift Classification Accuracy (DCA) measures the share of AI drift classifications that match ground-truth assessment by domain experts.
In plain language: take a sample of AI-classified drift events. Have a platform engineer who knows the context assess each one independently. Count the cases where the AI's classification matches the expert's. DCA is the percentage that match. Split the metric into DCA_intentional (accuracy on cases the AI called intentional) and DCA_accidental (accuracy on cases the AI called accidental), because the operational consequences of misclassification differ: misclassifying intentional as accidental causes harmful auto-remediation (the Friday afternoon scenario), while misclassifying accidental as intentional leaves real config errors in place.
The formula:
DCA = Correctly_classified_drift / Total_sampled_drift × 100
Split into:
- DCA_intentional = correctness rate on events the AI classified as intentional
- DCA_accidental = correctness rate on events the AI classified as accidental
A worked example. The platform engineering team runs DCA on a sample of 60 drift events from the trailing quarter, balanced 30 intentional / 30 accidental per the AI's classification.
Expert validation:
- Of the 30 AI-classified intentional: 26 confirmed intentional, 4 actually accidental (AI was wrong)
- Of the 30 AI-classified accidental: 23 confirmed accidental, 7 actually intentional (AI was wrong. Including the Friday afternoon case)
DCA = (26 + 23) / 60 = 81.7%
DCA_intentional = 26 / 30 = 86.7%
DCA_accidental = 23 / 30 = 76.7%
Per the catalog guidance (above 85% overall, DCA_accidental above 80%) this team is below both thresholds, and the failure mode is on the operationally dangerous side (DCA_accidental at 76.7% means 23.3% of AI-claimed accidental drift was actually intentional and got harmfully auto-remediated). The action trigger applies: DCA_accidental below 75% would disable auto-remediation entirely; at 76.7% the team is one bad quarter from that. The recommendation is to disable auto-remediation now and revert to human review until DCA_accidental is consistently above 80%.
Three edge cases need explicit treatment. First, confirm via change records (incident channels, runbook documentation, change tickets) not via self-report from the team. Self-reports are biased toward "yes, that was intentional" because admitting accidental drift implies oversight failure. Second, genuinely ambiguous drift (the change was operationally reasonable but not documented and not clearly intentional) should be tracked as a separate "process gap" category; it represents a documentation problem rather than a classification problem. Third, the minimum sample is 40 events per quarter, balanced between intentional and accidental classifications.
Cross-references. Downstream of DCA is
Blast Radius Prediction Accuracy (BRPA): drift classification and blast radius prediction depend on the same underlying dependency model and the same AI judgment about what changes mean.
Proposed formula, pending validation pilots.