Definition
Accuracy of AI dependency-upgrade impact predictions against actual post-upgrade outcomes, measuring whether auto-merge decisions rest on assessments that hold.
The Measurement Gap
Measurement Gap and Decision
The Gap: AI-powered dependency upgrade tools (Dependabot, Renovate, Snyk auto-fix) analyze changelogs, commit history, and semantic versioning to predict impact. But their assessments stop at the boundary of what they can statically analyze -- real breakage often happens in transitive dependencies, downstream consumers, and behavioral contracts not captured in type signatures.
Traditional metrics track whether upgrades succeed or fail but do not distinguish between predicted failures (AI warned about X, X broke) and surprise failures (AI said safe, Y broke). This distinction matters for trust calibration. Without it, every green-lit upgrade is a gamble disguised as confidence.
The Decision:
UCA measures the percentage of AI-recommended dependency upgrades that complete without triggering unexpected breaking changes beyond what the AI predicted. When UCA falls below threshold for a specific category, the intervention is targeted manual review for that category while allowing auto-merge where UCA exceeds threshold.
Formula
Formula:
UCA = Upgrades_no_surprise_breakage / Upgrades_total_AI_recommended
Where:
- Upgrades_no_surprise_breakage = AI-recommended upgrades where all breakage (if any) was within the AI predicted impact scope. Includes clean upgrades and upgrades with predicted breakage.
- Upgrades_total_AI_recommended = all AI-recommended upgrades actually executed
- Surprise breakage = test failures, build failures, or runtime errors in packages NOT identified in the AI impact assessment
Worked Example:
Q3: 45 AI-recommended upgrades merged across the monorepo.
- 32 upgrades: no breakage at all, AI predicted safe. Correct.
- 5 upgrades: breakage occurred in packages the AI predicted would be affected. Correct prediction.
- 8 upgrades: breakage in packages the AI did NOT predict would be affected. Surprise breakage.
UCA = (32 + 5) / 45 = 37/45 = 0.822
Segmented: Patch upgrades UCA = 0.93 (2 surprises in 28). Minor version UCA = 0.71 (5 surprises in 14). Major version UCA = 0.33 (1 surprise in 3).
Proposed formula, pending validation pilots.