Upgrade Cascade Accuracy (UCA)

GMI-SWD-011Software DeliveryIntegration-RequiredPattern O-6: Autonomous Coherence

Definition

Accuracy of AI dependency-upgrade impact predictions against actual post-upgrade outcomes, measuring whether auto-merge decisions rest on assessments that hold.

The Measurement Gap

Measurement Gap and Decision

The Gap: AI-powered dependency upgrade tools (Dependabot, Renovate, Snyk auto-fix) analyze changelogs, commit history, and semantic versioning to predict impact. But their assessments stop at the boundary of what they can statically analyze -- real breakage often happens in transitive dependencies, downstream consumers, and behavioral contracts not captured in type signatures.

Traditional metrics track whether upgrades succeed or fail but do not distinguish between predicted failures (AI warned about X, X broke) and surprise failures (AI said safe, Y broke). This distinction matters for trust calibration. Without it, every green-lit upgrade is a gamble disguised as confidence.

The Decision:

UCA measures the percentage of AI-recommended dependency upgrades that complete without triggering unexpected breaking changes beyond what the AI predicted. When UCA falls below threshold for a specific category, the intervention is targeted manual review for that category while allowing auto-merge where UCA exceeds threshold.

Formula

Formula:

UCA = Upgrades_no_surprise_breakage / Upgrades_total_AI_recommended

Where:

  • Upgrades_no_surprise_breakage = AI-recommended upgrades where all breakage (if any) was within the AI predicted impact scope. Includes clean upgrades and upgrades with predicted breakage.
  • Upgrades_total_AI_recommended = all AI-recommended upgrades actually executed
  • Surprise breakage = test failures, build failures, or runtime errors in packages NOT identified in the AI impact assessment

Worked Example:

Q3: 45 AI-recommended upgrades merged across the monorepo.

  • 32 upgrades: no breakage at all, AI predicted safe. Correct.
  • 5 upgrades: breakage occurred in packages the AI predicted would be affected. Correct prediction.
  • 8 upgrades: breakage in packages the AI did NOT predict would be affected. Surprise breakage.

UCA = (32 + 5) / 45 = 37/45 = 0.822

Segmented: Patch upgrades UCA = 0.93 (2 surprises in 28). Minor version UCA = 0.71 (5 surprises in 14). Major version UCA = 0.33 (1 surprise in 3).

Proposed formula, pending validation pilots.

At a Glance

Unit
Percentage
Trend Direction
Higher
Reporting Frequency
Quarterly
Category
Dependency and Security Intelligence
← Reachability Score Accuracy Index (RSAI)Audit Trail Actionability Ratio (ATAR) →

Assess Your Organization

The AI-GPS Assessment evaluates your AI governance maturity and recommends KPIs like this one for your implementation roadmap.

Take the Free AI-GPS Assessment