Effective Review Depth Index (ERDI)

GMI-SWD-005Software DeliveryInfrastructure-DependentPattern O-5: Override Value / Trust

Definition

Combined verification depth actually achieved by AI and human review gates on each change, measuring whether two approvals verify complementary concerns or the same shallow ones twice.

The Measurement Gap

Measurement Gap and Decision

The Gap: Two approval gates on every PR -- one AI, one human -- looks like doubled review capacity. But the dashboard does not show what those two reviews actually verify. The AI checks style, null dereferences, unused imports, and idiomatic patterns at depth level 2-3 on those dimensions. The human reviewer, seeing the AI has covered the surface, scans the PR, confirms the AI suggestions look reasonable, and approves. The human review becomes depth level 1 across most dimensions.

Before AI review, a human reviewer spent 25 minutes on a complex PR producing comments across 4-5 dimensions at depth level 2. Now the same reviewer spends 8 minutes with fewer substantive comments. Two approvals may represent less total verification depth than one thorough human approval pre-AI. No metric counts review substance rather than review occurrences.

The Decision:

ERDI estimates the net verification depth when AI and human review combine. It produces a single score answering: how deeply is this PR actually being scrutinized across all dimensions that matter? When ERDI declines from baseline, the intervention is structured checklists restoring depth on degraded dimensions.

Formula

Formula:

ERDI = (1/N) x SUM_PR [ (1/D) x SUM_d min(depth_human_d + depth_AI_d, 3) ]

Where:

  • depth_human_d = depth score (0-3) of human review on dimension d
  • depth_AI_d = depth score (0-3) of AI review on dimension d
  • D = 8 (number of dimensions)
  • N = number of PRs in the measurement period
  • min(..., 3) caps combined depth at maximum meaningful level

Depth anchors: 0 = not addressed; 1 = mentioned superficially; 2 = specific observations with justification; 3 = thoroughly investigated with alternatives considered.

Worked Example:

Payment retry service PR. AI review depths: bug=2, arch=0, business=0, ops=1, style=3, perf=1, security=2, test=0. Human review depths: bug=1, arch=1, business=0, ops=0, style=1, perf=0, security=0, test=1.

Combined (capped at 3): bug=3, arch=1, business=0, ops=1, style=3, perf=1, security=2, test=1.

ERDI = (1/8) x (3+1+0+1+3+1+2+1) = 1.50

Pre-AI comparison for same reviewer on similar PR: bug=2, arch=2, business=2, ops=1, style=2, perf=1, security=1, test=1. Pre-AI ERDI = 1.50.

Same total score but different distribution. Pre-AI had more even coverage. Post-AI has peaks (bug, style) and valleys (business logic = 0). Track per-dimension depth alongside the composite.

Proposed formula, pending validation pilots.

At a Glance

Unit
Index
Trend Direction
Higher
Reporting Frequency
Quarterly
Category
Review and Verification Integrity
← AI Test Environmental Sensitivity Rate (ATESR)Review Attention Displacement Index (RADI) →

Assess Your Organization

The AI-GPS Assessment evaluates your AI governance maturity and recommends KPIs like this one for your implementation roadmap.

Take the Free AI-GPS Assessment