Semantic Defect Origin Rate (SDOR)

GMI-SWD-001Software DeliveryIntegration-RequiredPattern O-1: Confidence Calibration

Definition

SDOR measures the distribution of production defect types originating from AI-generated code compared to the defect type distribution from human-authored code, revealing whether AI assistance introduces a structurally different pattern of failure that requires targeted guardrails.

The Measurement Gap

There is no metric that tells you whether AI-generated code introduces semantic errors at a different rate or with a different profile than human-written code, which means defect risk from AI assistance accumulates unattributed until production failures reveal it. DORA's change failure rate treats all failures equally and does not differentiate by code origin. Code coverage reports that AI-generated lines are tested, but the tests may not cover domain-specific conditions the AI consistently misses. Static analysis passes because the AI's code is syntactically correct even when semantically wrong. Velocity metrics are up because AI generates code faster. Every metric leadership watches moves in the right direction while a specific category of defect accumulates unattributed to its source. The defects exist. The source attribution exists. But no measurement connects them in a way that produces an operational decision.

Decision Supported:

Should AI code generation guardrails be tuned by defect type rather than applying uniform review to all AI output?

Formula

Plain language: Take all production defects from a given period, split them by code origin (AI vs. human), calculate the percentage of each defect type within each origin group, and subtract the human percentage from the AI percentage for each type. The result shows whether AI over-produces or under-produces specific defect types relative to human developers.

Notation:

SDOR(type_t) = (Defects_AI_t / Defects_AI_total) - (Defects_Human_t / Defects_Human_total)

Variables:

Variable Definition Unit
Defects_AI_t Count of defects of type t in AI-generated code during the measurement period Count
Defects_AI_total Total count of defects in AI-generated code during the measurement period Count
Defects_Human_t Count of defects of type t in human-written code during the measurement period Count
Defects_Human_total Total count of defects in human-written code during the measurement period Count
type_t A specific category from the defect taxonomy (e.g., null handling, concurrency, business logic violations) Category

Proposed formula, pending validation pilots.

At a Glance

Unit
Ratio
Trend Direction
Lower; approaching 0
Reporting Frequency
Quarterly
Category
AI-Generated Code Quality
← Dependency Selection Quality Score (DSQS)Architecture-Team Fitness Score (ATFS) →

Assess Your Organization

The AI-GPS Assessment evaluates your AI governance maturity and recommends KPIs like this one for your implementation roadmap.

Take the Free AI-GPS Assessment