Definition
SDOR measures the distribution of production defect types originating from AI-generated code compared to the defect type distribution from human-authored code, revealing whether AI assistance introduces a structurally different pattern of failure that requires targeted guardrails.
The Measurement Gap
There is no metric that tells you whether AI-generated code introduces semantic errors at a different rate or with a different profile than human-written code, which means defect risk from AI assistance accumulates unattributed until production failures reveal it. DORA's change failure rate treats all failures equally and does not differentiate by code origin. Code coverage reports that AI-generated lines are tested, but the tests may not cover domain-specific conditions the AI consistently misses. Static analysis passes because the AI's code is syntactically correct even when semantically wrong. Velocity metrics are up because AI generates code faster. Every metric leadership watches moves in the right direction while a specific category of defect accumulates unattributed to its source. The defects exist. The source attribution exists. But no measurement connects them in a way that produces an operational decision.
Decision Supported:
Should AI code generation guardrails be tuned by defect type rather than applying uniform review to all AI output?
Formula
Plain language: Take all production defects from a given period, split them by code origin (AI vs. human), calculate the percentage of each defect type within each origin group, and subtract the human percentage from the AI percentage for each type. The result shows whether AI over-produces or under-produces specific defect types relative to human developers.
Notation:
SDOR(type_t) = (Defects_AI_t / Defects_AI_total) - (Defects_Human_t / Defects_Human_total)
Variables:
| Variable | Definition | Unit |
|---|---|---|
| Defects_AI_t | Count of defects of type t in AI-generated code during the measurement period | Count |
| Defects_AI_total | Total count of defects in AI-generated code during the measurement period | Count |
| Defects_Human_t | Count of defects of type t in human-written code during the measurement period | Count |
| Defects_Human_total | Total count of defects in human-written code during the measurement period | Count |
| type_t | A specific category from the defect taxonomy (e.g., null handling, concurrency, business logic violations) | Category |
Proposed formula, pending validation pilots.