Definition
The measured change in output quality characteristics when the platform's model pool changes (through version updates, new model additions, or routing rule modifications) compared to the baseline period before the change.
The Measurement Gap
There is no metric that tells you whether model substitutions or version updates have changed the defect profile of AI-generated code, which means development teams calibrate trust to models they are no longer using.
Formula
Plain language: After a model pool change event, compare the defect rate, checkpoint rejection rate, and rework rate for the 14-day period after the change to the 14-day baseline period before the change. Express as a percentage deviation from baseline. Zero means no drift. Positive means quality degraded. Negative means quality improved.
Notation:
MSDS = ((D_post / T_post) - (D_base / T_base)) / (D_base / T_base) × 100
Where D is a composite quality deviation score:
D = (defect_count × 3) + (checkpoint_rejections × 2) + (rework_requests × 1)
Variables:
- D_post = weighted quality deviation events in the 14-day post-change period (count, DERIVED)
- D_base = weighted quality deviation events in the 14-day pre-change baseline (count, DERIVED)
- T_post = total tasks completed in post-change period (count, NATIVE)
- T_base = total tasks completed in baseline period (count, NATIVE)
- defect_count = production defects traced to AI-generated artifacts in the period (count, INTEGRATION from defect tracker)
- checkpoint_rejections = checkpoint reviews that rejected AI-generated output (count, NATIVE)
- rework_requests = post-acceptance modifications to AI-generated artifacts (count, INTEGRATION from source control)
Proposed formula, pending validation pilots.
At a Glance
Decision Supported
When the platform updates its model pool or routing rules, did the change preserve output quality, or did it introduce a shift in defect profile, style, or reliability that the development team hasn't calibrated to?