Model Substitution Drift Score (MSDS)

GMI-SWD-021Software DeliveryIntegration-Required

Definition

The measured change in output quality characteristics when the platform's model pool changes (through version updates, new model additions, or routing rule modifications) compared to the baseline period before the change.

The Measurement Gap

There is no metric that tells you whether model substitutions or version updates have changed the defect profile of AI-generated code, which means development teams calibrate trust to models they are no longer using.

Formula

Plain language: After a model pool change event, compare the defect rate, checkpoint rejection rate, and rework rate for the 14-day period after the change to the 14-day baseline period before the change. Express as a percentage deviation from baseline. Zero means no drift. Positive means quality degraded. Negative means quality improved.

Notation:

MSDS = ((D_post / T_post) - (D_base / T_base)) / (D_base / T_base) × 100

Where D is a composite quality deviation score:

D = (defect_count × 3) + (checkpoint_rejections × 2) + (rework_requests × 1)

Variables:

  • D_post = weighted quality deviation events in the 14-day post-change period (count, DERIVED)
  • D_base = weighted quality deviation events in the 14-day pre-change baseline (count, DERIVED)
  • T_post = total tasks completed in post-change period (count, NATIVE)
  • T_base = total tasks completed in baseline period (count, NATIVE)
  • defect_count = production defects traced to AI-generated artifacts in the period (count, INTEGRATION from defect tracker)
  • checkpoint_rejections = checkpoint reviews that rejected AI-generated output (count, NATIVE)
  • rework_requests = post-acceptance modifications to AI-generated artifacts (count, INTEGRATION from source control)

Proposed formula, pending validation pilots.

At a Glance

Category
Model Routing Intelligence & Economics

Decision Supported

When the platform updates its model pool or routing rules, did the change preserve output quality, or did it introduce a shift in defect profile, style, or reliability that the development team hasn't calibrated to?

← Model Routing Accuracy Index (MRAI)Routing Cost Efficiency Ratio (RCER) →

Assess Your Organization

The AI-GPS Assessment evaluates your AI governance maturity and recommends KPIs like this one for your implementation roadmap.

Take the Free AI-GPS Assessment