Model Routing Accuracy Index (MRAI)

GMI-SWD-020Software DeliveryIntegration-Required

Definition

The proportion of model routing decisions where the selected model produced output quality at or above the quality the task classification warranted, measured by downstream outcome validation.

The Measurement Gap

There is no metric that tells you whether the multi-model routing orchestrator is selecting the right model for each task, which means tasks may be handled by models that are too weak (quality loss) or too expensive (cost waste) without anyone knowing.

Formula

Plain language: For each task routed by the orchestrator, compare the output quality (measured by acceptance without rework, defect-free deployment, or peer review score) against the quality expectation for that task's complexity tier. Count how many routing decisions produced adequate quality versus total routing decisions in the measurement period.

Notation:

MRAI = (Σ tasks where Q_actual ≥ Q_expected for assigned tier) / T_total × 100

Variables:

  • Q_actual = quality outcome of the routed task, measured by a composite of: (a) accepted without modification at checkpoint (boolean), (b) no defect traced to this artifact within 30 days (boolean), (c) peer review score if applicable (0–5 scale). Composite: (a × 0.4) + (b × 0.4) + (c/5 × 0.2). (dimensionless, DERIVED)
  • Q_expected = quality threshold for the task's complexity tier. Platform-defined: Tier 1 (simple completion) = 0.6, Tier 2 (standard implementation) = 0.75, Tier 3 (complex reasoning/architecture) = 0.85. (dimensionless, NATIVE from routing metadata)
  • T_total = total routing decisions in measurement period (count, NATIVE from platform routing logs)

Proposed formula, pending validation pilots.

At a Glance

Category
Model Routing Intelligence & Economics

Decision Supported

Is the routing orchestrator selecting the right model for each task type, or is it systematically mis-routing tasks in ways that degrade quality or waste compute?

← AI Orchestration Net Value Index (AONVI)Model Substitution Drift Score (MSDS) →

Assess Your Organization

The AI-GPS Assessment evaluates your AI governance maturity and recommends KPIs like this one for your implementation roadmap.

Take the Free AI-GPS Assessment