Definition
The proportion of model routing decisions where the selected model produced output quality at or above the quality the task classification warranted, measured by downstream outcome validation.
The Measurement Gap
There is no metric that tells you whether the multi-model routing orchestrator is selecting the right model for each task, which means tasks may be handled by models that are too weak (quality loss) or too expensive (cost waste) without anyone knowing.
Formula
Plain language: For each task routed by the orchestrator, compare the output quality (measured by acceptance without rework, defect-free deployment, or peer review score) against the quality expectation for that task's complexity tier. Count how many routing decisions produced adequate quality versus total routing decisions in the measurement period.
Notation:
MRAI = (Σ tasks where Q_actual ≥ Q_expected for assigned tier) / T_total × 100
Variables:
- Q_actual = quality outcome of the routed task, measured by a composite of: (a) accepted without modification at checkpoint (boolean), (b) no defect traced to this artifact within 30 days (boolean), (c) peer review score if applicable (0–5 scale). Composite: (a × 0.4) + (b × 0.4) + (c/5 × 0.2). (dimensionless, DERIVED)
- Q_expected = quality threshold for the task's complexity tier. Platform-defined: Tier 1 (simple completion) = 0.6, Tier 2 (standard implementation) = 0.75, Tier 3 (complex reasoning/architecture) = 0.85. (dimensionless, NATIVE from routing metadata)
- T_total = total routing decisions in measurement period (count, NATIVE from platform routing logs)
Proposed formula, pending validation pilots.
At a Glance
Decision Supported
Is the routing orchestrator selecting the right model for each task type, or is it systematically mis-routing tasks in ways that degrade quality or waste compute?