Definition
CQRD measures the maximum performance degradation of a compressed (quantized, distilled, or pruned) model versus its full-precision counterpart, measured across defined production traffic subpopulations: specifically identifying where compression concentrates quality loss rather than averaging it across the full population.
The Measurement Gap
The platform team has been pushing to reduce inference costs. GPU serving is expensive, and the CFO has noticed. The ML engineer compresses the recommendation model using INT8 quantization, a standard technique that reduces model precision from 32-bit floats to 8-bit integers, cutting memory footprint and improving throughput. Offline benchmarks on the standard evaluation set show 0.4% accuracy degradation. The threshold for acceptable accuracy loss is 1%. The compression passes the gate.
Six weeks later, a data analyst notices something in the recommendation engagement dashboard: click-through rate among users who joined the platform within the last 30 days has dropped roughly 4% compared to the pre-compression baseline. Overall CTR is unchanged. The compression apparently hit new users harder than existing users: possibly because new users have sparser interaction histories, and quantization affects the model's handling of sparse inputs more than dense ones.
The 0.4% benchmark accuracy loss was real. It was also misleading. The degradation wasn't distributed uniformly across users: it concentrated in a segment that the benchmark's evaluation set underrepresented. New users are 12% of total traffic. In the evaluation set, they were 7%. The benchmark diluted the visible impact. The compression passed the accuracy gate while systematically degrading recommendations for users in their first month on the platform.
This failure mode is specific to model compression, and it's structurally invisible to standard metrics. Overall accuracy after compression tells you the population-average quality. It doesn't tell you where the quality loss concentrated. The benchmark evaluation set is usually a random sample designed to be representative of overall traffic, not representative of each subpopulation, especially minority segments that the model may be least confident about.
Quantization, distillation, and pruning all reduce model capacity. That capacity reduction doesn't fall evenly across inputs: it falls hardest on the inputs the model is least certain about, which tend to be the rarer, more complex, and less frequently seen patterns. These are often the edge cases, the minority classes, or the underrepresented user segments. Compression's worst-case performance lands on the cases that are often most important to get right.
Legacy metrics miss this because they measure central tendency. Accuracy, AUC, F1, MAPE. These are averages or macro-level statistics. Compression-induced quality loss that concentrates in a 10% subpopulation while leaving the other 90% unchanged barely registers in a population-average metric, yet may be causing substantial harm to that 10%.
The measurement gap: there is no standard metric that measures the worst-case performance degradation of a compressed model across production traffic subpopulations. That's what Compression Quality Regression Depth is designed to fill.
Formula
Compression Quality Regression Depth (CQRD) is proposed. It doesn't appear in model serving platforms, compression toolkits, or ML monitoring dashboards as a standard metric. It's an extension of fairness evaluation methodology applied to the operational context of model compression: measuring not just average quality loss, but where the quality loss concentrates.
Definition:
CQRD measures the maximum performance degradation of a compressed (quantized, distilled, or pruned) model versus its full-precision counterpart, measured across defined production traffic subpopulations. Specifically identifying where compression concentrates quality loss rather than averaging it across the full population.
The Formula
Plain language: Serve both the full-precision model and the compressed model on production traffic. Either via A/B testing or shadow mode. For each defined subpopulation (user segment, geographic region, input category, value tier), compute the performance difference. Report the worst case across all subpopulations.
Notation:CQRD = max_k |P_full,k − P_compressed,k|Where:
- P_full,k = performance of full-precision model on subpopulation k, measured on the same production traffic
- P_compressed,k = performance of compressed model on subpopulation k, measured on the same traffic
- Maximum is taken over all defined subpopulations k (minimum two; more is better)
Sign convention: CQRD is always non-negative. It's a maximum absolute difference. Higher CQRD indicates worse regression depth. CQRD = 0 means no subpopulation is measurably degraded by compression relative to the full-precision model.
Units: Same as the underlying performance metric (percentage points of accuracy, AUC delta, MAPE change). State the unit.
Worked Example
An NLP classification model is quantized from FP32 to INT8. The team defines four subpopulations based on input length: short inputs (under 10 tokens), medium inputs (10–50 tokens), long inputs (50–200 tokens), and very long inputs (over 200 tokens). They run A/B traffic for 30 days with equal traffic split. Results:
- Short inputs: FP32 accuracy 91.2%, INT8 accuracy 90.8%. Delta 0.4 pp
- Medium inputs: FP32 accuracy 88.4%, INT8 accuracy 87.9%. Delta 0.5 pp
- Long inputs: FP32 accuracy 84.1%, INT8 accuracy 83.7%. Delta 0.4 pp
- Very long inputs: FP32 accuracy 79.3%, INT8 accuracy 76.1%. Delta 3.2 ppOverall: FP32 accuracy 87.8%, INT8 accuracy 87.4%. Delta 0.4 pp (passes the standard benchmark gate). CQRD = 3.2 percentage points. The worst-case subpopulation degraded 8× more than the overall benchmark suggested. The compression should not have been approved without a remediation plan for very long inputs: either keeping the full-precision model for that segment or accepting the degradation with explicit business sign-off.
Edge Cases
Subpopulations with small sample sizes produce unreliable estimates. Require a minimum sample size per subpopulation: 500 predictions is a practical minimum; 2,000 gives acceptable precision for a 1-point accuracy measurement. Subpopulations below the minimum sample size should be flagged as "insufficient sample" rather than included in the CQRD maximum. Multiple compression levels (INT8, INT4, weight pruning): compute CQRD separately for each compression configuration. Report as a comparison table: CQRD by compression level. Often the incremental CQRD from INT8 to INT4 is more revealing than the INT8 result alone. No defined subpopulations: define them at minimum by model confidence score decile (high-confidence vs. low-confidence predictions). Compression quality loss tends to concentrate in low-confidence predictions. This is a natural and practically relevant dimension for any model.
Cross-References
CQRD is downstream of CSS (GMI-IT-021). If the canary window was too small to detect the subpopulation regression that CQRD measures, CQRD will go undetected at deployment time: the canary will pass, the compression will ship, and CQRD will only be discovered via dedicated measurement after the fact. This is the chain: CSS tells you whether your canary could have caught CQRD; CQRD tells you what CSS may have missed. CQRD flows upstream to MRII (GMI-IT-023). If compression degrades quality on high-value subpopulations, the inference cost savings are offset by business value loss. The MRII calculation for a compressed model must account for CQRD's impact on the value side: reduced cost minus reduced value, not just reduced cost.
Proposed formula, pending validation pilots.