Definition
Training Experiment Efficiency measures the ratio of total compute cost consumed by a training experiment campaign to the performance improvement achieved by the final selected model versus the baseline: capturing the cost per unit of improvement across the full search process.
The Measurement Gap
Your ML team is training a new demand forecasting model. The product manager wants it in production by end of quarter. The team fires up their hyperparameter optimization framework (Optuna, Ray Tune, or one of the managed AutoML services) and kicks off a search campaign. The first 15 experiments run over two days. They find a configuration that improves MAE by 3.2% over baseline. The team is satisfied and could ship.
They keep running. The campaign hits 40 experiments. The best model has now improved by 3.4% over baseline: the additional 25 experiments bought 0.2 percentage points. At $80 per experiment in mixed GPU and CPU costs, that's $2,000 spent for a marginal improvement that may not even be statistically distinguishable from the 15-experiment result in production conditions. Nobody noticed. Nobody was measuring.
This is the pattern that Training Experiment Efficiency is designed to surface. Experiment campaigns run to completion because "we had the budget" or because stopping early feels premature. The question "has this campaign reached the point of diminishing returns?" doesn't get asked systematically because there's no metric that answers it.
Legacy metrics fail here in a specific way. Pipeline success rate measures whether experiments ran and completed. Training cost tracks the total GPU spend for the period. Accuracy or MAE tracks model performance at the campaign endpoint. None of these metrics reveal the shape of the improvement curve: how much improvement was achieved per dollar of compute at each stage of the campaign.
The fundamental blind spot: you can see where you ended up (final model performance) and what you spent (total training cost). You can't see whether you spent the last 60% of the budget on the first 10% of the improvement. The improvement curve that would make the waste obvious is not a standard output of any ML training platform. You'd have to reconstruct it manually from experiment logs, which nobody does, because nobody knows to look.
This matters more as organizations scale their ML operations. A team running three training campaigns per month across 15 models, spending $200–$500 per campaign, can easily burn $20,000–$30,000 per month on training experiments. If 30–40% of that spend falls past the diminishing returns inflection point, the waste is real and structurally invisible to every standard metric the team tracks.
The measurement gap: there is no standard metric that measures the cost per marginal improvement in a training experiment campaign, or that tells you when a campaign has passed the point of economic rationality. That's what Training Experiment Efficiency is designed to address.
Formula
Training Experiment Efficiency (TEE) is proposed. It doesn't appear in MLOps tooling documentation as a standard metric. No AutoML platform computes it automatically. It's a measurement approach that takes existing data from experiment tracking and billing systems and reorganizes it to answer a question those systems weren't designed to answer: was this campaign economically rational?
Definition:
Training Experiment Efficiency measures the ratio of total compute cost consumed by a training experiment campaign to the performance improvement achieved by the final selected model versus the baseline. Capturing the cost per unit of improvement across the full search process.
The Formula
Plain language: Sum the GPU cost of every experiment in the campaign, including failed runs and configurations that didn't improve over baseline. Compute the performance improvement of the final selected model over whatever baseline you're beating. Divide total cost by total improvement. Lower TEE is better: less cost per unit of improvement.
Notation:TEE = Σ C_experiment,j / ΔP_final
Where:
- C_experiment,j = compute cost of experiment j (GPU_hours × GPU_rate), summed over all j experiments in the campaign- ΔP_final = performance improvement of selected final model over baseline, in the relevant metric (accuracy percentage points, AUC delta, MAPE reduction, etc.)
- Summation includes all experiments, including those that failed to improve over baseline
Sign convention: TEE is denominator-sensitive. If ΔP_final is zero (the campaign failed to improve over baseline), TEE is undefined: flag as a "no-improvement campaign" and track the cost separately. TEE cannot be negative. Lower values indicate higher efficiency.
Units: Dollars per percentage-point improvement (or per unit of your performance metric). Always state the unit alongside the value: "TEE of $340 per percentage point of AUC" is meaningful; "TEE of 340" without a unit is not.
Worked Example
A fraud detection team runs a hyperparameter search campaign to improve their transaction classification model. They run 38 experiments over 5 days. Total GPU cost: $3,120 across all experiments, including 8 failed runs. The baseline model has AUC of 0.872. The final selected model has AUC of 0.889, a 0.017 improvement, expressed as 1.7 percentage points of AUC.TEE = $3,120 / 1.7 = $1,835 per percentage point of AUC.Whether $1,835 per percentage point is acceptable depends on context: what's a percentage point of AUC worth for this model? If a 1 percentage point AUC improvement prevents $200,000 in annual fraud losses, $1,835 is well-justified. If it prevents $1,500, the campaign consumed more than it returned. The value of TEE is in comparison and trend: is this campaign more or less efficient than the last one? Are some teams consistently achieving better TEE than others?Now plot the cumulative improvement curve: after experiment 10, the best model had AUC 0.885 (1.3 percentage points of improvement) at a cost of $810. Experiments 11–38 added 0.4 percentage points at $2,310: the marginal cost rate nearly tripled. The campaign should have had a stopping checkpoint at experiment 10. TEE makes the elbow visible.
Edge Cases
Campaigns that fail to improve over baseline: report cost as "no-improvement campaign cost" and track separately. A $3,000 campaign that found nothing is as important to know about as one that found marginal improvements. Both represent waste that TEE is designed to surface. AutoML campaigns with built-in early stopping: TEE works with these too. The early stopping mechanism is the right behavior; TEE measures whether the stopping criterion was calibrated correctly relative to actual marginal returns. Multi-objective campaigns optimizing for both accuracy and inference cost: use a composite performance metric or report TEE for each objective separately.
Cross-References
TEE has no upstream dependencies in this catalog: it's a foundational metric about training efficiency. It flows downstream into REY (GMI-IT-019): if training is wasteful, the C_retrain denominator inflates and REY drops even on productive retrains. And TEE feeds into MRII (GMI-IT-023): training cost is a component of total model cost of ownership, and chronically high TEE across a portfolio signals structural training waste that erodes portfolio economics. Track TEE alongside REY for each model; the combination reveals whether waste is happening at the experiment level, the retrain trigger level, or both.
Proposed formula, pending validation pilots.