Model Accuracy Trend (MAT)

GMI-WT-006Water TreatmentIntegration-Required

Definition

Measures whether a predictive model's accuracy is improving, stable, or degrading over time, reported as the percentage change from baseline accuracy established at deployment or last retraining.

The Measurement Gap

Predictive models degrade as underlying data distributions shift. Climate change alters precipitation patterns, population growth changes demand, aging infrastructure changes failure modes, treatment upgrades change process dynamics. No KPI tracks model accuracy over time. Break rate per mile tells you about pipes, not about your pipe model. The model could become progressively wrong over 18 months while break rate stays flat because the utility happened to replace some genuinely bad pipes. Water quality prediction models may become less accurate as source water chemistry evolves seasonally and year over year. Without tracking accuracy trend, the organization continues trusting outputs from an increasingly inaccurate model.

Decision Supported:

Is the model that was validated 18 months ago still performing at the level it was validated at, and when should you retrain?

Formula

Plain language: Calculate the model's accuracy metric (appropriate to model type) for each rolling time window. Monthly, for example. Compare each window's accuracy to the accuracy measured when the model was deployed or last retrained (the baseline). Report the percentage change from baseline and the trend direction.

Notation:

MAT = (Acc_current − Acc_baseline) / Acc_baseline × 100%

MAT_slope = linear regression slope of Acc_w across trailing W windows

Variables:

Variable Definition Unit
Acc_current Model accuracy in the most recent time window (metric appropriate to model type) Varies by model type
Acc_baseline Model accuracy measured at deployment or last retrain Same as Acc_current
Acc_w Model accuracy in time window w Same as Acc_current
W Number of trailing time windows for trend calculation (typically 6–12 monthly windows) Count

Accuracy metric by model type: classification models (pipe failure, overflow prediction) use F1 score or balanced accuracy; regression models (turbidity prediction, demand forecasting) use RMSE or MAE normalized by baseline; anomaly detection models (leak detection, quality events) use precision and recall separately.

Proposed formula, pending validation pilots.

At a Glance

Unit
Percentage change
Trend Direction
Band; Staying within ±5% of baseline
Reporting Frequency
Monthly with rolling 6-month slope
Category
AI Decision Quality
← Inspection Classification Equity Index (ICEI)Model Calibration Accuracy (MCA) →

Assess Your Organization

The AI-GPS Assessment evaluates your AI governance maturity and recommends KPIs like this one for your implementation roadmap.

Take the Free AI-GPS Assessment