Water Quality Prediction Confidence Alignment (WQPC)

GMI-WT-019Water TreatmentIntegration-RequiredPattern O-1: Confidence Calibration

Definition

WQPC measures whether a water quality prediction model's stated confidence levels match its actual prediction accuracy across defined parameter bands and prediction horizons.

The Measurement Gap

Measurement Gap and Decision Supported

The Invisible Failure Mode: A drinking water plant deploys a machine learning model to predict finished water turbidity 4 hours ahead, enabling proactive coagulant dosing adjustments. The model reports confidence intervals with every prediction, and operators have learned to trust predictions flagged as "high confidence." But nobody has checked whether the model's confidence claims are actually calibrated. When the model says 90% confidence, it might only be correct 74% of the time for turbidity predictions during spring runoff conditions. The operator trusts the number, reduces manual sampling frequency, and misses a turbidity excursion that triggers a boil-water advisory. The model was not wrong in concept: it was wrong about how right it was.

What Existing Metrics Miss:

Standard model performance metrics like RMSE or R² measure overall accuracy but ignore confidence calibration. A model can have excellent average accuracy while being systematically overconfident during the specific conditions that matter most: high-turbidity events, seasonal transitions, or unusual source water chemistry. Without WQPC, utilities cannot distinguish between a model that knows what it does not know and a model that confidently makes poor predictions during critical operating conditions.

The Decision WQPC Enables:

Should operators trust this model's confidence claims for treatment decisions, or does the confidence calibration gap require maintaining manual verification? WQPC quantifies this trust question by parameter band and prediction horizon, allowing utilities to set evidence-based thresholds for when model-guided decisions are safe and when human verification remains necessary.

Cross-References:

WQPC connects upstream to source water monitoring variability and model training data representativeness. Downstream, WQPC calibration quality affects Chemical Dosing Lag and Impact Score (CLDIS, GMI-WT-016) because overconfident predictions drive incorrect dosing adjustments. WQPC also feeds Corrective Action Margin Erosion Index (CMEI, GMI-WT-015): overconfident models that drive process changes without warranted certainty erode the margin available for corrective action. On the maintenance KPI side, WQPC parallels Model Calibration Accuracy (MCA, GMI-WT-005), which measures the same fundamental calibration problem for asset condition and maintenance prediction models. The measurement approach is identical; the domain data differs.

Formula

Plain language: Group model predictions into confidence bands (e.g., 70-79%, 80-89%, 90-100%). For each band, compute the actual prediction accuracy (percentage of predictions that fell within the model's stated error bounds). Perfect calibration means the 90% confidence band has 90% actual accuracy.

Notation:
WQPC(b, p) = Actual_accuracy(b, p) / Stated_confidence(b)

Where:

  • b = confidence band (e.g., 70-79%, 80-89%, 90-100%)
  • p = water quality parameter (turbidity, TOC, pH, temperature, algal indicators)
  • Actual_accuracy(b, p) = fraction of predictions in band b for parameter p where the actual measurement fell within the model's predicted range
  • Stated_confidence(b) = midpoint of confidence band b

Units: Ratio per band per parameter. 1.0 = perfectly calibrated. >1.0 = underconfident (conservative, generally safe). <1.0 = overconfident (dangerous: the model claims more certainty than it delivers).

Proposed formula, pending validation pilots.

At a Glance

Category
Water Quality Prediction Reliability
← Prediction-Measurement Substitution Gap (PMSG)

Assess Your Organization

The AI-GPS Assessment evaluates your AI governance maturity and recommends KPIs like this one for your implementation roadmap.

Take the Free AI-GPS Assessment