Definition
Measures whether an AI model's stated confidence or uncertainty corresponds to its actual prediction accuracy, reported as a score from 0 (complete miscalibration) to 1 (perfect calibration where 90%-confident predictions are right 90% of the time).
The Measurement Gap
AI water quality prediction and pipe failure models report confidence scores, but no metric tracks whether 90%-confident predictions are actually correct 90% of the time. During edge conditions (algal blooms, reservoir turnover, industrial discharge, freeze-thaw cycles) models may be confidently wrong. Turbidity, compliance rate, and break rate only catch miscalibration after the predicted event either happens or does not. The gap is how long the model was confidently wrong before anyone noticed. No legacy KPI measures model output quality: they measure process output quality, which is downstream and delayed.
Decision Supported:
When the water quality model says "90% confident tomorrow's raw water turbidity will be below 10 NTU," can operators trust that confidence level, or is the model systematically overconfident under certain conditions?
Formula
Plain language: Collect all predictions the model has made over the reporting period along with their confidence scores. Bin the predictions by confidence level (50–60%, 60–70%, etc.). For each bin, check what fraction of predictions were actually correct. If the model is well-calibrated, the 80–90% bin should have about 85% accuracy. Calculate the weighted deviation between expected and observed accuracy across all bins. Subtract from 1.
Notation:
MCA = 1 − Σ ( Nᵦ / N ) × |Expectedᵦ − Observedᵦ|
This is 1 minus the Expected Calibration Error (ECE).
Variables:
| Variable | Definition | Unit |
|---|---|---|
| Nᵦ | Number of predictions in confidence bin b | Count |
| N | Total predictions in reporting period | Count |
| Expectedᵦ | Midpoint of confidence bin b (e.g., 0.85 for the 80–90% bin) | Fraction |
| Observedᵦ | Fraction of predictions in bin b that were correct | Fraction |
| b | Confidence bins, typically 10 bins of 10% width from 0–100% | n/a |
Proposed formula, pending validation pilots.