Model Calibration Accuracy (MCA)

GMI-WT-005Water TreatmentIntegration-Required

Definition

Measures whether an AI model's stated confidence or uncertainty corresponds to its actual prediction accuracy, reported as a score from 0 (complete miscalibration) to 1 (perfect calibration where 90%-confident predictions are right 90% of the time).

The Measurement Gap

AI water quality prediction and pipe failure models report confidence scores, but no metric tracks whether 90%-confident predictions are actually correct 90% of the time. During edge conditions (algal blooms, reservoir turnover, industrial discharge, freeze-thaw cycles) models may be confidently wrong. Turbidity, compliance rate, and break rate only catch miscalibration after the predicted event either happens or does not. The gap is how long the model was confidently wrong before anyone noticed. No legacy KPI measures model output quality: they measure process output quality, which is downstream and delayed.

Decision Supported:

When the water quality model says "90% confident tomorrow's raw water turbidity will be below 10 NTU," can operators trust that confidence level, or is the model systematically overconfident under certain conditions?

Formula

Plain language: Collect all predictions the model has made over the reporting period along with their confidence scores. Bin the predictions by confidence level (50–60%, 60–70%, etc.). For each bin, check what fraction of predictions were actually correct. If the model is well-calibrated, the 80–90% bin should have about 85% accuracy. Calculate the weighted deviation between expected and observed accuracy across all bins. Subtract from 1.

Notation:

MCA = 1 − Σ ( Nᵦ / N ) × |Expectedᵦ − Observedᵦ|

This is 1 minus the Expected Calibration Error (ECE).

Variables:

Variable Definition Unit
Nᵦ Number of predictions in confidence bin b Count
N Total predictions in reporting period Count
Expectedᵦ Midpoint of confidence bin b (e.g., 0.85 for the 80–90% bin) Fraction
Observedᵦ Fraction of predictions in bin b that were correct Fraction
b Confidence bins, typically 10 bins of 10% width from 0–100% n/a

Proposed formula, pending validation pilots.

At a Glance

Unit
Score (0 to 1)
Trend Direction
Toward 1.0
Reporting Frequency
Monthly, segmented by operating condition
Category
AI Decision Quality
← Model Accuracy Trend (MAT)Autonomous System Coherence Score (ASCS) →

Assess Your Organization

The AI-GPS Assessment evaluates your AI governance maturity and recommends KPIs like this one for your implementation roadmap.

Take the Free AI-GPS Assessment