Definition
Weighted deviation between AI-predicted asset failure probabilities and observed failure rates across score bands: the utility-specific equivalent of Expected Calibration Error applied to asset health models.
The Measurement Gap
Utility asset management has historically relied on three tracking mechanisms: age-based replacement schedules, dissolved gas analysis (DGA) interpreted by engineers, and visual inspection condition scores on a 1–5 scale. These approaches share a structural limitation: they produce judgments about individual assets without systematically comparing model outputs to outcomes. When an engineer rates a transformer as "condition 3," there is no feedback loop that checks whether condition-3 transformers actually fail at the rate the rating implies.
AI health models were supposed to improve on this. And they do improve discrimination: they correctly identify the worst assets better than age alone. What they do not improve, without explicit calibration measurement, is the accuracy of the failure probabilities those scores imply. Vendors validate models at deployment using AUC-ROC, precision, and recall: metrics that measure ranking quality, not probability accuracy. A model can have an AUC of 0.85 and be miscalibrated by 20 percentage points in the bands that drive the most capital decisions.
The gap is that no system in a utility's standard toolset compares health score outputs to failure outcomes at the band level over multi-year windows. OMS records failures. CMMS records replacements. The analytics platform records scores. None of these systems performs the cross-system join that constitutes calibration measurement.
The decision HSCE enables: determining whether the capital plan's input data is trustworthy before the next $100M+ replacement cycle is approved. If HSCE reveals the model overstates risk in high-score bands, the utility is replacing assets it didn't need to replace. If it understates risk, the utility is deferring assets that are more dangerous than their scores suggest. Without HSCE, the planning team cannot tell which scenario they're in.
AI changes this by making it possible to join health scores, operational outcomes, and consequence data at scale: turning calibration measurement from a manual research exercise into a computable operational metric.
Formula
HSCE = Σ (n_b / N) × |P_predicted_b − P_observed_b|
Variables:
- n_b: number of assets in score band b (count). Source: Asset analytics platform
- N: total number of scored assets in the measurement population (count). Source: Asset registry / CMMS
- P_predicted_b: model-implied failure probability for score band b, derived from the vendor's scoring guide or the historical calibration table (probability, 0–1). Source: Asset analytics platform documentation or vendor validation report
- P_observed_b: actual observed failure rate for assets in score band b over the measurement horizon, calculated as failures divided by asset-years of exposure (probability, 0–1). Source: OMS (in-service failures) + CMMS (condition-based removals)
Score bands are defined by equal intervals across the health model's output range (e.g., 1.0–1.5, 1.5–2.0 for a 1–5 scale) or by the vendor's own risk tier definitions if documented.
Calculation Frequency:
Annual (minimum 3-year failure observation window; 5-year preferred for power transformers)
Output:
0 to 1 scale; 0 = perfectly calibrated. Below 0.05 = acceptable. 0.05–0.10 = marginal, monitor closely. Above 0.10 = recalibrate before next capital cycle. Compute HSCE_over and HSCE_under separately to identify bias direction: overstating risk drives excess capital spend; understating risk drives dangerous deferrals.
Proposed formula, pending validation pilots.
At a Glance
Decision Supported
When the AI scores a transformer at "3.2 out of 5," what failure probability does that actually correspond to, and does the model's implied probability match reality?