Confidence Interval Honesty Index (CIHI)

GMI-INS-012InsuranceInfrastructure-Dependent

Definition

Measures whether ML reserve models' stated confidence intervals are honest, specifically whether a 90% confidence interval actually contains the true development outcome 90% of the time or whether overfitting produces artificially narrow intervals.

The Measurement Gap

ML reserve models produce confidence intervals that the reserving committee uses to set reserve ranges and allocate capital. The problem is that ML models, particularly gradient boosting and neural network architectures, tend to produce overconfident intervals. A stated 90% confidence interval may actually contain the true development outcome only 70% of the time. Legacy actuarial model validation focuses on point-estimate accuracy but does not systematically test whether stated confidence intervals are calibrated. An actuary sees a tight interval and interprets it as high certainty, when the interval is simply too narrow. No existing metric detects this, which means capital allocation decisions rest on dishonest uncertainty estimates.

Decision Supported:

Is the actuarial team basing reserve range estimates on confidence intervals that are honest about uncertainty, or are ML models producing intervals that look precise but are systematically overconfident?

Formula

Plain Language: For each historical reserve estimate where the development outcome is now known, check whether the actual ultimate loss fell within the model's stated 90% confidence interval at the time of the estimate. The percentage of times the actual falls within the stated interval should be close to 90%. If it is substantially lower, the intervals are overconfident.

Formula:

CIHI(p) = (1/M) x Sum of I(Actual_j is in CI_p_j) / p

Where I(Actual_j is in CI_p_j) = 1 if actual ultimate loss j falls within the p% confidence interval, 0 otherwise.

Variables:

  • p: stated confidence level (e.g., 90%)
  • M: number of historical reserve estimates with known ultimate outcomes
  • CI_p_j: the model's stated p% confidence interval for reserve estimate j at the time of estimation
  • Actual_j: the actual ultimate loss for reserve estimate j (known after development)
  • CIHI(p) near 1.0 means intervals are honest at confidence level p. Below 1.0 means intervals are overconfident (too narrow). Above 1.0 means intervals are underconfident (too wide).

Units:

Ratio. Compute at multiple confidence levels (50%, 75%, 90%, 95%) to build a calibration curve.

Data Requirements:

Data Element Source System Classification
ML model reserve point estimates and confidence intervals (at time of estimation) Actuarial reserving platform / model documentation NATIVE (if intervals stored) / NOT CAPTURED (if only point estimates stored)
Actual ultimate losses by accident year/segment Actuarial loss development triangles NATIVE
Traditional actuarial method intervals (for comparison) Actuarial work papers MANUAL
Model training vintage and development data cutoff Model documentation MANUAL

Proposed formula, pending validation pilots.

At a Glance

Unit
Ratio
Trend Direction
Toward 1.0
Reporting Frequency
Annually (with development lag)
Category
Actuarial Model Trust and Reserve Integrity
Emerging Liability Blind Spot Index (ELBSI) →

Assess Your Organization

The AI-GPS Assessment evaluates your AI governance maturity and recommends KPIs like this one for your implementation roadmap.

Take the Free AI-GPS Assessment