Definition
Measures the systematic bias in AI computer vision damage estimates by property type, construction class, or damage category, revealing segments where the model overestimates or underestimates relative to expert assessment.
The Measurement Gap
Computer vision models are increasingly used to estimate property damage from photographs and aerial imagery. These models can process claims at a speed human adjusters cannot match, enabling straight-through processing for routine damage estimates. But speed is not accuracy, and aggregate accuracy is not segment accuracy.
A model can achieve 95% accuracy on average while systematically overestimating hail damage on wood frame structures by 18% and underestimating fire damage on manufactured housing by 22%. The average hides the bias. Segments where the model overestimates create claims leakage: the carrier pays more than the damage warrants. Segments where the model underestimates create customer harm and errors-and-omissions litigation risk: the carrier pays less than the damage warrants.
Legacy metrics for AI damage estimation report aggregate accuracy against expert assessment. They do not decompose performance by property segment to reveal systematic directional bias. A claims director reviewing aggregate model accuracy sees a number that looks acceptable while specific property types receive materially incorrect estimates every time.
DESB closes this gap by computing the mean signed error by property segment, revealing not just where the model is inaccurate but in which direction it is wrong. Overestimation and underestimation have different business consequences and require different remediation strategies.
Decision Supported:
For which property types and damage categories is the computer vision model accurate enough to rely on, and for which should the carrier require human inspection to supplement or override the AI estimate?
Formula
Plain Language: For a sample of AI-estimated claims, compare the AI damage estimate against expert assessment. Compute the mean signed error (bias) by property segment. Segments with consistent positive bias indicate overestimation; segments with consistent negative bias indicate underestimation.
Formula:
DESB(s) = (1/N_s) x Sum of (Estimate_AI_i - Estimate_expert_i) / Estimate_expert_i x 100
Variables:
- s: property segment (construction class multiplied by damage type, e.g., "wood frame / hail," "commercial flat roof / wind," "manufactured housing / fire")
- Estimate_AI_i: AI computer vision damage estimate for claim i in segment s (dollars)
- Estimate_expert_i: expert assessor's damage estimate for the same claim (dollars)
- N_s: number of sampled claims in segment s
- Positive DESB(s) indicates overestimation in segment s. Negative indicates underestimation.
Units:
Signed percentage. Report both segment bias and absolute magnitude.
Data Requirements:
| Data Element | Source System | Classification |
|---|---|---|
| AI damage estimates with property/damage classification | Computer vision estimation platform (Tractable, Verisk, Cape Analytics) | NATIVE |
| Expert field assessments for sampled claims | Structured audit program / independent adjusting firms | MANUAL |
| Property characteristics (construction, occupancy, age) | Underwriting data / property databases | NATIVE |
| Damage type classification | Claims management system | NATIVE |
Proposed formula, pending validation pilots.