Definition
Measures the change in underwriter pricing accuracy on full-distribution risks over time, revealing whether handling only AI-referred outlier cases is degrading the underwriter's ability to price the routine risks they no longer see.
The Measurement Gap
When AI handles routine risk selection and only refers outliers to human underwriters, the underwriters' exposure narrows to the hardest 20% of submissions. Over time, they lose calibration on the routine risks that make up 80% of the book. If the AI model fails or needs to be overridden at scale (regulatory change, model degradation, catastrophe recalibration), the underwriting team may lack the skill to price the full distribution accurately. Legacy metrics cannot detect this skill erosion. Loss ratio tracks aggregate outcomes but cannot isolate underwriter pricing accuracy from model accuracy. Combined ratio measures overall profitability without distinguishing human contribution from AI contribution. Hit ratio (quote-to-bind) measures conversion efficiency, not pricing accuracy. Referral rate tracks volume of human review but not quality of human judgment. Time to quote measures speed, not skill. None of these metrics assess whether the human underwriting workforce retains the capability to function independently of AI support.
Decision Supported:
Do underwriters retain the calibration to price the full risk distribution, or has AI-only-the-exceptions handling created a workforce that can only evaluate the hardest 20% of submissions, and what training investment is needed to maintain full-spectrum capability?
Formula
Plain language: Periodically present underwriters with a blind sample of risks that includes both AI-auto-decided cases and referred cases, without telling them which is which. Compare their pricing to actual loss outcomes with lag. Track whether their accuracy on the "easy" risks degrades over time.
Notation:
USRI = 1 - |MAPE_UW_current - MAPE_UW_baseline| / MAPE_UW_baseline
Where: MAPE_UW = (1/N) x Sum of |Price_UW_i - Loss_actual_i| / Loss_actual_i
Variables:
| Variable | Definition | Unit |
|---|---|---|
| Price_UW_i | Underwriter's blind-assessed price for risk i | Dollars (premium) |
| Loss_actual_i | Actual incurred loss for risk i over the exposure period | Dollars |
| MAPE_UW_current | Mean absolute percentage error of underwriter pricing in the current assessment period | % |
| MAPE_UW_baseline | MAPE from the first assessment period post-AI deployment (or pre-AI if available) | % |
| N | Number of blind-sampled risks in the assessment | Count |
Values near 1.0 indicate no degradation from baseline. Declining values indicate skill atrophy. Values above 1.0 indicate improvement.
Confidence Grading (required alongside every USRI value):
| Grade | Basis | Interpretation |
|---|---|---|
| A: Empirical | 5+ observed events in measurement window provide direct tail observation | Measurement grounded in observed outcomes; highest confidence |
| B: Hybrid | 1-4 observed events plus Monte Carlo simulation with Bayesian updating | Blends observation and simulation; moderate confidence |
| C: Simulated | Zero observed events; derived entirely from Monte Carlo simulation | Modeled, not observed; treat as indicative, not validated |
A USRI of any value at Grade C is a simulation that agrees with itself. It tells you the model is internally consistent but not that it matches reality. Grade C scores should trigger proactive stress-testing against historical near-miss events and cross-asset analogues rather than acceptance as calibrated.
Proposed formula, pending validation pilots.