Underwriter Skill Retention Index (USRI)

GMI-INS-003InsuranceInfrastructure-Dependent

Definition

Measures the change in underwriter pricing accuracy on full-distribution risks over time, revealing whether handling only AI-referred outlier cases is degrading the underwriter's ability to price the routine risks they no longer see.

The Measurement Gap

When AI handles routine risk selection and only refers outliers to human underwriters, the underwriters' exposure narrows to the hardest 20% of submissions. Over time, they lose calibration on the routine risks that make up 80% of the book. If the AI model fails or needs to be overridden at scale (regulatory change, model degradation, catastrophe recalibration), the underwriting team may lack the skill to price the full distribution accurately. Legacy metrics cannot detect this skill erosion. Loss ratio tracks aggregate outcomes but cannot isolate underwriter pricing accuracy from model accuracy. Combined ratio measures overall profitability without distinguishing human contribution from AI contribution. Hit ratio (quote-to-bind) measures conversion efficiency, not pricing accuracy. Referral rate tracks volume of human review but not quality of human judgment. Time to quote measures speed, not skill. None of these metrics assess whether the human underwriting workforce retains the capability to function independently of AI support.

Decision Supported:

Do underwriters retain the calibration to price the full risk distribution, or has AI-only-the-exceptions handling created a workforce that can only evaluate the hardest 20% of submissions, and what training investment is needed to maintain full-spectrum capability?

Formula

Plain language: Periodically present underwriters with a blind sample of risks that includes both AI-auto-decided cases and referred cases, without telling them which is which. Compare their pricing to actual loss outcomes with lag. Track whether their accuracy on the "easy" risks degrades over time.

Notation:

USRI = 1 - |MAPE_UW_current - MAPE_UW_baseline| / MAPE_UW_baseline

Where: MAPE_UW = (1/N) x Sum of |Price_UW_i - Loss_actual_i| / Loss_actual_i

Variables:

Variable Definition Unit
Price_UW_i Underwriter's blind-assessed price for risk i Dollars (premium)
Loss_actual_i Actual incurred loss for risk i over the exposure period Dollars
MAPE_UW_current Mean absolute percentage error of underwriter pricing in the current assessment period %
MAPE_UW_baseline MAPE from the first assessment period post-AI deployment (or pre-AI if available) %
N Number of blind-sampled risks in the assessment Count

Values near 1.0 indicate no degradation from baseline. Declining values indicate skill atrophy. Values above 1.0 indicate improvement.

Confidence Grading (required alongside every USRI value):

Grade Basis Interpretation
A: Empirical 5+ observed events in measurement window provide direct tail observation Measurement grounded in observed outcomes; highest confidence
B: Hybrid 1-4 observed events plus Monte Carlo simulation with Bayesian updating Blends observation and simulation; moderate confidence
C: Simulated Zero observed events; derived entirely from Monte Carlo simulation Modeled, not observed; treat as indicative, not validated

A USRI of any value at Grade C is a simulation that agrees with itself. It tells you the model is internally consistent but not that it matches reality. Grade C scores should trigger proactive stress-testing against historical near-miss events and cross-asset analogues rather than acceptance as calibrated.

Proposed formula, pending validation pilots.

At a Glance

Unit
Index (0 to 1)
Trend Direction
Toward 1.0
Reporting Frequency
Quarterly
Category
AI Underwriting and Pricing Integrity
← Telematics Score Calibration Index (TSCI)Reserve Anchor Independence Score (RAIS) →

Assess Your Organization

The AI-GPS Assessment evaluates your AI governance maturity and recommends KPIs like this one for your implementation roadmap.

Take the Free AI-GPS Assessment