Risk Model Discrimination Currency (RMDC)

GMI-HC-015HealthcareIntegration-RequiredPattern O-2: Model Drift / Currency

Definition

Current C-statistic (area under ROC curve) of deployed risk models measured on the current patient population, compared to the C-statistic at model deployment, measuring whether the model's ability to discriminate high-risk from low-risk patients has degraded as the population has changed.

The Measurement Gap

The Decision That Legacy Metrics Cannot Support

Is the readmission risk model still good enough to drive clinical decisions, or has population drift degraded its discrimination to the point where high-risk scores are no more meaningful than a coin flip?

What Goes Unmeasured Today

Healthcare organizations deploy risk prediction models and assume they maintain their validated performance over time. A readmission risk model validated at C=0.82 on the development population is assumed to perform at C=0.82 on next year's patients. But patient populations change (acuity shifts, payer mix evolves, service lines expand or contract, community demographics shift) and the model's discrimination degrades without anyone measuring the decline.

Most organizations do not recompute C-statistics on their current patient populations. They rely on the vendor's original validation report or an implementation validation performed at go-live. The model's discriminative power may have degraded significantly by the time anyone questions its performance, and the question usually arises only after outcomes fail to improve, a lagging signal that could have been caught earlier with prospective discrimination monitoring.

Legacy model governance metrics (if they exist at all) typically track model utilization (is the score being displayed?) and operational metrics (how many patients are flagged?). These tell the organization that the model is running, not that it is still performing. A model can generate scores, display them in the workflow, and flag patients at the same rate even after its discrimination has degraded to clinically useless levels.

The Invisible Failure Mode

Risk model discrimination degrades silently. The model continues to produce scores, the scores continue to appear in the EHR, and clinicians continue to see them. Because no one recomputes the C-statistic on the current population, the degradation is invisible until outcomes analysis reveals that the model is not working, typically months or years after the degradation began.

Formula

Primary Formula

RMDC = C_current / C_deployment

Variable Definitions

Variable Definition Unit
C_current C-statistic of the risk model computed on trailing 12-month patient population AUC (0.5–1.0)
C_deployment C-statistic reported at model deployment or last formal validation AUC (0.5–1.0)

Data Requirements

Data Element Source System Classification
Risk model predictions (patient, score, model version) EHR risk model output NATIVE
Patient outcomes (readmission, sepsis, fall, etc.) EHR, claims data NATIVE
Deployment C-statistic Model documentation / vendor validation report MANUAL
Current population characteristics EHR demographic and clinical data NATIVE

Edge Cases and Exclusions

Outcome data lag varies by model: readmission outcomes have a 30-day lag, sepsis outcomes are faster. Match the recomputation schedule to outcome availability. If the vendor pushes model updates, reset C_deployment to the new version's validation statistic. Compute RMDC for each deployed model separately. Minimum sample: 500 patients with known outcomes per model per recomputation.

Proposed formula, pending validation pilots.

At a Glance

Category
AI Governance - Risk Model Governance
← Schedule Equity Distribution Index (SEDI)Risk Score Action Rate (RSAR) →

Assess Your Organization

The AI-GPS Assessment evaluates your AI governance maturity and recommends KPIs like this one for your implementation roadmap.

Take the Free AI-GPS Assessment