Critical Alert Override Discrimination (CAOD)

GMI-HC-004HealthcarePilot-ReadyPattern O-4: Alert Quality / Signal-to-Noise Ratio

Definition

Ratio of override rates for high-clinical-significance alerts versus low-clinical-significance alerts, measuring whether clinicians can distinguish important alerts from noise in their override behavior.

The Measurement Gap

The Decision That Legacy Metrics Cannot Support

Are clinicians overriding important CDS alerts at the same rate as trivial ones (meaning alert fatigue has eliminated their ability to discriminate) or is the override pattern clinically rational?

What Goes Unmeasured Today

Clinical decision support systems fire thousands of alerts per provider per month: drug interaction warnings, duplicate order checks, sepsis screening triggers, dosing range alerts. Healthcare organizations measure aggregate override rates and sometimes track them by alert category. What they do not measure is whether clinicians are discriminating between alert significance levels in their override behavior. A 90% override rate is concerning if it applies equally to critical drug interaction alerts and trivial informational notices; it is less concerning if critical alerts are overridden at 40% and informational alerts at 95%.

Alert fatigue is one of the most studied problems in health IT, yet the standard metrics (total alert volume, aggregate override rate, and alert-to-action ratio) measure the quantity of the problem without measuring the quality of clinician response. Legacy metrics cannot answer the discrimination question because they do not stratify override behavior by clinical significance tier. An organization reporting a 92% aggregate override rate may have excellent discrimination (critical alerts at 30%, informational at 98%) or catastrophic discrimination (both tiers at 91-93%).

The clinical significance tier itself is part of the measurement challenge. Most CDS systems assign severity levels internally, but these system-assigned levels may not reflect actual clinical importance. CAOD requires clinical informatics committees to independently classify alerts by potential patient harm, a governance step that most organizations have not formalized.

The Invisible Failure Mode

When alert fatigue progresses to the point where critical alerts are dismissed at the same rate as trivial ones, the CDS system has become a negative contributor to patient safety: it consumes attention without directing it. The failure is invisible in aggregate override reporting because the overall rate may remain stable even as discrimination collapses.

Formula

Primary Formula

CAOD = Override_Rate_Critical / Override_Rate_Informational

Variable Definitions

Variable Definition Unit
Override_Rate_Critical Overridden critical alerts / total critical alerts Ratio (0–1)
Override_Rate_Informational Overridden informational alerts / total informational alerts Ratio (0–1)
Clinical significance tier Assigned by clinical informatics committee based on potential patient harm (not by CDS system) Category (Critical / Moderate / Informational)

Data Requirements

Data Element Source System Classification
CDS alert fired (type, content, severity) EHR CDS engine NATIVE
Alert action (overridden, accepted, modified) EHR CDS log NATIVE
Clinical significance tier Clinical informatics classification (manual, reviewed annually) MANUAL
Override reason (if documented) EHR alert override reason field NATIVE (if configured)
Patient outcome post-override EHR adverse event reports, clinical outcomes INTEGRATION

Edge Cases and Exclusions

Alert classification is subjective and should use clinical committee consensus, not AI-assigned severity. Segment CAOD by provider type (physician, pharmacist, nurse) and specialty, as alert fatigue varies by role. Time-of-day effects should be segmented if sample size permits, as overnight override rates may differ. Minimum sample: 100 critical alerts and 100 informational alerts per measurement period per provider type.

Proposed formula, pending validation pilots.

At a Glance

Category
Clinical Decision Support Signal Quality
← CDS Cognitive Yield Ratio (CDSYR)AI Worklist Prioritization Equity Score (WPES) →

Assess Your Organization

The AI-GPS Assessment evaluates your AI governance maturity and recommends KPIs like this one for your implementation roadmap.

Take the Free AI-GPS Assessment