Diagnostic Confidence Calibration Index (DCCI)

GMI-HC-001HealthcareIntegration-RequiredPattern O-1: Confidence Calibration

Definition

Ratio of the diagnostic AI's actual accuracy to its stated confidence at each confidence band, measuring whether clinicians can trust the confidence signal to calibrate their verification effort.

The Measurement Gap

The Decision That Legacy Metrics Cannot Support

Should a radiologist invest additional verification time on a study the AI scored at 95% confidence, or is that 95% stated confidence only worth 78% actual accuracy, meaning high-confidence reads need the same scrutiny as low-confidence ones?

What Goes Unmeasured Today

Healthcare organizations deploying AI diagnostic tools track overall accuracy (sensitivity, specificity, positive predictive value) but almost none measure whether the confidence scores attached to individual findings are calibrated. A diagnostic AI might report 95% confidence on a chest CT finding, but if findings in the 90-100% confidence band are only correct 78% of the time, radiologists are making verification decisions based on a misleading signal. The confidence score becomes a trust proxy that directs clinical effort (high confidence reads get less scrutiny, low confidence reads get more) and if that proxy is miscalibrated, verification effort is systematically misallocated.

Legacy quality metrics like diagnostic accuracy rates, peer review discordance rates, and quality assurance random sampling capture average performance across all reads. They cannot detect that the AI performs differently across confidence bands, that overconfidence in the highest band (where radiologists invest the least additional effort) is the most dangerous failure mode, or that calibration may drift over time as patient populations and imaging protocols change. Without band-level calibration measurement, organizations cannot distinguish between an AI that honestly reports uncertainty and one that inflates confidence, and radiologists cannot calibrate their trust accordingly.

The Invisible Failure Mode

Miscalibrated diagnostic confidence creates a compounding problem. When high-confidence AI findings turn out to be wrong more often than the confidence score implies, clinicians who trusted the score may have missed findings they would have caught with standard verification. Over time, this erodes the diagnostic safety margin that AI assistance was supposed to improve. The failure is invisible in aggregate accuracy metrics because the errors are distributed across individual reads: each one defensible in isolation, collectively representing a systematic misallocation of expert attention.

Formula

Primary Formula

DCCI(b) = Actual_Accuracy(b) / Stated_Confidence_Midpoint(b)

Aggregate:

DCCI_overall = Σ [DCCI(b) × n(b)] / Σ n(b)

where n(b) = number of findings in band b.

Variable Definitions

Variable Definition Unit
b Confidence band (e.g., 90–100%, 80–89%, 70–79%) Band range
Actual_Accuracy(b) Confirmed-correct findings / total findings in band b Ratio (0–1)
Stated_Confidence_Midpoint(b) Midpoint of confidence band (e.g., 0.95 for 90–100%) Ratio (0–1)
n(b) Number of findings in band b Count
Confirmation source Final radiologist read + clinical follow-up outcome (pathology, surgery, subsequent imaging) n/a

Data Requirements

Data Element Source System Classification
AI confidence score per finding AI diagnostic platform NATIVE
AI finding type and location AI diagnostic platform NATIVE
Final radiologist/pathologist interpretation PACS/RIS NATIVE
Clinical outcome confirmation EHR: pathology module, surgical notes INTEGRATION
Finding-level concordance Derived: match AI findings to final interpretation DERIVED

Edge Cases and Exclusions

Bands with fewer than 30 findings should be reported as "insufficient sample: do not interpret." Ground truth ambiguity cases should use consensus read or pathology confirmation where available; equivocal cases should be excluded from accuracy calculation. Multi-finding studies should assess calibration per finding, not per study. If the AI does not report confidence scores, DCCI is undefined: advocate for confidence output in procurement specifications.

Proposed formula, pending validation pilots.

At a Glance

Category
Diagnostic AI Calibration and Clinical Trust
← Automation Bias Differential (ABD)Energy-Environment AI Coherence Score (EEACS) →

Assess Your Organization

The AI-GPS Assessment evaluates your AI governance maturity and recommends KPIs like this one for your implementation roadmap.

Take the Free AI-GPS Assessment