Definition
Ratio of the diagnostic AI's actual accuracy to its stated confidence at each confidence band, measuring whether clinicians can trust the confidence signal to calibrate their verification effort.
The Measurement Gap
The Decision That Legacy Metrics Cannot Support
Should a radiologist invest additional verification time on a study the AI scored at 95% confidence, or is that 95% stated confidence only worth 78% actual accuracy, meaning high-confidence reads need the same scrutiny as low-confidence ones?
What Goes Unmeasured Today
Healthcare organizations deploying AI diagnostic tools track overall accuracy (sensitivity, specificity, positive predictive value) but almost none measure whether the confidence scores attached to individual findings are calibrated. A diagnostic AI might report 95% confidence on a chest CT finding, but if findings in the 90-100% confidence band are only correct 78% of the time, radiologists are making verification decisions based on a misleading signal. The confidence score becomes a trust proxy that directs clinical effort (high confidence reads get less scrutiny, low confidence reads get more) and if that proxy is miscalibrated, verification effort is systematically misallocated.
Legacy quality metrics like diagnostic accuracy rates, peer review discordance rates, and quality assurance random sampling capture average performance across all reads. They cannot detect that the AI performs differently across confidence bands, that overconfidence in the highest band (where radiologists invest the least additional effort) is the most dangerous failure mode, or that calibration may drift over time as patient populations and imaging protocols change. Without band-level calibration measurement, organizations cannot distinguish between an AI that honestly reports uncertainty and one that inflates confidence, and radiologists cannot calibrate their trust accordingly.
The Invisible Failure Mode
Miscalibrated diagnostic confidence creates a compounding problem. When high-confidence AI findings turn out to be wrong more often than the confidence score implies, clinicians who trusted the score may have missed findings they would have caught with standard verification. Over time, this erodes the diagnostic safety margin that AI assistance was supposed to improve. The failure is invisible in aggregate accuracy metrics because the errors are distributed across individual reads: each one defensible in isolation, collectively representing a systematic misallocation of expert attention.
Formula
Primary Formula
DCCI(b) = Actual_Accuracy(b) / Stated_Confidence_Midpoint(b)
Aggregate:
DCCI_overall = Σ [DCCI(b) × n(b)] / Σ n(b)
where n(b) = number of findings in band b.
Variable Definitions
| Variable | Definition | Unit |
|---|---|---|
| b | Confidence band (e.g., 90–100%, 80–89%, 70–79%) | Band range |
| Actual_Accuracy(b) | Confirmed-correct findings / total findings in band b | Ratio (0–1) |
| Stated_Confidence_Midpoint(b) | Midpoint of confidence band (e.g., 0.95 for 90–100%) | Ratio (0–1) |
| n(b) | Number of findings in band b | Count |
| Confirmation source | Final radiologist read + clinical follow-up outcome (pathology, surgery, subsequent imaging) | n/a |
Data Requirements
| Data Element | Source System | Classification |
|---|---|---|
| AI confidence score per finding | AI diagnostic platform | NATIVE |
| AI finding type and location | AI diagnostic platform | NATIVE |
| Final radiologist/pathologist interpretation | PACS/RIS | NATIVE |
| Clinical outcome confirmation | EHR: pathology module, surgical notes | INTEGRATION |
| Finding-level concordance | Derived: match AI findings to final interpretation | DERIVED |
Edge Cases and Exclusions
Bands with fewer than 30 findings should be reported as "insufficient sample: do not interpret." Ground truth ambiguity cases should use consensus read or pathology confirmation where available; equivocal cases should be excluded from accuracy calculation. Multi-finding studies should assess calibration per finding, not per study. If the AI does not report confidence scores, DCCI is undefined: advocate for confidence output in procurement specifications.
Proposed formula, pending validation pilots.