Definition
Percentage of ambient AI-generated clinical documentation elements (history, physical exam findings, assessment, plan) that accurately reflect what occurred during the encounter, measured by structured audit against audio recording or clinician attestation.
The Measurement Gap
The Decision That Legacy Metrics Cannot Support
Can the organization trust that AI-generated clinical notes (which become the legal medical record) accurately represent what happened, or is hallucinated and omitted content entering the chart at a rate that creates clinical and legal risk?
What Goes Unmeasured Today
Ambient AI documentation tools (such as those that listen to clinician-patient conversations and generate structured notes) are among the fastest-adopted AI technologies in healthcare. These tools promise to reduce documentation burden (a leading contributor to clinician burnout) by automatically producing encounter notes that the physician reviews and signs. Once signed, the AI-generated note becomes the legal medical record.
Healthcare organizations measure documentation completeness (are all required fields populated?), coding accuracy (do the codes match the narrative?), and physician satisfaction with the documentation tool. What they do not measure is documentation fidelity: does the AI-generated narrative accurately represent what occurred during the encounter? The distinction matters because completeness and accuracy can coexist with fabrication. An AI note that includes all required sections, supports the assigned codes, and satisfies the physician may still contain hallucinated physical exam findings, misattributed patient statements, or fabricated clinical details.
Legacy documentation quality metrics (dictation error rate, transcription accuracy, template completion rate) were designed for human-generated documentation where the failure modes are typographical errors, omissions, and template misuse. AI-generated documentation introduces a fundamentally new failure mode: hallucination. The AI may confidently produce a physical exam finding that was never performed, attribute a symptom to the patient that was never stated, or fabricate a clinical detail that is plausible but false.
The Invisible Failure Mode
Hallucinated clinical content is uniquely dangerous because it is often plausible. A fabricated "lungs clear to auscultation" finding looks exactly like a real one in the medical record. The physician reviewing the note may not catch it, especially if the note is largely accurate and the hallucinated element is clinically unremarkable. Over time, uncaught hallucinations accumulate in the medical record, creating a false clinical history that may affect future treatment decisions.
Formula
Primary Formula
DFI = Accurate_elements / Total_audited_elements
Error Decomposition:
DFI_hallucination = Hallucinated_elements / Total_audited_elements
DFI_omission = Omitted_elements / Total_audited_elements
Variable Definitions
| Variable | Definition | Unit |
|---|---|---|
| Accurate_elements | Documentation elements confirmed as correctly reflecting encounter content | Count |
| Total_audited_elements | Total elements audited across sampled encounters | Count |
| Hallucinated_elements | Elements present in AI note that did not occur (fabricated findings, misattributed statements) | Count |
| Omitted_elements | Clinically relevant elements that occurred but are absent from AI note | Count |
| Audited elements | HPI components, physical exam findings, assessment items, plan items | Categories |
Data Requirements
| Data Element | Source System | Classification |
|---|---|---|
| AI-generated encounter note | EHR: ambient AI module | NATIVE |
| Encounter audio recording | Ambient AI platform | NATIVE (if consent obtained) |
| Clinician attestation / structured audit | Manual review tool | MANUAL |
| Element-level concordance scoring | Audit framework (structured checklist per note section) | NOT CAPTURED (requires new workflow) |
Edge Cases and Exclusions
If audio is unavailable, use structured clinician attestation within 24 hours: less reliable but scalable. Ambiguous dictation should be scored as "interpretation variance," not hallucination. Exclude template-based content populated from structured data (medications, allergies): focus on narrative AI-generated sections. Minimum sample: 50 audited encounters per provider per quarter.
Confidence Grading (required alongside every DFI value):
| Grade | Basis | Interpretation |
|---|---|---|
| A: Empirical | 5+ observed events in measurement window provide direct tail observation | Measurement grounded in observed outcomes; highest confidence |
| B: Hybrid | 1-4 observed events plus Monte Carlo simulation with Bayesian updating | Blends observation and simulation; moderate confidence |
| C: Simulated | Zero observed events; derived entirely from Monte Carlo simulation | Modeled, not observed; treat as indicative, not validated |
A DFI of any value at Grade C is a simulation that agrees with itself. It tells you the model is internally consistent but not that it matches reality. Grade C scores should trigger proactive stress-testing against historical near-miss events and cross-asset analogues rather than acceptance as calibrated.
Proposed formula, pending validation pilots.