Definition
RECA measures the proportion of regulatory events where the AI-assisted or automated classification matches expert human adjudication. It targets the growing use of natural language processing and machine learning to auto-classify incident reports, safety events, and regulatory submissions. Higher RECA indicates that automated classification can be trusted for reporting; lower RECA means human review remains essential to prevent misclassification that could trigger regulatory consequences.
The Measurement Gap
Measurement Gap and Decision
Decision Question: When the AI system classifies a regulatory event, is the classification accurate enough to trust for reporting, or does misclassification create regulatory exposure that nobody detects until an audit?
What Goes Unmeasured Today:
Healthcare organizations are adopting NLP-based tools to classify incident reports, patient safety events, and regulatory submissions. These tools read narrative text from event reports and assign classification codes: event type, severity level, harm category, and reporting obligations. The promise is efficiency: faster classification means faster reporting and better analytics. The risk is misclassification. An event classified as a near-miss rather than an adverse event may not trigger the required reporting pathway. A sentinel event misclassified as a less severe category may not receive the investigation it demands. Current implementations report throughput (how many events were classified and how quickly) but do not systematically measure whether the classifications are correct. Without accuracy measurement, organizations cannot determine whether their automated classification saves time or creates regulatory liability. The false negative problem is particularly dangerous: events that should have been classified as reportable but were downgraded by the algorithm create exposure that surfaces only during external review.
Invisible Failure Mode:
Automated misclassification creates a false sense of regulatory compliance. The organization believes events are properly categorized and reported while systematic classification errors suppress the visibility of reportable events. Without measuring classification accuracy against expert adjudication, the organization has no way to know whether its automated system is a compliance asset or a compliance liability.
Formula
Plain language: Of the events that expert adjudication confirms were reportable, what fraction did the automated classification system miss? That false-negative rate is the headline metric. Overall classification accuracy is reported alongside it for model health, but missed reportable events drive compliance risk.
Primary metric: Reportable Event False-Negative Rate:
RECA_FNR = False_negatives / (True_positives + False_negatives)
Secondary metric: Overall Classification Accuracy:
RECA_OA = Correctly_Classified_Events / Total_Sampled_Events
Variables:
| Variable | Definition | Unit |
|---|---|---|
| False_negatives | Events classified as non-reportable by the system that expert review determined were reportable | Count |
| True_positives | Events correctly classified as reportable, confirmed by expert review | Count |
| Correctly_Classified_Events | Events where automated classification matches expert adjudication across all dimensions (type, severity, reporting category) | Count |
| Total_Sampled_Events | All events in the adjudication sample | Count |
RECA produces a value between 0 and 1. Stratify by event type (patient safety, sentinel, HAC) because aggregate accuracy can mask category-specific failures. Minimum 50 events per adjudication period.
Why false-negative rate is primary:
In healthcare, false positives (flagging routine events as reportable) create investigation burden. False negatives (missing actual reportable events) create unreported sentinel events, CMS Condition of Participation violations, Joint Commission survey findings, and patient safety failures that remain invisible to root cause analysis. The asymmetry is a consequence structure, not a preference. A 95% overall accuracy that misses 10% of sentinel events is a patient safety liability, not a success.
Proposed formula, pending validation pilots.