Definition
ECCAR measures the percentage of NLP-processed employee complaints correctly classified by regulatory significance.
The Measurement Gap
Measurement Gap and Decision Impact
It uses a human expert audit sample to compute overall classification accuracy and, critically, a downgrade rate: the percentage of complaints that human experts identify as regulatory-significant but the NLP classified as routine. These false downgrades are the dangerous errors because they route legally consequential complaints away from compliance review.
The Decision It Supports:
Is the NLP complaint triage system correctly identifying complaints with regulatory, legal, or compliance significance? If the system is routing genuine harassment complaints, whistleblower reports, or safety concerns into the routine queue, the organization faces both legal exposure and employee harm.
Why It Requires a New Metric:
Legacy complaint management relied on human intake and triage. Experienced HR professionals read each complaint and made routing decisions based on judgment and training. NLP-powered case management systems now perform initial classification at scale, routing complaints to appropriate queues based on text analysis. No legacy metric evaluates whether the NLP's regulatory significance classification is accurate, particularly whether it correctly identifies the complaints that must reach compliance reviewers.
The measurement gap ECCAR addresses is the absence of any accuracy assessment for NLP-driven complaint classification. Organizations are using NLP to triage employee complaints without verifying whether the classification engine correctly identifies complaints with regulatory and legal significance.
NLP-powered case management represents a genuine efficiency gain: classifying incoming complaints, assigning severity, routing to queues, and flagging urgent items. But the efficiency gain is only safe if classification is accurate, particularly for regulatory-significant complaints.
The stakes are asymmetric. Misclassifying a routine complaint as regulatory-significant wastes compliance review time. Misclassifying a regulatory-significant complaint as routine creates legal exposure, regulatory risk, and potential employee harm. These false downgrades are the high-consequence errors.
NLP systems are particularly vulnerable to this error type because regulatory significance often depends on context and legal framing rather than explicit keywords. An employee describing "pressure to adjust numbers before the quarterly report" may be reporting normal deadline stress or financial fraud. The NLP's ability to distinguish these requires contextual understanding that keyword-based approaches handle unreliably.
ECCAR quantifies classification accuracy through structured human expert audit. The critical metric is the downgrade rate: what percentage of expert-identified regulatory-significant complaints did the NLP classify as routine? Without ECCAR, the first signal of misrouting may be a regulatory investigation or lawsuit.
Formula
Plain language: Of the complaints that human experts independently classify as regulatory-significant (EEOC, ADA, FMLA, Title VII, whistleblower), what fraction did the NLP classify as routine? That downgrade rate is the primary metric: it measures buried regulatory exposure. Overall classification accuracy is secondary.
Primary metric: Regulatory Complaint Downgrade Rate (False-Negative Rate):
ECCAR_FNR = NLP_routine_but_regulatory / Total_regulatory_in_sample × 100
Secondary metric: Overall Classification Accuracy:
ECCAR_OA = Correctly_classified / Total_sampled × 100
Variables:
| Variable | Definition | Unit |
|---|---|---|
| NLP_routine_but_regulatory | Complaints the NLP classified as routine that experts classified as regulatory-significant (false negatives) | Count |
| Total_regulatory_in_sample | Total complaints classified as regulatory-significant by human experts | Count |
| Correctly_classified | Complaints where NLP classification matches expert classification | Count |
| Total_sampled | Total complaints in the audit sample | Count |
Stratify by complaint type (discrimination, harassment, retaliation, accommodation, safety) since NLP accuracy varies across categories. Minimum 100 complaints per audit cycle, drawn from the full case management system.
Why false-negative rate is primary:
In human capital management, overclassification (routing routine complaints to the legal/compliance queue) creates review burden. Downgrading regulatory-significant complaints to routine means potential EEOC charges, ADA accommodation failures, or whistleblower retaliation claims are handled as standard HR matters, missing statutory response deadlines and destroying the organization's legal position. The asymmetry is a consequence structure, not a preference. A 90% overall accuracy that downgrades 12% of Title VII complaints is a litigation liability, not acceptable performance.
Proposed formula, pending validation pilots.