Definition
Percentage of AI-classified driver qualification file documents where the classification matches human auditor determination, specifically measuring accuracy on date-sensitive items (medical certificate expiration, CDL renewal) and endorsement-match items (hazmat endorsement vs. assigned loads).
The Measurement Gap
DQ file management systems produce efficiency metrics that measure throughput without validating the accuracy of the AI's compliance judgments. The blind metrics include:
DQ file processing speed: measures how fast the AI classifies documents but says nothing about whether the classifications are correct. A system that processes 500 files per hour looks impressive until you discover it is passing expired medical certificates as compliant.
DQ compliance rate: reports the percentage of driver files the AI has marked compliant. This metric reflects the AI's judgment, not reality. If the classifier has a systematic blind spot for endorsement mismatches, the compliance rate looks healthy while the fleet operates with non-qualified drivers on restricted cargo.
Audit pass rate: measures how many audits the compliance team completed, not whether those audits agreed with the AI's classification. High audit throughput with low classification accuracy is worse than no AI at all: it creates false confidence.
The core problem: DQ file compliance is binary in regulatory terms (a driver is either qualified or not) but the consequences of a false "compliant" classification are asymmetric. A false non-compliant flag creates administrative friction; a false compliant flag creates a driver operating without proper qualification, which is a regulatory violation with safety and liability implications.
Decision Supported:
Can the fleet trust the AI's "compliant" classification of DQ files, or does the false compliance rate require mandatory human review of specific document categories?
Formula
Plain language: Of the DQ files that a human auditor flags as non-compliant, how many did the AI classify as compliant? That False Compliance Rate is the number that matters most: it directly measures regulatory exposure. Overall classification accuracy is reported alongside it for model health.
Primary metric: False Compliance Rate (False-Negative Rate):
FCR = AI_compliant_Auditor_noncompliant / Total_AI_compliant_in_sample × 100
Secondary metric: Overall Classification Accuracy:
DQCA = AgreedClassifications / TotalReviewed × 100
Variables:
| Variable | Definition | Unit |
|---|---|---|
| AI_compliant_Auditor_noncompliant | Files the AI passed as compliant but the auditor flagged as non-compliant (false negatives) | Count |
| Total_AI_compliant_in_sample | Total files the AI classified as compliant in the audit sample | Count |
| AgreedClassifications | Files where AI and auditor classifications matched | Count |
| TotalReviewed | Total DQ files in the audit sample | Count |
A DQCA of 95% sounds acceptable, but if the 5% error is concentrated entirely in false compliance (non-compliant drivers marked as compliant), every missed file is a driver operating without proper qualification under 49 CFR 391.
Why false compliance rate is primary:
In fleet compliance, false flags (compliant files marked non-compliant) create rework. False compliance (non-compliant files marked compliant) means unqualified drivers operate vehicles, creating FMCSA enforcement exposure and liability in any crash investigation. The asymmetry is a consequence structure, not a preference.
Proposed formula, pending validation pilots.