DQ File Classification Accuracy (DQCA)

GMI-TF-030Fleet & TransportationPilot-Ready

Definition

Percentage of AI-classified driver qualification file documents where the classification matches human auditor determination, specifically measuring accuracy on date-sensitive items (medical certificate expiration, CDL renewal) and endorsement-match items (hazmat endorsement vs. assigned loads).

The Measurement Gap

DQ file management systems produce efficiency metrics that measure throughput without validating the accuracy of the AI's compliance judgments. The blind metrics include:

DQ file processing speed: measures how fast the AI classifies documents but says nothing about whether the classifications are correct. A system that processes 500 files per hour looks impressive until you discover it is passing expired medical certificates as compliant.

DQ compliance rate: reports the percentage of driver files the AI has marked compliant. This metric reflects the AI's judgment, not reality. If the classifier has a systematic blind spot for endorsement mismatches, the compliance rate looks healthy while the fleet operates with non-qualified drivers on restricted cargo.

Audit pass rate: measures how many audits the compliance team completed, not whether those audits agreed with the AI's classification. High audit throughput with low classification accuracy is worse than no AI at all: it creates false confidence.

The core problem: DQ file compliance is binary in regulatory terms (a driver is either qualified or not) but the consequences of a false "compliant" classification are asymmetric. A false non-compliant flag creates administrative friction; a false compliant flag creates a driver operating without proper qualification, which is a regulatory violation with safety and liability implications.

Decision Supported:

Can the fleet trust the AI's "compliant" classification of DQ files, or does the false compliance rate require mandatory human review of specific document categories?

Formula

Plain language: Of the DQ files that a human auditor flags as non-compliant, how many did the AI classify as compliant? That False Compliance Rate is the number that matters most: it directly measures regulatory exposure. Overall classification accuracy is reported alongside it for model health.

Primary metric: False Compliance Rate (False-Negative Rate):

FCR = AI_compliant_Auditor_noncompliant / Total_AI_compliant_in_sample × 100

Secondary metric: Overall Classification Accuracy:

DQCA = AgreedClassifications / TotalReviewed × 100

Variables:

Variable Definition Unit
AI_compliant_Auditor_noncompliant Files the AI passed as compliant but the auditor flagged as non-compliant (false negatives) Count
Total_AI_compliant_in_sample Total files the AI classified as compliant in the audit sample Count
AgreedClassifications Files where AI and auditor classifications matched Count
TotalReviewed Total DQ files in the audit sample Count

A DQCA of 95% sounds acceptable, but if the 5% error is concentrated entirely in false compliance (non-compliant drivers marked as compliant), every missed file is a driver operating without proper qualification under 49 CFR 391.

Why false compliance rate is primary:

In fleet compliance, false flags (compliant files marked non-compliant) create rework. False compliance (non-compliant files marked compliant) means unqualified drivers operate vehicles, creating FMCSA enforcement exposure and liability in any crash investigation. The asymmetry is a consequence structure, not a preference.

Proposed formula, pending validation pilots.

At a Glance

Unit
Percentage
Trend Direction
Mixed; Increasing (higher accuracy); FCR decreasing
Reporting Frequency
Quarterly
Category
Compliance and Risk Calibration
← Relay Outcome Differential (ROD)ETA Promise Calibration Gap (EPCG) →

Assess Your Organization

The AI-GPS Assessment evaluates your AI governance maturity and recommends KPIs like this one for your implementation roadmap.

Take the Free AI-GPS Assessment