NLP Classification Accuracy Rate (NCAR)

GMI-MIN-036MiningIntegration-Required

Definition

Proportion of NLP-classified work orders where the AI-assigned failure code matches engineering ground truth.

The Measurement Gap

Measurement Gap

Failure coding is the foundation of every reliability decision: bad actor identification, failure trending, root cause analysis, and investigation prioritization all depend on work orders being coded correctly. In legacy environments, failure codes are assigned by technicians at work order closure, and the well-documented result is inconsistent, incomplete, or outright wrong coding. NLP-based classification promises to fix this by reading work order descriptions and assigning codes automatically.

The problem is that no one checks whether the NLP is actually getting it right. Once deployed, the AI classifier runs autonomously, assigning codes to hundreds or thousands of work orders per month. Operations assume the codes are correct because they were generated by an AI model that was validated during implementation. But model performance drifts as equipment changes, new failure modes emerge, and work order language evolves. Without ongoing accuracy measurement, the reliability database accumulates misclassified records that corrupt every downstream analysis.

Legacy CMMS/EAM systems have no built-in mechanism to validate AI-assigned failure codes against expert judgment on an ongoing basis. NCAR creates that validation loop by requiring systematic sampling, engineer review, and accuracy tracking over time.

Decision Supported:

Can the reliability database be trusted for bad actor analysis, failure trending, and investigation prioritization, or is NLP misclassification corrupting the foundation that every reliability decision depends on?

Formula

Plain language: Of the work orders with safety-critical or MSHA-reportable failure codes as determined by a reliability engineer, what fraction did the NLP assign to the wrong code? That misclassification rate on safety-critical events is the primary metric. Overall classification accuracy is reported for model health.

Primary metric: Safety-Critical Misclassification Rate (False-Negative Rate):

NCAR_FNR = Safety_critical_misclassified / Total_safety_critical_sampled

Secondary metric: Overall NLP Classification Accuracy:

NCAR_OA = (Correctly_classified / Total_sampled) × 100

Variables:

Variable Definition Unit
Safety_critical_misclassified Work orders with safety-critical failure codes per engineer assessment that NLP assigned to non-critical categories (false negatives) Count
Total_safety_critical_sampled Total work orders in the sample with safety-critical failure codes per engineer assessment Count
Correctly_classified Work orders where NLP-assigned code matches engineer assessment (exact or same parent category) Count
Total_sampled Total work orders in the validation sample Count

Track a confusion matrix showing which failure codes are most frequently confused to guide NLP retraining.

Why false-negative rate is primary:

In mining operations, miscoding a routine work order inflates a non-critical failure category but does not create safety exposure. Miscoding a safety-critical failure as routine means MSHA-reportable conditions go untracked, hazard patterns are invisible in trend analysis, and precursor signals are lost. The asymmetry is a consequence structure, not a preference. A 90% overall accuracy that misclassifies 15% of ground control or ventilation failures is a safety liability, not acceptable performance.

Proposed formula, pending validation pilots.

At a Glance

Unit
Percentage
Trend Direction
Higher
Reporting Frequency
Quarterly
Category
Work Order Intelligence & Reliability Data Quality
← Condition Monitoring Analyst Skill Index (CMASI)

Assess Your Organization

The AI-GPS Assessment evaluates your AI governance maturity and recommends KPIs like this one for your implementation roadmap.

Take the Free AI-GPS Assessment