Work Order Feedback Quality and Model Learning Gain (WOFQ)

GMI-MRO-017MROIntegration-Required

Definition

A two-part measure: (1) the quality of technician feedback on closed work orders, scored on entity completeness, inter-rater coding agreement, and code-text alignment, and (2) the change in model F1 score per 100 corrected work orders used for retraining.

The Measurement Gap

Predictive maintenance models train on structured failure codes (Problem/Cause/Remedy hierarchy) while ignoring the unstructured free text where the real diagnostic intelligence lives. F1 score cannot detect noisy labels, a model trained on inconsistent codes will converge on the inconsistency and score well against future labels that carry the same systematic error. The model is confidently wrong in a consistent direction. PM Compliance, OEE, and MTBF have no concept of feedback quality. The industry measures model outputs obsessively and model inputs almost never. Inter-technician coding agreement (Cohen's kappa) frequently falls below 0.5 (meaning techs disagree on the failure code more than half the time, adjusted for chance) and the model trains on that.

Decision Supported:

Whether to invest in technician coaching, failure-code library redesign, NLP extraction of free-text intelligence, or model architecture changes, and whether feedback corrections are actually moving model performance.

Formula

Plain language: Part 1 scores the quality of completed-work-order feedback across three dimensions: how complete the free-text entities are, how consistently different technicians code the same failure, and whether the codes match what the free text describes. Part 2 measures how much model accuracy actually improves per 100 corrected work orders fed back into retraining.

Notation:

FQS = α · NER_completeness + β · κ_coding + γ · Code_Text_Alignment

MLG = (F1_after − F1_before) / (N_corrections / 100)

Variables:

Variable Definition Unit
NER_completeness Fraction of key entities (failure mode, root cause, component, action) extractable from free text Ratio (0–1)
κ_coding Cohen's kappa for inter-technician failure-code agreement on a sampled set Score (−1 to 1)
Code_Text_Alignment Fraction of WOs where structured codes match what the technician wrote in free text Ratio (0–1)
α, β, γ Weights summing to 1.0 (start at 0.33 each) Weights
MLG Model F1 improvement per 100 corrected work orders F1 points / 100

Proposed formula, pending validation pilots.

At a Glance

Unit
Composite; Two-part composite (FQS index + ΔF1 per 100 corrections)
Trend Direction
Mixed; FQS increasing; MLG positive
Reporting Frequency
Monthly FQS sample; per-retraining-cycle MLG
Category
Human-AI Symbiosis & Workflow
← Time to Calibrated Trust (TTCT)

Assess Your Organization

The AI-GPS Assessment evaluates your AI governance maturity and recommends KPIs like this one for your implementation roadmap.

Take the Free AI-GPS Assessment