Definition
A two-part measure: (1) the quality of technician feedback on closed work orders, scored on entity completeness, inter-rater coding agreement, and code-text alignment, and (2) the change in model F1 score per 100 corrected work orders used for retraining.
The Measurement Gap
Predictive maintenance models train on structured failure codes (Problem/Cause/Remedy hierarchy) while ignoring the unstructured free text where the real diagnostic intelligence lives. F1 score cannot detect noisy labels, a model trained on inconsistent codes will converge on the inconsistency and score well against future labels that carry the same systematic error. The model is confidently wrong in a consistent direction. PM Compliance, OEE, and MTBF have no concept of feedback quality. The industry measures model outputs obsessively and model inputs almost never. Inter-technician coding agreement (Cohen's kappa) frequently falls below 0.5 (meaning techs disagree on the failure code more than half the time, adjusted for chance) and the model trains on that.
Decision Supported:
Whether to invest in technician coaching, failure-code library redesign, NLP extraction of free-text intelligence, or model architecture changes, and whether feedback corrections are actually moving model performance.
Formula
Plain language: Part 1 scores the quality of completed-work-order feedback across three dimensions: how complete the free-text entities are, how consistently different technicians code the same failure, and whether the codes match what the free text describes. Part 2 measures how much model accuracy actually improves per 100 corrected work orders fed back into retraining.
Notation:
FQS = α · NER_completeness + β · κ_coding + γ · Code_Text_Alignment
MLG = (F1_after − F1_before) / (N_corrections / 100)
Variables:
| Variable | Definition | Unit |
|---|---|---|
| NER_completeness | Fraction of key entities (failure mode, root cause, component, action) extractable from free text | Ratio (0–1) |
| κ_coding | Cohen's kappa for inter-technician failure-code agreement on a sampled set | Score (−1 to 1) |
| Code_Text_Alignment | Fraction of WOs where structured codes match what the technician wrote in free text | Ratio (0–1) |
| α, β, γ | Weights summing to 1.0 (start at 0.33 each) | Weights |
| MLG | Model F1 improvement per 100 corrected work orders | F1 points / 100 |
Proposed formula, pending validation pilots.