Definition
Failure Code Fidelity Rate measures what percentage of AI-assigned failure codes actually match the real failure modes in the field. That is the whole idea. Every complexity in the formula exists to make that measurement honest.
The Measurement Gap
Your AI failure coding system is lying to you, and you have no metric that tells you so.
A reliability engineer at an automotive parts manufacturer was preparing for a quarterly RCM review on his stamping press line. He pulled the failure mode distribution from Maximo: mechanical wear at 43%, electrical fault at 22%, hydraulic leak at 18%. Looked clean. Looked defensible. The RCM team was ready to reallocate condition monitoring resources toward wear-based inspection techniques: ultrasonic thickness gauging, surface profilometry, die wear trending. The business case wrote itself on paper. Then a senior technician made a passing comment during the pre-meeting walkthrough. Half those mechanical wear codes, he said, were entered by the new AI system, and he wasn't sure it knew the difference between a worn die and a cracked frame.
The reliability engineer did what good engineers do when they get a bad feeling: he pulled the data and looked at it himself. He manually reviewed 60 AI-coded work orders, a quick audit, not a formal one. Of the 34 coded as mechanical wear, only 19 were actually mechanical wear. Eight were hydraulic failures where the technician had described the condition in wear-adjacent language: "seals worn," "cylinder dragging," "rod showing wear marks." The NLP saw "wear" and coded accordingly. Four were electrical failures with phrases like "worn contacts" or "brush wear" in the description. Three were complete misattributions: the failure description didn't support mechanical wear by any reasonable reading. The true mechanical wear rate was closer to 28%, not 43%. Hydraulic failures were actually running at 26%, not 18%. The team was about to redirect resources based on a phantom failure mode distribution built on language pattern matching, not actual failure physics.
Here is what makes this dangerous: none of the traditional metrics caught it. Failure rate by mode showed mechanical wear leading the distribution, which is exactly what you'd expect if mechanical wear were actually leading. The corrupted distribution looked completely plausible. An experienced reliability engineer reviewing the numbers without the technician's comment would have had no reason to question it. RCM completion rate was tracking fine: the team was completing RCM analyses on schedule, progressing through the equipment hierarchy, ticking the governance boxes. Data quality scores from the EAM were green because the system was checking whether fields were populated and codes were valid entries in the taxonomy, not whether the codes were correct. Every administrative metric said the program was healthy.
This is the specific failure mode that none of your existing KPIs address: the AI assigned a code that exists in your taxonomy, passes all validation rules, and populates every required field, and it is still wrong. The system did not break. The system did exactly what it was designed to do. It matched language patterns to failure codes. The problem is that maintenance technicians do not write failure descriptions the way NLP models are trained to expect them. They use shorthand, site-specific jargon, abbreviated equipment names, and casual language that happens to contain words associated with different failure modes than the one actually present. The NLP does not know that "worn seals" on a hydraulic cylinder is a hydraulic failure, not a mechanical wear event. It knows that "worn" correlates with a mechanical wear code in the training data.
Failure rate by mode shows you the distribution of coded failures. It cannot detect that the distribution is corrupted because the codes do not correspond to actual failure modes. RCM completion rate measures program throughput: it has no visibility into whether the failure data feeding the RCM analysis is accurate. You can complete every RCM task on schedule and build a fully documented RCM strategy on a foundation of miscoded data. Data quality scores measure structural integrity of records: populated fields, valid code entries, complete hierarchies. A wrong code that exists in the taxonomy passes every data quality check.
There is no metric that tells you whether AI-assigned failure codes match actual failure modes in the field.
That is the gap this chapter closes. You need a way to measure the fidelity of AI failure coding against ground truth, not as a one-time validation exercise during system implementation, but as an ongoing operational metric that tells you, every quarter, whether the NLP is still producing reliable classifications across every failure mode category in your taxonomy. Because the automotive engineer's situation is not unusual. It is what happens when you deploy AI auto-coding, watch the completion metrics improve, assume the quality improved with it, and never build a mechanism to verify the assumption.
Formula
Failure Code Fidelity Rate measures what percentage of AI-assigned failure codes actually match the real failure modes in the field. That is the whole idea. Every complexity in the formula exists to make that measurement honest.
The Formula
In plain language: take a stratified sample of work orders where the AI assigned a failure code. Have qualified reviewers evaluate each one against the available evidence. Count how many codes were correct. Divide by the sample size. Multiply by 100 to express as a percentage. In notation:FCFR = (C_m / S_t) × 100
Where:C_m = the number of AI-assigned failure codes confirmed as correct by qualified review (count, derived from audit results)S_t = the total number of AI-assigned failure codes in the audit sample (count, derived from the stratified audit sample)
Worked Example
A petrochemical plant audits its NLP auto-coding quarterly. In Q2, the system auto-coded 1,240 work orders. The reliability team pulls an audit sample of 120 work orders, stratified by failure mode category, not randomly drawn from the full population, but selected to ensure meaningful representation across failure types. Two independent reviewers (a reliability engineer and a senior technician) evaluate each work order against the available evidence: technician notes, parts replaced, condition monitoring data, and any photos attached to the work order. Each reviewer makes an independent determination before discussing with the other. The results: 94 codes confirmed correct, 26 incorrect. Breaking down those 26 errors: 11 had the wrong failure mechanism coded (the event was an electrical fault coded as mechanical wear), 8 had the right mechanism but the wrong component, 4 were coded at the wrong level of the cause hierarchy (the symptom was coded instead of the root cause), and 3 were completely unrelated to the actual failure. FCFR = 94/120 × 100 = 78.3%.That number sits below the 80% threshold. The NLP is miscoding roughly one in five failure events in the sample, a rate that will materially corrupt failure mode distributions, bad actor lists, and any RCM analysis drawing on this data. But the overall number is only the starting point. Segment by failure mode category and the picture sharpens considerably:Mechanical failures: 38 in the sample, 35 confirmed correct, FCFR 92.1%. The NLP handles mechanical failures well. Technicians tend to describe mechanical failures in language that is relatively unambiguous: "bearing failed," "shaft cracked," "gear tooth broken." The words match the codes reliably. Electrical failures: 29 in the sample, 25 correct, FCFR 86.2%. Acceptable but below the 90% threshold for production reliance. The errors are concentrated in failures where electrical and instrumentation symptoms overlap in technician descriptions. Instrumentation failures: 22 in the sample, 19 correct, FCFR 86.4%. Similar pattern to electrical: the language boundary between instrumentation and control failures is blurry in field notes. Hydraulic failures: 18 in the sample, 10 correct, FCFR 55.6%. This is the problem category. Hydraulic failures in this plant are consistently described using language that the NLP associates with mechanical wear: "worn seals," "cylinder dragging," "rod degradation." The NLP is miscoding hydraulic failures more often than it codes them correctly. Process failures: 13 in the sample, 5 correct, FCFR 38.5%. Worse still. Process-related failures in the petrochemical environment involve complex interactions that technicians describe in highly contextual, site-specific language the NLP was not trained on adequately. This segmented view is the actionable output. The plant does not need to suspend NLP auto-coding for mechanical failures: 92.1% is solid. It needs immediate human review of hydraulic and process failures specifically, and it needs to retrain the NLP on hydraulic and process failure descriptions from its own work order history.
Edge Cases
Partially correct codes (where the failure mechanism is right but the component is wrong, or the cause hierarchy level is off) count as incorrect. I know this feels harsh. It is not. A partially correct code still corrupts downstream FMEA crosswalks and bad actor analysis. If your RCM analysis categorizes failures by mechanism and component, a right-mechanism/wrong-component code puts the event in the wrong bucket. Count it as incorrect and record what was wrong specifically: the error taxonomy gives you the retraining signal. Multiple failure modes per event are evaluated independently. If a work order documents a primary failure and a contributing failure, each AI-assigned code gets its own determination. One event can contribute two correct codes or one correct and one incorrect to the count. Reviewer disagreement is handled through escalation. Require two independent reviews for every audited work order: reviewers should not see each other's determination before completing their own. If they agree, that is your result. If they disagree, escalate to a third reviewer and take the majority. Track disagreement rate as a secondary indicator: persistent disagreement above 15% suggests the failure code taxonomy itself is ambiguous, which is a separate problem that will limit both NLP performance and FCFR achievability. Sample size: minimum 50 events per audit cycle, stratified by failure mode category. The stratification requirement is not optional, a random draw from 1,240 work orders could easily undersample hydraulic failures if they represent only 15% of events, leaving you with 18 hydraulic samples and no statistical power to detect systematic miscoding in that category. Aim for at least 10 per failure mode category. If a category has fewer than 10 events in the period, note that explicitly and defer judgment on that category until the next cycle.
Cross-References
FCFR is upstream of two KPIs that depend on the integrity of failure coding. SOSI (GMI-AR-008) tracks FMEA crosswalk completion. When failure codes are wrong, the FMEA crosswalk maps the event to the wrong failure mode, and the crosswalk completion metric looks healthy while the underlying analysis is corrupted. AWSNR (GMI-AR-005) tracks whether AI-generated work orders are appropriate. If the failure code is wrong, the AI generates work orders based on the wrong failure mode, and AWSNR has no way to know the upstream input was bad. Fix FCFR first. The downstream KPIs only tell you the truth when the failure coding does.
Proposed formula, pending validation pilots.
At a Glance
Decision Supported
Is the failure mode distribution in my EAM accurate, or is NLP miscoding corrupting my bad actor lists, RCM inputs, and spare parts planning?