Failure Mode Classification Accuracy (FMCA)

GMI-RE-034Renewable EnergyPilot-ReadyPattern O-1: Confidence Calibration

Definition

FMCA measures the percentage of auto-classified failure modes on condition-generated work orders that match the actual failure mode identified by the technician during repair. This metric validates whether the automated front door of the work management process is sending the right information downstream for planning, parts staging, and technician assignment.

The Measurement Gap

The Measurement Gap

Condition monitoring systems in renewable energy operations generate work orders automatically when sensor data indicates a developing fault. The system assigns a failure mode classification based on the sensor signature: vibration patterns suggest bearing degradation, temperature anomalies suggest electrical connection issues, power curve deviations suggest blade damage. This classification drives everything downstream: the planner assigns a technician with the right skill set, stages the predicted parts, and estimates the job duration based on the classified failure mode.

If the classification is wrong, the entire downstream chain is compromised. A work order classified as a bearing issue dispatches a mechanical technician with bearing replacement parts. The technician arrives, diagnoses the problem as an electrical connection fault, and must either return for the correct parts and tools or attempt a repair with inadequate preparation. The first-visit fix fails, a second trip is required, and the asset remains down longer than necessary.

Without FMCA, the organization trusts the classification without validation. Work orders are generated, dispatched, and completed. If the technician finds a different failure mode than classified, that information may be recorded in the work order completion notes but is not systematically compared against the original classification. The mismatch is treated as a normal part of maintenance rather than a failure of the classification system.

The Decision This Metric Supports

FMCA answers whether the planner can trust the AI's failure mode classification to assign the right technician, stage the right parts, and estimate the right duration. When FMCA is above 75%, the classification is useful for planning purposes. When FMCA drops below 60%, every auto-classified work order should be reviewed by the planner before execution planning begins.

Formula

Primary metric (miss rate on safety-critical failure modes):

MR_critical = Safety_critical_misclassified / Total_safety_critical

Secondary metric (overall weighted accuracy):

FMCA = (Full_Match + 0.5 x Partial_Match) / Total_Classified x 100

Why miss rate on safety-critical failure modes is primary: Misclassifying a non-critical failure mode as critical triggers conservative maintenance action but no safety gap. Misclassifying a safety-critical failure mode (structural fatigue, blade delamination, generator bearing degradation) as non-critical delays intervention on conditions that can cause catastrophic failure, turbine collapse, or blade throw. Set MR_critical below 5%; overall FMCA above 75%.

Variable Definition
Full_Match Work orders where AI-classified failure mode exactly matched technician-confirmed actual failure mode
Partial_Match Work orders where AI classification was in the right system/subsystem but wrong specific component (e.g., "gearbox bearing" when actual was "gearbox seal")
Total_Classified All auto-classified work orders with confirmed actual failure modes in the period
Safety_critical_misclassified Safety-critical failures the AI classified as non-critical categories
Total_safety_critical All confirmed safety-critical failures in the period

Measurement frequency: Monthly, with quarterly trend analysis.

Segmentation: Track by failure mode category. Electrical faults often achieve FMCA above 85% due to clear SCADA signatures. Mechanical faults may achieve only 45%.

Edge cases:

  • No actual failure mode recorded by technician: track separately as data quality metric
  • Multiple failure modes found: credit AI if its classification was among actual modes
  • "No problem found" outcomes: count as mismatch (false positive)
  • If "no actual recorded" rate exceeds 30%, FMCA itself becomes unreliable

Proposed formula, pending validation pilots.

At a Glance

Unit
Percentage
Trend Direction
Higher; more accurate
Reporting Frequency
Monthly
Category
Work Order and Failure Classification Accuracy
← Dynamic Criticality Alignment Score (DYCAS)

Assess Your Organization

The AI-GPS Assessment evaluates your AI governance maturity and recommends KPIs like this one for your implementation roadmap.

Take the Free AI-GPS Assessment