Definition
Accuracy of NLP classification distinguishing planned maintenance from unplanned failures in maritime logs.
The Measurement Gap
What This Metric Measures
NLP Maintenance Event Classification Accuracy measures how correctly an AI natural language processing system categorizes maintenance events extracted from unstructured chief engineer logs, free-text work orders, and deficiency reports into their proper maintenance types: planned preventive, corrective failure, condition-based intervention, or emergency repair. The metric tracks classification precision against human-validated ground truth samples across the fleet.
The Decision It Supports
This metric answers a fundamental trust question: can the fleet rely on AI-classified maintenance data to drive reliability analytics, failure pattern detection, and fleet-wide maintenance campaigns? When classification accuracy drops below actionable thresholds, every downstream analysis built on that classified data becomes unreliable: from failure mode trending to spare parts demand forecasting. Operations teams use this metric to decide whether NLP-derived insights require manual validation before acting on them.
Why Legacy Metrics Cannot See This
Traditional maritime maintenance tracking relies on structured PMS completion records where event types are manually selected from dropdown menus by crew members. This approach has two blind spots. First, free-text maintenance narratives (where engineers describe what actually happened) contain richer failure context than any structured field, but legacy systems cannot parse them. Second, manual classification by crew is inconsistent across watches, vessels, and nationalities. Legacy metrics measure whether maintenance was recorded, not whether AI systems correctly understand what was recorded. Without measuring NLP classification accuracy, fleets deploying AI analytics have no way to know whether their AI is reading maintenance history correctly or systematically misclassifying events in ways that corrupt every downstream decision.
Formula
Primary metric (false-planned rate on actual failures):
FPR = Failures_Classified_as_Planned / Total_Actual_Failures
Secondary metric (overall accuracy):
NMECA = Correctly_Classified / Total_Sampled
Why false-planned rate is primary: Classifying planned maintenance as corrective inflates failure counts but triggers conservative action. Classifying actual corrective failures as planned maintenance conceals breakdowns, distorts reactive-vs-preventive ratios, and hides equipment degradation from reliability analysis. The consequence is invisible asset deterioration. Set FPR below 3%; overall NMECA above 90%.
Variable Definitions
Correctly_Classified: Maintenance events where NLP classification matches human-verified ground truth. Unit: count.
Total_Sampled: All maintenance events in the validation sample. Unit: count.
Failures_Classified_as_Planned: Actual corrective failures NLP incorrectly classified as planned maintenance. Unit: count.
Total_Actual_Failures: All corrective failures in the sample as determined by expert review. Unit: count.
Planned_Classified_as_Failure: Planned events NLP incorrectly classified as corrective failures. Unit: count.
Units:
Ratio expressed as percentage (0-100%).
Edge Cases
Multilingual maintenance logs present the primary edge case. Mixed events where planned maintenance discovered an unrelated failure during execution create legitimate classification ambiguity. Require minimum 100 classified events per vessel per quarter. Events with fewer than 10 words of narrative text should be excluded.
Proposed formula, pending validation pilots.