Definition
Change in analyst performance on novel anomaly identification tasks over time.
The Measurement Gap
Measurement Gap
AI-powered condition monitoring platforms generate severity scores, alert classifications, and recommended actions for every data collection route. Analysts increasingly rely on these outputs to prioritize their review queue: high-severity alerts get immediate attention, low-severity readings get a glance or no review at all. Over time, this workflow shift means analysts spend less time performing manual spectrum interpretation, waveform analysis, and diagnostic reasoning from raw data.
The risk is not that the AI is wrong on familiar failure signatures. The risk emerges when the AI encounters a failure mode it has never seen, a novel anomaly outside its training distribution. In that scenario, the AI assigns a low or moderate severity score because the pattern does not match any known alarm condition. The analyst, conditioned to trust the severity score, does not investigate further. The failure progresses undetected.
Legacy condition monitoring programs track analyst certification status and route completion rates but never assess whether analysts can still perform independent diagnostic reasoning on unfamiliar data. CMASI fills this gap by periodically testing analysts against curated cases where the AI missed the anomaly, creating a direct measurement of the skill most at risk of atrophy.
Decision Supported:
Are condition monitoring analysts maintaining the manual interpretation skills needed when the AI encounters a failure signature it has not been trained on, or is reliance on AI severity scores degrading the team's diagnostic capability?
Formula
Plain Language: Establish a baseline proficiency score for each analyst by administering a standardized assessment using real historical cases where the AI failed to detect the anomaly. Periodically re-administer the same type of assessment. Calculate the percentage change from baseline.
Notation:
CMASI = ((Score_current - Score_baseline) / Score_baseline) x 100
Variables:
| Variable | Description | Source |
|---|---|---|
| Score_current | Average analyst proficiency score on the novel-anomaly assessment in the current period, scored 0 to 100 based on correct identification, diagnosis accuracy, and recommended action appropriateness | Proficiency assessment results (manual administration) |
| Score_baseline | Same proficiency score from the initial assessment, established at AI deployment or at the earliest available measurement point | Historical assessment records (manual) |
| Assessment | A curated library of 10 to 15 real historical cases where the AI condition monitoring system failed to detect or correctly classify the anomaly; cases should span vibration, thermal, oil analysis, and electrical monitoring technologies as applicable | CM platform historical data and investigation records (manual curation) |
A CMASI of 0% means no change from baseline. A negative value indicates skill degradation. A positive value indicates improvement, potentially from targeted training interventions.
Proposed formula, pending validation pilots.