Definition
Percentage of AI-identified safety trends (emerging hazard patterns detected by NLP analysis of ASAP and Line Operations Safety Audit narratives) that are validated as genuine hazards upon investigation: as opposed to reporting artifacts, seasonal noise, or model-generated false patterns.
The Measurement Gap
Safety analytics platforms using NLP to detect emerging hazard trends in voluntary reporting data represent a significant advancement over manual report review. The ability to detect a subtle increase in reports mentioning unusual hydraulic behavior, specific approach instability patterns, or equipment-related crew workload concerns (before those events aggregate into an incident) is the core promise of AI-assisted safety management.
The problem is false patterns. An NLP system can detect a cluster of reports mentioning the same hazard category and flag it as an emerging trend, but the cluster may be caused by a reporter culture shift at one base, a change in safety prompting questions that made specific language more common, or a seasonal effect with no safety significance. Without validation, safety teams spend investigation capacity on false patterns while real emerging hazards may go undetected.
Decision supported:
Should the safety team investigate every AI-flagged trend, or are too many of them false patterns that are consuming investigation capacity better spent elsewhere? HTVR answers this with a validation rate that determines whether the AI trend detection threshold is calibrated correctly. If HTVR is 35%, the system is generating more false patterns than real ones: raising the detection threshold (requiring more evidence before flagging a trend) will improve investigation efficiency without materially increasing the rate of missed real hazards. HTVR converts the pattern detection system from a black box that generates investigation requests into a measurable quality signal that can be tuned.
Formula
Plain language: Count all trends flagged by the NLP system as "emerging hazard patterns" in a measurement period. For each, determine whether a subsequent investigation confirmed it as a genuine hazard: validated through operational evidence, increased incident rates, or subject matter expert confirmation. Divide confirmed genuine hazards by total flagged trends.
Notation:
HTVR = Validated_trends / Total_AI_flagged_trends
Variables:
| Variable | Definition | Source | Unit |
|---|---|---|---|
| Validated_trends | AI-flagged trends confirmed as genuine safety hazards after investigation | Safety investigation records | Count |
| Total_AI_flagged_trends | All trends the NLP system flagged as emerging hazard patterns in the measurement period | Safety analytics / NLP system output | Count |
| Inconclusive_investigations | Investigations that neither confirmed nor denied the trend | Safety investigation records | Count (track separately) |
Units: Percentage. 100% = every AI-flagged trend was a genuine hazard. Lower = more false patterns. Minimum meaningful sample: requires at least 10 flagged trends per measurement period. If fewer are flagged, extend the measurement window to accumulate sufficient events. Track the source of false patterns separately: reporting culture shifts, seasonal effects, and model failures require different interventions.
Proposed formula, pending validation pilots.