Intelligence Source False Discovery Rate (ISFDR)

GMI-RAIL-004Rail TransportationInfrastructure-Dependent

Definition

The percentage of actionable alerts, trends, or recommendations generated by NLP, text-mining, or pattern-recognition intelligence systems that, upon investigation, turn out to be artifacts rather than genuine operational findings.

The Measurement Gap

The NLP system detects a "trend" in wheel flat mentions across maintenance logs. Investigation reveals a new crew supervisor uses "wheel flat" where predecessors wrote "tread damage." The trend is linguistic, not mechanical. Resources are diverted to investigate a phantom problem. No legacy metric distinguishes NLP-detected trends from linguistic pattern changes. Maintenance backlog shows the wasted effort but does not attribute it to intelligence source quality.

Decision Supported:

When the NLP or pattern recognition system flags a trend, how often is it a genuine operational finding versus an artifact? Should we investigate every alert or apply a quality filter first?

Formula

Plain language: Over an evaluation period, count investigations triggered by intelligence system outputs. Of those, count how many confirmed genuine operational findings versus how many were artifacts (linguistic changes, data entry pattern shifts, seasonal noise). Divide artifacts by total.

Notation:

ISFDR = A_false / (A_false + A_true)

Variables:

Variable Description Unit
A_false Investigations triggered by intelligence output closed as artifact / no operational finding Count
A_true Investigations that confirmed a genuine finding and resulted in corrective action Count

Sign Convention: Lower is better. 0 = every intelligence output was genuine. 1.0 = every output was an artifact.

Proposed formula, pending validation pilots.

At a Glance

Unit
Ratio (0 to 1)
Category
AI/Model Trustworthiness
← Failure Mode Coverage Ratio (FMCR)Model Currency Index (MCI) →

Assess Your Organization

The AI-GPS Assessment evaluates your AI governance maturity and recommends KPIs like this one for your implementation roadmap.

Take the Free AI-GPS Assessment