Prediction Threshold Value Ratio (PTVR)

GMI-AR-004Asset ReliabilityIntegration-Required

Definition

Prediction Threshold Value Ratio (PTVR) is the ratio of costs avoided by true positive predictions to costs incurred by false positive predictions at the current detection threshold.

The Measurement Gap

A district heating utility in the upper Midwest was running vibration-based prediction on 340 circulation pumps across its distribution network. The detection threshold had been set during commissioning by the vendor's data science team, optimized to maximize detection accuracy on the training dataset. The system was producing results, and the PdM Hit Rate was reported as 28%, meaning about 28% of predictions led to confirmed bearing degradation on inspection.

The maintenance supervisor wanted a better number. His intuition was correct: if the hit rate was only 28%, the system was catching real failures but also sending technicians on a lot of unnecessary inspections. He suggested lowering the detection threshold to catch more failures earlier. The data science team complied. The new threshold caught roughly 15% more true positives: the PdM Hit Rate climbed to 32%. The supervisor was pleased.

Three months later, the planning team was drowning. Work order volume had increased 40%. Planners were reviewing and closing unnecessary inspection work orders at a pace that consumed a significant portion of their weekly capacity. The overtime budget for inspection technicians was exhausted by October. The 15% improvement in hit rate had been purchased with a much larger increase in false positives: the exact number the PdM Hit Rate metric didn't show.

PdM Hit Rate captures true positives: the detections that were correct. That's one side of the ledger. It says nothing about the cost of the false positives that came along with them. When you lower the detection threshold, you catch more real failures and you generate more false inspections. PdM Hit Rate counts the first without accounting for the second. A metric that only measures value creation while ignoring value destruction is incomplete at best and actively misleading when you're making threshold decisions with it.

PM Compliance showed 100% of inspections completed, regardless of necessity. Total Maintenance Cost captured both the savings from prevented failures and the waste from unnecessary inspections, but couldn't attribute either to the threshold setting or tell you which direction to adjust.

The threshold optimization problem is real and it's common. Every organization with a vibration prediction system, a heat exchanger fouling model, or any other condition-based prediction capability has a detection threshold that was set at some point in the past based on some combination of vendor defaults, training data optimization, and management intuition. Almost none of them have a metric that tells them whether that setting is generating more value than it costs.

There is no metric that tells you the realized cost trade-off between false positives and false negatives at a given detection threshold.

Formula

Prediction Threshold Value Ratio (PTVR) is the ratio of costs avoided by true positive predictions to costs incurred by false positive predictions at the current detection threshold. For every dollar the prediction system wastes on false alarms, how many dollars does it save by catching real failures?

The Formula

PTVR = Σ(Cₐᵥ × TP) / Σ(Cᵢₙ × FP)

Variables: Cₐᵥ is the average cost avoided per true positive. The differential between what an emergency repair costs and what a planned intervention costs for the same failure mode. TP is the count of true positive predictions in the evaluation period. Cᵢₙ is the average cost incurred per false positive: labor, materials, and any production impact from an unnecessary inspection or intervention. FP is the count of false positive predictions in the evaluation period.

Worked Example

A worked example using the utility's actual data. Over 12 months, the vibration system generated 892 predictions. 248 were true positives: the flagged pump showed genuine bearing degradation on inspection. 644 were false positives: the pump was inspected and the bearing was fine. Each true positive saved approximately $8,400: a $3,200 planned bearing replacement versus an $11,600 emergency replacement after failure. Each false positive cost approximately $1,850 in technician time, travel, and non-value-added inspection labor. PTVR = ($8,400 × 248) / ($1,850 × 644) = $2,083,200 / $1,191,400 = 1.75

For every dollar spent on false alarm inspections, the system saves $1.75 by catching real failures. The system is net positive, but the margin isn't compelling. Simulating the effect of threshold adjustments:Threshold +10%: PTVR = 2.84. True positives drop to 203 (45 fewer), false positives drop to 312 (332 fewer). Net value increases to $1,126,200: $234,400 more than the current setting. The 45 additional missed failures need severity assessment before implementing, but for predominantly bearing replacements in a non-critical application, this trade-off is strongly favorable. Threshold +20%: PTVR = 3.41. True positives 167, false positives 198. Net value $1,035,900: slightly less than +10%. This is the diminishing returns zone; the threshold has moved too far. Threshold −10%: PTVR = 1.12. True positives 279, false positives 1,041. Net value $413,250: less than half the current setting. This is what happened when the maintenance supervisor lowered the threshold chasing PdM Hit Rate improvements.

Sign convention: PTVR above 1.0 means the threshold creates net value. Below 1.0 means the prediction system costs more than it saves at current settings. Higher is better, but very high PTVR (above 10) may indicate an overly conservative threshold that's missing many preventable failures at low cost. Very high PTVR warrants checking whether false negatives are accumulating.

Edge Cases

Three edge cases. Cₐᵥ estimation uncertainty: where actual emergency versus planned cost differentials aren't available from records, use industry benchmarks for the specific failure mode, and label them as estimates. True negatives (correct "no failure" predictions) are excluded from PTVR. They create no cost and avoid no cost. Track them separately as operational confirmation but don't include them in the ratio. False negatives (missed failures that weren't predicted) are not captured in PTVR. Track them separately as Missed Failure Rate. If false negatives are frequent and high-severity, PTVR alone gives an incomplete picture of threshold economics.

Cross-References

Upstream, Prediction Confidence Calibration Index (GMI-AR-017) is a prerequisite. You can't optimize thresholds on miscalibrated confidence scores: you'd be adjusting a setting based on numbers that don't mean what you think they mean. Model Drift Detection Rate (GMI-AR-018) shifts the TP/FP ratio over time even with a stable threshold. A drifting model will make PTVR decline without any threshold change. Downstream, Autonomy Boundary Calibration Score (GMI-AR-006) uses PTVR as economic evidence for boundary settings. If the prediction system's economics are strong (PTVR well above 3.0), expanding AI autonomous authority has economic justification. If PTVR is marginal, autonomous action based on those predictions is harder to justify. AWSNR is directly affected by threshold settings: any threshold adjustment that changes the TP/FP balance also changes the work order signal-to-noise ratio. PTVR and AWSNR can pull in opposite directions: a threshold optimized for PTVR might still produce AWSNR that frustrates planners. Both metrics are needed to manage the trade-off.

Proposed formula, pending validation pilots.

At a Glance

Category
Operational Response and Work Execution

Decision Supported

Is my current detection threshold setting creating more economic value than it destroys, and in which direction should I adjust it?

← Override Outcome Index (OOI)Prediction-to-Execution Cycle Time (PECT) →

Assess Your Organization

The AI-GPS Assessment evaluates your AI governance maturity and recommends KPIs like this one for your implementation roadmap.

Take the Free AI-GPS Assessment