Definition
Degree to which the CV anomaly detection system confidence scores correspond to actual anomaly presence, measured by comparing detection confidence to expert validation outcomes.
The Measurement Gap
The Measurement Gap
CV inspection platforms report confidence scores with every classification. A section of pit wall might be labeled "clear, 94% confidence" or "anomaly detected, 87% confidence." Operations use these confidence scores to prioritize follow-up: high-confidence detections get investigated first, while high-confidence clear classifications are assumed safe. But confidence scores are only useful if they are calibrated. A model that says "94% confident this is clear" should be correct approximately 94% of the time. If it is actually correct only 78% of the time, the confidence score is misleading, and decisions based on it carry hidden risk.
Legacy metrics for CV inspection performance focus on detection count, false positive rate, and overall accuracy. These aggregate metrics hide the calibration problem. A system with 90% overall accuracy might have 99% accuracy at the 95%+ confidence level but only 60% accuracy at the 80-85% level. Worse, aggregate accuracy metrics weight false positives and false negatives equally, when in practice a false negative (missed anomaly near critical infrastructure) carries far greater consequence than a false positive (unnecessary follow-up investigation).
Without per-band calibration analysis, operations cannot make risk-informed decisions about which "clear" classifications to trust and which require supplementary verification.
Decision Supported:
When the drone survey says a pit wall section is clear with 95% confidence, is that confidence trustworthy, or are high-confidence clear classifications masking missed anomalies?
Formula
Plain Language
Group all CV classifications into confidence bands (e.g., 50-60%, 60-70%, 70-80%, 80-90%, 90-100%). Within each band, compare what the model predicted against expert-validated ground truth. For positive detections (model says "anomaly"), calculate the proportion that experts confirmed. For negative detections (model says "clear"), calculate the proportion that experts confirmed. Then compare the actual accuracy within each band to the stated confidence level to assess calibration.
Notation
DCAS_positive(b) = True_positives(b) / (True_positives(b) + False_positives(b))
DCAS_negative(b) = True_negatives(b) / (True_negatives(b) + False_negatives(b))
DCAS_calibration(b) = DCAS_actual(b) / Stated_confidence(b)
Variables
| Variable | Description | Source |
|---|---|---|
| b | Confidence band (e.g., 90-100%, 80-90%) used to group classifications for analysis | Defined by analysis methodology |
| True_positives(b) | Classifications in band b where the CV model flagged an anomaly and expert review confirmed the anomaly | CV platform classifications cross-referenced with expert validation |
| False_positives(b) | Classifications in band b where the CV model flagged an anomaly but expert review found no anomaly | CV platform classifications cross-referenced with expert validation |
| True_negatives(b) | Classifications in band b where the CV model reported "clear" and expert review confirmed no anomaly present | CV platform classifications cross-referenced with expert validation |
| False_negatives(b) | Classifications in band b where the CV model reported "clear" but expert review found an anomaly that was missed | CV platform classifications cross-referenced with expert validation |
| Stated_confidence(b) | Midpoint or representative confidence value for band b | CV platform output metadata |
| DCAS_calibration(b) | Ratio of actual to stated accuracy; 1.0 means perfectly calibrated, below 1.0 means overconfident | Calculated |
The critical output is DCAS_negative, which measures false negative calibration. A DCAS_calibration value below 1.0 for negative classifications means the model is overconfident when declaring areas clear.
Confidence Grading (required alongside every DCAS value):
| Grade | Basis | Interpretation |
|---|---|---|
| A: Empirical | 5+ observed events in measurement window provide direct tail observation | Measurement grounded in observed outcomes; highest confidence |
| B: Hybrid | 1-4 observed events plus Monte Carlo simulation with Bayesian updating | Blends observation and simulation; moderate confidence |
| C: Simulated | Zero observed events; derived entirely from Monte Carlo simulation | Modeled, not observed; treat as indicative, not validated |
A DCAS of any value at Grade C is a simulation that agrees with itself. It tells you the model is internally consistent but not that it matches reality. Grade C scores should trigger proactive stress-testing against historical near-miss events and cross-asset analogues rather than acceptance as calibrated.
Proposed formula, pending validation pilots.