Detection Confidence Accuracy Score (DCAS)

GMI-MIN-013MiningInfrastructure-Dependent

Definition

Degree to which the CV anomaly detection system confidence scores correspond to actual anomaly presence, measured by comparing detection confidence to expert validation outcomes.

The Measurement Gap

The Measurement Gap

CV inspection platforms report confidence scores with every classification. A section of pit wall might be labeled "clear, 94% confidence" or "anomaly detected, 87% confidence." Operations use these confidence scores to prioritize follow-up: high-confidence detections get investigated first, while high-confidence clear classifications are assumed safe. But confidence scores are only useful if they are calibrated. A model that says "94% confident this is clear" should be correct approximately 94% of the time. If it is actually correct only 78% of the time, the confidence score is misleading, and decisions based on it carry hidden risk.

Legacy metrics for CV inspection performance focus on detection count, false positive rate, and overall accuracy. These aggregate metrics hide the calibration problem. A system with 90% overall accuracy might have 99% accuracy at the 95%+ confidence level but only 60% accuracy at the 80-85% level. Worse, aggregate accuracy metrics weight false positives and false negatives equally, when in practice a false negative (missed anomaly near critical infrastructure) carries far greater consequence than a false positive (unnecessary follow-up investigation).

Without per-band calibration analysis, operations cannot make risk-informed decisions about which "clear" classifications to trust and which require supplementary verification.

Decision Supported:

When the drone survey says a pit wall section is clear with 95% confidence, is that confidence trustworthy, or are high-confidence clear classifications masking missed anomalies?

Formula

Plain Language

Group all CV classifications into confidence bands (e.g., 50-60%, 60-70%, 70-80%, 80-90%, 90-100%). Within each band, compare what the model predicted against expert-validated ground truth. For positive detections (model says "anomaly"), calculate the proportion that experts confirmed. For negative detections (model says "clear"), calculate the proportion that experts confirmed. Then compare the actual accuracy within each band to the stated confidence level to assess calibration.

Notation

DCAS_positive(b) = True_positives(b) / (True_positives(b) + False_positives(b))

DCAS_negative(b) = True_negatives(b) / (True_negatives(b) + False_negatives(b))

DCAS_calibration(b) = DCAS_actual(b) / Stated_confidence(b)

Variables

Variable Description Source
b Confidence band (e.g., 90-100%, 80-90%) used to group classifications for analysis Defined by analysis methodology
True_positives(b) Classifications in band b where the CV model flagged an anomaly and expert review confirmed the anomaly CV platform classifications cross-referenced with expert validation
False_positives(b) Classifications in band b where the CV model flagged an anomaly but expert review found no anomaly CV platform classifications cross-referenced with expert validation
True_negatives(b) Classifications in band b where the CV model reported "clear" and expert review confirmed no anomaly present CV platform classifications cross-referenced with expert validation
False_negatives(b) Classifications in band b where the CV model reported "clear" but expert review found an anomaly that was missed CV platform classifications cross-referenced with expert validation
Stated_confidence(b) Midpoint or representative confidence value for band b CV platform output metadata
DCAS_calibration(b) Ratio of actual to stated accuracy; 1.0 means perfectly calibrated, below 1.0 means overconfident Calculated

The critical output is DCAS_negative, which measures false negative calibration. A DCAS_calibration value below 1.0 for negative classifications means the model is overconfident when declaring areas clear.

Confidence Grading (required alongside every DCAS value):

Grade Basis Interpretation
A: Empirical 5+ observed events in measurement window provide direct tail observation Measurement grounded in observed outcomes; highest confidence
B: Hybrid 1-4 observed events plus Monte Carlo simulation with Bayesian updating Blends observation and simulation; moderate confidence
C: Simulated Zero observed events; derived entirely from Monte Carlo simulation Modeled, not observed; treat as indicative, not validated

A DCAS of any value at Grade C is a simulation that agrees with itself. It tells you the model is internally consistent but not that it matches reality. Grade C scores should trigger proactive stress-testing against historical near-miss events and cross-asset analogues rather than acceptance as calibrated.

Proposed formula, pending validation pilots.

At a Glance

Unit
Score (0 to 1)
Trend Direction
Toward 1.0
Reporting Frequency
Monthly
Category
Inspection Fidelity & Anomaly Detection Quality
← Path Concentration Degradation Rate (PCDR)Inspection Failure Mode Coverage Ratio (FMCR-M) →

Assess Your Organization

The AI-GPS Assessment evaluates your AI governance maturity and recommends KPIs like this one for your implementation roadmap.

Take the Free AI-GPS Assessment