Definition
The Drilling Prediction Confidence Calibration (DPCC) measures the ratio of an AI drilling prediction's actual accuracy to its stated confidence, computed separately for each confidence band and each prediction type.
Plain language.
The Measurement Gap
Diego is the drilling supervisor on a deepwater rig in the Gulf of Mexico. The operator has been running an AI drilling advisor for nine months: the platform monitors WITSML real-time data and produces predictions for stuck pipe, kick risk, hole cleaning issues, and ROP variation, each tagged with a stated confidence. The advisor produces a few dozen predictions per shift; Diego and his team have learned to treat the high-confidence predictions seriously and the medium-confidence ones as flags for closer monitoring.
On a Tuesday night, the advisor reports "92% confident no stuck pipe in next 6 hours" during a hole cleaning trip in a 14,800-foot lateral. Based on that prediction, Diego authorizes a faster pull rate to make up time on the trip. Forty-three minutes later the pipe is stuck. The recovery operation takes 18 hours, costs $312,000 in rig time and tooling, and produces formation damage that requires sidetrack consideration. The post-event review pulls the advisor's prediction history. In the 90 prior predictions in the "90%+ confidence, no stuck pipe" band, six had ended in stuck pipe events. The band's actual accuracy was 93%. So the prediction was honest in the aggregate. But Diego had been treating "92% confident" as if it meant "nearly certain" (closer to 99%) and his decisions assumed that mental calibration.
The investigation goes further. The advisor's "80–89% confident" band shows 81% actual accuracy: well calibrated. The "70–79%" band shows 74%: well calibrated. The "60–69%" band shows 68%: well calibrated. The "90%+" band shows 93%. Every band is well calibrated except the highest, and that one is the band where Diego and his team have been changing their behavior the most. The advisor was not lying. The team was reading the high-confidence label as more certain than the math supported.
This is not a story about an unreliable model. The advisor's overall performance is credible: most bands track their stated accuracy within a few percentage points. The structural problem is that confidence labels carry behavioral weight that the underlying math may not support, especially in the high-confidence band where operational decisions get less conservative. Diego made a defensible decision against the information presented. The information was, in a precise sense, mislabeled.
Legacy drilling metrics cannot see this directly. Rate of penetration (ROP) reports productivity during normal operations and does not change when a prediction was wrong. Non-productive time (NPT) captures the consequence (the 18 hours stuck) but does not report which predictions led to which NPT events or whether the prediction confidence was honest. Stuck pipe rate as a population metric is too coarse and too lagging: it tells you whether your wells get stuck, not whether your AI's confidence labels mean what they say. The vendor's stated accuracy is a population-average claim and does not test calibration band by band.
What is missing is a measurement of whether the stated confidence in each prediction band matches the actual outcome rate for predictions in that band. That measurement is what calibration analysis does in the machine-learning literature. The question DPCC proposes to answer is the one Diego could not ask before authorizing the faster trip: when the AI says "92% confident no stuck pipe," does 92% confidence actually mean 92% accuracy?
Formula
The Drilling Prediction Confidence Calibration (DPCC) measures the ratio of an AI drilling prediction's actual accuracy to its stated confidence, computed separately for each confidence band and each prediction type.
Plain language. Bucket every AI prediction by its stated confidence band: 60–69%, 70–79%, 80–89%, 90%+, however the platform reports it. For each bucket, compute what fraction of predictions in that bucket turned out to be correct. Compare that fraction to the bucket's stated confidence midpoint. The ratio is DPCC for that bucket. A perfectly calibrated AI has DPCC = 1.0 in every band: the predictions in the 80% band turn out to be right 80% of the time. A miscalibrated AI has DPCC well below 1.0 in some bands, meaning the stated confidence is higher than the actual accuracy. The same calculation done for each prediction type (stuck pipe risk, kick risk, hole cleaning, ROP variation) captures whether the calibration holds across the AI's outputs or only for some of them.
Notation:
DPCC(b) = Accuracy_actual(b) ÷ Confidence_stated(b)
Accuracy_actual(b) = correct_predictions(b) ÷ total_predictions(b)
Where:
- b = a confidence band (e.g., 80–89%)
- correct_predictions(b) = the count of predictions in band b that matched the eventual outcome
- total_predictions(b) = the total count of predictions in band b
- Confidence_stated(b) = the band midpoint (84.5% for the 80–89% band)
- DPCC is computed per band per prediction type per rig (or per asset, depending on the unit of analysis)
The "correct" determination requires defining what it means for a prediction to match the outcome. For "92% confident no stuck pipe in next 6 hours," the prediction is correct if the well does not get stuck in the next 6 hours. For "78% confident kick within 24 hours," the prediction is correct if a kick occurs in the next 24 hours. The definition has to be applied consistently across the prediction population.
Worked example. The drilling AI advisor on Diego's rig has produced 1,240 predictions across the prior 6 months, split across 4 confidence bands and 5 prediction types. For the stuck-pipe-risk prediction type, the breakdown is: 90%+ band has 90 predictions, 84 of which were correct (Accuracy_actual = 93%, DPCC = 93% ÷ 95% = 0.98); 80–89% band has 135 predictions, 110 correct (81%, DPCC = 0.96); 70–79% band has 180 predictions, 133 correct (74%, DPCC = 0.99); 60–69% band has 220 predictions, 150 correct (68%, DPCC = 1.05). Three out of four bands are well calibrated (DPCC near 1.0). The 90% band is mildly under-calibrated: the stated confidence is 95% midpoint but actual accuracy is 93%. In this case the issue is small. The metric becomes diagnostic when a band shows DPCC well below 0.9 or above 1.15, indicating systematic miscalibration that operational decisions will be making against.
Edge cases. Low-probability correct predictions are trivially right. If the base rate of stuck pipe in 6-hour windows is 3%, a prediction of "97% confident no stuck pipe" will be right 97% of the time by simply matching the base rate. DPCC will read 1.0 even though the prediction added no information. Segment by prediction type: predictions about rare events need to be evaluated against the base rate, not just against the stated confidence. A separate "skill score" calculation (Brier score, or equivalent) handles this in the ML literature; for DPCC, the practical fix is to compute and report base rates alongside the metric.
Segment by prediction type. The AI may be well calibrated on stuck pipe (because it has seen many stuck pipe events) and poorly calibrated on kicks (because kicks are rarer). Aggregating across types hides the diagnosis.
Minimum sample size: 50 predictions per band per prediction type. Below that, the actual accuracy estimate has too much variance to compute a meaningful DPCC.
Proposed formula, pending validation pilots.