Dose Escalation Uncertainty Incorporation Score (DEUIS)

GMI-PH-049Pharmaceutical OperationsResearch-Forward

Definition

Ratio of observed coverage to stated coverage for AI dose prediction confidence intervals. Below 1.0 means the model's uncertainty is narrower than reality, the dangerous direction in first-in-human escalation.

The Measurement Gap

An AI dose optimization platform recommends dose levels for a first-in-human Phase I study of a novel bispecific antibody. At each level, the AI presents predictions with confidence intervals. The safety review committee approves escalation through four dose levels with no dose-limiting toxicities. The model appears accurate. The committee trusts it.

At the fifth dose level, two patients experience DLTs: observed rate 33.3%. The model predicted 12.7% DLT probability with 90% CI topping at 24.1%. The observed rate falls well outside the interval. The model was overconfident. If the committee had known actual uncertainty extended to 35-40%, they might have used a smaller cohort or intermediate dose.

Legacy dose escalation methods like 3+3 design do not have this problem because they use simple rules rather than probabilistic predictions. AI-based dose escalation replaces crude but transparent rules with predictions that are more informative when calibrated correctly and more dangerous when miscalibrated, presenting false precision that influences the committee's risk tolerance.

No current metric measures whether AI dose escalation models appropriately represent uncertainty, revealing whether safety committees make decisions on calibrated or overconfident information.

Formula

DEUIS = observed_coverage / stated_coverage

Where:

  • observed_coverage = fraction of actual outcomes falling within the model's stated confidence interval
  • stated_coverage = the nominal confidence level (e.g., 0.90 for 90% CI)

Worked Example:

Aggregated across 4 dose escalation programs over 3 years, 23 dose levels with AI predictions:

Stated confidence level: 90%. Actual outcomes within the 90% interval: 15 of 23.

Observed_coverage = 15/23 = 65.2%.
DEUIS = 65.2% / 90% = 0.724.

The 90% confidence intervals contain only 65% of actual outcomes. The safety committee has been making decisions based on intervals too narrow by roughly 25 percentage points.

Decomposing by program:

  • Program A (established mechanism, 7 levels): DEUIS = 0.857. Reasonable calibration.
  • Program B (novel bispecific, 6 levels): DEUIS = 0.556. Substantially overconfident.
  • Program C (ADC, 5 levels): DEUIS = 0.667. Moderately overconfident.
  • Program D (mRNA, 5 levels): DEUIS = 0.778. Slightly overconfident.

DEUIS degrades for novel modalities where training data is thinnest. Of 8 dose levels outside the interval, 6 were worse than predicted (higher DLT rate), 2 were better. The asymmetry is the patient safety concern.

Proposed formula, pending validation pilots.

At a Glance

Unit
Percentage
Trend Direction
Higher
Reporting Frequency
Per dose-escalation decision; per study retrospective
Category
Clinical Trials
← Adaptive Trial Generalizability Index (ATGI)Patient Matching Homogenization Score (PMHS) →

Assess Your Organization

The AI-GPS Assessment evaluates your AI governance maturity and recommends KPIs like this one for your implementation roadmap.

Take the Free AI-GPS Assessment