Causality Assessment Novelty Bias Index (CANBI)

GMI-PH-042Pharmaceutical OperationsResearch-Forward

Definition

Excess not-related classification rate the AI applies to novel products relative to established ones, after subtracting the same gap in expert assessments. Isolates AI novelty bias from legitimate clinical caution.

The Measurement Gap

A pharmaceutical company's AI PV system processes causality assessments across 42 products. When benchmarked against expert assessments, it achieves 87% agreement overall. But the portfolio contains 28 established small molecules with extensive case databases and 14 newer products including 3 with fewer than 2 years of post-market data.

For established products, AI-expert agreement is 89%. For novel products, agreement drops to 74%, and disagreement is almost exclusively one-directional: the AI says not related where experts say possibly or definitely related. The model defaults to not related because that is the base rate in the overall training data, roughly 65% of all adverse events.

This creates systematic under-attribution for novel products during the critical early post-market period. Signal detection is delayed because individual cases classified as unrelated cannot accumulate into a signal. Periodic safety update reports underestimate relatedness. The safety database creates a misleading picture that could influence benefit-risk assessments. Legacy metrics showing 87% overall agreement mask the directional bias for novel products where accurate classification matters most.

Formula

CANBI_adjusted = (NR_rate_novel - expert_NR_rate_novel) - (NR_rate_established - expert_NR_rate_established)

Where:

  • NR_rate = AI not-related classification rate by product group
  • expert_NR_rate = expert PV scientist not-related rate for the same cases

Worked Example:

Twelve-month sample, 200 cases per group:

Established products:

  • AI classified not related: 134/200 = 67.0%
  • Expert classified not related: 128/200 = 64.0%
  • AI excess: 67.0% - 64.0% = 3.0%

Novel products:

  • AI classified not related: 156/200 = 78.0%
  • Expert classified not related: 131/200 = 65.5%
  • AI excess: 78.0% - 65.5% = 12.5%

CANBI_adjusted = 12.5% - 3.0% = 9.5%

For a novel product receiving 600 adverse event reports per year, this bias means approximately 57 cases annually are classified as not related when an expert would classify them as at least possibly related. Those 57 cases are invisible to signal detection. Of the 25 misclassified cases examined: 11 involved known class effects the model had not learned, 7 involved delayed-onset events, 4 involved MedDRA terms associated only with disease progression, and 3 were rare events indistinguishable from noise.

Proposed formula, pending validation pilots.

At a Glance

Unit
Percentage
Trend Direction
Higher
Reporting Frequency
Quarterly; stratified by product maturity
Category
Pharmacovigilance
← Case Seriousness Classification Accuracy (CSCA)Signal Sensitivity Parity Index (SSPI) →

Assess Your Organization

The AI-GPS Assessment evaluates your AI governance maturity and recommends KPIs like this one for your implementation roadmap.

Take the Free AI-GPS Assessment