AI-Human Discordance Resolution Consistency (AHDRC)

GMI-PH-007Pharmaceutical OperationsIntegration-Required

Definition

Consistency of how AI-human discordance events are resolved, computed as normalized entropy across resolution categories. A high score means similar discordances receive similar treatment regardless of who is on shift.

The Measurement Gap

The Gap: When the PAT model predicts one value and the lab measures another, different QA supervisors on different shifts make different calls. One releases the batch immediately, another holds it for four days of additional testing. Neither violates an SOP because there is no SOP for AI-human discordance resolution. Deviation Rate does not capture this because neither supervisor raised a deviation. Batch cycle time sees the difference but attributes it to shift efficiency, not inconsistent decision-making. The inconsistency is invisible to every standard metric.

The Decision:

AHDRC below 0.50 triggers development and deployment of a discordance resolution SOP. The metric drives standardization of the one decision point that AI implementation creates but nobody governs: what happens when the human and the AI disagree. AHDRC becomes a standing quality system effectiveness indicator, reported quarterly.

Formula

Notation:

AHDRC = 1 − [H(R) / H_max]

Where:
H(R) = −Σ p(r_i) × ln(p(r_i)) for each resolution category r_i
H_max = ln(N_categories)

Variables:

  • p(r_i) = proportion of discordance events resolved in category r_i
  • Resolution categories: Trust AI, Trust Human, Additional Testing, Escalated, Other (N = 5)

Worked Example:

83 PAT-lab discordance events last quarter. Resolution breakdown: Trust AI 14.5%, Trust Human 49.4%, Additional Testing 20.5%, Escalated 10.8%, Other 4.8%. H(R) = 1.340. H_max = ln(5) = 1.609. AHDRC = 1 − (1.340/1.609) = 0.167. An AHDRC of 0.167 is low: resolution is nearly random. Compare to a well-governed process: 78% Additional Testing, 12% Escalated, 10% Trust Lab. AHDRC = 0.350: much more consistent, with the scientifically defensible path dominating.

Edge Cases:

  • AHDRC near 1.0 means nearly identical resolution for all events. If that resolution is always "trust the AI," that is consistent rubber-stamping, not good governance. Always report which category dominates alongside the score.
  • Minimum 20 discordance events per period for meaningful entropy calculation
  • If discordance is rare (high PMCR), AHDRC will be noisy, a good problem indicating strong model calibration

Sign convention: Higher AHDRC indicates more consistency. Consistency must be paired with appropriateness: consistently wrong is worse than inconsistently right.

Proposed formula, pending validation pilots.

At a Glance

Unit
Percentage
Trend Direction
Higher
Reporting Frequency
Monthly; per discordance event
Category
Human-AI Trust & Override Governance
← Validated Envelope Exceedance Duration (VEED)Batch Record AI Reliance Gap (BRARG) →

Assess Your Organization

The AI-GPS Assessment evaluates your AI governance maturity and recommends KPIs like this one for your implementation roadmap.

Take the Free AI-GPS Assessment