Assessment Disagreement Resolution Index (ADRI)

GMI-HCM-004Human Capital ManagementIntegration-RequiredPattern O-5: Override Value / Trust

Definition

The comparative predictive accuracy of AI-generated assessment scores versus human evaluator scores when the two disagree on a candidate or employee evaluation.

The Measurement Gap

When an AI assessment tool and a human evaluator agree on a candidate, the question of which one is right is irrelevant. The critical information lives in the disagreements: the cases where the algorithm says yes and the human says no, or vice versa. These disagreement cases reveal the actual complementarity (or redundancy) of human and AI judgment, and they are the only data points that can tell you how to weight the two sources in a combined decision.

Most organizations resolve AI-human disagreements through informal escalation, seniority, or default deference to one source. No legacy metric tracks which source was right when they disagreed. Interview-to-offer ratios measure pipeline efficiency but not assessment accuracy. Interviewer calibration exercises test agreement among humans but never evaluate human accuracy against AI accuracy on the same candidates. The result is that organizations invest in AI assessment tools without any evidence about whether the AI adds predictive value over human judgment alone, or whether humans add value over the AI alone.

Without this metric, organizations cannot make rational decisions about how to weight AI versus human scores in combined assessment models. They default to arbitrary weighting or gut-feel override policies that may systematically discard the more accurate signal.

Decision Supported:

When our AI assessment and human evaluators disagree, which source more accurately predicts actual outcomes, and how should we weight each source?

Formula

Plain language: Identify all cases where the AI score and human score disagree on a candidate assessment. For each disagreement case, determine which source more accurately predicted the actual outcome. ADRI is the proportion of disagreements where the AI was correct.

Notation:

ADRI = AI_correct_in_disagreements / Total_disagreements

Variables:

Variable Definition Unit
AI_correct_in_disagreements Number of disagreement cases where the AI score more accurately predicted the actual outcome (hire success, performance rating, retention) Count
Total_disagreements Total number of cases where AI and human assessment scores disagreed beyond a defined threshold Count

Proposed formula, pending validation pilots.

At a Glance

Unit
Ratio (0 to 1)
Trend Direction
Band; indicates calibrated weighting between AI and human
Reporting Frequency
Quarterly
Category
Human-AI Assessment Calibration
← AI Assessment Anchoring Bias Index (AABI)Calibration Predictive Validity Index (CPVI) →

Assess Your Organization

The AI-GPS Assessment evaluates your AI governance maturity and recommends KPIs like this one for your implementation roadmap.

Take the Free AI-GPS Assessment