Definition
Degree to which AI-driven dispute auto-resolution outcomes systematically favor one party (merchant or cardholder) compared to human-adjudicated outcomes on comparable dispute types.
The Measurement Gap
The Measurement Gap
No metric tells you whether AI dispute auto-resolution develops systematic biases favoring one party over another. As AI handles an increasing share of dispute adjudication (deciding whether the merchant or cardholder prevails) the system may develop category-specific biases that are invisible to aggregate outcome metrics. The AI auto-resolves disputes based on pattern matching against historical outcomes, and if the training data contains systematic biases (merchants in certain categories always lose, cardholders with certain dispute patterns always win), the model reproduces and potentially amplifies those biases at scale.
Dispute resolution time improves. This is the metric organizations celebrate. Chargeback rate is tracked as an aggregate. Neither party to the dispute typically knows whether the decision was made by an AI system or a human analyst. But outcome consistency (the degree to which comparable cases receive comparable outcomes regardless of which system adjudicates them) goes entirely unmeasured.
DARFI uses human adjudication outcomes as the fairness baseline. This is a proxy, not an absolute standard: human adjudicators carry their own biases. For disputes with objectively verifiable outcomes (tracking data confirms non-delivery, authorization code mismatch), DARFI measures accuracy divergence. For subjective judgment-call disputes (service quality complaints, partial-use claims), DARFI measures policy alignment divergence. Both are operationally useful, but they answer different questions and should be interpreted separately.
The gap has regulatory consequences. Regulation E and Regulation Z require accurate dispute resolution for electronic fund transfers and credit transactions respectively. If an AI system systematically favors cardholders for certain categories while systematically favoring merchants for others, the organization faces regulatory exposure that no current metric detects. The gap also has commercial consequences: systematic cardholder-favoring increases merchant chargeback losses and drives merchant attrition, while systematic merchant-favoring generates consumer complaints and regulatory scrutiny.
Decision Supported
Is the AI dispute auto-resolution system producing outcomes consistent with human adjudication across dispute categories, and where it diverges, is the divergence on objective-outcome disputes (accuracy problem) or subjective-judgment disputes (policy alignment problem)?
Formula
Plain Language
For each dispute category, compare the percentage of cases where the AI favored the cardholder to the percentage where human analysts favored the cardholder. The difference is the bias for that category. Aggregate across categories using absolute deviations weighted by volume.
Notation
Per-category: DARFI(d) = CardholderFavorRate_AI(d) − CardholderFavorRate_human(d)
Aggregate: DARFI_agg = Σ(w_d × |DARFI(d)|) / Σ w_d
Variables
| Variable | Definition | Unit |
|---|---|---|
| CardholderFavorRate_AI(d) | Percentage of AI-resolved disputes favoring the cardholder for dispute category d | % |
| CardholderFavorRate_human(d) | Percentage of human-adjudicated disputes favoring the cardholder for dispute category d | % |
| w_d | Volume weight for dispute category d | Count |
| d | Dispute category (by reason code) | Category |
Proposed formula, pending validation pilots.