Definition
Win-rate differential between AI-assembled and human-assembled chargeback representment packages on comparable dispute types.
The Measurement Gap
The Measurement Gap
No metric tells you whether AI-assembled chargeback evidence is as effective as the human-assembled evidence the representment model was trained on. Chargeback representment (the process of contesting a cardholder dispute with supporting evidence) is increasingly handled by AI systems that select which chargebacks to fight and assemble the evidence packages submitted to the card network. The selection decision and the evidence quality are different problems, but they are conflated in the only metric most organizations track: win rate.
Win rate on contested chargebacks may decline for two entirely different reasons. The representment model may be selecting the wrong disputes to fight, choosing cases with low win probability. Or the model may be selecting the right disputes but assembling weaker evidence packages than human analysts would have produced for the same cases. The distinction matters because the remediation is completely different: one requires retraining the selection model while the other requires improving the evidence assembly pipeline.
Chargeback rate measures dispute volume. Win rate measures representment outcomes. Neither measures evidence quality: the specific contribution of the AI evidence assembly step to the representment outcome. Without a controlled comparison between AI-assembled and human-assembled evidence on comparable dispute types, there is no way to determine whether AI representment is performing at, below, or above human capability. The gap is compounded by the fact that as AI handles an increasing share of representments, the human comparison baseline shrinks, making future measurement progressively harder unless a control group is maintained now.
Decision Supported
Is the AI evidence assembly pipeline producing representment packages that win at the same rate as human-assembled packages for comparable dispute types?
Formula
Plain Language
For each dispute type and amount range, divide the win rate of AI-assembled representment packages by the win rate of human-assembled packages. A ratio of 1.0 means AI evidence is equally effective. Below 1.0 means AI evidence is weaker. Aggregate across dispute types weighted by volume.
Notation
Per-type: REQI = WinRate_AI(d,a) / WinRate_human(d,a)
Aggregate: REQI_agg = Σ(w_da × REQI_da) / Σ w_da
Variables
| Variable | Definition | Unit |
|---|---|---|
| WinRate_AI(d,a) | Representment win rate for AI-assembled evidence, by dispute type d and amount range a | % |
| WinRate_human(d,a) | Representment win rate for human-assembled evidence, by dispute type d and amount range a | % |
| w_da | Volume weight for dispute type d and amount range a | Count |
| d | Dispute type category (e.g., Visa reason code, Mastercard reason code) | Category |
| a | Transaction amount range | $ range |
Proposed formula, pending validation pilots.