Definition
Degree to which the fraud model's stated probability scores correspond to actual fraud rates at each score band, a 90% fraud score should mean approximately 90% of transactions at that score are actually fraudulent.
The Measurement Gap
The Measurement Gap
No metric tells you whether the fraud model's stated probabilities match actual fraud rates at each score band. This is the foundational calibration question that determines whether every downstream decision (threshold setting, step-up targeting, retry logic) is operating on reliable signals. A model that assigns a score of 0.85 to a transaction is implicitly claiming an 85% probability of fraud. If the actual fraud rate for transactions scored at 0.85 is only 40%, every threshold and decision rule built on that score is operating on a false premise.
Score calibration degrades without detection because the metrics that operations teams monitor are aggregates. Fraud basis points measures total losses divided by total volume: it cannot reveal that the model is systematically overconfident in the 0.60–0.80 range and underconfident in the 0.90+ range. Authorization decisions are made on confidence scores whose relationship to reality nobody verifies, and the mismatch only surfaces when net economic outcomes (FDNEY) degrade enough to trigger investigation.
The problem is not that fraud models are inaccurate at ranking transactions: most modern models have strong AUC scores. The problem is that good ranking does not imply good calibration. A model can perfectly separate fraud from legitimate transactions in relative order while systematically misstating the probability of fraud at every score level. Threshold optimization, cost-sensitive decision-making, and challenge targeting all require calibrated probabilities, not just good rank ordering.
Decision Supported
Are the fraud model's probability scores trustworthy enough to make dollar-valued threshold and targeting decisions, or do they need recalibration?
Formula
Plain Language
For each score band, compare the model's average stated probability to the actual confirmed fraud rate. A perfectly calibrated model shows a 1:1 relationship. The aggregate FSCI averages the absolute deviation across all bands and subtracts from 1, yielding a 0-to-1 index where 1.0 is perfect calibration.
Notation
Per-band: FSCI(b) = ActualFraudRate(b) / StatedProbability(b)
Aggregate: FSCI_agg = 1 − (1/B) × Σ|ActualFraudRate(b) − StatedProbability(b)|
Variables
| Variable | Definition | Unit |
|---|---|---|
| b | Score band (e.g., 0.00–0.10, 0.10–0.20, etc.) | Band identifier |
| ActualFraudRate(b) | Confirmed fraud transactions divided by total transactions in band b | Ratio (0–1) |
| StatedProbability(b) | Average model output score for transactions in band b | Ratio (0–1) |
| B | Total number of score bands | Count |
Proposed formula, pending validation pilots.