Fraud Score Calibration Index (FSCI)

GMI-PAY-004Payments & Transaction ProcessingPilot-Ready

Definition

Degree to which the fraud model's stated probability scores correspond to actual fraud rates at each score band, a 90% fraud score should mean approximately 90% of transactions at that score are actually fraudulent.

The Measurement Gap

The Measurement Gap

No metric tells you whether the fraud model's stated probabilities match actual fraud rates at each score band. This is the foundational calibration question that determines whether every downstream decision (threshold setting, step-up targeting, retry logic) is operating on reliable signals. A model that assigns a score of 0.85 to a transaction is implicitly claiming an 85% probability of fraud. If the actual fraud rate for transactions scored at 0.85 is only 40%, every threshold and decision rule built on that score is operating on a false premise.

Score calibration degrades without detection because the metrics that operations teams monitor are aggregates. Fraud basis points measures total losses divided by total volume: it cannot reveal that the model is systematically overconfident in the 0.60–0.80 range and underconfident in the 0.90+ range. Authorization decisions are made on confidence scores whose relationship to reality nobody verifies, and the mismatch only surfaces when net economic outcomes (FDNEY) degrade enough to trigger investigation.

The problem is not that fraud models are inaccurate at ranking transactions: most modern models have strong AUC scores. The problem is that good ranking does not imply good calibration. A model can perfectly separate fraud from legitimate transactions in relative order while systematically misstating the probability of fraud at every score level. Threshold optimization, cost-sensitive decision-making, and challenge targeting all require calibrated probabilities, not just good rank ordering.

Decision Supported

Are the fraud model's probability scores trustworthy enough to make dollar-valued threshold and targeting decisions, or do they need recalibration?

Formula

Plain Language

For each score band, compare the model's average stated probability to the actual confirmed fraud rate. A perfectly calibrated model shows a 1:1 relationship. The aggregate FSCI averages the absolute deviation across all bands and subtracts from 1, yielding a 0-to-1 index where 1.0 is perfect calibration.

Notation

Per-band: FSCI(b) = ActualFraudRate(b) / StatedProbability(b)

Aggregate: FSCI_agg = 1 − (1/B) × Σ|ActualFraudRate(b) − StatedProbability(b)|

Variables

Variable Definition Unit
b Score band (e.g., 0.00–0.10, 0.10–0.20, etc.) Band identifier
ActualFraudRate(b) Confirmed fraud transactions divided by total transactions in band b Ratio (0–1)
StatedProbability(b) Average model output score for transactions in band b Ratio (0–1)
B Total number of score bands Count

Proposed formula, pending validation pilots.

At a Glance

Unit
Index (0 to 1)
Trend Direction
Toward 1.0
Reporting Frequency
Monthly (with 30-90 day lag)
Category
Fraud Model Economic Integrity
← Fraud Decision Net Economic Yield (FDNEY)Merchant Hold Precision Rate (MHPR) →

Assess Your Organization

The AI-GPS Assessment evaluates your AI governance maturity and recommends KPIs like this one for your implementation roadmap.

Take the Free AI-GPS Assessment