Definition
Ratio of actual release failure rates to model-predicted risk within each risk band, measuring whether the release risk score deserves the gating authority it has been given.
The Measurement Gap
Measurement Gap and Decision
The Gap: Traditional deployment metrics reveal symptoms but not causes. Change failure rate tells you 12% of releases fail. It doesn't tell you the model scored 8 of those 12 failing releases as "low risk." MTTR tells you recovery speed. It doesn't tell you recovery took longer because the deployment strategy put the failing release in front of more users before detection. Nobody measures HOW wrong the model is, in what direction, and for which release types.
The Decision:
Can release managers trust the AI risk score to set deployment strategy (canary percentage, rollout speed, monitoring intensity), or is the model miscalibrated in ways that expose users to preventable failures? The danger is specifically in the low-risk band. Everyone expects high-risk releases to sometimes fail. But "low-risk" failures are operationally expensive because the team isn't expecting them, monitoring isn't intensified, and rollout proceeds aggressively based on false confidence.
What Changes When You Have This Number:
With RRSCI above 0.75, you have empirical justification for trusting the model to drive deployment strategy. Below 0.50, you decouple deployment strategy from the AI risk score entirely and use conservative defaults until the model is recalibrated. Between those, you use more conservative defaults for all bands while investigating which release characteristics the model underweights.
Formula
Per-Band Formula:
RRSCI(band_b) = Actual_failure_rate(b) / Expected_failure_rate(b)
Aggregate Formula:
RRSCI = 1 − (1/B) × Σ |Actual_failure_rate(b) − Expected_failure_rate(b)| / Expected_failure_rate(b)
Where Actual_failure_rate(b) is the proportion of releases in risk band b that resulted in rollback, hotfix, or incident within 24 hours, Expected_failure_rate(b) is the model's implied failure rate for band b (derived from model documentation or first quarter's actuals as baseline), and B is the number of risk bands.
Sign Convention:
1.0 means perfect calibration. Lower scores mean worse calibration. Negative values (rare) mean the model is systematically inverted.
Worked Example:
Over one quarter with three risk bands:
- Low-risk band: 180 releases, 18 failures → 10% actual failure rate vs. 5% expected → ratio 2.0, deviation 1.0
- Medium-risk band: 60 releases, 12 failures → 20% actual vs. 15% expected → ratio 1.33, deviation 0.33
- High-risk band: 20 releases, 8 failures → 40% actual vs. 35% expected → ratio 1.14, deviation 0.14
RRSCI = 1 − (1/3) × (1.0 + 0.33 + 0.14) = 1 − 0.49 = 0.51
This score falls in the moderate calibration band. The low-risk band is severely miscalibrated (twice the expected failure rate), driving most of the penalty. The model is labeling risky releases as safe.
Key Distinction:
RRSCI measures calibration (does confidence match reality?), not discrimination (can the model separate good from bad?). A model can correctly rank releases by risk but assign wrong probabilities to each rank.
Proposed formula, pending validation pilots.