Definition
CRSA is the Pearson product-moment correlation coefficient between AI-assigned risk scores and observed adverse outcome rates, computed per entity type over a trailing 90-day window. It answers one question: when the model says this entity is high-risk, does it actually produce worse outcomes than the entities it scores low?
The Measurement Gap
Most security teams are making triage decisions based on AI-generated risk scores they have never validated. The SIEM or SOAR platform outputs a "Critical" score of 87 or a "High" of 64, and analysts respond accordingly, but no one has checked whether the 87s actually lead to more breaches or higher-severity incidents than the 64s. The score feels authoritative because it is a number. But a number without calibration is just a number.
The failure mode surfaces slowly. Your team is de-prioritizing alerts scored "Medium" because the AI has been consistently over-scoring routine authentication noise as "High." After six months of responding to High-scored alerts that turned out to be nothing, analysts develop their own mental calibration: they discount the official score and use gut feel instead. At that point the risk score is not reducing cognitive load. It is adding a step that everyone skips mentally. You have a shadow triage process running on top of the official one, and neither is documented.
The second failure mode runs in the opposite direction: a risk scoring model that is systematically wrong about a specific threat category. Your model may have strong aggregate correlation but terrible calibration on insider threat indicators, meaning that class of risk is being systematically underweighted in every triage queue. You will not catch that without measuring correlation by category. Aggregate CRSA can look fine while an entire threat class is effectively invisible to your model.
There is also a vendor trust problem. Your SIEM or UEBA vendor validated their risk model on training data from their client base, not your environment. Their validation numbers are real, but they reflect their data distribution, not yours. Financial services, healthcare, retail, and manufacturing all have different threat landscapes, different user behavior baselines, and different attacker profiles. A model validated on one does not transfer to another without calibration verification.
Without CRSA, you do not know whether your risk scoring model is making your team faster at the right decisions or just faster at the wrong ones. Those are very different outcomes, and the operational cost of conflating them is not visible until something gets missed.
Formula
CRSA measures the predictive validity of your AI-generated cyber risk scores: specifically, how well the scores assigned to assets, users, or events correlate with observed adverse outcomes in your environment.
Definition:
CRSA is the Pearson product-moment correlation coefficient between AI-assigned risk scores and observed adverse outcome rates, computed per entity type over a trailing 90-day window. It answers one question: when the model says this entity is high-risk, does it actually produce worse outcomes than the entities it scores low?
Formula: plain language:
For each tracked entity (user account, asset, or event class), compare the AI-assigned risk score to whether that entity experienced a confirmed adverse outcome within a defined follow-up window. Compute the Pearson correlation across all scored entities in the period. Higher correlation means the scores are doing what they are supposed to do.
Notation:
CRSA = Pearson r({R_i}, {O_i}) = [Σ(R_i − R̄)(O_i − Ō)] / √[Σ(R_i − R̄)² × Σ(O_i − Ō)²]
Where:
- R_i = risk score assigned to entity i by the AI system, normalized to 0–100
- O_i = adverse outcome indicator for entity i: 1 if the entity was involved in a confirmed security incident within the follow-up window, 0 otherwise
- R̄, Ō = mean risk score and mean outcome rate across all n entities in the measurement window
- Follow-up window: 30 days for event-level scoring, 90 days for user and asset scoring
- Measurement window: trailing 90 days of score-outcome pairsWorked Example Your SIEM assigns risk scores to 1,200 user accounts over Q3. You track whether each account was involved in a confirmed security incident within 30 days of scoring. Mean risk score across accounts: 42. Mean outcome rate: 0.04 (4% of accounts had an incident). Computing Pearson r across all 1,200 pairs gives r = 0.58. That is moderate-to-strong positive correlation. The higher the AI scores an account, the more likely that account actually produces an incident. Not perfect, but useful enough to act on. Now break it by category. Privileged accounts (n=150): r = 0.71. Strong, the model is well-calibrated for this group. Contractor accounts (n=200): r = 0.21. Weak, the model barely predicts contractor incidents. That is where you focus remediation, not on the aggregate number.
Secondary metric: Calibration Bias by Tier (CBT_r):
CBT_r = Actual_failure_rate_r − Expected_failure_rate_rWhere r = risk tier (Low / Medium / High / Critical), Actual_failure_rate_r is the observed incident rate for entities scored in tier r, and Expected_failure_rate_r is the incident rate the model's tier thresholds implicitly predict (derivable from vendor documentation or threshold configuration). CBT_r > 0 means the model is under-predicting risk for that tier: more incidents are happening than the tier label implies. That is the dangerous direction. CBT_r < 0 means over-prediction: the tier is noisier than expected. Track CBT_r by tier to catch systematic drift even when aggregate CRSA looks acceptable. A model can have an aggregate r of 0.55 while having CBT_r of +0.12 for the Critical tier, meaning your Critical-scored entities are producing far more incidents than the model's own threshold design anticipates.
Sign convention:
Higher CRSA is better. Range is theoretically −1.0 to +1.0. In practice, values below 0 mean the model is inversely predictive, a red flag that should not survive more than a week of live monitoring.
Edge Cases
Low incident rates: With base incident rates under 1%, Pearson correlation is sensitive to small-sample noise. Apply Fisher's Z transformation to compute 95% confidence intervals before acting on quarter-over-quarter changes. Do not interpret a change from r=0.52 to r=0.48 as meaningful without checking the confidence interval. Score compression: If your model outputs mostly scores in the 60–80 range and rarely uses the extremes, Pearson correlation will be artificially suppressed regardless of actual predictive quality. Check your score distribution histogram before interpreting CRSA. A model that only uses 20 points of a 100-point scale has restricted range; that is a model problem, not a measurement problem, but it affects how you read the number. Entity type mixing: Do not compute a single CRSA across users, assets, and events without stratification. The predictive dynamics, base rates, and follow-up windows differ enough across entity types that a combined CRSA hides more than it reveals. Always compute by entity type first.
Cross-references:
- Upstream: TMCD (GMI-IT-011, Threat Model Coverage Drift). If recall is dropping on an uncovered threat category, CRSA for that category will eventually follow; a model that cannot detect a threat class cannot score it accurately
- Downstream: CED (GMI-IT-034, Compliance Evidence Density). Compliance workflows that use risk scores to gate evidence collection depend on those scores being calibrated; poor CRSA means the compliance filter is applying the wrong inclusion criteria
Proposed formula, pending validation pilots.