Definition
Analyst Trust Calibration Index (ATCI) measures the correlation between analyst response depth at each AI alert confidence tier and the AI system's actual precision rate at that same tier: quantifying whether analysts are appropriately calibrating their vigilance to the system's actual reliability. In Plain Language the AI outputs alerts at different confidence levels. ATCI answers this question: do analysts act more thoroughly at the confidence levels where the AI is actually more accurate? A well-calibrated analyst should have high response depth at high-precision tiers and appropriate skepticism at low-precision tiers. ATCI measures whether that match exists.
The Measurement Gap
The SOC analyst's screen shows a "Critical" severity alert from the SIEM, a 94% confidence score from CrowdStrike Falcon on what it classifies as a lateral movement event. The analyst opens it, checks the source IP, looks at the timing, and closes it in eleven minutes with the note "FP: internal scanning tool." Their fourteenth false positive this week at the Critical tier.
Next Thursday, another Critical alert arrives at 96% confidence. The analyst opens it, glances at the source IP, scans the reasoning panel, and dismisses it in four minutes. It's also a lateral movement flag: same category as the fourteen false positives before it.
It's not a false positive. It's a real intrusion that an attacker spent two weeks establishing. The analyst dismissed it not because the evidence was weak but because their accumulated experience with the system's "Critical" label has taught them to discount it. By the time a second analyst catches the same indicator through a different detection vector three days later, the dwell time is significant.
This is a failure mode that MTTD cannot see. The SIEM detected the threat immediately: the day it generated the lateral movement signature. MTTD logs it as under an hour, technically accurate. What MTTD doesn't measure is what happened next: whether the detection was actually acted on, investigated, and confirmed, or was opened, glanced at, and dismissed based on a mental model formed by weeks of false positives at the same confidence tier.
Standard security metrics are built around detection and containment windows, not analyst calibration. False positive rate is commonly tracked at the aggregate level: overall FP rate across all tiers, all alert types, all time periods. That aggregate hides the crucial structural problem: the false positive rate at the Critical tier specifically, over the period when the analyst has formed a mental model about how reliable that tier actually is.
An analyst who has seen 87% of "Critical" alerts turn out to be false positives over the last 30 days is not making an irrational decision when they dismiss the next one in under five minutes. They're acting on the most relevant experience they have. The problem is that nobody measured that 87% figure, nobody told the analyst the actual precision rate at each tier, and nobody tracked whether analyst behavior was in sync with actual system accuracy at each confidence level.
The measurement gap is specific: there is no metric that tells you whether security analysts are calibrating their trust in AI alerts to the AI system's actual accuracy at each confidence tier, which means the human-AI loop in your SOC has an unknown calibration state. One that determines whether real threats get investigated or dismissed based on informal patterns analysts have built through experience you've never formally measured.
Analyst Trust Calibration Index (ATCI) closes this gap by measuring whether analyst response behavior aligns with actual system accuracy, confidence tier by confidence tier.
Formula
Analyst Trust Calibration Index (ATCI) measures the correlation between analyst response depth at each AI alert confidence tier and the AI system's actual precision rate at that same tier: quantifying whether analysts are appropriately calibrating their vigilance to the system's actual reliability.
In Plain Language
the AI outputs alerts at different confidence levels. ATCI answers this question: do analysts act more thoroughly at the confidence levels where the AI is actually more accurate? A well-calibrated analyst should have high response depth at high-precision tiers and appropriate skepticism at low-precision tiers. ATCI measures whether that match exists.
Formula
ATCI = Pearson correlation between {AI_precision_c} and {Analyst_action_rate_c} across confidence tiers c
Where:
- AI_precision_c = true positive rate (precision) of the AI system at confidence tier c, computed over the measurement period: AI_precision_c = TP_c / (TP_c + FP_c), where TP and FP are determined by analyst verdict at alert close
- Analyst_action_rate_c = percentage of alerts at confidence tier c that received substantive investigation (opened, evidence reviewed, determination documented) within the defined response SLA for that tier
- Confidence tiers c are typically: Critical, High, Medium, Low, or the equivalent in your SIEM's scoring schema
- Correlation is computed across tiers, not individual alertsSign Convention ATCI ranges from -1.0 to +1.0. +1.0 = perfect calibration: analysts are most thorough at exactly the tiers where the AI is most accurate. 0 = no calibration: analyst response is uniform regardless of system reliability. -1.0 = inverted calibration: analysts are most dismissive at the tiers where the AI is most accurate. The classic alert fatigue signature, where the AI has trained analysts to distrust its best work. Secondary metric: Calibration Gap by Tier (CGT):CGT_c = AI_precision_c − Analyst_action_rate_cPositive CGT means the AI is more reliable at that tier than analysts are treating it (underreaction). Negative CGT means analysts are acting more frequently at that tier than the AI's accuracy justifies.
Worked Example
A SOC running CrowdStrike and Splunk Enterprise Security reviews 90 days of alert data. For each confidence tier, they compute precision from confirmed outcomes and analyst action rates from investigation records:
- Critical tier: AI precision = 42%, Analyst action rate = 51% → CGT = −9%
- High tier: AI precision = 71%, Analyst action rate = 58% → CGT = +13%
- Medium tier: AI precision = 68%, Analyst action rate = 52% → CGT = +16%
- Low tier: AI precision = 55%, Analyst action rate = 34% → CGT = +21%
ATCI = Pearson correlation between [42%, 71%, 68%, 55%] and [51%, 58%, 52%, 34%] = 0.34A 0.34 correlation is weakly positive: analysts are somewhat more active at higher-precision tiers, but the calibration is imprecise. The Critical tier's poor precision (42%) has partially eroded analyst trust there, while the High and Medium tiers (71% and 68% precision) are getting less investigation depth than the AI's accuracy justifies. The practical implication: sharing the per-tier precision data directly with analysts, and resetting expectations about the Critical tier's actual accuracy, would likely improve both calibration and response effectiveness.
Edge Cases
If any tier has fewer than 30 alerts in the measurement period, that tier produces unreliable precision estimates and should be excluded from the correlation or flagged as insufficient volume. If the AI system doesn't expose confidence tier data (some SIEMs produce only a single priority), compute ATCI using score deciles instead of named tiers. Minimum requirement: four distinct confidence bands with 30+ alerts each to produce a meaningful correlation. Substantive investigation should be defined clearly: alert opened AND at least one evidence artifact reviewed (log query, network telemetry, endpoint data) AND a verdict filed (TP, FP, or under investigation). Alert opened and closed in under 60 seconds without evidence review should not count as substantive. Cross-references: TMCD (Threat Model Coverage Drift, GMI-IT-011) is upstream. If model coverage is shrinking in certain attack categories, precision calibration for those categories will be unreliable regardless of analyst behavior. AFA (Authentication Friction Accuracy, GMI-IT-029) is a parallel calibration metric in the IAM domain: the same trust miscalibration dynamic applies when identity analysts must decide whether to trust adaptive authentication risk scores. Downstream, poorly calibrated ATCI leads directly to extended dwell times on real threats: the mechanism is precisely documented in the worked example above.
Proposed formula, pending validation pilots.