Threat Model Coverage Drift (TMCD)

GMI-IT-011IT OperationsIntegration-Required

Definition

Threat Model Coverage Drift (TMCD) measures the change in an ML threat detection model's recall rate across attack categories over time, weighted by the current frequency of those categories in the live threat landscape: tracking whether the model's coverage is expanding or contracting relative to how the threat environment is actually evolving. In Plain Language segment your threat categories using MITRE ATT&CK or your own taxonomy. Measure how well the detection model recalls threats in each category this quarter compared to last quarter. Weight each category by how common that attack type is in the current threat intelligence landscape. Sum the weighted changes. Positive means coverage is expanding; negative means it's contracting.

The Measurement Gap

Your security team deployed an ML-based SIEM model 18 months ago. At launch, it performed well on the threat categories that had generated the most incidents over the prior two years: phishing, ransomware, and credential stuffing. Detection rates in quarterly red team exercises were solid: 89% recall on phishing-derived attacks, 84% on ransomware indicators, 91% on credential stuffing patterns. The security director presented these numbers to the board. The model was declared production-ready.

Over the following 18 months, three things happened that didn't surface in any metric. Supply chain compromise attacks increased significantly in the threat landscape: the SolarWinds-style attack vector where malicious code enters through legitimate software update channels. AI-generated spear phishing evolved past the syntactic patterns the model was trained to detect. And living-off-the-land techniques (where attackers use legitimate system tools like PowerShell and WMI rather than custom malware) became the dominant post-compromise methodology in the financial sector.

The model wasn't retrained on these patterns. The security team didn't request a retraining because the quarterly metrics showed no deterioration. Those metrics were measuring detection performance on the attack categories the model already knew. They weren't measuring (couldn't measure) whether the model's coverage of the evolving threat landscape was shrinking.

Legacy security metrics are built around detection rate and false positive rate at the aggregate level. They measure how the model performs on threats it encounters. They say nothing about threats it misses because the attack category wasn't represented in its training distribution. A model that detects 91% of the threat types it was trained on while missing entire categories of emerging attacks has excellent detection rate metrics and catastrophically incomplete coverage.

MTTD and MTTC measure speed on known threats. Neither asks whether the model generating your detections has any capability on threat types that weren't common two years ago. Your MTTD can be four hours and the model can still have near-zero recall on supply chain compromise. The metrics describe timing, not coverage.

The measurement gap is precise: there is no metric that tells you whether the threat detection model's coverage of the current threat landscape is expanding or contracting, which means the security AI is continuously optimizing for yesterday's threats while today's threat actors iterate past its detection boundaries without any dashboard showing you the gap is growing.

Threat Model Coverage Drift (TMCD) is the metric that asks whether the map still matches the territory, not whether you're navigating the known parts efficiently, but whether the territory has changed in ways the map no longer reflects.

Formula

Threat Model Coverage Drift (TMCD) measures the change in an ML threat detection model's recall rate across attack categories over time, weighted by the current frequency of those categories in the live threat landscape: tracking whether the model's coverage is expanding or contracting relative to how the threat environment is actually evolving.

In Plain Language

segment your threat categories using MITRE ATT&CK or your own taxonomy. Measure how well the detection model recalls threats in each category this quarter compared to last quarter. Weight each category by how common that attack type is in the current threat intelligence landscape. Sum the weighted changes. Positive means coverage is expanding; negative means it's contracting.

Formula

TMCD = Σ (w_c × ΔRecall_c) / Σ w_c

Where:

  • w_c = weight of threat category c, based on its current frequency in the threat intelligence landscape (percentage of observed sector attacks using this technique, from threat intelligence feeds or sector ISAC reports)- ΔRecall_c = Recall_c(current period) − Recall_c(prior period): the change in detection recall for category c between measurement windows
  • Recall_c = TP_c / (TP_c + FN_c): true positive rate for category c, measured through confirmed incident data, red team results, or threat simulation exercisesSign Convention Positive TMCD = the model is improving on weighted-important categories. Negative TMCD = coverage is shrinking on categories that currently matter most in the threat landscape. The magnitude matters: −2% per quarter in a stable environment is background noise; −10% per quarter with an actively evolving threat landscape warrants immediate model review.

Worked Example

A financial services security team categorizes threats using five MITRE ATT&CK tactics and measures recall quarterly via red team exercises. Threat intelligence weighting is derived from FS-ISAC quarterly threat reporting. Comparing Q4 to Q3:

  • Phishing / Initial Access: recall 89% → 92%; weight 0.28 → weighted change: +0.84%
  • Credential Access: recall 84% → 83%; weight 0.22 → weighted change: −0.22%
  • Supply Chain Compromise: recall 34% → 29%; weight 0.19 → weighted change: −0.95%
  • Living-off-the-Land (Defense Evasion): recall 61% → 57%; weight 0.16 → weighted change: −0.64%
  • Ransomware (Impact): recall 91% → 90%; weight 0.15 → weighted change: −0.15%

TMCD = (0.84 − 0.22 − 0.95 − 0.64 − 0.15) / 1.00 = −1.12 percentage pointsThe −1.12 aggregate looks modest. The category breakdown reveals the urgent finding: supply chain compromise has the third-highest threat intelligence weighting (19% of current observed sector attacks) and the worst single-quarter recall decline (−5 percentage points). That combination (deteriorating recall on a growing threat category) is exactly the signal TMCD is designed to surface. Without TMCD, the quarterly security review shows "detection rate stable" because the model's strong performance on phishing and ransomware dominates the aggregate.

Edge Cases

New threat categories with no prior measurement period: assign Recall = 0% for the baseline period. This creates a visible "new gap" signal in the first period measured, which is the correct behavior. Categories with fewer than 10 confirmed incidents or red team exercises in the period: flag the recall estimate as low-confidence and report separately as "untested categories." Threat intelligence weighting requires a quantitative source. Sector-specific ISACs, commercial threat intelligence platforms, or CISA advisories. If only qualitative sources are available, use equal weighting and document the limitation explicitly. Cross-references: TMCD has no upstream dependencies in this book. It's a direct measure of the threat model state. ATCI (GMI-IT-010) is downstream: if TMCD is negative for a specific attack category, analyst calibration at that category's confidence tier may be unreliable. The model is degrading in ways analysts haven't experienced yet, so their calibration lags the actual accuracy change. CRSA (GMI-IT-012) is also downstream: security-related changes that involve tactics in drifting threat categories may be under-scored for risk by the change risk model, which may share training data limitations with the detection model.

TMCD above 0 each quarter means coverage is expanding: the model is keeping up with threat evolution. TMCD below −5% per quarter warrants a formal model review. Any single category where recall drops below 50% while its threat intelligence weight exceeds 0.15 should trigger immediate targeted red team testing and potential model supplementation, regardless of the aggregate TMCD.

Proposed formula, pending validation pilots.

At a Glance

Category
AI-Era IT Operations
← Test Adequacy Illusion Index (TAII)Tier Routing Economic Accuracy (TREA) →

Assess Your Organization

The AI-GPS Assessment evaluates your AI governance maturity and recommends KPIs like this one for your implementation roadmap.

Take the Free AI-GPS Assessment