Definition
Chaos Coverage Gap (CCG) measures the share of high-risk system components that have never been chaos-tested.
In plain language: rank your system components by risk independently of the chaos platform's selection algorithm.
The Measurement Gap
An e-commerce company's reliability team rolled out an AI-driven chaos engineering platform 14 months ago. The platform observes service behavior, identifies promising chaos experiment targets, schedules experiments during business windows, and reports results. The team has run over 800 experiments in the past year. Pass rate is 94%. The platform's quarterly reports are impressive. Leadership cites the chaos program as evidence of operational maturity.
Then a regional outage hits a third-party payment processor on a Black Friday. The company's payment service fails in a way the team had not seen before. The retry logic that was supposed to fall back to a secondary processor times out, then cascades into the cart service, which serializes its writes against the failing payment service, which fills the queue, which exhausts connection pools, which takes down the inventory service. End-to-end outage: 47 minutes. Revenue impact: estimated $4.2 million.
The post-incident review surfaces an uncomfortable fact. The cart service's serialized-write behavior under payment latency was a known fragility: listed in the team's risk register from a quarterly architecture review eight months ago. The reliability team had not run a chaos experiment against that behavior. The AI chaos platform had not selected the cart-payment interaction for experimentation because the platform's selection algorithm reward observed-failure-rate; the cart-payment interaction had been stable for over a year and ranked low on the experiment-priority list. The platform was running experiments on the parts of the system that had failed before. The parts that had not failed before were not being tested. They were the parts that failed during the outage.
The legacy chaos metrics on the team's dashboard report a successful program.
Experiments run per quarter: 200+, growing.
Experiment pass rate: 94%.
Mean time between experiment-discovered issues: stable.
Coverage breadth: described as "broad" but not quantified.
Service coverage percentage: 89%. Most services have had at least one experiment run against them.
Failure mode coverage: tracked qualitatively, not quantitatively.
None of these report on whether the program is testing the highest-risk components. The "service coverage percentage" is the closest to a coverage metric, but a service can have had one minor experiment run against it and count as "covered" by that metric while the most consequential failure modes for that service remain untested. The AI platform's selection algorithm optimizes for observed-failure-rate or for predicted-experiment-value, both of which tend to concentrate experiments on already-understood failure modes. The untested risk surface grows in parallel.
These metrics were designed when chaos engineering was a human-curated practice. The reliability engineer running the program kept the risk register in mind and chose experiments deliberately to cover the highest-risk areas. The coverage question was held in human judgment. With AI-driven chaos platforms, the selection happens automatically and the coverage question becomes a measurement question that nobody is yet measuring.
There is no metric that tells you what percentage of the actual risk surface has been chaos-tested versus only the portion the AI selected, which means chaos engineering generates impressive results on tested systems while untested systems accumulate fragility.
Formula
Chaos Coverage Gap (CCG) measures the share of high-risk system components that have never been chaos-tested. In plain language: rank your system components by risk independently of the chaos platform's selection algorithm. Identify the top two risk quartiles: the components most likely to fail in ways that matter. CCG is the percentage of those high-risk components that have not had any chaos experiment run against them in the past 12 months.
The formula:
CCG = 1 − (High_risk_components_tested / High_risk_components_total)
Where:
- High_risk_components_total = the count of components in the top two quartiles of an independent composite risk score
- High_risk_components_tested = the subset that received at least one chaos experiment in the trailing 12 monthsUnits are a ratio from 0 to 1. Zero means every high-risk component has been tested. One means none have been tested. Lower is better. A worked example. The e-commerce company ranks 88 services in their inventory by composite risk score (combining business criticality, blast radius, change frequency, and incident history). The top two quartiles include 44 services. Of those 44, 26 have had at least one chaos experiment run in the trailing 12 months. 18 have not.
CCG = 1 − (26 / 44) = 1 − 0.591 = 0.409
Per the proposed starting threshold (below 0.30 means 70%+ high-risk tested, above 0.60 means concentration on already-understood systems) this CCG of 0.41 sits in the middle band. The action trigger at 0.50 has not been crossed yet, but the trend warrants attention. The team should mandate that at least 30% of upcoming experiments target the 18 untested high-risk services, even if the AI platform's selection algorithm ranks them lower. Three edge cases need explicit treatment. First, the risk ranking must be independent of the chaos AI's targeting model. If you use the AI's own risk assessment to rank components, you will reproduce the AI's blind spots and CCG will look artificially low. Use incident data, architecture review findings, change-frequency analysis, and explicit human judgment to construct the risk ranking. Second, "tested" means an experiment was actually executed, not that it was considered or scheduled. A component that was on the experiment backlog for six months and never run is not tested. Third, you need at least 50 components in the risk ranking for the quartile-based selection to be meaningful. Below that, the top two quartiles are too small for stable CCG calculation. Cross-references. Upstream of CCG is Dependency Discovery Coverage Rate (DDCR). If the dependency map is incomplete, the risk ranking is also incomplete, and CCG underestimates the untested surface. Downstream is
Operational Self-Awareness Index (OSAI, GMI-IT-006): chaos coverage is one of the dimensions OSAI rolls up to measure how well the system reasons about its own blind spots. The misinterpretation to watch for: a team reports CCG = 0.20 (low. Good) and concludes the chaos program is comprehensive. They miss that the risk ranking used to compute CCG was derived primarily from the chaos AI's own assessment, which is the same model that prioritizes experiments. The ranking and the selection are correlated; CCG looks low because the components the AI did not test are also the ones it did not flag as high-risk. The remedy is to recompute CCG against a ranking derived independently (from incident history, from architecture review findings, from external resilience assessments) and accept that the independent ranking may produce a much higher CCG.
Proposed formula, pending validation pilots.