Definition
Rate at which AI-generated tests fail or flake due to timing, network, and load sensitivities, measuring the environmental fragility experienced test engineers would have designed out.
The Measurement Gap
Measurement Gap and Decision
The Gap: AI test generation tools produce tests involving timing, network conditions, and system load dependencies without the environmental awareness experienced test engineers bring: mocking external dependencies, using deterministic time controls, isolating network interactions, setting generous timeouts for CI environments. The result is disproportionate flakiness in AI-generated tests.
Existing metrics see the symptom but miss the attribution. CI pipeline success rate is declining but reports do not break down by test authorship. Mean time to green is increasing but nobody separates AI-test contribution from human-test contribution. Flaky test dashboards list which tests are flaky but rarely tag them by generation method. Teams lose trust in the entire test suite because of the AI-generated subset, and eventually real failures hide behind the noise.
The Decision:
ATESR measures the ratio of flaky test rates between AI-generated and human-authored tests. When ATESR exceeds threshold, the intervention is mandatory environmental sensitivity review for AI-generated tests before merge -- mocking external dependencies, deterministic time controls, state isolation.
Formula
Formula:
ATESR = Flaky_rate_AI / Flaky_rate_human
Where:
- Flaky_rate_AI = Tests_flaky_AI / Tests_total_AI
- Flaky_rate_human = Tests_flaky_human / Tests_total_human
- A test is "flaky" if it produces different results (pass/fail) across multiple CI runs on the same code commit
Worked Example:
End of Q3 test suite:
- Total AI-generated tests: 284
- AI tests flagged as flaky (failed at least once on retry in last 30 days): 19
-
Flaky_rate_AI = 19/284 = 6.69%
-
Total human-authored tests: 1,247
- Human tests flagged as flaky: 31
- Flaky_rate_human = 31/1247 = 2.49%
ATESR = 0.0669 / 0.0249 = 2.69
Proposed formula, pending validation pilots.