AI Test Environmental Sensitivity Rate (ATESR)

GMI-SWD-007Software DeliveryPilot-Ready

Definition

Rate at which AI-generated tests fail or flake due to timing, network, and load sensitivities, measuring the environmental fragility experienced test engineers would have designed out.

The Measurement Gap

Measurement Gap and Decision

The Gap: AI test generation tools produce tests involving timing, network conditions, and system load dependencies without the environmental awareness experienced test engineers bring: mocking external dependencies, using deterministic time controls, isolating network interactions, setting generous timeouts for CI environments. The result is disproportionate flakiness in AI-generated tests.

Existing metrics see the symptom but miss the attribution. CI pipeline success rate is declining but reports do not break down by test authorship. Mean time to green is increasing but nobody separates AI-test contribution from human-test contribution. Flaky test dashboards list which tests are flaky but rarely tag them by generation method. Teams lose trust in the entire test suite because of the AI-generated subset, and eventually real failures hide behind the noise.

The Decision:

ATESR measures the ratio of flaky test rates between AI-generated and human-authored tests. When ATESR exceeds threshold, the intervention is mandatory environmental sensitivity review for AI-generated tests before merge -- mocking external dependencies, deterministic time controls, state isolation.

Formula

Formula:

ATESR = Flaky_rate_AI / Flaky_rate_human

Where:

  • Flaky_rate_AI = Tests_flaky_AI / Tests_total_AI
  • Flaky_rate_human = Tests_flaky_human / Tests_total_human
  • A test is "flaky" if it produces different results (pass/fail) across multiple CI runs on the same code commit

Worked Example:

End of Q3 test suite:

  • Total AI-generated tests: 284
  • AI tests flagged as flaky (failed at least once on retry in last 30 days): 19
  • Flaky_rate_AI = 19/284 = 6.69%

  • Total human-authored tests: 1,247

  • Human tests flagged as flaky: 31
  • Flaky_rate_human = 31/1247 = 2.49%

ATESR = 0.0669 / 0.0249 = 2.69

Proposed formula, pending validation pilots.

At a Glance

Unit
Percentage
Trend Direction
Lower
Reporting Frequency
Monthly
Category
Review and Verification Integrity
← Release Risk Score Calibration Index (RRSCI)Effective Review Depth Index (ERDI) →

Assess Your Organization

The AI-GPS Assessment evaluates your AI governance maturity and recommends KPIs like this one for your implementation roadmap.

Take the Free AI-GPS Assessment