Test Specification Alignment Score (TSAS)

GMI-SWD-006Software DeliveryIntegration-RequiredPattern O-23: Specification Fidelity

Definition

Degree to which AI-generated tests verify intended specification behavior rather than merely confirming what the implementation already does, separating genuine validation from tautology.

The Measurement Gap

Measurement Gap and Decision

The Gap: AI test generation produces tests by reading the implementation and generating assertions that match the code behavior. This is tautological validation -- testing that the code does what the code does. Coverage goes up. Safety does not. A specification-derived test validates a business requirement regardless of implementation. An implementation-derived test verifies that the code does what it currently does -- if the implementation has a bug, the test confirms the bug rather than catching it.

Code coverage reports treat all assertions equally -- 87% coverage from tautological tests is indistinguishable from 87% coverage from genuinely protective tests. Mutation testing might catch some tautological tests, but most organizations do not run it in CI. Test pass rates are always 100% because tautological tests always pass against the code they were derived from.

The Decision:

TSAS measures the percentage of AI-generated test assertions that validate specified behaviors versus assertions that merely exercise the current implementation. When TSAS falls below threshold, the intervention is requiring acceptance criteria as explicit input to AI test generation before it runs.

Formula

Formula:

TSAS = Assertions_spec_derived / Assertions_total

Where:

  • Assertions_spec_derived = count of assertions traceable to requirements, acceptance criteria, or documented behavioral specifications
  • Assertions_total = total count of assertions in the sampled AI-generated test suites

Worked Example:

Order fulfillment module with 6 documented acceptance criteria. AI generates 34 assertions.

Spec-derived examples:

  • "orderTotal should include shipping when items exceed weight threshold" traces to AC-3
  • "order status should be PENDING_PAYMENT after creation" traces to AC-1
  • "discount should not exceed 40% of subtotal" traces to AC-5
    Count: 14 assertions trace to documented acceptance criteria.

Implementation-derived examples:

  • "processOrder should call validateInventory exactly once" -- tests internal implementation detail
  • "order.metadata.createdAt should be a valid ISO timestamp" -- implementation detail not in any requirement
  • "calculateShipping should return 0 for digital items" -- AI inferred from implementation, not specified
    Count: 20 assertions verify implementation behavior without requirement traceability.

TSAS = 14 / 34 = 0.412

Proposed formula, pending validation pilots.

At a Glance

Unit
Score
Trend Direction
Higher
Reporting Frequency
Quarterly
Category
Review and Verification Integrity
← Review Attention Displacement Index (RADI)Policy Freshness Score (PFS) →

Assess Your Organization

The AI-GPS Assessment evaluates your AI governance maturity and recommends KPIs like this one for your implementation roadmap.

Take the Free AI-GPS Assessment