DR TEST VALIDITY INDEX (DTVI)

GMI-IT-054IT OperationsIntegration-Required

Definition

DR Test Validity Index (DTVI) measures the share of automated DR test scenario elements that still match the current production architecture.

The Measurement Gap

A telecommunications company has a mature DR program. Quarterly DR tests run automatically against a documented test scenario library. The test orchestration platform reports test pass rates above 95% consistently. The CIO presents the DR posture as a strength to the board annually. The IT auditors note the program as a positive control.

Then a production incident exposes a gap. A regional cloud outage takes down the primary site for a customer-facing portal. The team initiates the documented failover. The failover begins normally and then hangs at a step that depends on a service that did not exist in the DR test scenario. The service had been added to the production stack 11 months earlier as part of a new feature launch; the DR test scenario had not been updated. The team works around the gap manually, restoring service in 4.5 hours instead of the documented RTO of 90 minutes. Customers experience extended outage.

The post-incident review surfaces the broader problem. The DR test scenario library was last comprehensively updated 18 months ago. Since then, the production stack has added 47 new services, retired 12, changed 23 service-to-service dependencies, and migrated 8 services to different infrastructure providers. The DR tests still pass: they pass against the old scenarios, which the orchestration platform runs faithfully. The pass rate above 95% means the platform can recover the system it was configured to test, not the system that exists in production.

The legacy DR metrics report a healthy program.

DR test pass rate: 96% trailing 12 months.

RPO compliance: 100% across tested services.

RTO compliance: 100% across tested scenarios.

DR test execution rate: every quarter as scheduled.

Number of DR runbooks current: tracked, mostly current at last review.

Audit findings: clean at last DR audit 8 months ago.

None of these report on whether the tested scenarios still match production. The pass rate measures whether the platform succeeds against the configured tests. The RPO and RTO compliance measure whether the tested services meet their targets. Neither tells you whether the tested services are the production services that would actually need to recover.

These metrics were designed when DR test scenarios were maintained alongside production changes through formal change management. Every production change went through a review that included DR impact assessment, and the DR scenarios were updated as part of the change. With production changes happening at cloud-native pace (CI/CD deployments, infrastructure-as-code modifications, microservice births and deaths) the DR scenario update cadence has fallen behind. The change management discipline that kept the scenarios current does not scale to current change rates, and AI-driven DR test generation has not caught up.

There is no metric that tells you whether automated DR tests are still testing recovery scenarios that match the current architecture, which means DR test pass rates stay green while actual recovery posture degrades as the architecture evolves.

Formula

DR Test Validity Index (DTVI) measures the share of automated DR test scenario elements that still match the current production architecture.

In plain language: take a DR test scenario, decompose it into the elements it includes (services, dependencies, infrastructure components, integration points), and compare each element against the current production architecture. Count the elements that still match. DTVI is the percentage that match. A high value means the test scenario still reflects production. A low value means the test is running against a stale picture of the system.

The formula:

DTVI = Matched_test_elements / Total_test_elements × 100

Where:

  • Matched_test_elements = the count of elements in the DR test scenario that accurately reflect the current production architecture
  • Total_test_elements = the count of all elements in the DR test scenario

Units are a percentage from 0 to 100. 100% means the test fully reflects current architecture.

A worked example. The telecommunications company runs DTVI on their customer portal DR scenario after the incident. The scenario includes 87 elements across services, dependencies, infrastructure components, and integration points.

Assessment of each element against current production architecture:

  • 54 elements current (the element accurately reflects the current production state)
  • 19 elements stale (the element existed at scenario creation but is no longer accurate)
  • 14 elements missing from the scenario but present in production (new services added since last scenario update)

For DTVI, the missing elements are also a problem but are counted differently in the formula. Strict interpretation: DTVI = 54 / (54 + 19) = 74% on present-but-stale elements. Comprehensive interpretation including missing elements: DTVI = 54 / (54 + 19 + 14) = 61.4%. Report both and label them.

DTVI (strict) = 54 / 73 = 74.0% DTVI (comprehensive) = 54 / 87 = 61.4%

Per the catalog guidance (above 85% reasonably current, below 70% significant untested recovery capability) the comprehensive DTVI of 61.4% is well into the danger zone. The action trigger at 75% has been crossed. The team should halt reporting DR test passes for this scenario, refresh the scenario against current production architecture, and re-test before reporting recovery posture.

Three edge cases need explicit treatment. First, the current-state source of truth must be accurate itself. If your CMDB or service catalog is out of date, the DTVI calculation will misidentify "matches" or "mismatches" based on stale source data. Validate the source of truth before computing DTVI. Second, new services not in the DR test count as mismatches; they represent untested recovery capability and should be highlighted prominently. Third, assess DTVI after every significant architecture change, not only at fixed intervals, a single major change can drop DTVI by 30% in a day.

Cross-references. Upstream of DTVI is Dependency Discovery Coverage Rate (DDCR). If the dependency map is incomplete, DTVI computed against the incomplete map will overstate validity. Downstream is Failover Necessity Rate (FNR). If DTVI is low, the failure prediction models that drive FNR are also trained on a production reality that has shifted.

Proposed formula, pending validation pilots.

At a Glance

Category
AI-Era IT Operations
← DRIFT CLASSIFICATION ACCURACY (DCA)EFFORT ESTIMATION ANCHORING BIAS INDEX (EEABI) →

Assess Your Organization

The AI-GPS Assessment evaluates your AI governance maturity and recommends KPIs like this one for your implementation roadmap.

Take the Free AI-GPS Assessment