Definition
Runbook Validity Score (RVS) measures the share of AI-generated runbook steps confirmed as correct and complete by domain expert review.
The Measurement Gap
It is 3:17 AM. The primary on-call engineer at a payment processing company gets paged. The pager message reads: "Authorization service P95 latency above 800ms threshold for 5 minutes." Engineer opens the incident link. The page links to an AI-generated runbook titled "Authorization Service Latency Spike Recovery."
The runbook has six steps. Step 1: Check the latest deployment to the authorization service. Step 2: If a deployment occurred within 30 minutes, roll back. Step 3: If no recent deployment, check the connection pool to the card network. Step 4: If pool exhausted, restart the authorization pods to reset connections. Step 5: If pool not exhausted, check the downstream fraud scoring service. Step 6: Escalate to the secondary on-call if not resolved in 20 minutes.
The engineer is half-awake and trusts the runbook. Steps 1-3 take three minutes. No deployment, pool not exhausted. Step 4 is therefore not the right path, but the engineer reads Step 4 as "if you're at this step, restart" and restarts the authorization pods. The restart drops in-flight transactions. Some retries succeed; some fail. Customers see declined transactions. Step 5 was the correct branch (fraud scoring was the bottleneck) but the runbook's flow control was confusing enough at 3 AM that the engineer ran the wrong step. The incident extends from a 12-minute latency spike to a 47-minute outage with thousands of declined transactions.
The post-incident review surfaces that the runbook had been auto-generated three weeks earlier by an AI runbook tool, in response to a similar incident pattern. The tool had correctly identified the symptoms and the likely causes. It had assembled steps from a library of known remediations. What it had not done well was sequence the steps with clear branching, validate the safety of each step against current state, or test the runbook against a recent incident before publishing it. The runbook was plausible. It was not correct enough for 3 AM.
The legacy metrics on the team's dashboard report a healthy runbook program.
Runbook coverage: 87% of recurring incident types.
Runbook generation rate: 12 new runbooks per month.
Mean runbook age: under 60 days (fresh).
Runbook execution rate: 73% of incidents follow a runbook.
MTTR for runbook-followed incidents: 12 minutes.
Engineer satisfaction with runbooks: tracked qualitatively, mostly positive.
None of these report on whether the runbook steps are correct. Coverage tells you a runbook exists. Generation rate tells you new ones are being produced. Mean age tells you they are not stale. Execution rate tells you engineers follow them. MTTR tells you incidents close. None of these tell you whether the steps in the runbook would actually solve the problem the runbook claims to solve, or whether the steps would make the problem worse.
These metrics were designed when runbooks were written by humans who had operated the relevant system, reviewed by peers, and tested in practice. The validity check was implicit in the production process. With AI-generated runbooks, the production process is automated and the validity check has not been reproduced as an explicit measurement.
There is no metric that tells you whether AI-generated runbook steps actually match the failure mode they claim to address, which means on-call engineers follow plausible but incorrect procedures during incidents.
Formula
Runbook Validity Score (RVS) measures the share of AI-generated runbook steps confirmed as correct and complete by domain expert review. In plain language: take a sample of AI-generated runbooks, have a domain expert evaluate each step, count the steps confirmed correct and complete, divide by total steps reviewed. RVS is that percentage. Track wrong steps and incomplete steps separately, because wrong steps are operationally worse than incomplete ones, a wrong step can make the incident worse.
The formula:
RVS = Steps_correct / Steps_total_sampled × 100
Where:
- Steps_correct = the count of runbook steps confirmed correct and complete by expert review
-
Steps_total_sampled = the total count of steps reviewed across the sampled runbooksTrack separately:
-
Wrong_step_count = steps that would actively make the situation worse if executed
-
Incomplete_step_count = steps that are correct but missing context or branchingUnits are a percentage. The overall RVS is the primary metric; the wrong-step count and incomplete-step count are secondary metrics that drive different responses. A worked example. The payment processing company runs RVS on a quarterly sample of 20 AI-generated runbooks, totaling 134 steps across the sample. Expert review of the 134 steps:
-
108 confirmed correct and complete
- 19 incomplete (correct as far as they go, but missing branching, missing validation, missing context)
- 7 wrong (would worsen the incident if executed: wrong commands, wrong order, wrong assumptions about state)
RVS = 108 / 134 = 80.6%Wrong_step_rate = 7 / 134 = 5.2%Incomplete_step_rate = 19 / 134 = 14.2%
Per the catalog guidance (above 90% with Wrong below 2% means trustworthy; below 80% or Wrong above 5% means mandatory human review before on-call rotation) this team is below both thresholds. RVS is below 90% and Wrong rate is above 5%. Any wrong step that would worsen production should trigger immediate removal of the runbook. The team needs to pause auto-publishing new runbooks, route them through human review before they reach the on-call rotation, and audit the existing runbook library for similar wrong steps. Three edge cases need explicit treatment. First, evaluate each step against the specific failure mode the runbook targets, not against general operational best practice. A step that would be wrong for one failure mode might be correct for another; the runbook's claimed scope matters. Second, runbooks for failure modes that have never been seen in production cannot be fully validated: mark them "unvalidatable" and track that count separately. Third, the minimum sample is 20 runbooks per quarter with at least 5 steps per runbook. Below that, the percentage is too noisy. Cross-references. Upstream of RVS is
Adversarial Confidence Audit (ACA, GMI-IT-004): runbook validity is one operational manifestation of AI confidence calibration; if ACA is poor, RVS will be poor as well. Downstream is
Operational Self-Awareness Index (OSAI, GMI-IT-006): runbook quality is one dimension of how well the system reasons about its own remediation capability. The misinterpretation to watch for: a team reports RVS = 92% (above the trustworthy threshold) and concludes the runbook program is healthy. They miss that the sample was drawn entirely from well-understood failure modes for mature services: exactly where AI-generated runbooks perform well. The runbooks for novel failure modes on newer services are not in the sample because there are fewer of them, and those are exactly the runbooks most likely to have wrong steps. Sample across both well-understood and novel failure modes, weighted by the operational risk each represents.
Proposed formula, pending validation pilots.