OBSERVABILITY ADEQUACY SCORE (OAS)

GMI-IT-063IT OperationsInfrastructure-Dependent

Definition

Observability Adequacy Score (OAS) measures the share of a service's top N failure modes that have observability specifically designed to detect them.

The Measurement Gap

An online marketplace's platform engineering team rolled out a uniform observability standard across their service catalog. Every service has metrics, logs, and traces. Every service has a default alerting suite. The observability platform reports 100% coverage on the standard. The CTO references observability coverage as a maturity indicator in board meetings.

Then a Tier-1 service (the search ranking pipeline) has an incident that takes 4 hours to diagnose. The team's metrics dashboard shows everything green throughout the incident window. The logs show normal log volumes. The traces show normal latency distributions. The incident is real (search results were significantly degraded) but the standard observability suite did not detect it. The team eventually finds the cause by manually inspecting the model's prediction distributions, which were skewed in a way that none of the standard observability surfaces would have caught. The skew was the failure mode that the search ranking pipeline most needed observability for. It was the failure mode the standard observability did not include.

Post-incident, the team enumerates the top failure modes for the search ranking pipeline based on incident history and FMEA: model drift, prediction distribution skew, feature pipeline lag, index staleness, query parser errors, ranking score calibration drift, recommendation diversity collapse, click-through rate anomaly, conversion rate decay. Of those nine failure modes, the standard observability suite specifically detects three (query parser errors, feature pipeline lag at the infrastructure layer, click-through rate anomaly at the aggregate layer). The other six modes either require custom instrumentation (which exists for some) or are not specifically detectable from standard signals (which is the gap that produced the 4-hour incident).

The legacy observability metrics report a healthy state.

Observability coverage: 100% of services on the standard.

Standard metrics deployed: 100%.

Standard alerts configured: 100%.

Trace coverage: 100%.

Log coverage: 100%.

Compliance with observability policy: 100%.

Time to deploy observability for new service: under 1 day.

None of these report on whether the observability deployed is the observability the service needs. Coverage measures presence of monitoring; it does not measure adequacy for the failure modes that matter. A service can have 100% standard coverage and have 50% adequacy if its dominant failure modes are not in the standard.

These metrics were designed when observability was scarce and the question was whether services were being monitored at all. With observability commodity-level for most services, the question has shifted, not "are we monitoring?" but "are we monitoring the right things?" The coverage metric does not answer the new question.

There is no metric that tells you whether instrumented services have observability adequate to detect their actual failure modes, not just presence of metrics/logs/traces, which means coverage dashboards report "monitored" while actual detection capability is unknown.

Formula

Observability Adequacy Score (OAS) measures the share of a service's top N failure modes that have observability specifically designed to detect them. In plain language: identify the top failure modes for a service (the ones most likely to occur or most impactful when they do). For each, check whether the observability includes a specific signal or alert designed to detect that failure mode. OAS is the percentage of top failure modes that have such specific detection.

The formula:
OAS = Failure_modes_with_adequate_observability / Top_N_failure_modes × 100

Where:

  • Failure_modes_with_adequate_observability = the count of failure modes that have a specific detection metric, alert, or trace
  • Top_N_failure_modes = the N most frequent or impactful failure modes per service (N = 10 recommended)

Units are a percentage from 0 to 100. 100% means every top failure mode has specific detection capability. A worked example. The online marketplace runs OAS on the search ranking pipeline after the incident:Top 10 failure modes for the search ranking pipeline:1. Model drift: no specific detection (only inferred from CTR/conversion lag)2. Prediction distribution skew: no specific detection (this was the incident)3. Feature pipeline lag: detected at infrastructure layer (PASS)4. Index staleness: detected via index timestamp metric (PASS)5. Query parser errors: detected via error rate (PASS)6. Ranking score calibration drift: no specific detection7. Recommendation diversity collapse: no specific detection8. CTR anomaly: detected via aggregate alert (PASS)9. Conversion rate decay: detected only at daily aggregation (too slow for SLO)10. Search latency spike: detected via P95 latency alert (PASS)5 of 10 have adequate specific detection.
OAS = 5 / 10 = 50%
Per the catalog guidance (above 80% for Tier-1, above 60% for Tier-2, below 50% Tier-1 is critical gaps) this OAS of 50% on a Tier-1 service is in the critical band. The team needs to trigger a failure mode analysis and observability remediation sprint, focusing on the five undetected failure modes (model drift, distribution skew, ranking calibration, diversity collapse, conversion rate latency). Three edge cases need explicit treatment. First, general health metrics (CPU, memory, disk) do not count as "specifically designed to detect" particular failure modes: be strict. They support diagnosis but do not constitute detection capability for specific application-layer failures. Second, services with no incident history must use FMEA instead: they are the highest-risk OAS computations because the failure modes are inferred rather than observed, but the metric is still meaningful and arguably more important. Third, assess Tier-1 services annually, Tier-2 bi-annually, lower tiers as resources permit. Cross-references. Upstream of OAS is

Telemetry Sampling Blind Spot Rate (TSBSR): even if a service has specific detection for a failure mode, sampling at that detection layer can render the detection ineffective. OAS and TSBSR together cover the two main observability adequacy questions. Downstream is

Operational Self-Awareness Index (OSAI, GMI-IT-006): observability adequacy is one dimension of how well a system reasons about its own monitoring capability. The misinterpretation to watch for: a team reports OAS = 85% (above the Tier-1 threshold) and concludes the service is well-monitored. They miss that the failure mode list was constructed from the trailing 12 months of incidents, which excludes failure modes the service hasn't yet experienced. New failure modes from architecture changes, integration additions, or load pattern shifts are not in the list. Refresh the failure mode list whenever the service architecture materially changes.

Proposed formula, pending validation pilots.

At a Glance

Category
AI-Era IT Operations
← MULTI-CLOUD PORTABILITY EROSION INDEX (MCPEI)Operator Skill Atrophy Index (OSAI) →

Assess Your Organization

The AI-GPS Assessment evaluates your AI governance maturity and recommends KPIs like this one for your implementation roadmap.

Take the Free AI-GPS Assessment