SLO-EXPERIENCE ALIGNMENT SCORE (SEAS)

GMI-IT-046IT OperationsIntegration-Required

Definition

SLO-Experience Alignment Score (SEAS) measures the correlation between SLO attainment and user-reported (or user-observed) experience quality, computed weekly.

In plain language: each week, compute the percentage of SLO target met for a service.

The Measurement Gap

A media streaming company's SRE team rolled out an AI-recommended SLO framework eight months ago. The platform analyzed historical performance data, recommended SLO targets for each service, and the team adopted them with minor tweaks. The SRE dashboard now shows all Tier-1 services consistently attaining their SLOs at 99.9% or better. Error budgets are healthy. The platform engineering team is using SLO attainment as their primary reliability narrative in monthly business reviews.

Then a customer success leader sends a chart to the head of SRE. Customer complaints about playback quality have been rising for three months. Net Promoter Score is down four points. Cancellation reasons trending up include "buffering" and "stream quality." The customer success team thinks something is wrong with the streaming infrastructure. The SRE team's response: "Our SLOs are green. The platform is healthy."

Both statements are correct. The streaming SLOs measure backend service health: API success rates, error rates, latency at the service boundary. These are all within target. The customer experience metrics measure something different: playback start time, rebuffering rate, video quality drift. These are degrading. The backend services are healthy. The user experience is not. The SLO framework recommends targets based on what is measurable at the service layer, not based on what produces a good experience for the human watching the video.

The SLO framework was working as designed. It was just measuring the wrong things.

The legacy reliability metrics on the team's dashboard report a healthy state.

SLO attainment percentage: 99.92% trailing 30 days.

Error budget remaining: 78%.

Burn rate: well below alerting threshold.

Service availability: 99.95%.

API success rate: 99.97%.

P95 latency: under target.

Change failure rate: low.

MTTR: under SLA.

None of these report on whether the user is having a good experience. They report on whether the services are doing what the SLOs say they should. The mapping from service-level SLO to user-level experience was set by the AI recommender based on what it could observe in the service telemetry, not based on what produces an outcome the user cares about. The user experience layer (playback start time, rebuffering, quality drift) was either not in the telemetry the recommender saw, or was de-weighted because it was harder to measure than backend metrics.

These metrics were designed for a world where SLOs were defined by humans who explicitly mapped service behavior to user experience. The SRE who set the SLO knew that playback start time was the experience metric that mattered and chose backend metrics that would predict it. The mapping was held in human judgment. With AI-recommended SLOs, the mapping is automated based on whatever the AI can observe. If the AI sees only backend telemetry, the SLOs it recommends will measure backend behavior. The user experience may or may not align.

There is no metric that tells you whether AI-recommended SLOs correlate with actual user-perceived experience, which means teams hit their SLOs while users experience degraded service.

Formula

SLO-Experience Alignment Score (SEAS) measures the correlation between SLO attainment and user-reported (or user-observed) experience quality, computed weekly. In plain language: each week, compute the percentage of SLO target met for a service. Each week, compute a user experience score for the same service (NPS-derived, error-report-derived, synthetic user success rate, CSAT, whatever signal you have). After 12 weeks, calculate the correlation between the two series. A correlation near 1 means SLO attainment is tracking user experience tightly. A correlation near 0 means SLO attainment is not predicting user experience at all: the SLO is measuring the wrong thing.

The formula:
SEAS = Corr(SLO_attainment_weekly, UX_score_weekly)

Where:

  • SLO_attainment_weekly = the percentage of SLO target met during the week for the service
  • UX_score_weekly = a user experience indicator for the same service during the same week (CSAT, inverted error report rate, synthetic user success rate, or whatever proxy is available)

Units are a correlation coefficient from −1.0 to 1.0. A value above 0.7 indicates strong alignment: the SLO is a reliable proxy for user experience. A value below 0.3 indicates that SLO attainment is measuring something other than user experience. Negative correlations are pathological and mean attainment increases as experience degrades; investigate immediately. A worked example. The streaming company computes SEAS for their playback service over a 12-week measurement window.| Week | SLO attainment | UX score (composite) ||------|----------------|----------------------|| 1 | 99.95% | 8.4 || 2 | 99.91% | 8.2 || 3 | 99.93% | 7.9 || 4 | 99.96% | 7.6 || 5 | 99.94% | 7.4 || 6 | 99.92% | 7.2 || 7 | 99.95% | 7.1 || 8 | 99.91% | 6.9 || 9 | 99.93% | 6.8 || 10 | 99.96% | 6.7 || 11 | 99.94% | 6.5 || 12 | 99.92% | 6.4 |SLO attainment hovers between 99.91% and 99.96%: effectively flat at the top of the band. UX score drops from 8.4 to 6.4 over the 12 weeks, a clear degradation. Correlation between the two series:
SEAS ≈ −0.18
Slightly negative correlation. SLO attainment has no predictive relationship to user experience. Per the catalog guidance (below 0.4 for any Tier-1 service triggers SLO review) this service needs an SLO redesign with explicit user journey mapping. The current SLO measures backend behavior that does not track user experience. Three edge cases need explicit treatment. First, if SLO attainment has no variance (always 99.9% or better), correlation is mathematically undefined or trivially uninformative. The target may be too loose, a service that attains 99.9% every week regardless of experience changes is being graded on a curve that doesn't matter. Second, the user experience measurement must be consistent across services. If you use CSAT for one service and synthetic user success rate for another, the SEAS values are not comparable across services. Third, the minimum sample is 12 weekly data points. Shorter windows produce correlations that are statistically meaningless. Cross-references. Upstream of SEAS is Error Budget Forecast Fidelity (EBFF). If the SLO does not align with experience, neither will the forecasts built on it. Downstream is True Remediation Rate (TRR, GMI-IT-007). When reliability work is driven by SLOs that do not align with experience, the remediation that gets done is not the remediation that improves the user's experience. The misinterpretation to watch for: a team reports SEAS = 0.8 (strong alignment) and concludes their SLO is well-designed. They miss that the high correlation comes from one or two weeks of incident-driven SLO breach that produced corresponding UX drops; the rest of the time, both metrics are flat. The correlation is real but it is dominated by the tail. SEAS should be read alongside the variance of each underlying series; if either is nearly flat, the correlation tells you less than the value suggests.

Proposed formula, pending validation pilots.

At a Glance

Category
AI-Era IT Operations
← SEMANTIC DATA QUALITY GAP (SDQG)STALE DEPENDENCY INFLATION RATE (SDIR) →

Assess Your Organization

The AI-GPS Assessment evaluates your AI governance maturity and recommends KPIs like this one for your implementation roadmap.

Take the Free AI-GPS Assessment