Definition
Automation Effectiveness Variance (AEV) is a proposed measure of how uniformly an MSP's automation performs across its client portfolio: specifically, the coefficient of variation of per-client automated remediation success rates.
The Measurement Gap
The MSP's automation platform was running at 84% success rate across the portfolio. The service delivery manager reported it in the monthly business review and nobody questioned it. By any reasonable standard, 84% sounded healthy. But when the team started looking at the same data by client rather than across the whole portfolio, the picture shifted considerably.
Eight of thirty-one clients had automation success rates above 90%. Nine were in the 75–85% range. Eleven were in the 60–75% range. Three were below 50%. Those three clients (a mid-sized financial services firm with a heavily customized ERP environment, a law firm running on-premises legacy infrastructure, and a manufacturing company still operating Windows Server 2012 across part of their environment) were receiving automation services that failed more often than they succeeded. Every failed automation attempt generated a ticket and required a technician to investigate. For those three clients, automation was producing work rather than eliminating it.
The portfolio average of 84% hid this entirely. Portfolio averages are appropriate when all clients are roughly equivalent. When automation performs similarly regardless of which client's systems you're touching. MSP portfolios are not homogeneous. Some clients have modern, standardized, cloud-first environments where automation scripts run cleanly every time. Others have legacy systems, custom configurations, and third-party integrations that break standard automation regularly.
The economic consequences were running silently beneath the surface metrics. The clients getting 90%+ automation success were receiving a premium service at standard pricing: faster resolution, lower technician touch rates, SLA attainment that was nearly automatic. The clients below 50% were receiving a manually-intensive service at the same standard pricing. Every automation failure for those three clients generated a follow-up human investigation, pushing their effective cost of service delivery well above what the contract supported. The MSP was subsidizing their service with labor that wasn't priced into the relationship.
Legacy metrics didn't surface this. Portfolio-level automation rate measured the average, which masked the variance. SLA attainment per client showed whether contractual targets were met but didn't break down how much of that time was automation failure investigation versus productive resolution work. MTTR per client showed resolution speed in aggregate, not the composition of that time. Technician utilization was high, but the utilization was concentrated on the clients with poor automation performance, not spread evenly across the portfolio.
Portfolio averages are designed to communicate to leadership, not to diagnose operations. When you need to understand whether your automation is performing consistently, the mean tells you the center of the distribution. What you actually need is the spread. An 84% portfolio rate with a coefficient of variation of 0.05 means reliable, consistent automation across clients. An 84% rate with a coefficient of variation of 0.25 means you have clients in serious trouble that the headline number is hiding.
There is no metric that captures the variance in automation effectiveness across clients: whether the portfolio is delivering equitable automated service or whether some clients are systematically receiving worse outcomes. That gap is what Automation Effectiveness Variance is designed to close.
Formula
Automation Effectiveness Variance (AEV) is a proposed measure of how uniformly an MSP's automation performs across its client portfolio: specifically, the coefficient of variation of per-client automated remediation success rates. The insight driving AEV is that a portfolio-level automation rate is meaningful only if the distribution around it is tight. A mean success rate of 80% with low variance means automation is delivering consistent value across clients. The same mean with high variance means some clients are receiving automation that works reliably and others are receiving automation that generates more support work than it prevents. AEV measures the spread, not the center.
The formula, in three layers:
Plain language: For each client in the portfolio, calculate their automated remediation success rate. The number of automation attempts that resolved the issue without human intervention, divided by total automation attempts for that client. Then calculate the standard deviation and mean of all the per-client success rates. Divide the standard deviation by the mean. Lower AEV means automation is delivering more uniform service across the portfolio.
Notation:AEV = σ(SR_c) / μ(SR_c)
Where:
- SR_c = automated remediation success rate for client c = (successful auto-remediations for c) / (total auto-remediation attempts for c)- σ = standard deviation of SR_c values across all eligible clients- μ = mean of SR_c values across all eligible clientsThe coefficient of variation is dimensionless: the standard deviation expressed as a fraction of the mean. This makes AEV comparable across periods and across MSPs of different sizes.
Worked Example
The MSP has 20 clients with automation enabled and eligible for the calculation. Per-client success rates (SR_c) range from 44% to 96% across those 20 clients. The mean is 81%. The standard deviation is 15.3 percentage points. AEV = 15.3 / 81 = 0.19. Now segment: the five clients with SR_c below 70% are the bottom quartile. What do they have in common? Two are running on-premises infrastructure without cloud standardization; one has a custom ERP that breaks standard restart scripts; two have legacy network configurations that cause automation timeouts. Those common characteristics drive the automation investment roadmap.
Sign convention:
Lower AEV is better. An AEV of 0 would mean every client has an identical automation success rate: the portfolio is perfectly uniform. In practice, some variance is always present. AEV below 0.10 means automation is performing consistently across clients. AEV between 0.10 and 0.20 indicates moderate variance worth monitoring. AEV above 0.30 indicates significant equity gaps requiring intervention.
Edge Cases
Clients with fewer than 20 automated remediation attempts in the measurement period should be excluded from AEV. Small samples produce extreme SR_c values, a client with 5 attempts and 1 failure shows 80%, but that's a meaningless estimate. Use 90-day rolling windows with minimum attempt thresholds per client. New clients (those with fewer than 90 days of managed service) should be excluded. Automation hasn't been configured and tuned for their environment yet, and lower initial success rates are expected during onboarding. Include them after the tuning period. Clients who have explicitly opted out of specific automation categories should have AEV calculated only on the automation types they've opted into. Comparing a full-automation client against a partial-automation client introduces definitional noise that isn't about automation quality.
Cross-references:
Upstream, AEV connects to Remediation Masking Rate (RMR, GMI-IT-005) from the existing IT KPI set. In clients where automation is masking root causes (closing alerts without addressing underlying problems) actual automation attempts may log as successes while the underlying issue recurs. If RMR is high in specific clients, their SR_c may be inflated, making those clients appear to have better automation performance than they actually do. Track RMR per client alongside AEV; a client with high SR_c and high RMR deserves scrutiny. Lateral connection to Cross-Client Pattern Detection Rate (CPDR, GMI-IT-031). When the MSP detects a cross-client pattern and propagates a fix to all affected clients, those clients should show AEV improvement: their previously low SR_c values move toward the portfolio mean. Better pattern detection produces better automation equity, and CPDR and AEV reinforce each other. Downstream, AEV connects to client churn. Clients in the bottom quartile (those receiving automation that fails more than it helps) have a materially worse service experience than the portfolio average. The causal path runs through service experience quality, not survey satisfaction scores: clients who generate more support contacts per incident than similar clients on the same tier are experiencing inequitable service even if they can't articulate it precisely in a survey. AEV is classified as Pilot-Ready. Per-client automation attempt and outcome data exists natively in every major RMM platform. The calculation requires only aggregating that data by client and computing a coefficient of variation. No new data collection, no external system integration. The practical work is the aggregation query and the operational definition of "successful auto-remediation."
Proposed formula, pending validation pilots.