Definition
Explanation Utility Score (EUS) measures the share of AI-generated explanations where a reviewer can correctly identify whether the model's reasoning is valid for the operational context.
The Measurement Gap
A health insurance company runs an AI model to recommend prior authorization decisions for specialty procedures. Regulatory and internal policy requires that every recommendation be reviewable by a clinical reviewer with an explanation of why the model produced its output. The model is integrated with an explanation engine that generates SHAP-based feature importance values for each recommendation. The clinical reviewers see the recommendation, the feature importance chart, and the patient context. They sign off, or they override.
The audit team reviews a quarterly sample of reviewer decisions. They find that reviewers sign off on the model's recommendation 91% of the time. The override rate is 9%, which the team considers a healthy oversight rate. Then they look at the overrides more carefully. They construct a sub-sample of 60 cases where the model's recommendation, in retrospect, was clinically inappropriate: cases that would have benefited from override. The reviewers overrode 14 of those 60. They signed off on the other 46. The explanation engine produced the same kind of feature importance chart for the inappropriate recommendations as it did for the appropriate ones. The reviewers could not tell from the explanation whether the reasoning was valid.
The audit report's conclusion is uncomfortable: the explainability requirement is being met on paper, but the explanations are not actually enabling oversight. The reviewers are not bad clinicians; they are responding rationally to an explanation format that does not give them the information needed to judge validity. The compliance program reports explainability as functional; in practice it is producing artifacts that do not affect decisions.
The legacy compliance metrics on the team's dashboard report a healthy explainability program.
Models with explanations: 100%.
Explanation generation rate: 100% of decisions.
Reviewer adoption: 100%. Every reviewer uses the explanation interface.
Mean explanation generation time: under 200ms.
Explanation completeness: every recommendation has a complete SHAP chart.
Audit log retention: 7 years.
None of these report on whether the explanations actually enable the reviewer to assess model validity. Generation, completeness, and adoption are process metrics. The operational question (does the human reviewing the explanation come to a more accurate conclusion about model correctness than they would without it) is not being measured. The explanation is treated as a compliance artifact rather than as a decision support tool.
These metrics were designed for compliance regimes that mandate explainability without specifying that the explanations must be useful. The legal text often says "the model must produce an explanation that allows a reviewer to understand the basis for the decision." The implementation reads "produce an explanation" and stops. The "understand the basis" part is harder to operationalize and is rarely measured.
There is no metric that tells you whether AI model explanations are meaningful to the humans reviewing them or just technically accurate descriptions of computation, which means explainability requirements are met on paper while reviewers can't use the explanations.
Formula
Explanation Utility Score (EUS) measures the share of AI-generated explanations where a reviewer can correctly identify whether the model's reasoning is valid for the operational context.
In plain language: construct a set of test cases where the model's reasoning is known (by an independent domain panel) to be either valid or invalid. Present each case to a reviewer with the model's recommendation and explanation. Ask the reviewer to assess whether the reasoning is valid. Count the cases the reviewer got right. EUS is that percentage.
The formula:
EUS = Correct_validity_assessments / Total_assessments
Where:
- Correct_validity_assessments = cases where the reviewer correctly identified valid reasoning as valid OR invalid reasoning as invalid
- Total_assessments = total cases presented to the reviewer
Units are a percentage from 0 to 100. 50% is random chance: the reviewer is doing no better than coin-flipping. 100% means the explanations fully enable oversight. The operational range of interest is 50% to 100%; below 50% indicates the explanations are actively misleading.
A worked example. The health insurance company runs EUS on their prior authorization model with 12 clinical reviewers. The audit team constructs 60 test cases: 30 where the model's recommendation is clinically appropriate (independent panel of clinicians agrees) and 30 where the recommendation is clinically inappropriate (the panel disagrees, with documented reasoning). The cases are presented to reviewers double-blind: reviewers do not know which cases are valid and which are not.
Aggregated results across 12 reviewers, 60 cases each (720 total assessments):
- 420 correct assessments (reviewer identified valid as valid or invalid as invalid)
- 300 incorrect assessments
EUS = 420 / 720 = 0.583 = 58.3%
Per the catalog guidance (above 75% means real oversight, below 60% means redesign needed, 50% is noise) this EUS of 58.3% sits in the "redesign" band. The explanation format is not enabling reviewers to differentiate valid from invalid reasoning much better than chance. The team needs to redesign the explanation format, likely with clinical-context-specific templates rather than generic SHAP charts.
Three edge cases need explicit treatment. First, the test cases require ground truth for "valid reasoning": determined by domain experts independent of the explanation system. If the same team that built the explanation system also determines ground truth, EUS is artificially inflated. Second, the study must be double-blind: reviewers must not know which cases are valid and which are not, or they will look for confirmation rather than evaluate honestly. Third, the minimum is 30 cases per model per measurement cycle, balanced between valid and invalid reasoning, distributed across multiple reviewers.
Cross-references. Upstream of EUS is Proxy Discrimination Detection Gap (PDDG). If the model is making decisions based on proxy variables, the explanations will reflect those proxies but the reviewer needs to recognize the proxy nature, which is harder than recognizing direct discriminatory reasoning. Downstream is essentially all governance decisions. If EUS is low, the governance program's oversight layer is not functioning, regardless of how complete the other components look.
Proposed formula, pending validation pilots.