Regulatory Model Regime Shift Index (RMRSI)

GMI-EU-047Energy & UtilitiesIntegration-Required

Definition

Measures the degree to which the NLP rate-case risk model's predicted disallowance rates deviate from actual commission disallowance rates, segmented by spending category, revealing regime-shift blind spots.

The Measurement Gap

Traditional rate case risk management relies on regulatory counsel reviewing recent commission orders and advising leadership on current commission priorities. This advisory process is relationship-dependent, qualitative, and not systematically updated when commission composition changes. When AI NLP tools are deployed for rate case analysis, they provide consistency and scale, but they also introduce a new failure mode: the model's training data reflects historical commission behavior, not the current commission's priorities.

The specific tracking failure is the absence of model calibration feedback. An AI rate-case tool that flags O&M spending as low-risk because the previous commission rarely disallowed it will continue flagging O&M as low-risk even if the new commission, with two new appointees who campaigned on utility cost accountability, has begun scrutinizing O&M more aggressively. The model doesn't know what it doesn't know about the new commission. RMRSI measures the gap between what the model predicted and what the commission actually did: category by category.

The operational decision this KPI enables is targeted retraining: RMRSI identifies which spending categories have the largest prediction-to-reality gap, enabling selective emphasis on recent decisions in those categories during the next model update rather than a full retraining that may dilute the signal.

What goes wrong without it: the utility files a rate case with capital additions classified as low-disallowance risk based on the AI model's output. The commission, operating under a new energy transition agenda, disallows 18% of capital spend on fossil-fuel plant upgrades that the previous commission routinely approved. The disallowance costs $40M. The regulatory affairs team never questioned the AI model's risk classification because no metric was tracking whether the model's predictions were still accurate against current commission behavior.

Formula

RMRSI(c) = Actual_disallowance_rate(c) − Predicted_disallowance_rate(c)

Variables:

  • c: Spending category (capital additions, O&M, fuel/purchased power, regulatory assets, storm cost recovery, grid modernization, vegetation management, etc.). Source: Rate case filing structure
  • Actual_disallowance_rate(c): Percentage of spending in category c that was disallowed in the most recent rate case decision(s). Source: Commission orders (public record), rate case settlement documents
  • Predicted_disallowance_rate(c): NLP model's predicted disallowance probability for category c at the time of filing. Source: Rate case analytics platform output log

Calculation Frequency: Per rate case (every 1–3 years for most utilities); supplement with peer-utility case monitoring for faster signal
Output: Percentage-point difference per category, −100 to +100 percentage points. Positive RMRSI means the commission was stricter than the model predicted (undiscovered risk). Negative RMRSI means the model was conservative (acceptable). RMRSI within ±5 percentage points per category represents adequate model calibration. Any category where RMRSI exceeds +15 points is a significant blind spot requiring model retraining with emphasis on recent decisions. Track separately by commissioner tenure: decisions made by the post-transition commission versus the pre-transition commission reveal the magnitude of the regime shift.

Proposed formula, pending validation pilots.

At a Glance

Category
Workforce Optimization and Regulatory Compliance

Decision Supported

Has the regulatory commission's posture shifted in ways that the NLP model hasn't captured, and which spending categories are now at higher disallowance risk than the model predicts?

← Optimizer Safety and Development Consumption Index (OSDCI)

Assess Your Organization

The AI-GPS Assessment evaluates your AI governance maturity and recommends KPIs like this one for your implementation roadmap.

Take the Free AI-GPS Assessment