Storm Readiness Prediction Accuracy (SRPA)

GMI-TEL-043TelecommunicationsIntegration-RequiredPattern O-1: Confidence Calibration

Definition

Percentage of sites declared "storm-ready" by the AI readiness assessment that actually survived the event without power-related service loss, measured per event and per readiness confidence tier. SRPA directly tests whether the readiness model's confidence calibration holds under real conditions: perfect calibration means 90% confidence equals 90% survival rate.

The Measurement Gap

AI-driven storm readiness assessments evaluate battery backup duration, generator reliability, and commercial power vulnerability at each site and produce a confidence-tiered readiness score. After the storm passes, operations teams report total sites affected but do not systematically compare pre-event readiness scores against actual survival outcomes. Without this comparison, the organization cannot determine whether the readiness model's high-confidence predictions are trustworthy for crew positioning and generator pre-deployment decisions, or whether the model is systematically overconfident about specific site types that fail despite being declared ready.

The Decision

Can operations trust the AI's pre-storm readiness assessment when deciding where to pre-position repair crews and portable generators, or is the assessment systematically overconfident about specific site types? The most valuable output is the false-ready rate by site characteristic: which site types does the model systematically overrate? Common finding: sites with aging generators that test well in normal conditions but fail under sustained load during prolonged outages.

Research Foundation

Pre-event infrastructure readiness models in utility and telecom sectors demonstrate systematic overconfidence for sites with standby generation that is infrequently exercised under load, with false-ready rates of 15-25% at the 90%+ confidence tier (IEEE PES General Meeting: Predictive Resilience Assessment for Critical Infrastructure, 2024). Hurricane season post-event analysis across Gulf Coast telecom networks found that 18% of sites rated "ready" by the AI assessment experienced power-related service loss, with aging generator start failure as the leading cause (CTIA Storm Preparedness Committee Report, 2023).

Formula

SRPA(b) = Sites_survived(b) / Sites_declared_ready(b) × 100

SRPA_overall = Σ [Sites_survived(b)] / Σ [Sites_declared_ready(b)] × 100

Variable Definitions (every variable resolved)

  • Sites_declared_ready(b) = sites the AI assessed as storm-ready with confidence in band b (e.g., 90-100%, 80-89%, 70-79%) before the event (count)
  • Sites_survived(b) = subset that maintained power and service throughout the event (count)
  • b = confidence band

Units: Percentage per confidence band. Perfect calibration means 90% confidence = 90% survival rate.

Edge Cases

Events less severe than forecast: sites that survived may have survived due to luck, not readiness; control by measuring only sites that experienced the forecasted conditions (wind speed, outage duration within predicted range). Sites that lost commercial power but battery/generator sustained service: count as "survived". The readiness assessment was correct. Sites that lost service for non-power reasons (physical damage, access road blocked): exclude from SRPA. These are outside the power readiness model's scope. Events with very few affected sites (< 20): may not produce statistically meaningful SRPA; accumulate across multiple events before drawing conclusions. Minimum sample: 20 sites per confidence band per event (or accumulated across events in a season).

These are proposed starting points: calibrate to your operational context during pilot: SRPA above 85% at the 90%+ confidence band means the readiness model is trustworthy for high-confidence assessments. Below 70% means the model is dangerously overconfident. If SRPA drops below 80% at any confidence band above 70%, pre-position portable generators at ALL sites in that band for subsequent events.

Proposed formula, pending validation pilots.

At a Glance

Unit
Percentage
Trend Direction
Toward 1.0
Reporting Frequency
Per storm season
Category
Power and Environmental Predictive Integrity
← RUL Prediction Accuracy by Equipment Class (RPAC)Prediction Economic Yield (PEY) →

Assess Your Organization

The AI-GPS Assessment evaluates your AI governance maturity and recommends KPIs like this one for your implementation roadmap.

Take the Free AI-GPS Assessment