Defect Severity Calibration Score (DSCS)

GMI-RE-012Renewable EnergyIntegration-RequiredPattern O-1: Confidence Calibration

Definition

Agreement between AI-assigned defect severity and what the defect actually did, established by tracking classified defects forward. A minor-rated defect that required emergency repair within six months was under-classified, and under-classification is the direction that costs money.

The Measurement Gap

AI-powered inspection has transformed renewable energy asset management. Drones survey blade surfaces for cracks, delamination, and erosion. Thermal cameras identify hotspots in solar modules. Automated image analysis classifies each detected defect into severity categories that drive repair prioritization and budget allocation. The problem is that the severity classification is only as valuable as its accuracy, and almost no one measures that accuracy systematically over time.

The measurement gap is the absence of outcome-based validation for AI severity classifications. When the AI labels a blade crack as "minor," the O&M manager defers repair. When it labels a hotspot as "critical," a repair crew is dispatched immediately. But nobody tracks whether the "minor" cracks actually stayed minor or progressed to "critical" within a year, and nobody tracks whether the "critical" hotspots actually warranted immediate intervention or could have waited two inspection cycles.

Without DSCS, the AI classification system operates without feedback. Over-classification of minor defects (calling everything "critical") wastes repair budget by dispatching crews to address cosmetic issues. Under-classification of genuine threats (calling truly progressing defects "minor") allows damage to advance until it requires emergency intervention or causes catastrophic failure. Both errors are expensive, but they are asymmetric in consequence: under-classification is a safety and asset destruction risk, while over-classification is an economic inefficiency.

The decision DSCS supports is whether the O&M manager can trust the AI's severity classification to set repair priority, or whether the system is over-classifying cosmetic issues (wasting budget) and under-classifying genuine threats (allowing undetected progression).

Formula

Sample defects that the AI classified at each severity level (minor, moderate, critical). Track them forward in time to see what actually happened. A "minor" defect that required emergency repair within 6 months was under-classified. A "critical" defect that showed no progression for 18 months was over-classified. Score the calibration at each severity level separately.

Formula (Notation):

DSCS(s) = Correctly_Progressed(s) / Total_Classified(s)

Variable Definitions:

Variable Definition
s severity level (minor, moderate, critical)
Correctly_Progressed(s) defects classified at severity s whose actual progression matched the expected trajectory for that severity within the tracking window
Total_Classified(s) all defects classified at severity s during the measurement period

Edge Cases and Measurement Boundaries:

Defects repaired before progression can be observed should be counted as "censored" since their calibration accuracy is unknown. Defects detected on assets that were subsequently decommissioned should be excluded. Minimum tracking windows should be established per severity level: 12 months for minor, 6 months for moderate, and 3 months for critical, reflecting the expected progression timeline for each level.

Proposed formula, pending validation pilots.

At a Glance

Unit
Percentage
Trend Direction
Higher; more accurate
Reporting Frequency
Annually per severity level
Category
AI Inspection and Defect Management Quality
Inspection Scope Coverage Ratio (ISCR) →

Assess Your Organization

The AI-GPS Assessment evaluates your AI governance maturity and recommends KPIs like this one for your implementation roadmap.

Take the Free AI-GPS Assessment