Reachability Score Accuracy Index (RSAI)

GMI-SWD-010Software DeliveryIntegration-RequiredPattern O-1: Confidence Calibration

Definition

Accuracy of AI reachability analysis on vulnerabilities labeled unreachable, measuring how safe it is to deprioritize security findings on the model's word.

The Measurement Gap

Measurement Gap and Decision

The Gap: AI-powered reachability analysis labels hundreds of vulnerabilities as unreachable -- no code path from your entry points reaches the vulnerable function. Teams deprioritize those findings and focus on the reachable subset. But the accuracy of that analysis varies enormously by vulnerability type, dependency depth, and invocation pattern. Some reachability assessments are highly reliable (direct function calls). Others are unreliable (transitive chains through reflection, dynamic dispatch, or runtime dependency injection).

Traditional vulnerability management metrics track time-to-remediation and backlog size but do not distinguish between genuinely unreachable and incorrectly classified. The AI confidence score becomes a security decision without calibration.

The Decision:

RSAI measures the correlation between AI-assigned reachability scores and actual reachability determined by manual analysis or production exploit evidence. When RSAI falls below threshold, the intervention is restricting reachability-based deprioritization to categories where the tool is accurate, and requiring manual confirmation for high-severity vulnerabilities.

Formula

Formula:

RSAI = 1 - (False_negatives / Total_low_reachability_sampled)

Where:

  • False_negatives = vulnerabilities scored as low/no reachability by AI that manual analysis confirms ARE reachable
  • Total_low_reachability_sampled = total vulnerabilities with AI low/no reachability scores that were manually verified

RSAI deliberately measures only the false negative rate for unreachable assessments. The consequence of errors is asymmetric: "reachable" when actually unreachable wastes effort; "unreachable" when actually reachable leaves exploitable vulnerabilities unpatched.

Worked Example:

SCA tool flags 847 vulnerabilities. AI labels 620 as unreachable. Security team samples 25 unreachable assessments for manual verification, stratified by severity and dependency depth:

  • 18 confirmed unreachable (AI correct)
  • 4 actually reachable through transitive dependency chains the AI missed
  • 3 actually reachable through reflection-based invocation patterns

RSAI = 1 - (7/25) = 0.72

Proposed formula, pending validation pilots.

At a Glance

Unit
Percentage
Trend Direction
Higher
Reporting Frequency
Quarterly
Category
Dependency and Security Intelligence
← Refactoring Semantic Fidelity Score (RSFS)Upgrade Cascade Accuracy (UCA) →

Assess Your Organization

The AI-GPS Assessment evaluates your AI governance maturity and recommends KPIs like this one for your implementation roadmap.

Take the Free AI-GPS Assessment