Definition
Accuracy of AI reachability analysis on vulnerabilities labeled unreachable, measuring how safe it is to deprioritize security findings on the model's word.
The Measurement Gap
Measurement Gap and Decision
The Gap: AI-powered reachability analysis labels hundreds of vulnerabilities as unreachable -- no code path from your entry points reaches the vulnerable function. Teams deprioritize those findings and focus on the reachable subset. But the accuracy of that analysis varies enormously by vulnerability type, dependency depth, and invocation pattern. Some reachability assessments are highly reliable (direct function calls). Others are unreliable (transitive chains through reflection, dynamic dispatch, or runtime dependency injection).
Traditional vulnerability management metrics track time-to-remediation and backlog size but do not distinguish between genuinely unreachable and incorrectly classified. The AI confidence score becomes a security decision without calibration.
The Decision:
RSAI measures the correlation between AI-assigned reachability scores and actual reachability determined by manual analysis or production exploit evidence. When RSAI falls below threshold, the intervention is restricting reachability-based deprioritization to categories where the tool is accurate, and requiring manual confirmation for high-severity vulnerabilities.
Formula
Formula:
RSAI = 1 - (False_negatives / Total_low_reachability_sampled)
Where:
- False_negatives = vulnerabilities scored as low/no reachability by AI that manual analysis confirms ARE reachable
- Total_low_reachability_sampled = total vulnerabilities with AI low/no reachability scores that were manually verified
RSAI deliberately measures only the false negative rate for unreachable assessments. The consequence of errors is asymmetric: "reachable" when actually unreachable wastes effort; "unreachable" when actually reachable leaves exploitable vulnerabilities unpatched.
Worked Example:
SCA tool flags 847 vulnerabilities. AI labels 620 as unreachable. Security team samples 25 unreachable assessments for manual verification, stratified by severity and dependency depth:
- 18 confirmed unreachable (AI correct)
- 4 actually reachable through transitive dependency chains the AI missed
- 3 actually reachable through reflection-based invocation patterns
RSAI = 1 - (7/25) = 0.72
Proposed formula, pending validation pilots.