AI Verification Thoroughness Ratio (AVTR)

GMI-NUC-028Nuclear OperationsInfrastructure-Dependent

Definition

Ratio of deficiencies detected by AI-based QA verification (sensor data, measurements, photos) to deficiencies detected by human QA inspectors on the same work, measured through parallel verification exercises.

The Measurement Gap

AI-based QA verification systems (using sensor data, dimensional measurements, and photographic comparison) are being evaluated as supplements or replacements for human QA inspectors at hold points. But no metric measures how the AI's detection capability compares to a human inspector on the same work. QA inspection completion rate tracks whether inspections were performed. Deficiency discovery rate tracks how many deficiencies are found. Neither measures the relative thoroughness of AI vs. human inspection for the same work scope. The invisible failure mode has two directions: the AI may systematically miss deficiency types that human inspectors detect (workmanship judgment, material substitution, subtle fit-up errors), giving false confidence in work quality; or the AI may detect deficiencies that humans miss (precise dimensional measurements, pattern recognition across thousands of inspection images), representing untapped value. Without parallel comparison, the plant cannot make informed decisions about where AI verification adds value and where it creates gaps.

Decision Supported

Can AI-based hold point verification replace or supplement human QA inspection, or does AI verification systematically miss deficiency types that human inspectors detect?

Formula

Plain language: On a sample of maintenance activities, perform both AI-based QA verification and independent human inspector verification of the same hold points. Compare what each method detected. The ratio tells you what fraction of human-detectable deficiencies the AI also detects.

Notation:
AVTR = Deficiencies_AI_detected / Deficiencies_human_detected

(where both are measured on the same work)

Variables:

  • Deficiencies_AI_detected = deficiencies identified by the AI verification system during hold point inspection (count)
  • Deficiencies_human_detected = deficiencies identified by a qualified human QA inspector performing the same hold point inspection independently (count)
  • Both counts are from the same set of work activities: the human inspector verifies the same hold points the AI assessed, without seeing the AI results
  • A deficiency detected by both methods counts in both numerators

Sign convention: AVTR = 1.0 means AI detects everything the human detects. Below 1.0 means the AI misses things the human catches (concerning). Above 1.0 means the AI catches things the human misses (valuable: the AI supplements human inspection). Values significantly below 1.0 indicate the AI cannot replace human verification.

Aggregation period: Quarterly (requires accumulation of parallel-verification events).

Edge Cases

  • AI detects deficiencies the human missed: this is AVTR > 1.0 and is a positive finding, but capture the specific deficiency types each method excels at. The goal is complementary detection, not replacement.
  • Neither method detects a deficiency that surfaces later in service: both methods failed. Track as an "escape" and investigate why. This is the most dangerous failure mode because it's invisible to AVTR.
  • Human inspector quality varies: use qualified, experienced inspectors for the benchmark. The comparison should be against the standard the AI is intended to match, not against the weakest inspector
  • Minimum sample: 50 parallel-verification hold points per quarter (resource-intensive. This is a structured program, not a continuous measurement)

Proposed formula, pending validation pilots.

At a Glance

Unit
Ratio
Trend Direction
Toward 1.0
Reporting Frequency
Quarterly
Category
Execution Monitoring Effectiveness
← Operating Experience Relevance Accuracy (OERA)Execution Alert Signal-to-Noise Ratio (EASNR) →

Assess Your Organization

The AI-GPS Assessment evaluates your AI governance maturity and recommends KPIs like this one for your implementation roadmap.

Take the Free AI-GPS Assessment