ROC Attention Allocation Effectiveness (RAAE)

GMI-MAR-013Maritime IndustriesInfrastructure-Dependent

Definition

Correlation between AI-assigned vessel attention priority and actual event significance across the ROC portfolio.

The Measurement Gap

What This Metric Measures

RAAE measures how well the AI anomaly scoring system directs ROC operator attention to the vessels that actually develop significant events. It computes the correlation between AI-assigned attention priority and actual event severity over rolling watch periods, revealing whether the system helps operators focus on the right vessels or creates attention fragmentation that causes monitoring gaps.

The Decision It Supports

Is the AI's attention-direction system helping the ROC operator focus on the right vessels, or is it creating attention fragmentation that causes monitoring gaps? When one operator monitors 15-25 vessels simultaneously, every minute spent on a false priority is a minute not spent on a vessel developing a real problem. The AI scoring system becomes the operator's triage nurse: RAAE measures whether that triage is competent.

Why Legacy Metrics Cannot See This

Traditional vessel monitoring metrics track alarm counts, response times, and incident rates per vessel. None of them measure the attention allocation decision itself. An ROC can have excellent per-vessel alarm response times while systematically directing attention to low-priority vessels because the AI scoring system ranks stable vessels above deteriorating ones. The operator responds quickly to whatever the system highlights: the metric never asks whether the system highlighted the right vessel. RAAE exposes the gap between attention speed and attention accuracy.

Formula

Primary Formula

RAAE = Spearman_rho(AI_priority_rank, Actual_significance_rank)

Variable Definitions

  • AI_priority_rank = vessel ranking by AI anomaly/priority score during the watch period (ordinal)
  • Actual_significance_rank = vessel ranking by actual event significance measured by subsequent incident severity, intervention level, or time to next significant event (ordinal)
  • Spearman_rho = Spearman rank correlation coefficient between the two rankings

Units: Correlation coefficient, -1 to +1. A value of +1 means perfect alignment where the AI always directed attention to the vessel that needed it most. A value of 0 means the AI's attention-direction is no better than random. Negative values mean the AI systematically directs attention away from the vessels that need it.

Edge Cases

Watch periods with no significant events on any vessel should be excluded because no discrimination is possible. Use midrank convention for tied rankings. Minimum sample of 50 watch periods containing at least one significant event is required for statistical reliability. Event significance scoring should be performed post-event by the watch leader to establish ground truth.

Proposed formula, pending validation pilots.

At a Glance

Unit
Correlation (-1 to 1)
Trend Direction
Higher; Increasing toward +1
Reporting Frequency
Rolling quarterly
Category
Autonomous Operations and Human-AI Authority
← Multi-System Optimization Coherence Index (MSOCI)Shore-Ship Override Value Index (SSOV) →

Assess Your Organization

The AI-GPS Assessment evaluates your AI governance maturity and recommends KPIs like this one for your implementation roadmap.

Take the Free AI-GPS Assessment