Safety Alert Credibility Index (SACI)

GMI-OG-001Oil and GasPilot-Ready

Definition

The Safety Alert Credibility Index (SACI) measures the fraction of CV-driven safety alerts that both identify a genuine hazard AND produce a documented, appropriate response within the response window defined for that alert type.

The Measurement Gap

It is 2:47 AM at a gas processing facility outside Midland. Sara, the night-shift control room operator, is watching her screen flash with another CV system alert: tank battery #3, restricted area, person without hard hat. She clicks acknowledge. Two hours earlier she clicked acknowledge on a different alert from the same camera, that one turned out to be a contractor walking through frame at shift change. She had walked over physically to confirm. Twenty minutes after that, the CV system flagged hot work in progress without a fire watch. The "hot work" was a steam plume the algorithm misread as welding sparks. By 3 AM Sara has acknowledged 47 alerts on her shift. Three of them were actually relevant.

When the real alert comes through at 4:13 AM (a contractor inside the maintenance shop without a flash hood during what should be an arc-rated task) Sara clicks acknowledge and goes back to her flow chart review. She does not call. The contractor takes a 2nd-degree burn the next hour. Post-incident review reveals the CV system flagged it correctly, six minutes before the burn occurred. The alert just was not credible anymore.

This is not a story about technology failure. The CV system worked the way it was sold to work: it generated an alert when its model detected something. What failed was the credibility relationship between the alert stream and the operator who had to act on it. And here is the structural problem: no metric in current use at most O&G facilities can detect that credibility decay before the incident.

TRIR will show the burn after it happens. It is a lagging measurement that catches the consequence, not the credibility decay that produced it. LTIR has the same blindness. Tier 1 and Tier 2 process safety events sit even further downstream: they require an actual loss of containment or injury before they register. Near-miss reporting captures what people choose to report, but Sara did not report the 4:13 AM acknowledge as a near-miss because she did not know it was the warning until after the burn.

What about CV-specific metrics? Most vendors report alert volume, which is useful for billing and not much else. Some report response time, but a fast acknowledge click without action is worse than no alert at all, because it logs as "responded" while doing nothing. Vendor-reported false positive rates are typically measured against curated internal test sets, not against the actual operational alert stream the operator sees at 2 AM.

The measurement gap is specific. When the volume and quality of AI safety alerts degrade past the point where the human in the loop will respond seriously, no current metric reports that degradation until something burns. The question SACI proposes to answer is the one Sara could not ask in real time: is the CV system's credibility degrading to where genuine hazards get dismissed?

Formula

The Safety Alert Credibility Index (SACI) measures the fraction of CV-driven safety alerts that both identify a genuine hazard AND produce a documented, appropriate response within the response window defined for that alert type.

Plain language first. Out of all the CV alerts that fired in your measurement period, how many were real AND got handled right? If 1,847 alerts came through, 312 of those turned out on review to be genuine hazards, and 184 of the 312 received a correct response within their response window, your SACI is 184 ÷ 1,847 = 0.10. The formula counts false positives in the denominator on purpose. False positives are what destroys credibility for the alerts that do matter, so they have to be in the score.

Notation:

SACI = Timely_response ÷ Total_alerts

Where:

  • Total_alerts = every CV/AI safety alert in the measurement period (count), pulled from the CV platform's raw alert log
  • Genuine_alerts = the subset of Total_alerts confirmed on review as genuine observations of the hazard the system was designed to detect (count), labeled by a human reviewer using a documented review protocol
  • Timely_response = the subset of Genuine_alerts that received an appropriate documented response action within the alert-type-specific response window (count), joined from the HSE management system response log

The formula is deliberately not Genuine_alerts ÷ Total_alerts. That ratio would measure detector precision in isolation: useful for tuning the CV model, not for measuring the operator-alert relationship. By dividing Timely_response by Total_alerts, the metric penalizes both false positives (which degrade trust) and missed responses (which degrade safety) in one number, and rewards only the joint condition of real-hazard-correctly-handled.

Worked example. A gas processing facility runs CV across three loading rack areas, two compressor station bays, and the flare deck. Over a 14-day window the system generates 1,847 alerts across 28 shifts. The HSE supervisor's monthly QA review classifies each alert as genuine or non-genuine by comparing the alert frame to video before and after, walking the area when necessary, and consulting the operator who handled the alert. After review, 312 alerts are confirmed genuine. Of those 312, the HSE management system shows 184 received a response action within the response window for that alert type: PPE alerts within 5 minutes, restricted-area alerts within 2 minutes, hot work alerts triggering an immediate halt. The remaining 128 genuine alerts received no documented action, late action, or an acknowledge-only click. SACI = 184 ÷ 1,847 = 0.10. The CV vendor's promotional material reports a 17% precision rate. The detector precision is 312 ÷ 1,847 = 0.17: the vendor's claim is honest, but operational credibility is far below it, because two-thirds of the alerts that mattered were not handled. That is the gap SACI exposes.

Edge cases. SACI requires a human alert review process. Without a structured review producing a binary genuine/non-genuine label per alert, SACI is undefined, and your real problem is upstream of measurement, because nobody is closing the QA loop on a safety system. Timeliness must be defined per alert type. The response window for a PPE alert is different from a restricted-area alert which is different from a hot work alert. Set the windows per class and document them in the procedure, otherwise reviewers will use different mental clocks and SACI will look like noise. Alert volume is a separate problem. More than 100 alerts per shift is a volume issue regardless of what SACI computes: at that rate the operator cannot give meaningful attention to any individual alert, and the metric will report the symptom while the disease is volume itself.

Proposed formula, pending validation pilots.

At a Glance

Category
Process Safety & HSE Intelligence
← PHA Safeguard Implementability Rate (PSIR)Contract Penalty Exposure Rate (CPER) →

Assess Your Organization

The AI-GPS Assessment evaluates your AI governance maturity and recommends KPIs like this one for your implementation roadmap.

Take the Free AI-GPS Assessment