MOC Risk Classification Accuracy (MRCA)

GMI-OG-018Oil and GasIntegration-Required

Definition

MOC Risk Classification Accuracy, or MRCA, is the percentage of AI-classified MOC requests where post-review by a qualified PSM coordinator confirms the AI''s risk classification and PHA trigger determination were correct under OSHA PSM 1910.119 requirements. The KPI exists to answer one decision: is the NLP triage system correctly distinguishing between changes that require full PHA evaluation and those that do not, particularly on the under-classification direction that produces both regulatory exposure and safety risk?

The Measurement Gap

Wendell is the PSM coordinator at an integrated chemicals and refining site running about eighteen process units under OSHA''s Process Safety Management standard. The site processes a couple of hundred management-of-change requests per month: everything from minor instrumentation swaps to genuine chemistry changes that require full process hazard analysis. Six months ago his company deployed an NLP-based MOC triage system that ingests the change request''s description, the affected equipment, the proposed modification, and the requester''s risk self-assessment, then classifies the change as routine, minor, or major, and flags whether a PHA is required. The intent was straightforward: clear the routine traffic out of the PSM coordinator''s queue so the coordinator could spend more time on the changes that actually needed full review.

The system shipped well. The classification confusion matrix from the validation set looked good. Most changes that the AI tagged routine were in fact routine: equipment-in-kind replacements, minor procedural updates, calibration adjustments. The PSM coordinator''s queue dropped by about forty percent in the first quarter. Wendell''s team caught up on the backlog and started getting ahead of the recurring PHA reviews that had been falling behind.

The trouble surfaced during a regulatory audit. An OSHA inspector pulled a sample of MOC records and walked through them with Wendell''s team. Three of the records the inspector flagged were ones the AI had classified as routine. The first was a chemical substitution where the requester had described the new chemical as functionally equivalent to the old one, but the new chemical had a meaningfully different flash point. Under OSHA PSM 1910.119, a flash point change inside the process envelope is a chemistry change that triggers PHA. The AI had keyed on the requester''s "functionally equivalent" language and missed the property delta. The second was an operating limit modification described as a "minor envelope adjustment" that raised the high-side setpoint on a reactor temperature. The temperature change was small in absolute terms but it crossed an alarm-versus-trip-point relationship that the unit''s PHA had specifically documented as a hazard boundary. The AI had not seen that relationship in the request and had classified the change as routine. The third was a safety-system bypass procedure update that the AI had treated as a procedural document edit. The procedure governed a layer of protection that the unit''s LOPA assigned a specific SIL credit to. Changes to safety system bypass procedures require PHA evaluation under the site''s PSM program. The AI had classified the change as a procedural update.

None of these were spectacular AI failures. The system was doing what NLP triage systems are designed to do: extracting features from text and matching them against patterns in the training data. The training data was real MOC history from the site and a peer site, and the patterns in the training data were dominated by the much larger volume of genuinely routine changes. The structural problem was that PHA-triggering changes are the small tail of the distribution and the model''s confidence on classifications in that tail was lower than the model''s overall validation accuracy suggested. The forty-percent queue reduction was real. So was the audit finding.

Legacy MOC metrics do not catch this. PHA completion rate measures whether PHAs that were initiated were completed on time. MOC cycle time measures throughput. Audit findings are a lagging indicator that arrives quarters after the misclassifications. None of these metrics report whether the AI is correctly triaging the small subset of changes whose misclassification matters most: the under-classification of PHA-triggering changes as routine.

What is missing is a metric that watches the AI''s classification accuracy specifically on the change types where misclassification produces regulatory and safety risk.

Formula

MOC Risk Classification Accuracy, or MRCA, is the percentage of AI-classified MOC requests where post-review by a qualified PSM coordinator confirms the AI''s risk classification and PHA trigger determination were correct under OSHA PSM 1910.119 requirements. The KPI exists to answer one decision: is the NLP triage system correctly distinguishing between changes that require full PHA evaluation and those that do not, particularly on the under-classification direction that produces both regulatory exposure and safety risk?

The structural form is a sampling-and-review process. A defined sample of AI-classified MOCs from the measurement period is pulled. A qualified PSM coordinator (someone authorized under the site''s PSM program to make classification calls) reviews each MOC in the sample independently of the AI''s classification. The PSM coordinator''s call is the ground truth. The AI''s classification is compared against the coordinator''s call. MRCA is the percentage of sample MOCs where the AI and the coordinator agreed on both the risk class and the PHA trigger determination.

The metric splits errors in two directions, and the asymmetry matters. Under-classification is when the AI classifies a change as routine or minor and the coordinator''s review determines the change should have triggered PHA. Under-classification is the dangerous error because it leaves PHA-required hazards unevaluated. Over-classification is when the AI flags a PHA trigger that the coordinator''s review determines was not required. Over-classification wastes PSM time but does not produce safety or regulatory exposure. The thresholds on the two error rates should not be the same.

The sampling method is the methodological core. Proposed approach: stratified random sampling across change types so that small-volume but high-consequence change types are represented in the sample at a rate that supports statistical meaning. A simple random sample over all MOCs in a quarter, where chemistry changes and operating-envelope changes are only a few percent of total volume, will produce a sample with too few of those types to evaluate MRCA on them specifically. The stratification ensures that change types where misclassification is most consequential are reviewed at adequate sample sizes.

Minimum sample size: thirty MOCs per quarter at the aggregate level, with the stratification ensuring at least ten MOCs in each of the high-consequence change types. Chemistry changes, operating-envelope modifications, safety-system modifications. Smaller samples produce numbers, but the numbers are too noisy to drive action.

Proposed formula, pending validation pilots.

At a Glance

Category
Process Safety & HSE Intelligence
← Time-Shifted Detection Value (TSDV)Operator Manual Competency Retention Index (OMCRI) →

Assess Your Organization

The AI-GPS Assessment evaluates your AI governance maturity and recommends KPIs like this one for your implementation roadmap.

Take the Free AI-GPS Assessment