Definition
Ratio of planner overrides that improved work package outcomes to total planner overrides of AI-generated work package recommendations, measuring whether the human review layer is adding value or introducing error.
The Measurement Gap
When experienced planners review and modify AI-generated work packages, the plant assumes this human review layer adds value. But no metric measures whether it actually does. Planning productivity metrics track throughput. Rework rates track execution failures. Neither measures whether the planner's modifications to the AI's original recommendation improved or degraded the work package outcome. The invisible failure mode has two sides: planners may be systematically degrading good AI packages (adding unnecessary conservatism, removing efficient task sequences, overriding correct parts selections based on outdated experience) or planners may be adding essential corrections the AI cannot make (catching context the AI missed, adding safety precautions the AI's training data didn't include, recognizing plant-specific conditions the model doesn't know). Without measuring override outcomes, the plant cannot distinguish between a value-adding review gate and a value-destroying bottleneck.
Decision Supported
Is the planner review of AI-generated work packages a value-adding quality gate, a neutral rubber stamp, or a value-destroying bottleneck where planner judgment is systematically worse than the AI's recommendation?
Formula
Plain language: Track every instance where a planner modifies an AI-generated work package element (adds a task, removes a task, changes a part, adjusts a labor estimate, modifies a prerequisite). After the work is executed, evaluate whether the override improved the outcome (work completed without rework, parts list was correct, labor estimate was more accurate) or degraded it (override introduced a scope gap, wrong part was substituted, labor estimate was less accurate).
Notation:
POVI = (Overrides_improved − Overrides_degraded) / Total_overrides
Variables:
- Overrides_improved = planner modifications to AI recommendations where the post-execution outcome was better than the AI's original recommendation would have produced (count)
- Overrides_degraded = planner modifications where the post-execution outcome was worse than the AI's original recommendation (count)
- Total_overrides = all planner modifications to AI-generated work package elements (count)
- Evaluation requires comparing actual execution outcome against what the AI's original recommendation would have produced. This requires preserving the AI's original output alongside the planner's modified version
Sign convention: POVI ranges from -1.0 (every override made things worse) to +1.0 (every override made things better). POVI = 0 means overrides are value-neutral (equal improvement and degradation). Positive is good. Negative means the human review layer is destroying value.
Aggregation period: Quarterly (requires post-execution outcome data, which takes weeks to months to collect).
Edge Cases
- Override outcome is indeterminate (e.g., planner removed a task, work completed successfully, but it's unknowable whether the removed task would have been needed): classify as "neutral" and exclude from both numerator terms
- Planner adds conservatism (extra parts "just in case," additional labor buffer): if the extra items weren't used, classify as "excess conservatism". Track separately. Mild excess conservatism may be appropriate in nuclear even if it doesn't improve the measurable outcome.
- Multiple overrides on one work package: count each override element separately
- Minimum sample: 50 overrides per quarter per planner category (mechanical, electrical, I&C)
Proposed formula, pending validation pilots.