Definition
Degree to which AI-generated work package scope matches the scope an independent experienced planner would specify for the same work order, measuring both excess scope (waste) and missing scope (risk).
The Measurement Gap
AI-generated maintenance work packages are evaluated for approval but not systematically measured for scope accuracy. Planner productivity metrics track how many work packages are produced. Work order completion metrics track execution. Rework rates capture some scope failures after the fact. But no metric measures the accuracy of the AI's scope determination at the point of planning, before the work package reaches the field. The invisible failure mode has two sides: excess scope (the AI includes unnecessary tasks, parts, or prerequisites, wasting resources and potentially creating additional hazards in radiological or confined-space work) and missing scope (the AI omits required tasks, clearance points, or prerequisite verifications, which are discovered during execution and cause delays, rework, or safety events). Legacy planning metrics like backlog age and planning rate measure throughput, not accuracy. The measurement gap is precise: no metric compares what the AI planned against what an experienced planner would have planned for the same work order.
Decision Supported
Can the plant accept AI-generated work packages with review-level planner oversight, or does every AI package require full independent re-planning because the AI systematically misses or over-includes scope items?
Formula
Plain language: Sample AI-generated work packages. Have an experienced planner (minimum 5 years nuclear maintenance planning, qualified on the system) independently scope the same work order without seeing the AI output. Compare the two scopes element by element: tasks, parts, tools, prerequisites, and special requirements. Score scope alignment on three dimensions: tasks that both agree on (concordant), tasks the AI included that the planner wouldn't (excess), and tasks the planner would include that the AI missed (deficit).
Notation:
WPSAI = Concordant / (Concordant + Excess + Deficit)
Sub-metrics:
WPSAI_excess = Excess / (Concordant + Excess)
WPSAI_deficit = Deficit / (Concordant + Deficit)
Variables:
- Concordant = scope elements where AI and expert planner agree (count of task/part/tool/prerequisite items)
- Excess = scope elements included by AI but not by expert planner (count)
- Deficit = scope elements included by expert planner but not by AI (count)
- Measurement is per work order, aggregated across the sample
Sign convention: WPSAI = 1.0 means perfect scope alignment. Lower is worse. WPSAI_excess and WPSAI_deficit should be tracked separately because they have different consequences: excess creates waste and unnecessary dose, deficit creates rework and potential safety issues. Deficit is the more dangerous failure mode.
Aggregation period: Monthly rolling, stratified by system type (safety-related, risk-significant, BOP) and work type (corrective, preventive, modification).
Edge Cases
- Legitimate planning judgment differences: two experienced planners may scope the same work order differently. Use a three-planner panel for disputed items; if all three agree the AI was wrong, count as excess or deficit; if the panel splits, count as "judgment-dependent" and exclude from WPSAI denominator
- Work orders with no AI equivalent (novel first-time activities): exclude from WPSAI. The AI has no basis for scoping work it hasn't seen before, and that's expected
- Minimum sample: 30 work packages per month per work type (corrective, preventive, modification)
Proposed formula, pending validation pilots.