Quality Prediction Confidence Index (QPCI)

GMI-PH-026Pharmaceutical OperationsPilot-Ready

Definition

Calibration of predictive quality confidence scores, computed per confidence band as actual pass rate against the band's stated confidence. Exposes models that appear accurate in aggregate by being overconfident on hard batches and underconfident on easy ones.

The Measurement Gap

The predictive quality model says this batch has a 92% probability of passing. The QA team nods and moves on. But a model that predicts 90% pass rate when the actual pass rate is 90% appears accurate overall, yet if it achieves this by predicting 95% on easy batches and 85% on hard batches when the actual rates are 99% and 65% respectively, the per-batch decisions based on those confidence scores are dangerously wrong.

Every manufacturing site using a predictive quality model treats the confidence score as a probability. Process engineers use it to decide whether to intervene. QA uses it to prioritize review. But almost nobody checks whether the model's confidence scores actually correspond to reality.

Decision enabled:

When per-band QPCI drops below 0.85 (overconfidence exceeding 15%), the model needs retraining for that confidence range. When aggregate QPCI exceeds 0.05, overall calibration has degraded and confidence thresholds for PIVI intervention triggers need adjustment.

Formula

Per-band: QPCI(b) = actual_pass_rate(b) / midpoint_confidence(b)
Aggregate: QPCI_aggregate = Σ(n_b × |actual_pass_rate(b) − midpoint_confidence(b)|) / Σ(n_b)

Per-band QPCI of 1.0 means perfect calibration. Below 1.0 means overconfident (dangerous). Above 1.0 means underconfident (wasteful but safer). Aggregate QPCI of 0 means perfect calibration; higher is worse.

Worked Example:

A solid oral dosage site produces 312 batches in Q2–Q3:

80–85% band: 28 batches, 19 passed (67.9% actual vs 82.5% midpoint). Per-band QPCI = 0.823. Calibration error = 14.6%.
85–90% band: 54 batches, 44 passed (81.5% vs 87.5%). Per-band QPCI = 0.931.
90–95% band: 118 batches, 109 passed (92.4% vs 92.5%). Per-band QPCI = 0.999.
95–100% band: 112 batches, 110 passed (98.2% vs 97.5%). Per-band QPCI = 1.007.

Aggregate QPCI = 0.026: healthy overall, but the 80–85% band at 0.823 is below the 0.85 retraining trigger. Engineers who saw "82% confident" and chose not to intervene were making decisions on false information.

Sign Convention:

Per-band: 1.0 is perfect, below 1.0 is overconfident. Aggregate: 0 is perfect, higher is worse. Aggregate below 0.05 is healthy.

Proposed formula, pending validation pilots.

At a Glance

Unit
Percentage
Trend Direction
Higher
Reporting Frequency
Monthly; stratified by batch difficulty
Category
Predictive Manufacturing Quality
← Predictive Intervention Value Index (PIVI)CAPA Contextual Effectiveness Rate (CCER) →

Assess Your Organization

The AI-GPS Assessment evaluates your AI governance maturity and recommends KPIs like this one for your implementation roadmap.

Take the Free AI-GPS Assessment