Concept Drift Velocity (CDV)

GMI-MRO-005MROIntegration-Required

Definition

A composite drift score fusing input-distribution shift, prediction-interval calibration error, and feature-importance divergence into a single triggering signal for whether a deployed model requires retraining.

The Measurement Gap

Predictive models drift silently when the real-world data distribution diverges from training. F1, precision, recall, and accuracy metrics are snapshots: they cannot detect that the relationship between inputs and outcomes has changed since deployment. OEE and PM Compliance touch neither model quality nor data shift. Univariate drift monitoring (PSI alone) over-triggers: seasonal swings and load changes shift input distributions for benign reasons, producing false retraining alarms; one study cited shows over 70% of PSI-only triggers are spurious. The failure mode is dual: retraining when not needed (burning compute and engineering time on a healthy model) and not retraining when needed (running a stale model into a missed failure). Calendar-based retraining is the time-based PM equivalent for AI models: uninformed by actual condition.

Decision Supported:

Has the relationship between this model's inputs and the world shifted enough to warrant retraining, or is observed variation benign?

Formula

Plain language: Combine three independent drift signals. How much the input distribution has shifted (PSI), how badly the model's uncertainty estimates miss their target coverage, and how much the SHAP feature-importance distribution has changed. Weight and sum them. Trigger retraining only when the composite stays above threshold for 72 hours.

Notation:

CDV = λ_1 · PSI(inputs) + λ_2 · | PICP − (1 − α) | + λ_3 · JSD(SHAP_ref ‖ SHAP_now)

Variables:

Variable Definition Unit
PSI(inputs) Population Stability Index on input feature distributions vs. training Index
PICP Prediction Interval Coverage Probability: fraction of outcomes inside the model's stated interval Ratio
α Target miscoverage level (e.g., 0.10 for 90% interval) Ratio
JSD(SHAP_ref ‖ SHAP_now) Jensen-Shannon divergence between training-time and current SHAP feature-importance distributions Ratio (0–1)
λ_1, λ_2, λ_3 Component weights (suggested 0.4 / 0.3 / 0.3) Weight
CDV Composite drift score Index

Proposed formula, pending validation pilots.

At a Glance

Unit
Index (0 to 1)
Trend Direction
Lower; Decreasing / staying below threshold
Reporting Frequency
Rolling 72-hour window, evaluated daily
Category
Algorithmic Efficacy & Model Health
← Alert Net Economic Yield (ANEY)Contextual Recall Across Operating Envelope (CROE) →

Assess Your Organization

The AI-GPS Assessment evaluates your AI governance maturity and recommends KPIs like this one for your implementation roadmap.

Take the Free AI-GPS Assessment