Vendor Score Temporal Calibration (VSTC)

GMI-FAC-024Facilities ManagementIntegration-RequiredPattern O-2: Model Drift / Currency

Definition

Degree to which the AI vendor scorer's recency weighting causes ranking inversions where long-term superior vendors rank below long-term inferior vendors with better recent scores.

The Measurement Gap

Measurement Gap and Decision

The Measurement Gap

Traditional vendor scorecards use simple averages or weighted averages with static weights chosen by procurement policy. AI-powered vendor scoring introduces dynamic time-weighting that adapts based on data patterns, but the weighting logic is often opaque. Without VSTC, procurement has no way to know whether the AI is appropriately weighting recent performance or has calibrated its recency bias so aggressively that one bad quarter erases three years of excellent service.

The consequence of excessive recency weighting is vendor relationship instability. Reliable vendors get deprioritized after a single rough period. New vendors with brief good track records leapfrog established performers. Contract decisions made on these rankings reallocate work to less proven vendors while penalizing partners who have demonstrated sustained quality.

The Decision It Supports

VSTC tells procurement leadership how much the AI's time-weighting changes vendor rankings compared to equal-weighted history. Some inversion is healthy: it means the system responds to genuine performance changes. Too much inversion means recency dominates, and the organization risks destabilizing vendor relationships based on short-term fluctuations. The metric enables procurement to set a policy threshold for acceptable inversion rates before AI rankings drive contract decisions.

Formula

Plain Language

For each vendor pair in the same service category with at least 12 months of data, compute both the recency-weighted ranking (from the AI scorer) and the equal-weighted ranking. Count pairs where the rankings are inverted. Divide by total comparable pairs.

Formal Notation

$$VSTC = \frac{Inverted_pairs}{Total_comparable_pairs} \times 100$$

Variable Definitions

Variable Definition Unit
Inverted_pairs Vendor pairs where the recency-weighted ranking reverses the equal-weighted ranking Count
Total_comparable_pairs All vendor pairs in the same category with 12+ months of performance data Count

Units: Percentage. A value of 0% means recency weighting changes no rankings relative to equal weighting. Some inversion is expected and desirable. The question is where the threshold falls between "responsive to change" and "overreacting to noise."

Proposed formula, pending validation pilots.

At a Glance

Category
Vendor & Contractor Intelligence Accuracy

Decision Supported

Is the AI producing vendor rankings that reflect sustained capability, or overreacting to recent events?

← Vendor Relationship Total Cost Index (VRTCI)

Assess Your Organization

The AI-GPS Assessment evaluates your AI governance maturity and recommends KPIs like this one for your implementation roadmap.

Take the Free AI-GPS Assessment