Calibration Predictive Validity Index (CPVI)

GMI-HCM-007Human Capital ManagementIntegration-RequiredPattern O-1: Confidence Calibration

Definition

The change in predictive power of performance ratings after AI-assisted calibration, measured as the difference in correlation with actual outcomes between post-calibration and pre-calibration ratings.

The Measurement Gap

AI calibration tools promise to reduce bias, eliminate leniency effects, and produce more accurate performance ratings by algorithmically adjusting manager-submitted scores. The assumption is that calibration improves the signal in performance ratings, making them better predictors of actual employee performance and potential. But almost no organization validates this assumption. Calibration is treated as inherently good because it reduces variance and produces cleaner distributions, without testing whether the cleaned-up ratings actually predict anything better than the raw ratings did.

This is a critical blind spot because calibration can destroy signal while improving consistency. If managers use different rating scales but each manager's relative rankings contain valid information, calibration that normalizes across managers may wash out the within-manager signal without adding cross-manager comparability. The ratings become more statistically uniform but less predictively useful. Legacy metrics cannot detect this failure. Rating distribution metrics (normal distribution compliance, forced ranking adherence) measure distributional shape but not predictive power. Calibration session completion rates measure process compliance but not outcome quality. Inter-rater agreement after calibration measures consistency but consistency and accuracy are different things, a perfectly consistent set of ratings can be consistently wrong.

The downstream cost of unchecked calibration is substantial: promotion decisions, compensation adjustments, succession planning, and reduction-in-force rankings all depend on calibrated ratings. If calibration degrades predictive validity, every downstream talent decision built on those ratings is less accurate than it would have been using the uncalibrated scores.

Decision Supported:

Does our AI calibration process improve the predictive accuracy of performance ratings, or does it produce cleaner distributions at the cost of predictive signal?

Formula

Plain language: Correlate pre-calibration ratings with actual outcomes (e.g., subsequent performance, promotion success, retention). Then correlate post-calibration ratings with the same outcomes. CPVI is the difference: positive means calibration improved the signal, negative means it degraded it.

Notation:

CPVI = r(post_calibration_rating, outcome) − r(pre_calibration_rating, outcome)

Variables:

Variable Definition Unit
post_calibration_rating The final performance rating after AI-assisted calibration adjustments Rating score
pre_calibration_rating The original manager-submitted performance rating before calibration Rating score
outcome Actual performance outcome measured 6–12 months post-review, such as subsequent rating trajectory, promotion, retention, or objective performance metrics Score or category

Proposed formula, pending validation pilots.

At a Glance

Unit
Difference (-1 to 1)
Trend Direction
Higher; calibration improves predictive signal
Reporting Frequency
Per review cycle (typically annual)
Category
Human-AI Assessment Calibration
← Assessment Disagreement Resolution Index (ADRI)Review Narrative Anchoring Index (RNAI) →

Assess Your Organization

The AI-GPS Assessment evaluates your AI governance maturity and recommends KPIs like this one for your implementation roadmap.

Take the Free AI-GPS Assessment