Definition
The change in predictive power of performance ratings after AI-assisted calibration, measured as the difference in correlation with actual outcomes between post-calibration and pre-calibration ratings.
The Measurement Gap
AI calibration tools promise to reduce bias, eliminate leniency effects, and produce more accurate performance ratings by algorithmically adjusting manager-submitted scores. The assumption is that calibration improves the signal in performance ratings, making them better predictors of actual employee performance and potential. But almost no organization validates this assumption. Calibration is treated as inherently good because it reduces variance and produces cleaner distributions, without testing whether the cleaned-up ratings actually predict anything better than the raw ratings did.
This is a critical blind spot because calibration can destroy signal while improving consistency. If managers use different rating scales but each manager's relative rankings contain valid information, calibration that normalizes across managers may wash out the within-manager signal without adding cross-manager comparability. The ratings become more statistically uniform but less predictively useful. Legacy metrics cannot detect this failure. Rating distribution metrics (normal distribution compliance, forced ranking adherence) measure distributional shape but not predictive power. Calibration session completion rates measure process compliance but not outcome quality. Inter-rater agreement after calibration measures consistency but consistency and accuracy are different things, a perfectly consistent set of ratings can be consistently wrong.
The downstream cost of unchecked calibration is substantial: promotion decisions, compensation adjustments, succession planning, and reduction-in-force rankings all depend on calibrated ratings. If calibration degrades predictive validity, every downstream talent decision built on those ratings is less accurate than it would have been using the uncalibrated scores.
Decision Supported:
Does our AI calibration process improve the predictive accuracy of performance ratings, or does it produce cleaner distributions at the cost of predictive signal?
Formula
Plain language: Correlate pre-calibration ratings with actual outcomes (e.g., subsequent performance, promotion success, retention). Then correlate post-calibration ratings with the same outcomes. CPVI is the difference: positive means calibration improved the signal, negative means it degraded it.
Notation:
CPVI = r(post_calibration_rating, outcome) − r(pre_calibration_rating, outcome)
Variables:
| Variable | Definition | Unit |
|---|---|---|
| post_calibration_rating | The final performance rating after AI-assisted calibration adjustments | Rating score |
| pre_calibration_rating | The original manager-submitted performance rating before calibration | Rating score |
| outcome | Actual performance outcome measured 6–12 months post-review, such as subsequent rating trajectory, promotion, retention, or objective performance metrics | Score or category |
Proposed formula, pending validation pilots.