Definition
The agreement between LLM-judge scores and human expert judgments on a sampled fraction of judged items, tracked over time and across judge versions.
The Measurement Gap
There is no metric that tells you whether LLM-judge scoring remains calibrated against human ground truth over time, which means gate decisions ride a moving ruler and cross-month comparisons are fiction.
Formula
LJCI = agreement(judge_score_i, human_score_i) for sampled items i, stratified by judge_version
Proposed formula, pending validation pilots.
At a Glance
Decision Supported
Whether promotion decisions gated on judge scores are riding a stable ruler, and when a judge model or judge prompt change requires re-baselining.