LLM Judge Calibration Index (LJCI)

GMI-MLOPS-002MLOps / AI Platform OperationsIntegration-RequiredPattern O-1: Confidence Calibration

Definition

The agreement between LLM-judge scores and human expert judgments on a sampled fraction of judged items, tracked over time and across judge versions.

The Measurement Gap

There is no metric that tells you whether LLM-judge scoring remains calibrated against human ground truth over time, which means gate decisions ride a moving ruler and cross-month comparisons are fiction.

Formula

LJCI = agreement(judge_score_i, human_score_i) for sampled items i, stratified by judge_version

Proposed formula, pending validation pilots.

At a Glance

Category
Evaluation Integrity

Decision Supported

Whether promotion decisions gated on judge scores are riding a stable ruler, and when a judge model or judge prompt change requires re-baselining.

← Eval Task Coverage Ratio (ETCR)Segment Quality Divergence Rate (SQDR) →

Assess Your Organization

The AI-GPS Assessment evaluates your AI governance maturity and recommends KPIs like this one for your implementation roadmap.

Take the Free AI-GPS Assessment