Definition
Degree of drift between sentiment model scores and independently validated customer sentiment over time, measuring whether the experience dashboard is still reading customers correctly.
The Measurement Gap
Sentiment analysis models power customer experience dashboards across retail. They score interactions, flag escalations, route complaints, and feed executive reports. These models are trained on historical language patterns, but language changes. Slang shifts, customer communication styles evolve, and new product categories introduce new vocabulary. The model's accuracy degrades silently. The failure mode is that sentiment dashboards never report their own accuracy. They report scores as ground truth. A customer who writes "yeah, that really fixed my problem" with heavy sarcasm may be scored positive. A detailed, neutral-tone complaint may be scored moderate when the actual sentiment is intensely negative. SCDI introduces a calibration layer that measures how closely NLP scores track reality by periodically comparing them to ground-truth feedback (surveys, ratings, explicit satisfaction signals) and tracking the correlation over time. Without this metric, organizations make strategic decisions about customer experience based on a model whose accuracy has never been audited.
Decision Supported:
Is the sentiment dashboard still reliable, or has language evolution caused the NLP model to drift from reality?
Formula
Formula:
SCDI = 1 - r(NLP_scores_t, Ground_truth_t)
Variables:
- r = Pearson or Spearman correlation coefficient
- NLP_scores_t = NLP-assigned sentiment scores for the sample in period t
- Ground_truth_t = direct customer feedback scores for the same interactions in period t
- Period t = rolling quarterly measurement
Units: 0 to 1. SCDI = 0 means perfect correlation (no drift). SCDI = 0.3 means moderate drift. SCDI above 0.5 means the NLP model is more wrong than right.
Edge Cases and Exclusions:
Selection bias in survey respondents may skew toward extremes (very satisfied or very dissatisfied). Use stratified sampling to ensure representation across satisfaction levels. Normalize NLP scores and feedback scores to a common 0-to-1 scale before computing correlation. Require a minimum of 100 matched pairs per quarter for stable correlation estimates. For multilingual customer bases, compute SCDI per language because drift rates differ by language. Social media language evolves faster than email language, so compute SCDI per channel as well.
Proposed formula, pending validation pilots.