Complaint Category Leakage Rate (CCLR)

GMI-RET-017Retail OperationsIntegration-RequiredPattern R-2: NLP Classification

Definition

Percentage of NLP-classified complaints assigned to the wrong category, tracked overall and for novel issue types separately, measuring how much complaint signal leaks out of the categories leadership watches.

The Measurement Gap

NLP complaint classification systems sort customer complaints into predefined categories (product quality, shipping, pricing, returns) and the sorting drives routing. When a new product defect emerges, the early complaints use language the model has never seen. The model assigns them to whatever existing category seems closest. A complaint about fabric pilling might be classified as "returns." A complaint about a defective zipper might be classified as "sizing" because the customer mentions the garment. The defect signal gets scattered across wrong categories instead of concentrating in the right one. This systemic risk is invisible because classification accuracy is rarely audited after deployment. The model is trained, tested on a validation set, and deployed. From that point forward, accuracy is assumed. CCLR provides the ongoing audit that reveals whether the model is still performing and whether it can handle problems it has never seen before. The sub-metric Novel Issue Leakage captures the more dangerous failure: the system's structural inability to surface new problem types.

Decision Supported:

Is the NLP complaint classification system hiding emerging product defects by scattering related complaints across wrong categories?

Formula

Primary Formula:

CCLR = Misclassified_complaints / Sampled_complaints

Sub-Metric:

Novel_issue_leakage = Novel_issues_misclassified / Total_novel_issues_in_sample

Variables:

  • Misclassified = complaints where the NLP category does not match the human-assessed category
  • Novel_issues = complaints where the human reviewer identifies the issue type as not fitting any existing category

Units: Percentage. Track CCLR overall and Novel_issue_leakage separately. The second is more dangerous because it measures the system's ability to detect new problems.

Edge Cases and Exclusions:

Ambiguous complaints where the customer mentions multiple issues should be scored against the primary issue, not secondary. If the human reviewer cannot find a matching category either, the problem is the taxonomy, not the NLP. Flag those cases for taxonomy update rather than counting as misclassification. Compute CCLR per language for multilingual customer bases. Require a minimum of 200 complaints per quarter for stable rates. Oversample recent complaints (last 30 days) to increase sensitivity to novel issues.

Proposed formula, pending validation pilots.

At a Glance

Unit
Percentage
Trend Direction
Lower
Reporting Frequency
Quarterly
Category
Customer Experience and AI Service Quality
← Pricing Override Value Score (POVS)Safety Complaint Classification Accuracy (SCCA) →

Assess Your Organization

The AI-GPS Assessment evaluates your AI governance maturity and recommends KPIs like this one for your implementation roadmap.

Take the Free AI-GPS Assessment