Complaint Classification Accuracy (CCA)

GMI-WT-039Water TreatmentPilot-Ready

Definition

CCA measures how accurately an NLP (natural language processing) classifier categorizes customer complaints into operational categories, and separately measures the recall rate specifically for water quality complaints.

The Measurement Gap

The Measurement Gap

What Legacy Metrics See: Customer service metrics track call volume, handle time, first-call resolution, and complaint category distribution. AI routing platforms report automation rates and satisfaction scores.

What Legacy Metrics Miss:

No standard metric measures whether the AI classifier is putting complaints in the right category. A platform reporting 78% automation rate does not reveal whether those complaints were routed correctly. The gap is particularly dangerous for water quality complaints, which arrive in ambiguous language ("my water looks funny," "the water tastes different"). If the classifier defaults ambiguous complaints to service issues (more common in training data), genuine water quality events get slow-tracked.

The Decision CCA Enables:

When CCA overall exceeds 85%, the utility can confidently automate most routing. When CCA WQ recall falls below 90%, too many water quality complaints are being misrouted. Options: retrain the classifier, lower the confidence threshold for WQ classification, or implement dual-path triage for ambiguous complaints.

Research Context:

NLP classification in customer service typically ranges 80-92% for well-defined categories. Water quality complaints are disproportionately affected because they represent 5-15% of total volume and use highly variable natural language.

Formula

Primary metric (water quality recall):

CCA_WQ_recall = Correctly_classified_WQ / Total_actual_WQ

Secondary metric (overall accuracy):

CCA_overall = Correctly_classified / Total_sampled

Why water quality recall is primary: Misclassifying a billing inquiry as water quality wastes lab and field resources but causes no public health risk. Misclassifying a water quality complaint as a billing or service inquiry means a potential contamination event, taste/odor complaint indicating treatment failure, or disinfection byproduct concern bypasses the water quality response protocol. The consequence is delayed investigation of conditions that may require public notification under SDWA. Set CCA_WQ_recall above 95%; CCA_overall above 90%.

Plain language: Sample AI-classified customer contacts. Have a trained agent re-classify each. Compute overall accuracy and the critical sub-metric.

Where:

  • Correctly_classified = contacts where AI category matches expert agent classification
  • Total_sampled = all sampled contacts
  • Correctly_classified_WQ = water quality contacts correctly classified as water quality
  • Total_actual_WQ = all contacts that should have been classified as water quality (per expert review)

Units: Percentage. CCA_WQ_recall is the critical metric because WQ contacts are a small fraction of total volume.

Proposed formula, pending validation pilots.

At a Glance

Category
Inspection/Classification Fidelity
← CCTV Defect Coding Agreement Rate (CDCAR)Complaint Pattern Aggregation Quality (CPAQ) →

Assess Your Organization

The AI-GPS Assessment evaluates your AI governance maturity and recommends KPIs like this one for your implementation roadmap.

Take the Free AI-GPS Assessment