Definition
CCA measures how accurately an NLP (natural language processing) classifier categorizes customer complaints into operational categories, and separately measures the recall rate specifically for water quality complaints.
The Measurement Gap
The Measurement Gap
What Legacy Metrics See: Customer service metrics track call volume, handle time, first-call resolution, and complaint category distribution. AI routing platforms report automation rates and satisfaction scores.
What Legacy Metrics Miss:
No standard metric measures whether the AI classifier is putting complaints in the right category. A platform reporting 78% automation rate does not reveal whether those complaints were routed correctly. The gap is particularly dangerous for water quality complaints, which arrive in ambiguous language ("my water looks funny," "the water tastes different"). If the classifier defaults ambiguous complaints to service issues (more common in training data), genuine water quality events get slow-tracked.
The Decision CCA Enables:
When CCA overall exceeds 85%, the utility can confidently automate most routing. When CCA WQ recall falls below 90%, too many water quality complaints are being misrouted. Options: retrain the classifier, lower the confidence threshold for WQ classification, or implement dual-path triage for ambiguous complaints.
Research Context:
NLP classification in customer service typically ranges 80-92% for well-defined categories. Water quality complaints are disproportionately affected because they represent 5-15% of total volume and use highly variable natural language.
Formula
Primary metric (water quality recall):
CCA_WQ_recall = Correctly_classified_WQ / Total_actual_WQ
Secondary metric (overall accuracy):
CCA_overall = Correctly_classified / Total_sampled
Why water quality recall is primary: Misclassifying a billing inquiry as water quality wastes lab and field resources but causes no public health risk. Misclassifying a water quality complaint as a billing or service inquiry means a potential contamination event, taste/odor complaint indicating treatment failure, or disinfection byproduct concern bypasses the water quality response protocol. The consequence is delayed investigation of conditions that may require public notification under SDWA. Set CCA_WQ_recall above 95%; CCA_overall above 90%.
Plain language: Sample AI-classified customer contacts. Have a trained agent re-classify each. Compute overall accuracy and the critical sub-metric.
Where:
- Correctly_classified = contacts where AI category matches expert agent classification
- Total_sampled = all sampled contacts
- Correctly_classified_WQ = water quality contacts correctly classified as water quality
- Total_actual_WQ = all contacts that should have been classified as water quality (per expert review)
Units: Percentage. CCA_WQ_recall is the critical metric because WQ contacts are a small fraction of total volume.
Proposed formula, pending validation pilots.