Safety Complaint Classification Accuracy (SCCA)

GMI-RET-018Retail OperationsIntegration-RequiredPattern R-2: NLP Classification

Definition

Percentage of product safety complaints correctly classified as safety issues rather than general quality feedback, measuring exposure to regulatory reporting failures buried in misrouted complaints.

The Measurement Gap

Product safety complaints are a special category of customer feedback. A complaint about a shirt that pilled is a quality issue. A complaint about a shirt that caught fire near a stovetop is a safety issue with CPSC reporting obligations. NLP classification systems treat both as customer complaints and assign categories based on language patterns. Safety complaints often arrive in language that does not match the model's training data for the safety category. A customer might write "the handle broke and cut my hand" without using the word "safety" or "injury." The model classifies it as general product quality. The safety signal is lost. If the defect is systematic, the delayed detection increases injury count before a recall can be initiated. CPSC reporting requirements create a legal obligation to identify safety issues within specific timeframes. An NLP system that misclassifies 20% of safety complaints means 20% of potentially reportable events are hiding in the wrong queue.

Decision Supported:

Is the NLP system reliably identifying product safety complaints that require regulatory review, or are reportable events hiding in general complaint categories?

Formula

Plain language: Of the complaints that a human safety reviewer identifies as potentially safety-related, what fraction did the NLP miss? That safety complaint false-negative rate is the headline metric. Overall classification accuracy is secondary.

Primary metric: Safety Complaint False-Negative Rate:

SCCA_FNR = Safety_complaints_missed / Total_safety_complaints_in_sample

Secondary metric: Overall Safety Classification Accuracy:

SCCA_OA = Safety_complaints_correctly_classified / Total_safety_complaints_in_sample

Variables:

Variable Definition Unit
Safety_complaints_missed Complaints identified as safety-related by human reviewer that the NLP did not route to the safety/compliance category (false negatives) Count
Safety_complaints_correctly_classified Complaints involving product safety issues that the NLP correctly assigned to the safety/compliance category Count
Total_safety_complaints_in_sample All complaints in the sample identified as potentially safety-related by human review, regardless of NLP classification Count

Score conservatively when customers do not use safety-specific language: if injury is plausible given the product failure described, classify as safety-related. Minimum 500 total complaints per quarter to capture enough safety-relevant cases. Weight SCCA by severity (injury outweighs near-miss, which outweighs defect-without-injury).

Why false-negative rate is primary:

In consumer retail, false positives (routing non-safety complaints to the safety queue) create review burden. False negatives (missing actual safety complaints) mean CPSC-reportable events hide in general complaint categories, product liability exposure accumulates, and recall signals are delayed. The asymmetry is a consequence structure, not a preference. A 90% overall accuracy that misses 15% of injury-related complaints is a product liability crisis, not acceptable performance.

Proposed formula, pending validation pilots.

At a Glance

Unit
Percentage
Trend Direction
Higher
Reporting Frequency
Quarterly
Category
Customer Experience and AI Service Quality
← Complaint Category Leakage Rate (CCLR)Sentiment Calibration Drift Index (SCDI) →

Assess Your Organization

The AI-GPS Assessment evaluates your AI governance maturity and recommends KPIs like this one for your implementation roadmap.

Take the Free AI-GPS Assessment