Definition
Percentage of product safety complaints correctly classified as safety issues rather than general quality feedback, measuring exposure to regulatory reporting failures buried in misrouted complaints.
The Measurement Gap
Product safety complaints are a special category of customer feedback. A complaint about a shirt that pilled is a quality issue. A complaint about a shirt that caught fire near a stovetop is a safety issue with CPSC reporting obligations. NLP classification systems treat both as customer complaints and assign categories based on language patterns. Safety complaints often arrive in language that does not match the model's training data for the safety category. A customer might write "the handle broke and cut my hand" without using the word "safety" or "injury." The model classifies it as general product quality. The safety signal is lost. If the defect is systematic, the delayed detection increases injury count before a recall can be initiated. CPSC reporting requirements create a legal obligation to identify safety issues within specific timeframes. An NLP system that misclassifies 20% of safety complaints means 20% of potentially reportable events are hiding in the wrong queue.
Decision Supported:
Is the NLP system reliably identifying product safety complaints that require regulatory review, or are reportable events hiding in general complaint categories?
Formula
Plain language: Of the complaints that a human safety reviewer identifies as potentially safety-related, what fraction did the NLP miss? That safety complaint false-negative rate is the headline metric. Overall classification accuracy is secondary.
Primary metric: Safety Complaint False-Negative Rate:
SCCA_FNR = Safety_complaints_missed / Total_safety_complaints_in_sample
Secondary metric: Overall Safety Classification Accuracy:
SCCA_OA = Safety_complaints_correctly_classified / Total_safety_complaints_in_sample
Variables:
| Variable | Definition | Unit |
|---|---|---|
| Safety_complaints_missed | Complaints identified as safety-related by human reviewer that the NLP did not route to the safety/compliance category (false negatives) | Count |
| Safety_complaints_correctly_classified | Complaints involving product safety issues that the NLP correctly assigned to the safety/compliance category | Count |
| Total_safety_complaints_in_sample | All complaints in the sample identified as potentially safety-related by human review, regardless of NLP classification | Count |
Score conservatively when customers do not use safety-specific language: if injury is plausible given the product failure described, classify as safety-related. Minimum 500 total complaints per quarter to capture enough safety-relevant cases. Weight SCCA by severity (injury outweighs near-miss, which outweighs defect-without-injury).
Why false-negative rate is primary:
In consumer retail, false positives (routing non-safety complaints to the safety queue) create review burden. False negatives (missing actual safety complaints) mean CPSC-reportable events hide in general complaint categories, product liability exposure accumulates, and recall signals are delayed. The asymmetry is a consequence structure, not a preference. A 90% overall accuracy that misses 15% of injury-related complaints is a product liability crisis, not acceptable performance.
Proposed formula, pending validation pilots.