Definition
Accuracy of AI seriousness triage on individual case safety reports, with the under-classification rate on serious cases as the primary measure and overall accuracy secondary. A serious case called non-serious is the error with patient consequences.
The Measurement Gap
A PV center receives 4,200 ICSRs monthly. The AI triage system auto-classifies each as serious or non-serious based on ICH E2A criteria. It classifies a narrative describing a patient admitted for observation following a severe allergic reaction with respiratory difficulty as non-serious because the narrative mentions observation, not hospitalization. Under ICH E2A, any inpatient admission constitutes hospitalization. The case requires expedited reporting within 15 calendar days but gets routed to the 90-day periodic queue.
Legacy PV metrics measure case processing speed, signal detection rates, case backlogs, and first-pass completion. None measure classification accuracy at the seriousness triage point. Seriousness classification requires nuanced clinical interpretation that NLP systems handle poorly, especially on consumer narratives written in informal language. Hospitalization includes observation admissions. Life-threatening means immediate risk at the time. Medically important is a judgment call requiring clinical context. Without CSCA, systematic misrouting of serious cases to non-expedited queues goes undetected until a regulatory inspection reveals the pattern.
Formula
Primary metric (under-classification rate on serious cases):
UCR_serious = serious_classified_as_nonserious / total_serious_in_sample
Secondary metric (overall accuracy):
CSCA = correctly_classified_cases / sampled_cases
Why under-classification rate on serious cases is primary: Over-classifying a non-serious case as serious triggers expedited reporting that wastes resources but ensures the case receives regulatory scrutiny. Under-classifying a serious adverse event as non-serious means the case misses the 15-day IND safety reporting window, creating FDA enforcement exposure and patient safety risk. Set UCR_serious below 2%; overall CSCA above 95%.
Over-classification rate = nonserious_classified_as_serious / total_nonserious_in_sample
Worked Example:
Q3 sample: 150 cases (75 AI-classified serious, 75 AI-classified non-serious).
Of 75 AI-classified serious: 72 confirmed serious, 3 over-classified.
Of 75 AI-classified non-serious: 69 confirmed non-serious, 6 under-classified.
CSCA = 141/150 = 94.0%
UCR_serious = 6/78 = 7.7%
Over-classification rate = 3/72 = 4.2%
Six serious cases in one quarter misrouted. Three involved observation-as-hospitalization, two involved medically important judgments, one involved indirect severity description.
Proposed formula, pending validation pilots.