Case Seriousness Classification Accuracy (CSCA)

GMI-PH-040Pharmaceutical OperationsIntegration-Required

Definition

Accuracy of AI seriousness triage on individual case safety reports, with the under-classification rate on serious cases as the primary measure and overall accuracy secondary. A serious case called non-serious is the error with patient consequences.

The Measurement Gap

A PV center receives 4,200 ICSRs monthly. The AI triage system auto-classifies each as serious or non-serious based on ICH E2A criteria. It classifies a narrative describing a patient admitted for observation following a severe allergic reaction with respiratory difficulty as non-serious because the narrative mentions observation, not hospitalization. Under ICH E2A, any inpatient admission constitutes hospitalization. The case requires expedited reporting within 15 calendar days but gets routed to the 90-day periodic queue.

Legacy PV metrics measure case processing speed, signal detection rates, case backlogs, and first-pass completion. None measure classification accuracy at the seriousness triage point. Seriousness classification requires nuanced clinical interpretation that NLP systems handle poorly, especially on consumer narratives written in informal language. Hospitalization includes observation admissions. Life-threatening means immediate risk at the time. Medically important is a judgment call requiring clinical context. Without CSCA, systematic misrouting of serious cases to non-expedited queues goes undetected until a regulatory inspection reveals the pattern.

Formula

Primary metric (under-classification rate on serious cases):

UCR_serious = serious_classified_as_nonserious / total_serious_in_sample

Secondary metric (overall accuracy):

CSCA = correctly_classified_cases / sampled_cases

Why under-classification rate on serious cases is primary: Over-classifying a non-serious case as serious triggers expedited reporting that wastes resources but ensures the case receives regulatory scrutiny. Under-classifying a serious adverse event as non-serious means the case misses the 15-day IND safety reporting window, creating FDA enforcement exposure and patient safety risk. Set UCR_serious below 2%; overall CSCA above 95%.

Over-classification rate = nonserious_classified_as_serious / total_nonserious_in_sample

Worked Example:

Q3 sample: 150 cases (75 AI-classified serious, 75 AI-classified non-serious).

Of 75 AI-classified serious: 72 confirmed serious, 3 over-classified.
Of 75 AI-classified non-serious: 69 confirmed non-serious, 6 under-classified.

CSCA = 141/150 = 94.0%
UCR_serious = 6/78 = 7.7%
Over-classification rate = 3/72 = 4.2%

Six serious cases in one quarter misrouted. Three involved observation-as-hospitalization, two involved medically important judgments, one involved indirect severity description.

Proposed formula, pending validation pilots.

At a Glance

Unit
Percentage
Trend Direction
Higher
Reporting Frequency
Monthly; per ICSR triage cohort
Category
Pharmacovigilance
← Reject Attribution Accuracy (RAA)Causality Assessment Novelty Bias Index (CANBI) →

Assess Your Organization

The AI-GPS Assessment evaluates your AI governance maturity and recommends KPIs like this one for your implementation roadmap.

Take the Free AI-GPS Assessment