Automated Threat Detection Novel-Signature Coverage Rate (ATDNCR)

GMI-AVI-021AviationInfrastructure-Dependent

Definition

Detection rate of the automated threat detection system on novel threat signatures (those not present in training data) compared to its detection rate on known signatures. Measured via controlled red-team testing with novel concealment methods and threat materials.

The Measurement Gap

The measurement gap ATDNCR addresses: standard ATD performance metrics track overall detection rate and false alarm rate, but only against the test objects currently used in covert testing programs, which largely replicate known threat signatures. There is no standard metric comparing detection performance on known-signature threats to detection performance on novel signatures. Without that comparison, ATD system owners cannot determine whether high overall detection rates reflect genuine generalization capability or narrow optimization for training-distribution threats.

The threat environment evolves. Adversarial actors develop novel concealment methods specifically to defeat AI detection systems trained on prior threat signatures. An ATD system optimized for performance on known signatures may be overtrained on those signatures and fail to generalize to novel methods, a classic out-of-distribution failure mode. Standard detection rate metrics do not make this failure visible because novel-signature test objects are rarely included in routine covert testing.

The decision ATDNCR supports: should ATD model retraining be accelerated? Should manual inspection rates be increased for items that ATD clears with low confidence scores? Should the ATD vendor be required to demonstrate novel-signature generalization capability? These decisions require a metric that separates known-signature performance from novel-signature performance, which is exactly what ATDNCR provides.

Formula

Plain language: During red-team exercises, introduce test objects with both known threat signatures (those in the ATD training data) and novel signatures (new concealment methods, materials, or configurations not in the training data). Compare the detection rate on novel-signature test objects to the detection rate on known-signature test objects.

Notation:

ATDNCR = Detection_rate_novel / Detection_rate_known

Variables:

Variable Definition Unit
Detection_rate_novel Percentage of novel-signature test objects flagged by ATD (count flagged / count introduced) Percentage
Detection_rate_known Percentage of known-signature test objects flagged by ATD (count flagged / count introduced) Percentage

Units: Ratio. 1.0 = ATD detects novel threats at the same rate as known threats. Below 1.0 = coverage gap on novel threats. Zero = ATD misses all novel signatures.

Confidence Grading (required alongside every ATDNCR value):

Grade Basis Interpretation
A: Empirical 5+ red-team exercises with novel-signature test objects provide direct detection observation Measurement grounded in observed outcomes; highest confidence
B: Hybrid 1-4 exercises with novel signatures plus simulation of detection sensitivity across signature types Blends observation and simulation; moderate confidence
C: Simulated Zero exercises with confirmed novel signatures; ATDNCR derived entirely from simulated signature variations Modeled, not observed; treat as indicative, not validated

Report the confidence grade alongside every ATDNCR score. An ATDNCR of 0.80 Grade A is a verified detection rate from live exercises. An ATDNCR of 0.80 Grade C is a simulation that agrees with itself: internally consistent but not reality-tested. Grade C scores should trigger expanded red-team programs rather than acceptance as calibrated.

Proposed formula, pending validation pilots.

At a Glance

Category
Security and Compliance AI Calibration
← Safety Classification Concordance Rate (SCCR)Fuel Tail-Risk Calibration Index (FTRCI) →

Assess Your Organization

The AI-GPS Assessment evaluates your AI governance maturity and recommends KPIs like this one for your implementation roadmap.

Take the Free AI-GPS Assessment