Definition
Detection rate of the automated threat detection system on novel threat signatures (those not present in training data) compared to its detection rate on known signatures. Measured via controlled red-team testing with novel concealment methods and threat materials.
The Measurement Gap
The measurement gap ATDNCR addresses: standard ATD performance metrics track overall detection rate and false alarm rate, but only against the test objects currently used in covert testing programs, which largely replicate known threat signatures. There is no standard metric comparing detection performance on known-signature threats to detection performance on novel signatures. Without that comparison, ATD system owners cannot determine whether high overall detection rates reflect genuine generalization capability or narrow optimization for training-distribution threats.
The threat environment evolves. Adversarial actors develop novel concealment methods specifically to defeat AI detection systems trained on prior threat signatures. An ATD system optimized for performance on known signatures may be overtrained on those signatures and fail to generalize to novel methods, a classic out-of-distribution failure mode. Standard detection rate metrics do not make this failure visible because novel-signature test objects are rarely included in routine covert testing.
The decision ATDNCR supports: should ATD model retraining be accelerated? Should manual inspection rates be increased for items that ATD clears with low confidence scores? Should the ATD vendor be required to demonstrate novel-signature generalization capability? These decisions require a metric that separates known-signature performance from novel-signature performance, which is exactly what ATDNCR provides.
Formula
Plain language: During red-team exercises, introduce test objects with both known threat signatures (those in the ATD training data) and novel signatures (new concealment methods, materials, or configurations not in the training data). Compare the detection rate on novel-signature test objects to the detection rate on known-signature test objects.
Notation:
ATDNCR = Detection_rate_novel / Detection_rate_known
Variables:
| Variable | Definition | Unit |
|---|---|---|
| Detection_rate_novel | Percentage of novel-signature test objects flagged by ATD (count flagged / count introduced) | Percentage |
| Detection_rate_known | Percentage of known-signature test objects flagged by ATD (count flagged / count introduced) | Percentage |
Units: Ratio. 1.0 = ATD detects novel threats at the same rate as known threats. Below 1.0 = coverage gap on novel threats. Zero = ATD misses all novel signatures.
Confidence Grading (required alongside every ATDNCR value):
| Grade | Basis | Interpretation |
|---|---|---|
| A: Empirical | 5+ red-team exercises with novel-signature test objects provide direct detection observation | Measurement grounded in observed outcomes; highest confidence |
| B: Hybrid | 1-4 exercises with novel signatures plus simulation of detection sensitivity across signature types | Blends observation and simulation; moderate confidence |
| C: Simulated | Zero exercises with confirmed novel signatures; ATDNCR derived entirely from simulated signature variations | Modeled, not observed; treat as indicative, not validated |
Report the confidence grade alongside every ATDNCR score. An ATDNCR of 0.80 Grade A is a verified detection rate from live exercises. An ATDNCR of 0.80 Grade C is a simulation that agrees with itself: internally consistent but not reality-tested. Grade C scores should trigger expanded red-team programs rather than acceptance as calibrated.
Proposed formula, pending validation pilots.