Micro-Signatures: The Effectiveness of Known Bad N-Grams for Network Anomaly Detection

Published: February 05, 2017


Richard Harang, Peter M. Mell


The field of intrusion detection is divided into signature detection and anomaly detection. The former involves identifying patterns associated with known attacks and the latter involves attempting to learn a 'normal' pattern of activity and then producing security alerts when behaviors outside of those norms is detected. The n-grams methodology has arguably been the most successful technique for anomaly detection (including for network packet inspection). In this work, we identify a new type of intrusion detection that neither uses typical signatures nor is anomaly based (though it is closely related to both). We generate n-grams from both malicious content and Snort signatures and use sets of these 'micro-signatures' to identify attacks. This micro-signature capability arises implicitly when the training sets for n-gram anomaly detection systems are scrubbed of malicious content and thus is not new. It was added explicitly by the seminal Anagram network anomaly approach, but was portrayed as a minor enhancement and its effect was not evaluated. In reproducing the Anagram results we find that for our data, the micro-signatures provide the vast majority of the detection capability. What appears on the surface to be an anomaly detection approach achieves most of its effectiveness from a (sometimes merely implicit) signature subsystem. We furthermore find that these micro- signatures enable highly effective standalone detection systems as well as hybrid micro- signature/anomaly systems that generalize to multiple attack classes. Our results thus shed new light into the functioning of n-gram anomaly detection systems, reveal the need to evaluate the micro-signature contribution within n-gram anomaly research, and open a new avenue of research into how to best use micro-signatures in future detection systems.
Proceedings Title: Foundations and Practice of Security: 9th International Symposium, FPS 2016
Conference Dates: October 24-25, 2016
Conference Location: Quebec City, -1
Conference Title: 9th International Symposium on Foundations and Practice of Security
Pub Type: Conferences


intrusion detection, anomaly detection, n-grams, micro-signatures, security
Created February 05, 2017, Updated November 10, 2018