Skip to main content
U.S. flag

An official website of the United States government

Official websites use .gov
A .gov website belongs to an official government organization in the United States.

Secure .gov websites use HTTPS
A lock ( ) or https:// means you’ve safely connected to the .gov website. Share sensitive information only on official, secure websites.

Micro-Signatures: The Effectiveness of Known Bad N-Grams for Network Anomaly Detection

Published

Author(s)

Richard Harang, Peter Mell

Abstract

The field of intrusion detection is divided into signature detection and anomaly detection. The former involves identifying patterns associated with known attacks and the latter involves attempting to learn a 'normal' pattern of activity and then producing security alerts when behaviors outside of those norms is detected. The n-grams methodology has arguably been the most successful technique for anomaly detection (including for network packet inspection). In this work, we identify a new type of intrusion detection that neither uses typical signatures nor is anomaly based (though it is closely related to both). We generate n-grams from both malicious content and Snort signatures and use sets of these 'micro-signatures' to identify attacks. This micro-signature capability arises implicitly when the training sets for n-gram anomaly detection systems are scrubbed of malicious content and thus is not new. It was added explicitly by the seminal Anagram network anomaly approach, but was portrayed as a minor enhancement and its effect was not evaluated. In reproducing the Anagram results we find that for our data, the micro-signatures provide the vast majority of the detection capability. What appears on the surface to be an anomaly detection approach achieves most of its effectiveness from a (sometimes merely implicit) signature subsystem. We furthermore find that these micro- signatures enable highly effective standalone detection systems as well as hybrid micro- signature/anomaly systems that generalize to multiple attack classes. Our results thus shed new light into the functioning of n-gram anomaly detection systems, reveal the need to evaluate the micro-signature contribution within n-gram anomaly research, and open a new avenue of research into how to best use micro-signatures in future detection systems.
Proceedings Title
Foundations and Practice of Security: 9th International Symposium, FPS 2016
Conference Dates
October 24-25, 2016
Conference Location
Quebec City, CA
Conference Title
9th International Symposium on Foundations and Practice of Security

Keywords

intrusion detection, anomaly detection, n-grams, micro-signatures, security

Citation

Harang, R. and Mell, P. (2017), Micro-Signatures: The Effectiveness of Known Bad N-Grams for Network Anomaly Detection, Foundations and Practice of Security: 9th International Symposium, FPS 2016, Quebec City, CA, [online], https://doi.org/10.1007/978-3-319-51966-1_3, https://tsapps.nist.gov/publication/get_pdf.cfm?pub_id=920217 (Accessed March 28, 2024)
Created February 4, 2017, Updated October 12, 2021