Viable Network Intrusion Detection in High-Performance Environments

175 Pages·2012·2.12 MB·English

Lehrstuhl fu(cid:127)r Netzwerkarchitekturen
Fakult(cid:127)at fu(cid:127)r Informatik
Technische Universit(cid:127)at Mu(cid:127)nchen
Viable Network Intrusion Detection in
High-Performance Environments
Robin Sommer
Vollst(cid:127)andiger Abdruck der von der Fakult(cid:127)at fu(cid:127)r Informatik
der Technischen Universit(cid:127)at Mu(cid:127)nchen zur Erlangung des akademischen Grades eines
Doktors der Naturwissenschaften (Dr. rer. nat.)
genehmigten Dissertation.
Vorsitzender: Univ.-Prof. Alfons Kemper, Ph.D.
Pru(cid:127)fer der Dissertation: 1. Univ.-Prof. Anja Feldmann, Ph.D.
2. Univ.-Prof. Dr. Dr. h.c. Manfred Broy
3. Adj. Prof. Vern Paxson, Ph.D.
University of California, Berkeley / USA
(schriftliche Beurteilung)
Die Dissertation wurde am 30.06.2005 bei der Technischen Universit(cid:127)at Mu(cid:127)nchen
eingereicht und durch die Fakult(cid:127)at fu(cid:127)r Informatik
am 29.08.2005 angenommen.

Abstract
Network intrusion detection systems (NIDS) continuously monitor network tra(cid:14)c for
malicious activity, raising alerts when they detect attacks. However, high-performance
Gbps networks pose major challenges for these systems. Despite vendor promises, they
often fail to work reliably in such environments.
In this work, we set out to understand the trade-o(cid:11)s involved in network intrusion
detection, andwemitigate theimpactoftheirchoiceonoperational securitymonitoring.
Webaseourstudyonextensiveexperiencewithseverallarge-scalenetworkenvironments,
includingtheMunichScienti(cid:12)cNetworkandthebackboneoftheUniversityofCalifornia
atBerkeley. Insuchnetworks,we(cid:12)ndanimmensetra(cid:14)cdiversitywhichrequiresaNIDS
to deal robustly with unexpected situations. However, to accommodate any conceivable
situation, a NIDS would need an unlimited supply of CPU cycles and memory. Thus,
the operator of the system needs to trade-o(cid:11) the quality of the detection with resource
demands. To provide the necessary tuning options, we devise several new mechanisms
which allow to choose this trade-o(cid:11) according to the policy of a particular environment.
Moreover, we enable a NIDS to transparently share its state across instances, thereby
multiplying the available amount of resources. Another major trade-o(cid:11) that a NIDS
faces is the decision when to alert: if it reports anything which could potentially be
malicious, it will generate an unmanageable numberof alerts; if it reports only the most
obviousattacks, itwillmisssome. To improvetheprecisionofthedetection, weenablea
NIDStoincorporate di(cid:11)erentkindsofnetworkcontext intoitsanalysis. Suchcontextual
information can either be derived during operation or provided externally, e.g., by host
applications.
The thesis starts with recapitulating concepts and limitations of network intrusion
detection, andthenpresentssevenoperationalenvironmentsinwhichwehaveconducted
our research. Next, we discuss our experiences with deploying four di(cid:11)erent NIDSs,
open-source systems as well as commercial. We examine the resource usage of one of the
systems in detail and improve the system’s resource management. Then we introduce
the concept of independent state, i.e., internal (cid:12)ne-grained state that can be shared
among instances. The implementation of this concept provides us with a wealth of new
operationally signi(cid:12)cant applications. Finally, we introduce contextual signatures which
supplement traditional byte-level pattern matching with contextual information. We
implement and evaluate all of our improvements within the framework provided by the
open-source Bro NIDS.

Acknowledgments
I would like to thank the many people who have contributed to this thesis. Without
them, this work would not have been possible.
First and foremost, I am indebted to my advisor, Anja Feldmann, for the outstanding
motivation, guidance, support, and knowledge she has provided throughout the course
of thiswork. Shekindlytook me intohergroupandprovidedme withagreat amountof
freedom to work on things that I like. She has become a good friend, and her feedback
has greatly improved the thesis.
IamextremelygratefultoVernPaxsonfortheassistance,generosity, andadvicethatI
havereceivedfromhim. Ihavebene(cid:12)tedgreatlyfromhisinsight,support,andfeedback.
His enthusiasmhas always been a great motivation, and I enjoyedmy stays at ICSIvery
much.
My gratitude also extends to Manfred Broy and Alfons Kemper for agreeing to be on
my thesis committee.
I also thank my colleagues and friends in our group at TU Mu(cid:127)nchen. It has been fun
working with them. In particular, many thanks go to Holger Dreger. His ideas, e(cid:11)orts,
and contributions have signi(cid:12)cantly added to my work. Thanks to Vinay Aggarwal
and J(cid:127)org Wallerich for commenting on drafts of the thesis, to J(cid:127)org also for mastering
FreeBSD, to Petra Lorenz for helping with bureaucracy, and to Gregor Maier for his
help with the DAG cards.
I also thank Scott Campbell for regular feedback and test-driving of my code; Jos(cid:19)e
Mar(cid:19)(cid:16)a Gonza(cid:19)lez for analyzing BPF-based sampling; Christian Kreibich for his terri(cid:12)c
work on Broccoli and his related contributions; Wolfgang Mu(cid:127)ller for arranging the op-
portunity to start the work; and to the people at ICSI for their help during my visits.
My work has been generously supported by the following institutions providing me
withaccesstotheirnetworkenvironments: theInternationalComputerScienceInstitute,
the Lawrence Berkeley National Laboratory, the Leibniz-Rechenzentrum, the National
Energy Research Scienti(cid:12)c Computing Center, the Saarland University, and the Univer-
sity of California at Berkeley. Without the help of many great people working at these
places, this work would have been impossible. In particular, I thank Victor Aposto-
lescu of the Leibniz-Rechenzentrum for his outright support. Moreover, my thanks go
to Mike Bennett, Christopher Chin, Mark Dedlow, Deti Fliegl, Jake F. Harwood, Mike
Hunter, David Johnson, Jay Krous, Alfred L(cid:127)apple, Ken Lindahl, Dietmar Reichert, Hel-
mut Tr(cid:127)obs, Claus Wimmer, as well as to LBNL’s BroLite group.
Last, but not the least, I thank my parents, my sister, as well as my grandparents. It
has been their lasting love and support that enabled me to reach this point.

Contents
1 Introduction 1
1.1 Motivation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1
1.2 Outline . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3
1.3 Published Work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4
2 Network Intrusion Detection 5
2.1 Historic Origins of Network Intrusion Detection . . . . . . . . . . . . . . . 5
2.2 Terminology . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6
2.2.1 Computer Security . . . . . . . . . . . . . . . . . . . . . . . . . . . 6
2.2.2 Policy and Mechanism . . . . . . . . . . . . . . . . . . . . . . . . . 6
2.2.3 Threats, Vulnerabilities, Intrusions . . . . . . . . . . . . . . . . . . 7
2.3 Concepts of Network Intrusion Detection . . . . . . . . . . . . . . . . . . 8
2.3.1 De(cid:12)nitions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9
2.3.2 Architecture . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9
2.3.3 Deployment Schemes . . . . . . . . . . . . . . . . . . . . . . . . . . 10
2.3.4 Detection Methods . . . . . . . . . . . . . . . . . . . . . . . . . . . 11
2.3.5 Alert Correlation . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13
2.3.6 Detection vs. Prevention . . . . . . . . . . . . . . . . . . . . . . . 15
2.3.7 Stateful vs. Stateless . . . . . . . . . . . . . . . . . . . . . . . . . . 15
2.3.8 User Interface . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16
2.3.9 License Scheme . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16
2.3.10 Hardware . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17
2.4 Limitations of Network Intrusion Detection . . . . . . . . . . . . . . . . . 17
2.4.1 Cost vs. Bene(cid:12)t . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17
2.4.2 False Positives . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18
2.4.3 Attack Vulnerability . . . . . . . . . . . . . . . . . . . . . . . . . . 19
2.4.4 State Expiration Heuristics . . . . . . . . . . . . . . . . . . . . . . 21
2.4.5 Evaluation Di(cid:14)culties . . . . . . . . . . . . . . . . . . . . . . . . . 21
2.4.6 User Privacy . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23
2.5 Selection of Network Intrusion Detection Systems . . . . . . . . . . . . . . 23
2.5.1 Snort . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24
2.5.2 Bro . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24
2.5.3 Emerald . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26
2.5.4 STAT . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27
2.5.5 Enterasys Dragon . . . . . . . . . . . . . . . . . . . . . . . . . . . 27
2.5.6 McAfee IntruShield. . . . . . . . . . . . . . . . . . . . . . . . . . . 28
i
Contents
3 High-Performance Environments 29
3.1 Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29
3.2 Environments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30
3.2.1 Munich Scienti(cid:12)c Network . . . . . . . . . . . . . . . . . . . . . . . 30
3.2.2 University of California, Berkeley . . . . . . . . . . . . . . . . . . . 33
3.2.3 Lawrence Berkeley National Laboratory . . . . . . . . . . . . . . . 34
3.2.4 National Energy Research Scienti(cid:12)c Computing Center . . . . . . 34
3.2.5 Saarland University . . . . . . . . . . . . . . . . . . . . . . . . . . 35
3.2.6 International Computer Science Institute . . . . . . . . . . . . . . 35
3.2.7 Work Group Network at TU Mu(cid:127)nchen . . . . . . . . . . . . . . . . 36
3.3 Characteristics of High-Performance Environments . . . . . . . . . . . . . 36
3.3.1 Policies . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 36
3.3.2 Threats . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 38
3.3.3 Tra(cid:14)c Diversity . . . . . . . . . . . . . . . . . . . . . . . . . . . . 40
3.4 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 44
4 Operational Deployment 51
4.1 Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 51
4.2 Experiences with Network Intrusion Detection Systems. . . . . . . . . . . 52
4.2.1 Logs, Alerts and Forensics . . . . . . . . . . . . . . . . . . . . . . . 53
4.2.2 Signature Quality . . . . . . . . . . . . . . . . . . . . . . . . . . . 55
4.2.3 User Interfaces . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 56
4.2.4 Resource Demands . . . . . . . . . . . . . . . . . . . . . . . . . . . 59
4.2.5 Evaluation Di(cid:14)culties . . . . . . . . . . . . . . . . . . . . . . . . . 60
4.2.6 Artifacts of the Monitoring Environment . . . . . . . . . . . . . . 62
4.2.7 Sensitivity to Programming Errors . . . . . . . . . . . . . . . . . . 63
4.3 Resource Measurement Methodology . . . . . . . . . . . . . . . . . . . . . 64
4.3.1 Memory Usage . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 64
4.3.2 CPU Usage . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 65
4.4 Analysis of Resource Usage . . . . . . . . . . . . . . . . . . . . . . . . . . 66
4.4.1 Evaluation Data . . . . . . . . . . . . . . . . . . . . . . . . . . . . 66
4.4.2 Connection State Management . . . . . . . . . . . . . . . . . . . . 67
4.4.3 User-Level State Management . . . . . . . . . . . . . . . . . . . . . 69
4.4.4 Packet Drops due to Network Load . . . . . . . . . . . . . . . . . . 71
4.4.5 Packet Drops due to Processing Spikes . . . . . . . . . . . . . . . . 71
4.4.6 Resources vs. Detection Rate . . . . . . . . . . . . . . . . . . . . . 73
4.5 High-Volume IDS Extensions . . . . . . . . . . . . . . . . . . . . . . . . . 74
4.5.1 Connection State Management . . . . . . . . . . . . . . . . . . . . 74
4.5.2 User-Level State Management . . . . . . . . . . . . . . . . . . . . . 76
4.5.3 Dynamically Controlling Packet Load . . . . . . . . . . . . . . . . 77
4.5.4 Using High-Performance Monitoring Hardware . . . . . . . . . . . 80
4.6 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 81
ii
Contents
5 Separating State From Processing 83
5.1 Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 83
5.2 Terminology . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 85
5.3 Bro’s State . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 85
5.3.1 Event Engine State . . . . . . . . . . . . . . . . . . . . . . . . . . . 86
5.3.2 User-Level State . . . . . . . . . . . . . . . . . . . . . . . . . . . . 87
5.4 Architecture for Independent State . . . . . . . . . . . . . . . . . . . . . . 88
5.4.1 Serialization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 88
5.4.2 Using Independent State . . . . . . . . . . . . . . . . . . . . . . . . 93
5.4.3 Robust and Secure Communication . . . . . . . . . . . . . . . . . . 97
5.4.4 Integrating External State . . . . . . . . . . . . . . . . . . . . . . . 99
5.5 Applications . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 99
5.5.1 Selective Checkpointing . . . . . . . . . . . . . . . . . . . . . . . . 99
5.5.2 Crash Recovery . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 101
5.5.3 Distributed Analysis . . . . . . . . . . . . . . . . . . . . . . . . . . 101
5.5.4 Dynamic Recon(cid:12)guration, Pro(cid:12)ling and Debugging . . . . . . . . . 105
5.6 Performance Evaluation . . . . . . . . . . . . . . . . . . . . . . . . . . . . 107
5.6.1 Benchmarks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 108
5.6.2 Performance on Realistic Data . . . . . . . . . . . . . . . . . . . . 109
5.7 Inclusion of Host-Based State . . . . . . . . . . . . . . . . . . . . . . . . . 112
5.7.1 Bene(cid:12)ts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 113
5.7.2 Integration into Bro . . . . . . . . . . . . . . . . . . . . . . . . . . 115
5.7.3 Examining HTTP Sessions . . . . . . . . . . . . . . . . . . . . . . 115
5.7.4 Deployment and Results . . . . . . . . . . . . . . . . . . . . . . . . 117
5.8 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 122
6 Enhancing Signatures with Context 125
6.1 Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 125
6.2 Contextual Signatures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 127
6.2.1 Regular Expressions . . . . . . . . . . . . . . . . . . . . . . . . . . 127
6.2.2 Improving Alert Quality by Using Context . . . . . . . . . . . . . 129
6.2.3 Signature Language . . . . . . . . . . . . . . . . . . . . . . . . . . 131
6.2.4 Applications . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 132
6.3 Evaluation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 137
6.3.1 Data . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 137
6.3.2 Di(cid:14)culties. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 138
6.3.3 Performance . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 139
6.3.4 Operational Experiences . . . . . . . . . . . . . . . . . . . . . . . . 141
6.4 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 143
7 Conclusion 145
7.1 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 145
7.2 Outlook . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 148
iii
Contents
iv