Minimization of DDoS false alarm rate in Network Security

177 Pages·2017·3.41 MB·English

University of Windsor
Scholarship at UWindsor
Electronic Theses and Dissertations
2012
Minimization of DDoS false alarm rate in Network
Security; Refining fusion through correlation
Faisal Mahmood
Follow this and additional works at:https://scholar.uwindsor.ca/etd
Recommended Citation
Mahmood, Faisal, "Minimization of DDoS false alarm rate in Network Security; Refining fusion through correlation " (2012).
Electronic Theses and Dissertations. 4829.
https://scholar.uwindsor.ca/etd/4829
This online database contains the full-text of PhD dissertations and Masters’ theses of University of Windsor students from 1954 forward. These
documents are made available for personal study and research purposes only, in accordance with the Canadian Copyright Act and the Creative
Commons license—CC BY-NC-ND (Attribution, Non-Commercial, No Derivative Works). Under this license, works must always be attributed to the
copyright holder (original author), cannot be used for any commercial purposes, and may not be altered. Any other use would require the permission of
the copyright holder. Students may inquire about withdrawing their dissertation and/or thesis from this database. For additional inquiries, please
contact the repository administrator via email ([e-mail]) or by telephone at 519-253-3000ext. 3208.
Minimization of DDoS false alarm rate in Network Security; Refining fusion through
correlation
by
Faisal Mahmood
A Thesis
Submitted to the Faculty of Graduate Studies
through Computer Science
in Partial Fulfillment of the Requirements for
the Degree of Master of Science at the
University of Windsor
Windsor, Ontario, Canada
2012
© 2012 Faisal Mahmood
Minimization of DDoS false alarm rate in Network Security; Refining fusion through
correlation
by
Faisal Mahmood
APPROVED BY:
______________________________________________
Dr. Kemal Ertugrul Tepe, External reader
Department of Electrical and Computer Engineering
______________________________________________
Dr. Arunita Jaekel, Internal reader
School of Computer Science
______________________________________________
Dr. Robert Kent, Advisor
School of Computer Science
_____________________________________________
Dr. Subir Bandyopadyhay, Chair of Defense
School of Computer Science
September 13, 2012
DECLARATION OF ORIGINALITY
I hereby certify that I am the sole author of this thesis and that no part of this thesis has
been published or submitted for publication.
I certify that, to the best of my knowledge, my thesis does not infringe upon anyone’s
copyright nor violate any proprietary rights and that any ideas, techniques, quotations, or
any other material from the work of other people included in my thesis, published or
otherwise, are fully acknowledged in accordance with the standard referencing practices.
Furthermore, to the extent that I have included copyrighted material that surpasses the
bounds of fair dealing within the meaning of the Canada Copyright Act, I certify that I
have obtained a written permission from the copyright owner(s) to include such
material(s) in my thesis and have included copies of such copyright clearances to my
appendix.
I declare that this is a true copy of my thesis, including any final revisions, as approved
by my thesis committee and the Graduate Studies office, and that this thesis has not been
submitted for a higher degree to any other University or Institution.

iii
ABSTRACT
Intrusion Detection Systems are designed to monitor a network environment and generate
alerts whenever abnormal activities are detected. However, the number of these alerts can
be very large making their evaluation a difficult task for a security analyst. Alert
management techniques reduce alert volume significantly and potentially improve
detection performance of an Intrusion Detection System.
This thesis work presents a framework to improve the effectiveness and efficiency of an
Intrusion Detection System by significantly reducing the false positive alerts and
increasing the ability to spot an actual intrusion for Distributed Denial of Service attacks.
Proposed sensor fusion technique addresses the issues relating the optimality of decision-
making through correlation in multiple sensors framework. The fusion process is based
on combining belief through Dempster Shafer rule of combination along with associating
belief with each type of alert and combining them by using Subjective Logic based on
Jøsang theory. Moreover, the reliability factor for any Intrusion Detection System is also
addressed accordingly in order to minimize the chance of false diagnose of the final
network state. A considerable number of simulations are conducted in order to determine
the optimal performance of the proposed prototype.

iv
DEDICATION
To the Open Source Community...

v
ACKNOWLEDGEMENTS
I am sincerely and heartily grateful to my advisor, Professor Dr. Robert Kent, for
the support and guidance he showed me throughout my research work and dissertation
writing. I am sure it would have not been possible without his help. I owe an earnest
thankfulness to Dr. Akshai Kumar Aggarwal for providing me the opportunity to start my
graduate degree under his supervision. Besides I would also like to thank to my lovely
wife and sweet kids who boosted me morally though out my study work.

vi
TABLE OF CONTENTS
DECLARATION OF ORIGINALITY .............................................................................. iii
ABSTRACT ....................................................................................................................... iv
DEDICATION .....................................................................................................................v
ACKNOWLEDGEMENTS ............................................................................................... vi
LIST OF TABLES ...............................................................................................................x
LIST OF FIGURES .......................................................................................................... xii
CHAPTER
I. INTRODUCTION
Overview ...............................................................................................1
Problem Statement ................................................................................2
Research Questions ...............................................................................3
Methods ................................................................................................4
Contribution ..........................................................................................4
Thesis Outline .......................................................................................5
II. REVIEW OF LITERATURE
Brief Introduction .................................................................................6
Network Types ......................................................................................7
Intrusion Detection System - IDS .........................................................9
Datasets available for experiments .....................................................11
Computer attacks in DARPA 1999 evaluation dataset .......................14
Selection of Network Intrusion Detection Systems ............................14
Intrusion Detection System used for research work. ..........................18
Standardized efforts for representing alerts in IDS - IDMEF .............26
Alert Management Techniques ...........................................................27
Data Fusion Process for IDS...............................................................30
Fusion-Based IDS ...............................................................................33
Probability Theory ..............................................................................37
Dempster Shafer Theory -DST ...........................................................40
Mathematics Computation for D – S Theory .....................................44
Rules for the Combination of Evidence ..............................................46
Subjective Logic Theory .....................................................................48
vii
The Consensus Operator .....................................................................50
III. DESIGN AND METHODOLOGY
Sensor output and Data Collection .....................................................53
Alert Normalization ............................................................................54
Alert Preprocessing .............................................................................56
Alert Filtering .....................................................................................57
Multilevel correlation and Alert prioritization ...................................58
Alert Fusion Engine based on Dempster Shafer Theory ....................61
Dempster-Shafer Applied to Distributed Intrusion Detection ............73
Final Decision Rule (FDR) for Dempster Shafer Combination rule. .75
Alert Fusion Engine based on Subjective Logic .................................76
The Consensus Operator .....................................................................77
Dealing with uncertainty of the sensor ...............................................79
IV. ANALYSIS OF RESULTS
Result Analysis: ..................................................................................84
Analysis of multilevel correlation component. ...................................85
Analysis of fusion component ............................................................90
Discussion ...........................................................................................94
V. CONCLUSIONS AND RECOMMENDATIONS
Conclusion and summary of contribution: .........................................96
Recommendations and Future Work ..................................................98
APPENDICES
System Configuration and Experimental Test Bed ..................................................101
IDS Installation and Configuration ...........................................................................104
Various Datasets and Research Institutes .................................................................119
DARPA 1999 Attacks...............................................................................................121
J.1 Alert Filtering component for Snort sensor ................................131
J.2 Alert Filtering component for Bro sensor ...................................133
J.3 Multilevel Correlation for Snort sensor ......................................134
J.4 Multilevel Correlation for Bro sensor .........................................136
J.5 DS Rule of Combination – Demspter Shafer Theory .................137
J.6 Attack scenario through Subjective Logic – Jøsang ...................141
List of Abbreviation ..................................................................................................151
REFERENCES ...............................................................................................................153
viii
VITA AUCTORIS .........................................................................................................163

ix