University of Windsor Scholarship at UWindsor Electronic Theses and Dissertations 2012 Minimization of DDoS false alarm rate in Network Security; Refining fusion through correlation Faisal Mahmood Follow this and additional works at:https://scholar.uwindsor.ca/etd Recommended Citation Mahmood, Faisal, "Minimization of DDoS false alarm rate in Network Security; Refining fusion through correlation " (2012). Electronic Theses and Dissertations. 4829. https://scholar.uwindsor.ca/etd/4829 This online database contains the full-text of PhD dissertations and Masters’ theses of University of Windsor students from 1954 forward. These documents are made available for personal study and research purposes only, in accordance with the Canadian Copyright Act and the Creative Commons license—CC BY-NC-ND (Attribution, Non-Commercial, No Derivative Works). Under this license, works must always be attributed to the copyright holder (original author), cannot be used for any commercial purposes, and may not be altered. Any other use would require the permission of the copyright holder. Students may inquire about withdrawing their dissertation and/or thesis from this database. For additional inquiries, please contact the repository administrator via email ([email protected]) or by telephone at 519-253-3000ext. 3208. Minimization of DDoS false alarm rate in Network Security; Refining fusion through correlation by Faisal Mahmood A Thesis Submitted to the Faculty of Graduate Studies through Computer Science in Partial Fulfillment of the Requirements for the Degree of Master of Science at the University of Windsor Windsor, Ontario, Canada 2012 © 2012 Faisal Mahmood Minimization of DDoS false alarm rate in Network Security; Refining fusion through correlation by Faisal Mahmood APPROVED BY: ______________________________________________ Dr. Kemal Ertugrul Tepe, External reader Department of Electrical and Computer Engineering ______________________________________________ Dr. Arunita Jaekel, Internal reader School of Computer Science ______________________________________________ Dr. Robert Kent, Advisor School of Computer Science _____________________________________________ Dr. Subir Bandyopadyhay, Chair of Defense School of Computer Science September 13, 2012 DECLARATION OF ORIGINALITY I hereby certify that I am the sole author of this thesis and that no part of this thesis has been published or submitted for publication. I certify that, to the best of my knowledge, my thesis does not infringe upon anyone’s copyright nor violate any proprietary rights and that any ideas, techniques, quotations, or any other material from the work of other people included in my thesis, published or otherwise, are fully acknowledged in accordance with the standard referencing practices. Furthermore, to the extent that I have included copyrighted material that surpasses the bounds of fair dealing within the meaning of the Canada Copyright Act, I certify that I have obtained a written permission from the copyright owner(s) to include such material(s) in my thesis and have included copies of such copyright clearances to my appendix. I declare that this is a true copy of my thesis, including any final revisions, as approved by my thesis committee and the Graduate Studies office, and that this thesis has not been submitted for a higher degree to any other University or Institution. iii ABSTRACT Intrusion Detection Systems are designed to monitor a network environment and generate alerts whenever abnormal activities are detected. However, the number of these alerts can be very large making their evaluation a difficult task for a security analyst. Alert management techniques reduce alert volume significantly and potentially improve detection performance of an Intrusion Detection System. This thesis work presents a framework to improve the effectiveness and efficiency of an Intrusion Detection System by significantly reducing the false positive alerts and increasing the ability to spot an actual intrusion for Distributed Denial of Service attacks. Proposed sensor fusion technique addresses the issues relating the optimality of decision- making through correlation in multiple sensors framework. The fusion process is based on combining belief through Dempster Shafer rule of combination along with associating belief with each type of alert and combining them by using Subjective Logic based on Jøsang theory. Moreover, the reliability factor for any Intrusion Detection System is also addressed accordingly in order to minimize the chance of false diagnose of the final network state. A considerable number of simulations are conducted in order to determine the optimal performance of the proposed prototype. iv DEDICATION To the Open Source Community... v ACKNOWLEDGEMENTS I am sincerely and heartily grateful to my advisor, Professor Dr. Robert Kent, for the support and guidance he showed me throughout my research work and dissertation writing. I am sure it would have not been possible without his help. I owe an earnest thankfulness to Dr. Akshai Kumar Aggarwal for providing me the opportunity to start my graduate degree under his supervision. Besides I would also like to thank to my lovely wife and sweet kids who boosted me morally though out my study work. vi TABLE OF CONTENTS DECLARATION OF ORIGINALITY .............................................................................. iii ABSTRACT ....................................................................................................................... iv DEDICATION .....................................................................................................................v ACKNOWLEDGEMENTS ............................................................................................... vi LIST OF TABLES ...............................................................................................................x LIST OF FIGURES .......................................................................................................... xii CHAPTER I. INTRODUCTION Overview ...............................................................................................1 Problem Statement ................................................................................2 Research Questions ...............................................................................3 Methods ................................................................................................4 Contribution ..........................................................................................4 Thesis Outline .......................................................................................5 II. REVIEW OF LITERATURE Brief Introduction .................................................................................6 Network Types ......................................................................................7 Intrusion Detection System - IDS .........................................................9 Datasets available for experiments .....................................................11 Computer attacks in DARPA 1999 evaluation dataset .......................14 Selection of Network Intrusion Detection Systems ............................14 Intrusion Detection System used for research work. ..........................18 Standardized efforts for representing alerts in IDS - IDMEF .............26 Alert Management Techniques ...........................................................27 Data Fusion Process for IDS...............................................................30 Fusion-Based IDS ...............................................................................33 Probability Theory ..............................................................................37 Dempster Shafer Theory -DST ...........................................................40 Mathematics Computation for D – S Theory .....................................44 Rules for the Combination of Evidence ..............................................46 Subjective Logic Theory .....................................................................48 vii The Consensus Operator .....................................................................50 III. DESIGN AND METHODOLOGY Sensor output and Data Collection .....................................................53 Alert Normalization ............................................................................54 Alert Preprocessing .............................................................................56 Alert Filtering .....................................................................................57 Multilevel correlation and Alert prioritization ...................................58 Alert Fusion Engine based on Dempster Shafer Theory ....................61 Dempster-Shafer Applied to Distributed Intrusion Detection ............73 Final Decision Rule (FDR) for Dempster Shafer Combination rule. .75 Alert Fusion Engine based on Subjective Logic .................................76 The Consensus Operator .....................................................................77 Dealing with uncertainty of the sensor ...............................................79 IV. ANALYSIS OF RESULTS Result Analysis: ..................................................................................84 Analysis of multilevel correlation component. ...................................85 Analysis of fusion component ............................................................90 Discussion ...........................................................................................94 V. CONCLUSIONS AND RECOMMENDATIONS Conclusion and summary of contribution: .........................................96 Recommendations and Future Work ..................................................98 APPENDICES System Configuration and Experimental Test Bed ..................................................101 IDS Installation and Configuration ...........................................................................104 Various Datasets and Research Institutes .................................................................119 DARPA 1999 Attacks...............................................................................................121 J.1 Alert Filtering component for Snort sensor ................................131 J.2 Alert Filtering component for Bro sensor ...................................133 J.3 Multilevel Correlation for Snort sensor ......................................134 J.4 Multilevel Correlation for Bro sensor .........................................136 J.5 DS Rule of Combination – Demspter Shafer Theory .................137 J.6 Attack scenario through Subjective Logic – Jøsang ...................141 List of Abbreviation ..................................................................................................151 REFERENCES ...............................................................................................................153 viii VITA AUCTORIS .........................................................................................................163 ix
Description: