Developing a high-accuracy cross platform Host-Based Intrusion Detection System capable of

215 Pages·2014·3.05 MB·English

Developing a high-accuracy cross platform
Host-Based Intrusion Detection System capable
of reliably detecting zero-day attacks
Gideon Creech
B.E.(Elec) Hons, M.Eng.Sc.
A thesis in fulfilment of the requirements for the degree of
Doctor of Philosophy
School of Engineering and Information Technology,
University College,
The University of New South Wales,
Australian Defence Force Academy.
2013
PLEASE TYPE
THE UNIVERSITY OF NEW SOUTH WALES
Thesis/Dissertation Sheet

Surname or Family name:

First name: Other name/s:

Abbreviation for degree as given in the University calendar:

School: Faculty:

Title:

Abstract 350 words maximum: (PLEASE TYPE)

Declaration relating to disposition of project thesis/dissertation

I hereby grant to the University of New South Wales or its agents the right to archive and to make available my thesis or dissertation in whole or in
part in the University libraries in all forms of media, now or here after known, subject to the provisions of the Copyright Act 1968. I retain all
property rights, such as patent rights. I also retain the right to use in future works (such as articles or books) all or part of this thesis or dissertation.

I also authorise University Microfilms to use the 350 word abstract of my thesis in Dissertation Abstracts International (this is applicable to doctoral
theses only).

.. . ... .
Signature Witness Date

The University recognises that there may be exceptional circumstances requiring restrictions on copying or conditions on use. Requests for
restriction for a period of up to 2 years must be made in writing. Requests for a longer period of restriction may be considered in exceptional
circumstances and require the approval of the Dean of Graduate Research.

FOR OFFICE USE ONLY Date of completion of requirements for Award:

THIS SHEET IS TO BE GLUED TO THE INSIDE FRONT COVER OF THE THESIS
c Copyright 2014
�
by
Gideon Creech
B.E.(Elec) Hons, M.Eng.Sc.
i
COPYRIGHT STATEMENT

‘I hereby grant the University of New South Wales or its agents the right to
archive and to make available my thesis or dissertation in whole or part in the
University libraries in all forms of media, now or here after known, subject to the
provisions of the Copyright Act 1968. I retain all proprietary rights, such as patent
rights. I also retain the right to use in future works (such as articles or books) all
or part of this thesis or dissertation.
I also authorise University Microfilms to use the 350 word abstract of my thesis in
Dissertation Abstract International (this is applicable to doctoral theses only).
I have either used no substantial portions of copyright material in my thesis or I
have obtained permission to use copyright material; where permission has not
been granted I have applied/will apply for a partial restriction of the digital copy of
my thesis or dissertation.'

Signed ...........................

Date ...........................

AUTHENTICITY STATEMENT

‘I certify that the Library deposit digital copy is a direct equivalent of the final
officially approved version of my thesis. No emendation of content has occurred
and if there are any minor variations in formatting, they are the result of the
conversion to digital format.’

Signed ...........................

Date ...........................
ORIGINALITY STATEMENT

‘I hereby declare that this submission is my own work and to the best of my
knowledge it contains no materials previously published or written by another
person, or substantial proportions of material which have been accepted for the
award of any other degree or diploma at UNSW or any other educational
institution, except where due acknowledgement is made in the thesis. Any
contribution made to the research by others, with whom I have worked at
UNSW or elsewhere, is explicitly acknowledged in the thesis. I also declare that
the intellectual content of this thesis is the product of my own work, except to
the extent that assistance from others in the project's design and conception or
in style, presentation and linguistic expression is acknowledged.’

Signed ..............

Date ..............
I hereby declare that this submission is my own work and to
the best of my knowledge it contains no materials previously
published or written by another person, or substantial pro-
portions of material which have been accepted for the award
of any other degree or diploma at UNSW or any other ed-
ucational institution, except where due acknowledgement is
made in the thesis. Any contribution made to the research by
others, with whom I have worked at UNSW or elsewhere, is
explicitly acknowledged in the thesis.
I also declare that the intellectual content of this thesis is the
product of my own work, except to the extent that assistance
from others in the project’s design and conception or in style,
presentation and linguistic expression is acknowledged.
Gideon Creech
ii
Abstract
Current anomaly host-based intrusion detection systems are limited in accuracy
with any increase in detection rate resulting in a corresponding increase in false
alarm rate. Furthermore, present technology is largely limited in scope to the Linux
operating system, with the popular Windows family of computers forced to rely on
signature-based protection schemes.
This thesis investigates the development of a new approach to host-based intru-
siondetectionsystemdesignwiththespecificaimsofimprovingperformancebeyond
that of existing technology and developing a cross platform approach to intrusion
detection. This research has made three original and significant contributions to the
field, and represents a marked advance in the body of knowledge.
The first major contribution is the development of a new semantic approach to
system call data processing, allowing the creation of host-based intrusion detection
systems for the Linux operating system which perform significantly better than
existing approaches. Performance was evaluated against existing datasets and also
against a new modern dataset designed as part of this research.
The second key contribution is the development of a new theory which allows
the deployment of traditional system call centric Linux anomaly-based intrusion de-
tection systems on the Windows operating system for the first time. This significant
technological advance means that protection against zero-day attacks is now possi-
ble on this operating system for the first time. These results were tested using a
second new dataset designed as part of this research.
The final key contribution of this thesis is the development of a new attack
methodology which is able to bypass traditional Windows signature-based defences
without any obfuscation. The revelation of this new attack technology is an impor-
tant contribution in and of itself as it allows the community of researchers worldwide
to address this important weakness in current approaches. Notwithstanding this
threat, host-based intrusion detection systems which use the first two new theories
outlined in this thesis are shown to be able to detect this new attack class with
a high degree of accuracy, allowing e↵ective protection and significantly mitigating
this threat.
iii
For my Mother.
iv
Acknowledgements
Research is a thrilling pursuit, but it does not occur in a vacuum and it is seldom
the named author alone who has invested large amounts of time and energy in
a published work. The people acknowledged here have played a huge role in the
conductofthisresearchproject,andtheyhavemyheartfeltthanksandappreciation.
I would like to take this opportunity to thank the following people for their
support and assistance:
Professor Jiankun Hu, my research supervisor, for agreeing to take me on as
a student and assisting throughout my research. I could not have asked for a better
introduction to the academic process or a better guide for those first critical steps
on the research path.
Dr Lawrie Brown, my co-supervisor, for his support and mentoring over many
years. I first encountered Dr Brown when a Masters student, and have consequently
been lucky enough to receive the benefit of his wisdom and experience for quite some
time. I could not have asked for a better scholarly role model than Dr Brown, and
am truly grateful for his involvement.
Lieutenant-Commander Stefaan De Brauwer, RAN, for taking a chance
on me and posting me to full time study. Stefaan is in many ways the military
counterpart to Dr Brown’s scholarly mentoring, and I am extremely lucky to have
had such an excellent career manager.
Beverley Humphrey and Dr George Humphrey, my grandparents and
career scientists of the highest calibre, for introducing me to the world of research
before I could talk, and encouraging me to always ask “Why?”.
Gabrielle Le Grand, my wife and partner, for her encouragement, love and
support, both for my research and in everything I do. I would not be the person I
am today without her, and my mind would still be asleep. Quite simply, I could not
have done this research without her by my side.
Finally, Sonia MacDougall, my late mother - never forgotten and always in
my heart.
v
Publications
The following papers were produced in the course of this Ph.D. research:
Accepted for Publication
Refereed Journal Articles
G. Creech and J. Hu. A Semantic Approach to Host-based Intrusion Detection
Systems Using Contiguous and Discontiguous System Call Patterns. Computers,
IEEE Transactions on, PP(99):11, 2013.
Conferences
G. Creech and J. Hu. Generation of a new IDS test dataset: Time to retire the
KDDcollection. InWireless Communications and Networking Conference (WCNC),
2013 IEEE, pages 44874492, 2013.
G. Creech and F. Jiang. The application of extreme learning machines to the net-
work intrusion detection problem. AIP Conference Proceedings, 1479(1):15061511,
2012.
G. Creech and F. Jiang. Semantics based multi-layered networks for spam email
detection. AIP Conference Proceedings, 1479(1):15181523, 2012.
Under Review or Pending Submission
G. Creech and J. Hu. A New Virtual Kernel Theory Allowing the Application of
System Call Intrusion Detection Analysis to the Windows OS. Manuscript submit-
ted to IEEE Transactions on Computers for peer review.
G. Creech and J. Hu. A New Theory of Hacking Attacks - Stealth Attack Tech-
niques and their Detection in the Windows OS. To be submitted to IEEE Transac-
tions on Computers for peer review.
vi