Reverse Engineering Packet Structures from Network Traces by Segment-based Alignment

222 Pages·2017·1.42 MB·English

by  Othman Esoul

Reverse Engineering Packet Structures
from Network Traces by Segment-based
Alignment
Thesis submitted for the degree of
Doctor of Philosophy
at the University of Leicester
by
Othman Esoul
Department of Informatics
University of Leicester
July 2018

To my loving parents

Reverse Engineering Packet Structures from Network
Traces by Segment-based Alignment
Othman Esoul
Manyapplicationsinsecurity, fromunderstandingunfamiliarprotocolstofuzz-testingand
guardingagainstpotentialattacks,relyonanalysingnetworkprotocols. Inmanysituations
wecannotrelyonaccesstoaspecificationorevenanimplementationoftheprotocol,and
mustinsteadrelyonrawnetworkdata“sniffed”fromthenetwork. Whenthisisthecase,one
ofthekeychallengesistodiscernfromtherawdatatheunderlyingpacketstructures–atask
thatiscommonlycarriedoutbytwosteps: messageclustering,andmessageAlignment.
Clusteringqualityiscriticallycontingentupontheselectionoftherightparameters. Inthis
thesis, weexperimentally investigated twoaspects: 1) theeffect ofdifferent parameterson
clustering,and2)whethersuitableparameterconfigurationforclusteringcanbeinferredfor
undocumented protocols (when messages classes are unavailable). In this thesis, we have
quantified the impact of specific parameters on clustering, and used clustering validation
measures to predict parameter configurations with high clustering accuracy. Our results
indicate that: 1) The choice of the distance measure and the message length has the most
substantial impact on cluster accuracy. 2) The Ball-Hall intrinsic validation measure has
yieldedthebestresultsinpredictingsuitableparameterconfigurationforclustering.
Whileclusteringisusedtodetectmessagetypes(similargroups)withinadataset,sequence
alignmentalgorithmsareoftenusedtodetecttheprotocolmessagestructure(fieldpartition-
ing). Forthis,mostapproacheshaveusedvariantsoftheNeedleman-Wunschalgorithmto
performbyte-wisealignment. However,theycansufferwhenmessagesareheterogeneous,
or in cases where protocol fields are separated by long variable fields. In this thesis, we
present analternative alignmentalgorithm knownas segment-based alignment. The results
indicatethatsegmented-basedalignmentcanproducehighlyaccurateresultsthantraditional
alignmenttechniquesespeciallywithlonganddiversenetworkpackets.

Preface
I hereby declare that except where specific reference is made to the work of others, the
contentsofthisdissertationareoriginalandhavenotbeensubmittedinwholeorinpartfor
considerationforanyotherdegreeorqualificationinthis,oranyotheruniversity.
Thisdissertationismyownworkandcontainsnothingwhichistheoutcomeofworkdonein
collaboration,exceptasspecifiedintheAcknowledgements.
Thenamesofallproductsreferredtointhisdissertationareacknowledgedasthetrademarks
oftheirrespectiveowners.
Othman Esoul
July 2018

Acknowledgements
Firstandforemost,Iwouldliketothankmysupervisor,Dr. NeilWalkinshaw. Itishisadvice,
encouragement,andenormouspatiencethathelpedmegetthisfar. Dr. Neilhastaughtme
howdoresearch,writeanacademicpaper,andgrowasareseracher. Hisenormousenergyfor
teaching, attending conferences,and performingresearchhavemade himaverybusyperson,
buthehasalwayshadtimeforourmeetings. Iwouldliketothankhimforhisoutstanding
supervision,andthetremendoussupportheofferedmethroughoutmyPhD.
I also would like to thank all my friends who helped during this period especially Salah
Albukhasheimandhiswifefortheirencouragementandendlesssupport. Itisimpossibleto
enumeratethegenerosityandwaysinwhichtheyhavehelpedduringthisjourny.
Finally, I am grateful to my family in Libya. To my parents for their unwavering love
and support for my education, and to my bothers and sisters for their encouragement and
dedicationtohelp.