Smart Log Data Analytics: Techniques for Advanced Security Analysis

210 Pages·2021·8.123 MB·English

by  Florian Skopik, Markus Wurzenberger, Max Landauer

Florian Skopik
Markus Wurzenberger
Max Landauer
Smart
Log Data
Analytics
Techniques for Advanced Security
Analysis
Smart Log Data Analytics
Florian Skopik • Markus Wurzenberger
Max Landauer
Smart Log Data Analytics
Techniques for Advanced Security Analysis
FlorianSkopik MarkusWurzenberger
CenterforDigitalSafety&Security CenterforDigitalSafety&Security
AustrianInstituteofTechnology AustrianInstituteofTechnology
Vienna,Austria Vienna,Austria
MaxLandauer
CenterforDigitalSafety&Security
AustrianInstituteofTechnology
Vienna,Austria
ISBN978-3-030-74449-6 ISBN978-3-030-74450-2 (eBook)
https://doi.org/10.1007/978-3-030-74450-2
©SpringerNatureSwitzerlandAG2021
Thisworkissubjecttocopyright.AllrightsarereservedbythePublisher,whetherthewholeorpartof
thematerialisconcerned,specificallytherightsoftranslation,reprinting,reuseofillustrations,recitation,
broadcasting,reproductiononmicrofilmsorinanyotherphysicalway,andtransmissionorinformation
storageandretrieval,electronicadaptation,computersoftware,orbysimilarordissimilarmethodology
nowknownorhereafterdeveloped.
Theuseofgeneraldescriptivenames,registerednames,trademarks,servicemarks,etc.inthispublication
doesnotimply,evenintheabsenceofaspecificstatement,thatsuchnamesareexemptfromtherelevant
protectivelawsandregulationsandthereforefreeforgeneraluse.
Thepublisher,theauthors,andtheeditorsaresafetoassumethattheadviceandinformationinthisbook
arebelievedtobetrueandaccurateatthedateofpublication.Neitherthepublishernortheauthorsor
theeditorsgiveawarranty,expressedorimplied,withrespecttothematerialcontainedhereinorforany
errorsoromissionsthatmayhavebeenmade.Thepublisherremainsneutralwithregardtojurisdictional
claimsinpublishedmapsandinstitutionalaffiliations.
ThisSpringerimprintispublishedbytheregisteredcompanySpringerNatureSwitzerlandAG.
Theregisteredcompanyaddressis:Gewerbestrasse11,6330Cham,Switzerland
Preface
Prudent event monitoring and logging are the only means that allow system
operatorsandsecurityteamstotrulyunderstandhowcomplexsystemsareutilized.
Logdataareessentialtodetectintrusionattemptsinrealtimeorforensicallywork
throughpreviousincidentstocreateavitalunderstandingofwhathashappenedin
thepast.
Today,almosteveryorganizationalreadylogsdatatosomeextent,andalthough
itmeansaconsiderableefforttoestablishasecureandrobustlogginginfrastructure
aswellasthegoverningmanagementpoliciesandprocesses,basicandrawlogging
is comparatively simple in contrast to log analysis. The latter is an art of its own,
whichnotmanyorganizationsknowhowtomaster.Logdataareextremelydiverse
and processing them is unfortunately quite complex. There is no standard that
dictatesthegranularity,structure,andlevelofdetailsthatlogeventsprovide.There
isnoagreementwhatlogscompriseandhowtheyareformatted.
Facingthesefacts,itisastonishingthatnotmuchliteraturethatconcernslogging
in computer networks exists. And although there are at least some great books
out there, it is not enough. On the one side some existing literature did not age
well (certain topics are simply outdated after several years as technologies evolve
and newer concepts such as bring your own device, cloud computing, and IoT hit
the market), and on the other side some relevant topics are simply not sufficiently
covered yet, especially when it comes to complex—and sometimes dry—log data
analytics.
WetakeDr.Chuvakin’s(etal.)book‘LoggingandLogManagement’from2013
as a starting point. This is a great book that covers all the essential basics from
a technical and management point of view, such as what log data actually are,
how to collect log data, and how to perform simple analysis, and also explains
filtering, normalization, and correlation as well as reporting of findings. It further
elaborates on available tools and helps the practitioner to adopt state-of-the-art
loggingtechnologiesquickly.However,whileitprovidesaprofoundandimportant
basisforeveryonewhoisinchargeofsettingupalogginginfrastructure,thisbook
does not go far enough for certain audiences. The authors essentially stop there,
where our book starts. We assume, the reader of our book knows the basics and
v
vi Preface
hasalreadycollectedexperiencewithloggingtechnologies.Wefurtherassume,the
reader spent some serious thoughts on what to log, how to log and why to log—
andthatcommonchallengesregardingthecollectionoflogdatahavebeensolved,
includingtimesynchronization,accesscontrolforlogagents,logbuffering/rotation,
and consistency assurance. For all these topics, technical (and vendor-specific)
documentationexists.
Wepickupthereaderatthispoint,wheretheyaskthequestionwhattodowith
thecollectedlogsbeyondsimpleoutlierdetectionandstaticrule-basedevaluations.
Here, we enter new territory and provide insights into latest research results and
promising approaches. We provide an outlook on what kind of log analysis is
actually possible with the appropriate algorithms and provide the accompanying
open-source software solution AMiner1 to try out cutting-edge research methods
fromthisbookonowndata!
This book discusses important extensions to the state of the art. Its content is
meant for academics, researchers, and graduate students—as well as any forward-
thinkingpractitionerinterestedto:
(cid:129) Learn how to parse and normalize log data in a scalable way, i.e., without
inefficientlinearlistsofregularexpressions
(cid:129) Learn how to efficiently cluster log events in real time, i.e., create clusters
incrementallywhilelogeventsarrive
(cid:129) Learn how to characterize systems and create behavior profiles with the use of
clustermaps
(cid:129) Learnhowtoautomaticallycreatecorrelationrulesfromlogdata
(cid:129) Learnhowtotracksystembehaviortrendsovertime
In the last decade, numerous people supported this project. We would like to
specifically thank Roman Fiedler as one of the founders of the AMiner project,
WolfgangHotwagnerfortheinvaluableinfrastructureandimplementationsupport,
GeorgHöldforhiscontributionstotheadvanceddetectors,andErnstLeierzopffor
softwarequalityimprovements.
Vienna,Austria FlorianSkopik
Vienna,Austria MarkusWurzenberger
Vienna,Austria MaxLandauer
March2021
1https://github.com/ait-aecid.
Acknowledgments
This work has been financially supported by the Austrian Research Promotion
Agency FFG and the European Union’s FP7 and H2020 programs in course of
severalresearchprojectsfrom2011to2021.
vii
Contents
1 Introduction .................................................................. 1
1.1 StateoftheArtinSecurityMonitoringandAnomalyDetection .... 1
1.2 CurrentTrendsand... ................................................. 3
1.3 ... FutureChallenges .................................................. 5
1.4 LogDataAnalysis:TodayandTomorrow ............................ 7
1.5 SmartLogDataAnalytics:StructureoftheBook.................... 9
1.6 TryItOut:Hands-onExamplesThroughouttheBook............... 10
2 SurveyonLogClusteringApproaches .................................... 13
2.1 Introduction............................................................. 13
2.2 SurveyBackground..................................................... 14
2.3 SurveyMethod ......................................................... 19
2.4 SurveyResults.......................................................... 22
2.5 Conclusion.............................................................. 41
3 Incremental Log Data Clustering for Processing Large
AmountsofDataOnline .................................................... 43
3.1 Introduction............................................................. 43
3.2 ConceptforIncrementalClustering ................................... 45
3.3 OutlookandFurtherDevelopment .................................... 54
3.4 TryItOut ............................................................... 55
4 GeneratingCharacter-BasedTemplatesforLogData................... 63
4.1 Introduction............................................................. 63
4.2 ConceptforGeneratingCharacter-BasedTemplates ................. 65
4.3 ClusterTemplateGeneratorAlgorithms .............................. 67
4.4 OutlookandFurtherDevelopment .................................... 76
4.5 TryitOut................................................................ 77
5 TimeSeriesAnalysisforTemporalAnomalyDetection................. 83
5.1 Introduction............................................................. 83
5.2 ConceptforDynamicClusteringandAD............................. 85
5.3 ClusterEvolution....................................................... 87
ix
x Contents
5.4 TimeSeriesAnalysis................................................... 91
5.5 Example................................................................. 94
6 AECID:ALight-WeightLogAnalysisApproachforOnline
AnomalyDetection........................................................... 99
6.1 Introduction............................................................. 99
6.2 TheAECIDApproach ................................................. 100
6.3 SystemDeploymentandOperation.................................... 106
6.4 ApplicationScenarios.................................................. 108
6.5 TryItOut ............................................................... 109
7 AConceptforaTree-BasedLogParserGenerator...................... 131
7.1 Introduction............................................................. 131
7.2 Tree-BasedParserConcept ............................................ 132
7.3 AECID-PG:Tree-BasedLogParserGenerator....................... 136
7.4 OutlookandFurtherApplication...................................... 141
7.5 TryitOut................................................................ 142
8 VariableTypeDetectorforStatisticalAnalysisofLogTokens ......... 151
8.1 Introduction............................................................. 151
8.2 VariableTypeDetectorConcept....................................... 152
8.3 VariableTypeDetectorAlgorithm..................................... 154
8.4 TryItOut ............................................................... 165
9 FinalRemarks................................................................ 173
A GettingStartedwithAIT’sAMiner ....................................... 175
A.1 Requirements........................................................... 175
A.2 Installation.............................................................. 176
A.3 FirstVerySimpleConfiguration....................................... 177
A.4 DetectingAnomaliesinCombinationsofDifferentLog
LineFields.............................................................. 183
B DescriptionoftheAITLogDataSet(AIT-LDSv1.1) .................... 189
B.1 TestbedDesign ......................................................... 189
B.2 LogFiles................................................................ 190
B.3 Attacks.................................................................. 191
C GoingFurther:IntegratingAMinerwithSIEMSolutions.............. 193
C.1 Interfaces................................................................ 193
C.2 ELKStack .............................................................. 195
C.3 QRadar.................................................................. 198
References......................................................................... 201
About the Authors
Florian Skopik is head of the cyber security research
program at the Austrian Institute of Technology (AIT)
with a team comprising around 30 people. He spent
10+yearsincybersecurityresearch,before,andpartly
in parallel, another 15 years in software development.
Nowadays, he coordinates national and large-scale
international research projects as well as the overall
research direction of the team. His main interests
are centered on critical infrastructure protection,
smart grid security, and national cyber security and
defense. Since 2018, Florian further works as ISO
27001 Lead Auditor. Before joining AIT in 2011,
Florian was with the Distributed Systems Group at
the Vienna University of Technology as a research
assistant and postdoctoral research scientist from
2007 to 2011, where he was involved in a number
of international research projects dealing with cross-
organizational collaboration over the Web. In context
of these projects, he also finished his PhD studies.
Florian further spent a sabbatical at IBM Research
India in Bangalore for several months. He published
morethan125scientificconferencepapersandjournal
articles and holds more than 50 industry recognized
security certifications, including CISSP, CISM, CISA,
CRISC, and CCNP Security. In 2017, he finished a
professional degree in advanced computer security at
the Stanford University, USA. Florian is member of
various conference program committees and editorial
boards and standardization groups, such as ETSI
TC Cyber and OASIS CTI. He frequently serves as
xi