Table Of ContentDissertation zur Erlangung des Grades eines Doktors der
Naturwissenschaften (Dr. rer. nat.)
Static Analysis of x86 Executables
Statische Analyse von Programmen in x86 Maschinensprache
Dipl.-Inf. Johannes Kinder
geb. in Mu¨nchen
Eingereicht am 24. September 2010
Referent: Prof. Dr. Helmut Veith
Korreferentin: Prof. Dr.-Ing. Mira Mezini
Pru¨fungstermin: 17. November 2010
Fachbereich Informatik
Technische Universita¨t Darmstadt
Darmstadt – 2010 – D17
Erkl¨arung zur Dissertation
Hiermit versichere ich, die vorliegende Dissertation ohne Hilfe Dritter nur mit
den angegebenen Quellen und Hilfsmitteln angefertigt zu haben. Alle Stellen,
die aus Quellen entnommen wurden, sind als solche kenntlich gemacht. Diese
Arbeithatingleicherodera¨hnlicherFormnochkeinerPru¨fungsbeho¨rdevorge-
legen.
Darmstadt,den24.September2010
JohannesKinder
Acknowledgments
Firstandforemost,Iwouldliketothankmyadvisor,HelmutVeith,forhiscon-
tinuing support and his valuable guidance in all aspects of academic life. He
gave me considerable freedom in developing my own research agenda and al-
ways trusted in my abilities. His uncomplicated way of leading our group al-
lowedeveryonetodotheirbestandmadeiteasytofocusonresearchandteach-
ingwithoutunnecessaryoverhead.
Furthermore, I thank my colleagues for fruitful scientific discussions and the
cheerfulhoursbothonandoffcampus. Evenintoughtimes,therewasalwaysan
exceptionalspiritofcompanionshipandmutualsupport. Inparticular,Iwould
liketothankFlorianZulegerforhiscontributionstoourworkoncontrolflowre-
construction;AndreasHolzerforourfrequentdiscussionsabouttheCPAframe-
work;him,VisarJanuzaj,andtheuntiringMichaelTautschnigforproofreading
onshortnotice.
Finally,IwanttothankmyparentsSusanneandHelmutKinderfortheirsup-
portandtheirfirmbeliefinme,andAnne-SophieDo¨rnbrackforbeingmyemo-
tional stronghold in the never-ending series of highs and lows that is graduate
research.
Darmstadt,November2010
JohannesKinder
3
Contents
Abstract 13
Zusammenfassung (German Abstract) 15
1 Introduction 17
1.1 BenefitsofBinaryAnalysis . . . . . . . . . . . . . . . . . . . . . . . 18
1.1.1 AlternativetoSourceCodeAnalysis . . . . . . . . . . . . . 19
1.1.2 AnalysiswithoutAccesstoSourceCode . . . . . . . . . . 22
1.2 ChallengesinBinaryAnalysis . . . . . . . . . . . . . . . . . . . . . 24
1.3 TraditionalDisassemblyandAnalysis . . . . . . . . . . . . . . . . 30
1.4 OverviewontheProposedMethod . . . . . . . . . . . . . . . . . . 34
1.5 Contributions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 39
2 An Intermediate Language for Executable Analysis 41
2.1 Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41
2.2 Syntax . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 43
2.2.1 Expressions . . . . . . . . . . . . . . . . . . . . . . . . . . . 44
2.2.2 BasicStatements . . . . . . . . . . . . . . . . . . . . . . . . 45
2.2.3 AbstractStatements . . . . . . . . . . . . . . . . . . . . . . 46
2.3 Types . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 47
2.4 Semantics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 48
2.5 ILPrograms . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 51
2.6 RelatedWork . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 53
5
Contents
3 Control Flow Analysis for Low Level Programs 57
3.1 Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 57
3.2 AWorkedExample . . . . . . . . . . . . . . . . . . . . . . . . . . . 60
3.3 ControlFlowSemantics . . . . . . . . . . . . . . . . . . . . . . . . 63
3.4 ControlFlowReconstructionbyAbstractInterpretation . . . . . . 65
3.4.1 TheResolveOperator . . . . . . . . . . . . . . . . . . . . . 66
3.4.2 AConstraintSystemforControlFlowAutomata . . . . . . 67
3.5 AlgorithmsforControlFlowReconstruction . . . . . . . . . . . . 74
3.5.1 GenericFixedPointAlgorithm . . . . . . . . . . . . . . . . 74
3.5.2 WorklistAlgorithm . . . . . . . . . . . . . . . . . . . . . . . 78
3.6 RelatedWork . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 82
4 Bounded Address Tracking 87
4.1 PrecisionRequirements . . . . . . . . . . . . . . . . . . . . . . . . 88
4.2 PartitionedMemoryModel . . . . . . . . . . . . . . . . . . . . . . 90
4.3 AbstractDomainofAddressValuations . . . . . . . . . . . . . . . 92
4.4 AbstractSemantics . . . . . . . . . . . . . . . . . . . . . . . . . . . 95
4.4.1 BoundedPathSensitivity . . . . . . . . . . . . . . . . . . . 96
4.4.2 AbstractExpressionEvaluation . . . . . . . . . . . . . . . . 98
4.4.3 AbstractPostOperator . . . . . . . . . . . . . . . . . . . . . 101
4.5 AbstractionofNondeterminism . . . . . . . . . . . . . . . . . . . . 104
4.6 ImplementationIssues . . . . . . . . . . . . . . . . . . . . . . . . . 106
4.6.1 RepresentingByte-AddressableAbstractMemory . . . . . 106
4.6.2 RegisterAliasing . . . . . . . . . . . . . . . . . . . . . . . . 108
4.7 RelatedWork . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 110
5 Disassembly and Static Analysis with Jakstab 113
5.1 GeneralArchitecture . . . . . . . . . . . . . . . . . . . . . . . . . . 113
5.1.1 SinglePassDisassemblyandAnalysis . . . . . . . . . . . . 114
5.1.2 SecondaryPost-ReconstructionAnalysis . . . . . . . . . . 116
5.1.3 ProgramRepresentation . . . . . . . . . . . . . . . . . . . . 117
5.1.4 ExecutionEnvironment . . . . . . . . . . . . . . . . . . . . 120
6
Contents
5.2 ModularImplementationofDifferentAnalyses . . . . . . . . . . . 123
5.2.1 ConfigurableProgramAnalysis . . . . . . . . . . . . . . . 124
5.2.2 ModificationstotheWorklistAlgorithm . . . . . . . . . . 127
5.2.3 BalancingSoundnessandCoverage . . . . . . . . . . . . . 129
5.2.4 CompositeAnalysiswithSelectiveMerging . . . . . . . . 132
5.3 AbstractDomainsinJakstab . . . . . . . . . . . . . . . . . . . . . . 134
5.3.1 LocationAnalysis . . . . . . . . . . . . . . . . . . . . . . . . 135
5.3.2 BoundedAddressTracking . . . . . . . . . . . . . . . . . . 136
5.3.3 ConstantPropagation . . . . . . . . . . . . . . . . . . . . . 139
5.3.4 StridedIntervalAnalysis . . . . . . . . . . . . . . . . . . . 141
5.3.5 CallStackAnalysis . . . . . . . . . . . . . . . . . . . . . . . 144
5.3.6 ForwardExpressionSubstitution . . . . . . . . . . . . . . . 145
5.3.7 LiveVariableAnalysis . . . . . . . . . . . . . . . . . . . . . 148
5.4 CodeTransformations . . . . . . . . . . . . . . . . . . . . . . . . . 149
5.5 RelatedWork . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 151
6 Experiments 155
6.1 AnalyzingUntrustedDriverBinaries . . . . . . . . . . . . . . . . . 155
6.1.1 Motivation . . . . . . . . . . . . . . . . . . . . . . . . . . . . 156
6.1.2 WindowsDriverModel . . . . . . . . . . . . . . . . . . . . 156
6.1.3 OSAbstractionandDriverHarness . . . . . . . . . . . . . 157
6.1.4 ExperimentalSetup . . . . . . . . . . . . . . . . . . . . . . . 161
6.1.5 Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 161
6.1.6 AnalysisofCOTSDriverBinaries . . . . . . . . . . . . . . . 168
6.2 Disassembly . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 170
6.2.1 ProcedureEntryPointHeuristic . . . . . . . . . . . . . . . 170
6.2.2 Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 171
7 Conclusions 175
Bibliography 179
Curriculum Vitae 199
7
List of Figures
1.1 Exampleofpossibleprocedurelayoutsinanexecutable. . . . . . . 25
1.2 Exampleofoverlappinginstructionsinx86machinecode. . . . . 28
1.3 Executiontraceoftheexampleforoverlappinginstructions. . . . 28
1.4 Challengesinbinaryanalysisandtheproposedsolutions. . . . . 35
3.1 Controlflowreconstructionexample. . . . . . . . . . . . . . . . . 61
3.2 GenericControlFlowReconstructionAlgorithm. . . . . . . . . . . 75
3.3 WorklistControlFlowReconstructionAlgorithm. . . . . . . . . . 79
3.4 Addinganunknownnode((cid:62))withunlabelededgesleadstoad-
ditionalpossiblevaluesfor attheindirectjump. . . . . . . . . . 83
x
4.1 Diagramofthelatticeofabstractaddressesandvalues Aˆ. . . . . . 93
4.2 Examplecodefragmentandfinalstatespace. . . . . . . . . . . . . 98
5.1 Unifieddisassemblyandanalysisarchitecture. . . . . . . . . . . . 114
5.2 SecondaryanalysisperformedonthereconstructedCFA. . . . . . 116
5.3 ThethreelevelsofprogramrepresentationinJakstab. . . . . . . . 118
5.4 DynamiclinkinginWindowsPEfiles. . . . . . . . . . . . . . . . . 120
5.5 CPA+algorithmfordeterminingthesetofreachablestates. . . . . 125
5.6 TheJakstabalgorithm,acontrolflowresolvingversionoftheCPA+
algorithm. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 128
6.1 Simplifiedcodefrom floppy.cwithabstractVSA/ESPstates. . . 166
9
ListofFigures
6.2 Resultsofanalyzing322driverbinariesfromastandardWindows
XP machine (a) using standard settings and (b) when ignoring
weakupdates. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 169
6.3 AverageresolverateofIDAProandJakstab(inheuristicmode). . 172
10
Description:of procedures. These building blocks are combined into an extensible program analysis archi- tecture, which is implemented in a novel binary analysis tool. The tool the software is compiled into binary format and shipped, however, users further From a specification of the instruction semantics,.
