The Art and Science of Analyzing Software Data

685 Pages·2015·14.71 MB·English

by  Christian Bird

The Art and Science of
Analyzing Software Data
This page intentionally left blank
The Art and Science of
Analyzing Software Data
Christian Bird
Microsoft Research, Redmond, WA, USA
Tim Menzies
North Carolina State University, Raleigh, NC, USA
Thomas Zimmermann
Microsoft Research, Redmond, WA, USA
AMSTERDAM (cid:129) BOSTON (cid:129) HEIDELBERG (cid:129) LONDON
NEW YORK (cid:129) OXFORD (cid:129) PARIS (cid:129) SAN DIEGO
SAN FRANCISCO (cid:129) SINGAPORE (cid:129) SYDNEY (cid:129) TOKYO
Morgan Kaufmann is an imprint of Elsevier
AcquiringEditor:ToddGreen
EditorialProjectManager:LindsayLawrence
ProjectManager:PunithavathyGovindaradjane
Designer:MarkRogers
MorganKaufmannisanimprintofElsevier
225WymanStreet,Waltham,MA02451,USA
Copyright©2015ElsevierInc.Allrightsreserved.
Nopartofthispublicationmaybereproducedortransmittedinanyformorbyanymeans,electronicor
mechanical,includingphotocopying,recording,oranyinformationstorageandretrievalsystem,without
permissioninwritingfromthepublisher.Detailsonhowtoseekpermission,furtherinformationaboutthe
Publisher’spermissionspoliciesandourarrangementswithorganizationssuchastheCopyrightClearanceCenter
andtheCopyrightLicensingAgency,canbefoundatourwebsite:www.elsevier.com/permissions.
ThisbookandtheindividualcontributionscontainedinitareprotectedundercopyrightbythePublisher(other
thanasmaybenotedherein).
Notices
Knowledgeandbestpracticeinthisfieldareconstantlychanging.Asnewresearchandexperiencebroadenour
understanding,changesinresearchmethods,professionalpractices,ormedicaltreatmentmaybecomenecessary.
Practitionersandresearchersmustalwaysrelyontheirownexperienceandknowledgeinevaluatingandusing
anyinformation,methods,compounds,orexperimentsdescribedherein.Inusingsuchinformationormethods
theyshouldbemindfuloftheirownsafetyandthesafetyofothers,includingpartiesforwhomtheyhavea
professionalresponsibility.
Tothefullestextentofthelaw,neitherthePublishernortheauthors,contributors,oreditors,assumeanyliability
foranyinjuryand/ordamagetopersonsorpropertyasamatterofproductsliability,negligenceorotherwise,or
fromanyuseoroperationofanymethods,products,instructions,orideascontainedinthematerialherein.
ISBN:978-0-12-411519-4
BritishLibraryCataloguinginPublicationData
AcataloguerecordforthisbookisavailablefromtheBritishLibrary
LibraryofCongressCataloging-in-PublicationData
AcatalogrecordforthisbookisavailablefromtheLibraryofCongress
ForinformationonallMKpublications
visitourwebsiteatwww.mkp.com
Contents
ListofContributors.......................................................................................................xix
CHAPTER 1 Past, Present, and Future of Analyzing Software Data..................1
1.1 Definitions...............................................................................................3
1.2 ThePast:Origins.......................................................................................5
1.2.1 Generation1:PreliminaryWork..........................................................6
1.2.2 Generation2:AcademicExperiments..................................................6
1.2.3 Generation3:IndustrialExperiments...................................................6
1.2.4 Generation4:DataScienceEverywhere...............................................7
1.3 PresentDay..............................................................................................7
1.4 Conclusion.............................................................................................11
Acknowledgments....................................................................................12
References.............................................................................................12
PART 1 TUTORIAL-TECHNIQUES 15
CHAPTER 2 Mining Patterns and Violations Using Concept Analysis............. 17
2.1 Introduction............................................................................................17
2.1.1 Contributions................................................................................19
2.2 PatternsandBlocks..................................................................................19
2.3 ComputingAllBlocks..............................................................................21
2.3.1 AlgorithminaNutshell...................................................................23
2.4 MiningShoppingCartswithColibri.............................................................25
2.5 Violations...............................................................................................27
2.6 FindingViolations....................................................................................28
2.7 TwoPatternsorOneViolation?...................................................................28
2.8 Performance...........................................................................................30
2.9 EncodingOrder.......................................................................................32
2.10 Inlining..................................................................................................32
2.11 RelatedWork..........................................................................................33
2.11.1 MiningPatterns.............................................................................33
2.11.2 MiningViolations..........................................................................34
2.11.3 PR-Miner.....................................................................................35
2.12 Conclusions............................................................................................36
Acknowledgments....................................................................................37
References.............................................................................................37
v
vi Contents
CHAPTER 3 Analyzing Text in Software Projects..................................... 39
3.1 Introduction............................................................................................39
3.2 TextualSoftwareProjectDataandRetrieval..................................................40
3.2.1 TextualData.................................................................................40
3.2.2 TextRetrieval................................................................................43
3.3 ManualCoding........................................................................................45
3.3.1 CodingProcess..............................................................................45
3.3.2 Challenges....................................................................................48
3.4 AutomatedAnalysis.................................................................................49
3.4.1 TopicModeling.............................................................................49
3.4.2 Part-of-SpeechTaggingandRelationshipExtraction..............................51
3.4.3 n-Grams.......................................................................................53
3.4.4 CloneDetection.............................................................................54
3.4.5 Visualization.................................................................................57
3.5 TwoIndustrialStudies..............................................................................60
3.5.1 NamingthePaininRequirementsEngineering:ARequirements
EngineeringSurvey........................................................................60
3.5.2 CloneDetectioninRequirementsSpecifications...................................65
3.6 Summary...............................................................................................70
References.............................................................................................71
CHAPTER 4 Synthesizing Knowledge from Software Development Artifacts..... 73
4.1 ProblemStatement...................................................................................73
4.2 ArtifactLifecycleModels..........................................................................74
4.2.1 Example:PatchLifecycle................................................................74
4.2.2 ModelExtraction...........................................................................75
4.3 CodeReview..........................................................................................76
4.3.1 MozillaProject..............................................................................76
4.3.2 WebKitProject..............................................................................77
4.3.3 BlinkProject.................................................................................77
4.4 LifecycleAnalysis...................................................................................77
4.4.1 MozillaFirefox..............................................................................77
4.4.2 WebKit........................................................................................80
4.4.3 Blink...........................................................................................81
4.5 OtherApplications...................................................................................83
4.6 Conclusion.............................................................................................83
References.............................................................................................83
Contents vii
CHAPTER 5 A Practical Guide to Analyzing IDE Usage Data ....................... 85
5.1 Introduction............................................................................................86
5.2 UsageDataResearchConcepts...................................................................87
5.2.1 WhatisUsageDataandWhyShouldWeAnalyzeit?............................87
5.2.2 SelectingRelevantDataontheBasisofaGoal.....................................88
5.2.3 PrivacyConcerns...........................................................................89
5.2.4 StudyScope..................................................................................90
5.3 HowtoCollectData.................................................................................90
5.3.1 EclipseUsageDataCollector............................................................91
5.3.2 MylynandtheEclipseMylynMonitor..............................................100
5.3.3 CodingSpectator...........................................................................102
5.3.4 BuilditYourselfforVisualStudio....................................................108
5.4 HowtoAnalyzeUsageData.....................................................................115
5.4.1 DataAnonymity...........................................................................115
5.4.2 UsageDataFormat.......................................................................116
5.4.3 MagnitudeAnalysis......................................................................116
5.4.4 CategorizationAnalysis.................................................................117
5.4.5 SequenceAnalysis........................................................................118
5.4.6 StateModelAnalysis....................................................................120
5.4.7 TheCriticalIncidentTechnique.......................................................122
5.4.8 IncludingDatafromOtherSources..................................................123
5.5 LimitsofWhatYouCanLearnfromUsageData...........................................124
5.6 Conclusion...........................................................................................125
5.7 CodeListings........................................................................................125
Acknowledgments..................................................................................136
References............................................................................................136
CHAPTER 6 Latent Dirichlet Allocation: Extracting Topics from Software
Engineering Data...........................................................139
6.1 Introduction..........................................................................................140
6.2 ApplicationsofLDAinSoftwareAnalysis..................................................141
6.3 HowLDAWorks...................................................................................142
6.4 LDATutorial.........................................................................................145
6.4.1 Materials....................................................................................145
6.4.2 AcquiringSoftware-EngineeringData...............................................146
6.4.3 TextAnalysisandDataTransformation.............................................146
6.4.4 ApplyingLDA.............................................................................149
6.4.5 LDAOutputSummarization...........................................................149
viii Contents
6.5 PitfallsandThreatstoValidity..................................................................153
6.5.1 CriterionValidity..........................................................................155
6.5.2 ConstructValidity.........................................................................155
6.5.3 InternalValidity...........................................................................155
6.5.4 ExternalValidity..........................................................................156
6.5.5 Reliability...................................................................................157
6.6 Conclusions..........................................................................................157
References............................................................................................157
CHAPTER 7 Tools and Techniques for Analyzing Product and Process Data ...161
7.1 Introduction..........................................................................................162
7.2 ARationalAnalysisPipeline....................................................................163
7.2.1 GettingtheData...........................................................................163
7.2.2 Selecting....................................................................................164
7.2.3 Processing..................................................................................165
7.2.4 Summarizing...............................................................................167
7.2.5 Plumbing....................................................................................168
7.3 SourceCodeAnalysis.............................................................................168
7.3.1 Heuristics...................................................................................169
7.3.2 LexicalAnalysis..........................................................................169
7.3.3 ParsingandSemanticAnalysis........................................................173
7.3.4 Third-PartyTools.........................................................................173
7.4 CompiledCodeAnalysis.........................................................................179
7.4.1 AssemblyLanguage......................................................................179
7.4.2 MachineCode.............................................................................181
7.4.3 DealingwithNameMangling.........................................................182
7.4.4 ByteCode...................................................................................183
7.4.5 DynamicLinking.........................................................................185
7.4.6 Libraries.....................................................................................186
7.5 AnalysisofConfigurationManagementData...............................................188
7.5.1 ObtainingRepositoryData.............................................................188
7.5.2 AnalyzingMetadata......................................................................190
7.5.3 AnalyzingTimeSeriesSnapshots....................................................193
7.5.4 AnalyzingaCheckedOutRepository...............................................195
7.5.5 CombiningFileswithMetadata.......................................................196
7.5.6 AssemblingRepositories................................................................198
7.6 DataVisualization..................................................................................199
7.6.1 Graphs.......................................................................................199
7.6.2 DeclarativeDiagrams....................................................................203
Contents ix
7.6.3 Charts........................................................................................204
7.6.4 Maps.........................................................................................206
7.7 ConcludingRemarks...............................................................................208
References............................................................................................209
PART 2 DATA/PROBLEM FOCUSSED 213
CHAPTER 8 Analyzing Security Data ..................................................215
8.1 Vulnerability.........................................................................................215
8.1.1 Exploits......................................................................................217
8.2 SecurityData“Gotchas”..........................................................................217
8.2.1 Gotcha#1.HavingVulnerabilitiesisNormal......................................217
8.2.2 Gotcha#2.“MoreVulnerabilities”DoesnotAlwaysMean“LessSecure”218
8.2.3 Gotcha#3.Design-LevelFlawsarenotUsuallyTracked.......................219
8.2.4 Gotcha#4.SecurityisNegativelyDefined.........................................219
8.3 MeasuringVulnerabilitySeverity...............................................................220
8.3.1 CVSSOverview...........................................................................220
8.3.2 ExampleCVSSApplication............................................................221
8.3.3 CriticismsoftheCVSS..................................................................221
8.4 MethodofCollectingandAnalyzingVulnerabilityData.................................222
8.4.1 Step1.TraceReportedVulnerabilitiesBacktoFixes............................222
8.4.2 Step2.AggregateSourceControlLogs.............................................223
8.4.3 Step3a.DetermineVulnerabilityCoverage........................................224
8.4.4 Step3c.ClassifyAccordingtoEngineeringMistake............................224
8.5 WhatSecurityDatahasToldUsThusFar....................................................225
8.5.1 VulnerabilitieshaveSocio-TechnicalElements...................................226
8.5.2 VulnerabilitieshaveLong,ComplexHistories....................................227
8.6 Summary..............................................................................................228
References............................................................................................228
CHAPTER 9 A Mixed Methods Approach to Mining Code Review Data:
Examples and a Study of Multicommit Reviews
and Pull Requests .........................................................231
9.1 Introduction..........................................................................................232
9.2 MotivationforaMixedMethodsApproach..................................................232
9.3 ReviewProcessandData.........................................................................233
9.3.1 SoftwareInspection......................................................................233
9.3.2 OSSCodeReview........................................................................234
9.3.3 CodeReviewatMicrosoft..............................................................234
9.3.4 Google-BasedGerritCodeReview...................................................234