DoubleTake: Fast and Precise Error Detection via Evidence-Based Dynamic Analysis

0.5 MB·

by  Tongping Liu

DOUBLETAKE: Fast and Precise Error Detection via
Evidence-Based Dynamic Analysis
Tongping Liu ∗ Charlie Curtsinger ∗ Emery D. Berger
Dept. ofComputerScience Dept. ofComputerScience CollegeofInformationand
UniversityofTexas GrinnellCollege ComputerSciences
atSanAntonio 11168thAve. UniversityofMassachusetts
SanAntonio,TX78249 Grinnell,IA50112 Amherst
[e-mail] [e-mail] Amherst,MA01003
[e-mail]
6
1 ABSTRACT Keywords
0
ProgramswritteninunsafelanguageslikeCandC++oftensuffer Dynamic Analysis, Software Quality, Testing, Debugging, Leak
2
fromerrorslikebufferoverflows, danglingpointers, andmemory Detection,BufferOverflowDetection,Use-After-FreeDetection

n leaks. Dynamic analysis tools like Valgrind can detect these er-
a rors,buttheiroverhead—primarilyduetothecostofinstrumenting 1. INTRODUCTION
J every memory read and write—makes them too heavyweight for
Dynamic analysis tools are widely used to find bugs in appli-
9 useindeployedapplicationsandmakestestingwiththempainfully
cations. They are popular among programmers because of their
2 slow. Theresultisthatmuchdeployedsoftwareremainssuscepti-
bletothesebugs,whicharenotoriouslydifficulttotrackdown. precision—formanyanalyses,theyreportnofalsepositives—and
] This paper presents evidence-based dynamic analysis, an ap- can pinpoint the exact location of errors, down to the individual
E lineofcode.Perhapsthemostprominentandwidelyuseddynamic
proachthatenableslightweightanalyses—under5%overheadfor
S analysistoolforC/C++binariesisValgrind[28]. Valgrind’smost
thesebugs—makingitpracticalforthefirsttimetoperformthese
s. analysesindeployedsettings. Thekeyinsightofevidence-based popularusecase,viaitsdefaulttool,MemCheck,canfindawide
rangeofmemoryerrors,includingbufferoverflows,use-after-free
c dynamicanalysisisthatforaclassoferrors,itispossibletoensure
[ thatevidencethattheyhappenedatsomepointinthepastremains errors,andmemoryleaks.
for later detection. Evidence-based dynamic analysis allows ex- Unfortunately, these dynamic analysis tools often impose sig-
1 nificantperformanceoverheadsthatmakethemprohibitiveforuse
ecution to proceed at nearly full speed until the end of an epoch
v
outside of testing scenarios. An extreme example is the widely-
(e.g.,aheavyweightsystemcall). Itthenexaminesprogramstate
2
usedtoolValgrind. AcrosstheSPECCPU2006benchmarksuite,
6 to check for evidence that an error occurred at some time during
Valgrind degrades performance by almost 17× on average (geo-
9 thatepoch. Ifso,itrollsbackexecutionandre-executesthecode
metricmean);itsoverheadrangesfrom4.5×and42.8×,makingit
7 withinstrumentationactivatedtopinpointtheerror.
oftentooslowtouseevenfortesting(seeTable1).
0 WepresentDOUBLETAKE,aprototypeevidence-baseddynamic
Whilefasterdynamicanalysisframeworksexistforfindingpar-
. analysis framework. DOUBLETAKE is practical and easy to de-
1 ticular errors (leveraging compiler support to reduce overhead),
ploy, requiring neither custom hardware, compiler, nor operating
0 theysacrificeprecisionwhilecontinuingtoimposesubstantialover-
system support. We demonstrate DOUBLETAKE’s generality and
6 headthatwouldimpedetheiruseindeployedsettings.Thecurrent
efficiencybybuildingdynamicanalysesthatfindbufferoverflows,
1 state-of-the-art,Google’sAddressSanitizer,detectsbufferoverflows
memory use-after-free errors, and memory leaks. Our evaluation
v: showsthatDOUBLETAKEisefficient,imposingjust4%overhead anduse-after-freeerrors,butslowsapplicationsbyaround30%[38].
i on average, making it the fastest such system to date. It is also AddressSanitizeralsoidentifiesmemoryleaksbutonlyattheend
X ofprogramexecution,whichisnotusefulforserversorotherlong-
precise:DOUBLETAKEpinpointsthelocationoftheseerrorstothe
r exactlineandmemoryaddresseswheretheyoccur,providingvalu- livedapplications.
a Becauseoftheiroverhead, thisclassofdynamicanalysistools
abledebugginginformationtoprogrammers.
cangenerallyonlybeusedduringtesting. However,theyarelim-
itedbydefinitiontotheexecutionsthataretestedpriortodeploy-
CategoriesandSubjectDescriptors ment. Even exhaustive testing regimes will inevitably fail to un-
covertheseerrors,whicharenotoriouslydifficulttodebug.
D.2.5[SoftwareEngineering]:TestingandDebugging–Debugging
Thispaperpresentsanapproachcalledevidence-baseddynamic
Aids, Monitors, Tracing; D.2.4 [Software Engineering]: Soft-
analysisthatisbasedonthefollowingkeyinsight:itisoftenpossi-
ware/ProgramVerification–Reliability;D.3.4[ProgrammingLan-
bletodiscoverevidencethatanerroroccurredorplantmarkersthat
guages]:Run-timeenvironments
ensure that such evidence exists. By combining evidence place-
mentwithcheckpointingandinfrequentchecking,wecanrunapp-
GeneralTerms plicationsatnearlyfullspeedinthecommoncase(noerrors). If
we find an error, we can use the checkpoint to roll back and re-
Performance,Reliability
executetheprogramwithinstrumentationactivatedtopinpointthe
∗This work was initiated and partially conducted while Liu and exactcauseoftheerror.
Curtsinger were PhD students at the University of Massachusetts Certainerrors,includingtheoneswedescribehere,naturallyex-
Amherst. hibitamonotonicityproperty:whenanerroroccurs,evidencethatit
happenedtendstoremainorevengrowsothatitcanbediscovered ValgrindExecutionTimeOverhead
Benchmark Overhead Benchmark Overhead
atalaterpointduringexecution. Whenthisevidenceisnotnatu-
400.perlbench 20.5× 458.sjeng 20.3×
rallyoccurringornotnaturallymonotonic,itcanbeforcedtoex-
401.bzip2 16.8× 471.omnetpp 13.9×
hibitthispropertybyplantingevidenceviawhatwecalltripwires 403.gcc 18.7× 473.astar 11.9×
toensurelaterdetection. Acanonicalexampleofsuchatripwire 429.mcf 4.5× 433.milc 11.0×
isarandomvalue,alsoknownasacanary,placedinunallocated 445.gobmk 28.9× 444.namd 24.9×
spacebetweenheapobjects[11]. Acorruptedcanaryisincontro- 456.hmmer 13.8× 450.dealII 42.8×
vertible evidence that a buffer overflow occurred at some time in
Table1: Valgrind’sexecutiontimeoverheadacrosstheSPEC
thepast.
benchmarksuite. Valgrindimposesonaverage17×overhead
Thispaperpresentsaprototypeevidence-baseddynamicanaly-
(geometric mean), making it prohibitively high for use in de-
sisframeworkcalled DOUBLETAKE thatlocatessucherrorswith
ploymentandquiteexpensiveevenfortestingpurposes.
extremely low overhead and no false positives. DOUBLETAKE
checkpointsprogramstateandperformsmostofitserroranalyses
Irrevocable)))
onlyatepochboundaries(whatwecallirrevocablesystemcalls)or
Snapshot) system)calls) Snapshot)
whensegfaultsoccur;theseoccurrelativelyinfrequently,amortiz-
Normal)execu2on) Check)errors)
ingDOUBLETAKE’soverhead.
IfDOUBLETAKEfindsevidenceofanerroratanepochbound-
aryorafterasegmentationviolation,itre-executestheapplication
Rollback) Error)detected)
fromthemostrecentcheckpoint. Duringre-execution, DOUBLE-
TAKE enablesinstrumentationtoletitpreciselylocatethesource
oftheerror.Forexample,forbufferoverflows,DOUBLETAKEsets Instrumented)re;execu2on)
hardwarewatchpointsonthetripwirememorylocationsthatwere
Report)errors)
found to be corrupted. During re-execution, DOUBLETAKE pin-
pointsexactlythepointwherethebufferoverflowoccurred.
We have implemented DOUBLETAKE as a drop-in library that
can be linked directly with the application, without the need to Figure1: Overviewof DOUBLETAKE inaction: executionis
modifycodeorevenrecompiletheprogram.DOUBLETAKEworks dividedintoepochsattheboundaryofirrevocablesystemcalls.
withouttheneedforcustomhardware,compiler,orOSsupport. Eachepochbeginsbytakingasnapshotofprogramstate.Exe-
UsingDOUBLETAKEasaframework,wehavebuiltthreediffer- cutionrunsatnearlyfull-speedduringepochs.Evidence-based
entanalysesthatattackthreeofthemostsalientproblemsforun- analysis takes place once an epoch ends, replaying execution
safecode: thebufferoverflowdetectordescribedaboveaswellas fromtheprevioussnapshotuntilitpinpointstheexactlocation
ause-after-freedetectorandmemoryleakdetector.Theseanalyses wheretheerrorisintroduced.Relatively-longepochsamortize
canallrunconcurrently. Byvirtueofbeingevidence-based,they thecostofsnapshotsandanalysis,keepingoverheadlow.
haveazerofalsepositiverate,preciselypinpointtheerrorlocation,
andoperatewithextremelylowoverhead:forexample,withDOU-
BLETAKE, buffer overflow analysis alone operates with just 3% implementationdetails.Section5evaluatesDOUBLETAKE’seffec-
overheadonaverage.Whenallthreeoftheseanalysesareenabled, tiveness,performance,andmemoryoverhead,andcomparesthese
DOUBLETAKE’saverageoverheadisunder5%. tothestateoftheart. Section6discusseslimitationsofevidence-
Foralloftheanalyseswehaveimplemented,DOUBLETAKEis basedanalysisandthedetectorsweimplement.Section7describes
the fastest detector of these errors to date, providing compelling keyrelatedworkandSection8concludes.
evidenceforthepromiseofevidence-baseddynamicanalyses. Its
overheadisalreadylowenoughtodramaticallyspeedtestingand 2. OVERVIEW
often low enough to enable the use of these formerly-prohibitive
DOUBLETAKE is an efficient dynamic analysis framework for
analysesindeployedsettings. Thisworkthuspromisestosignifi-
a class of errors that exhibit or can be forced to exhibit a mono-
cantlyextendthereachofdynamicanalyses.
tonicity property: evidence of the error is persistent and can be
Contributions gatheredafter-the-fact.WithDOUBLETAKE,programexecutionis
dividedintoepochs,duringwhichexecutionproceedsatfullspeed
Thecontributionsofthispaperarethefollowing:
(Figure 1). At the beginning of an epoch, DOUBLETAKE check-
pointsprogramstate.Epochsendonlywhentheapplicationissues
1. Itintroducesevidence-baseddynamicanalysis, anewanal-
anirrevocablesystemcall(e.g.,asocketread);mostsystemcalls
ysis technique that combines checkpointing with evidence
arenotirrevocable(see4.3forfulldetails). Onceanepochends,
gatheringandinstrumentedreplaytoenablepreciseerrorde-
tectionwithextremelylowoverhead. DOUBLETAKE checkstheprogramstateforevidenceofmemory
errors.Becauseepochsarerelativelylong-lived,thecostofcheck-
2. It presents DOUBLETAKE, a prototype framework that im- pointing and error analysis is amortized over program execution.
plements evidence-based dynamic analyses for C/C++ pro- IfDOUBLETAKEfindsanerror,itre-executescodeexecutedfrom
grams: eachoftheanalyseswehavebuiltusing DOUBLE- thepreviousepochwithadditionalinstrumentationtopinpointthe
TAKE–detectingbufferoverflows,use-after-frees,andmem- exactcauseoftheerror.
oryleaks–arethefastestreportedtodate. Todemonstrate DOUBLETAKE’seffectiveness, wehaveimple-
mented detection tools for three of the most important classes of
Outline
errorsinCandC++code:heapbufferoverflows,use-after-freeer-
This paper first provides an overview of the basic operation of rors, andmemoryleaks(Section3describestheseindetail). All
DOUBLETAKE in Section 2. Section 3 details the dynamic anal- detection tools share the following core infrastructure that DOU-
yseswehavebuiltusing DOUBLETAKE. Section4describeskey BLETAKEprovides.
2.1 EfficientRecording
header& requested& canary& header& requested& canary&
Atthebeginningofeveryepoch, DOUBLETAKE savesasnap-
shotofprogramregistersandallwritablememory. Anepochends allocated(object(1( allocated(object(2(
whentheprogramattemptstoissueanirrevocablesystemcall,but
most system calls do not end the current epoch. DOUBLETAKE
also records a small amount of system state at the beginning of Figure2:Heaporganizationusedtoprovideevidenceofbuffer
eachepoch(e.g.,fileoffsets),whichletsitunrolltheeffectofsys- overflowerrors. Objectheadersandunrequestedspacewithin
temcallsthatmodifythisstatewhenre-executionisrequired. allocated objects are filled with canaries; a corrupted canary
Duringexecution,DOUBLETAKEmanagesvarioustypesofsys- indicatesanoverflowoccurred.
temcallsinanefforttoreducethenumberofepochs,whichSec-
tion4.3discusses.Inpractice,DOUBLETAKElimitsthenumberof
epochboundaries,amortizingthecostofprogramstatechecks.The Quaran&ne((FIFO)(
kindofchecksemployeddependontheparticulardynamicanaly-
canary& canary& canary&
sisbeingperformed;Section3describesthedetailsoftheanalyses
wehavebuiltontopofDOUBLETAKE. freed(object(1( freed(object(2( freed(object(3(
2.2 LightweightReplay
Whenprogramstatechecksindicatethatanerroroccurreddur- Figure 3: Evidence-based detection of dangling pointer (use-
ing the current epoch, DOUBLETAKE replays execution from the after-free)errors. Freedobjectsaredeferredinaquarantine
last epoch to pinpoint the error’s root cause. DOUBLETAKE en- in FIFO order and filled with canaries. A corrupted canary
sures that all program-visible state, including system call results indicatesthatawritewasperformedafteranobjectwasfreed.
andmemoryallocationsanddeallocations,isidenticaltotheorig-
inalrun. Duringreplay,DOUBLETAKEreturnscachedreturnval-
ues for most system calls, with special handling for some cases.
Section 4 describes in detail how DOUBLETAKE records and re- Section 4.3 presents full details of DOUBLETAKE’s heap imple-
executessystemcalls. mentation.
2.3 Deterministic Memory Management and 2.4 PinpointingErrorLocations
TripwireSupport
Duringreplay, DOUBLETAKEletsdetectiontoolssethardware
Onekeychallengetousingreplaytofindtheexactlocationof watchpoints during re-execution to pinpoint error locations (i.e.,
errors is that we cannot rely on the default system-supplied heap on an overwritten canary). Modern architectures make available
allocator.Thereasonforthisisthatitdoesnotprovideareplayable a small number of watchpoints (four on x86). Each watchpoint
sequenceofaddresses. Thedefaultheapgrowsondemandbyin- canbeconfiguredtopauseprogramexecutionwhenaspecificbyte
vokingmmap(orasimilarcallonotheroperatingsystems)toob- orwordofmemoryisaccessed. Whilewatchpointsareprimarily
tainmemoryfromthesystem. However,becauseofaddress-space usedbydebuggers, DOUBLETAKEusesthemtospeederrorloca-
layout randomization, now implemented on all modern operating tionduringre-execution.
systemstoincreasesecurity,mmapalmostalwaysreturnsdifferent DOUBLETAKE’swatchpointsareparticularlyusefulincombina-
addresseswheninvoked. Thiseffectmeansthatheapaddressesin tionwithheapcanaries. Forexample,duringre-execution, DOU-
areplayedexecutionwouldlikelydifferfromtheoriginal. BLETAKE’sbufferoverflowanduse-after-freedetectorsplaceawatch-
DOUBLETAKEthereforereplacesthedefaultheapallocatorwith pointatthelocationoftheoverwrittencanarytotraptheinstruc-
a heap built with the HEAP LAYERS framework [4]. In addition tion(s)responsiblefortheerror.
to providing repeatable sequences of addresses, DOUBLETAKE’s
heapprovidesanumberofotherusefulfeaturesthatimproveDOU- 3. ANALYSES
BLETAKE’sefficiencyandsimplifybuildinganalysesusingit:
To demonstrate DOUBLETAKE’s generality and efficiency, we
• Efficiencyvialargechunkallocation. TheDOUBLETAKE implement a range of error-detection tools as evidence-based dy-
heapobtainsmemoryfromtheoperatingsysteminlargechunks namic analyses. In particular, we implement the following three
and satisfies all memory allocations from it, reducing the detectiontoolswithDOUBLETAKE:
number of system calls that DOUBLETAKE must track and
thusloweringitsoverhead. • Heapbufferoverflowdetection(§3.1):whenanapplication
writesoutsidetheboundsofanallocatedobject,
• Simplifiedtripwireinstallation.DOUBLETAKE’sheapalso
makestheprocessofimplantingtripwireseasier. Forexam-
• Use-after-freedetection(§3.2): whenanapplicationwrites
ple,detectiontoolscaneasilyinterposeonheapoperationsto
tofreedmemory(i.e.,throughadanglingpointer),and
altermemoryallocationrequestsordeferthereuseoffreed
memory,andcanmarkthestatusofeachobjectinmetadata
• Memoryleakdetection(§3.3):whenaheapobjectbecomes
(e.g.,viaadedicatedobjectheaderthattheheapprovidesfor
inaccessiblebuthasnotbeenexplicitlyfreed.
thispurpose).
• Efficienttripwirechecking. Finally,DOUBLETAKE’sheap For each of these tools, we describe the evidence that DOU-
makes tripwire checking far more efficient. It maintains a BLETAKE observes or places to detect these errors, and how re-
shadow bitmap to identify the locations and status of heap execution and error isolation proceeds once an error is detected.
canaries,whichallowsittousevectorizedbitoperationsto Note that because these analyses are orthogonal, they can all be
performefficientcheckingattheendofeachepoch. usedsimultaneously.
3.1 HeapBufferOverflowDetection ofallocatedobjects)introducessubstantialoverheadduringnormal
Heap buffer overflows occur when programs write outside the execution,butisunlikelytocatchanyadditionalerrors.Thisisbe-
boundsofanallocatedobject.DOUBLETAKEreportsanerrorwhen causelargeobjectsoftenconsistofaheaderfollowedbyabuffer.
itdiscoversthatacanaryvaluehasbeenoverwritten.Whenitfinds Aprematurelyreusedobjectislikelytohaveitsprologuescram-
an overwritten canary, the detector places watchpoints during re- bledbyaconstructor,whiletheremainderoftheobject(thebuffer
executiontoidentifytheinstructionresponsiblefortheoverflow. contents)mayremainunmodifiedforalongtime.
Beforeanobjectcanbereturnedtotheprogramheap,DOUBLE-
Evidence-BasedErrorDetection TAKEverifiesthatnocanarieshavebeenoverwritten.Italsochecks
allcanariesintheentireheapatepochboundaries.Ineithercase,if
Figure2presentsanoverviewoftheapproachusedtolocatebuffer
acanaryhasbeenoverwritten,thedetectorknowsthatause-after-
overflows. Our buffer overflow detector places canaries between
freeerrorhasoccurred. Itthenimmediatelytriggersre-execution
heapobjectssothatanoverflowfromoneobjectintoanadjacent
toidentifythecauseofthiserror.
onecanbedetected.
Inaddition,theoverflowdetectorfillsanyremainingemptyspace
Re-ExecutionandErrorIsolation
inside allocated objects with canaries; DOUBLETAKE’s allocator
roundsallobjectsizerequestsuptothenearestpoweroftwo.This Duringre-execution,theuse-after-freedetectorinterposesonmalloc
approach lets DOUBLETAKE identify small overflows that would and free calls to find the allocation and deallocation sites of the
otherwisebemissedbecausetheydidnotactuallygobeyondthe overwrittenobject. Thedetectorrecordsacallstackforbothsites
object’sallocatedspace. usingthebacktracefunction. Thedetectoralsoinstallsawatch-
Atmemorydeallocationtime(callstofreeordelete), DOU- pointattheaddressoftheoverwrittencanary.Aswithbufferover-
BLETAKEchecksforbufferoverflowsinobjectswhoserequested flow detection, any writes to the watched address will generate a
sizeislessthanapoweroftwo.Itdefersthecheckingofpower-of- SIGTRAPsignal. Whenthissignalistriggered,thedetectorreports
twosizedobjectstotheendofthecurrentepoch. informationabouttheobject’sallocationanddeallocationsites,as
At the end of each epoch, DOUBLETAKE checks whether any wellascallstackandlinenumberinformationfortheinstructions
canaries have been overwritten (including those for exact power- responsiblefortheuse-after-freeerror.
of-tworequests). Ifitfindsanyoverwrittencanaries,ithasincon-
3.3 MemoryLeakDetection
trovertibleevidencethatabufferoverflowhasoccurred.DOUBLE-
TAKE thentriggersare-executiontolocatetheexactpointinthe Heap memory is leaked when it becomes inaccessible without
programwhentheoverflowhappened. beingfreed. Memoryleakscansignificantlydegradeprogramper-
formanceduetoanincreasedmemoryfootprint. Ourleakdetec-
Re-ExecutionandErrorIsolation
tor identifies possible unreachable allocated objects at the end of
DOUBLETAKEinstallsawatchpointattheaddressofthecorrupted eachepoch. Allocationsitescanhelpusersfixmemoryleaks,but
canarybeforere-execution. Whentheprogramisre-executed,any collectingthisinformationforallmalloccallsinnormalexecution
instruction that writes to this address will trigger the watchpoint. wouldunnecessarilyslowdowntheprogramforthecommoncase
The operating system will deliver a SIGTRAP signal to DOUBLE- (no memory leaks). Instead, DOUBLETAKE only records the al-
TAKE beforetheinstructionisexecuted. Byhandlingthissignal, locationsitesofleakedmemoryduringre-execution,andaddsno
DOUBLETAKE reports the complete call stack of the trapped in- overheadfornormalexecution.
structionbyinvokingthebacktracefunction.
Evidence-BasedErrorDetection
3.2 Use-After-FreeDetection
Unlikethepreviously-describeddetectors,memoryleakdetection
Use-after-freeordanglingpointeroverflowerrorsoccurwhenan doesnotneedtripwires. Instead,theevidenceofamemoryleakis
applicationcontinuestoaccessmemorythroughpointersthathave latentintheheaporganizationitself.
been passed to free() or delete. Writes to freed memory can Our detector finds memory leaks using the same marking ap-
overwritethecontentsofotherliveobjects,leadingtounexpected proachasconservativegarbagecollection[42].Themarkingphase
programbehavior.Likethebufferoverflowdetector,ouruse-after- performs a breadth-first scan of reachable memory using a work
freedetectorusescanariestodetectwritestofreedmemory.When queue. Initially,allvaluesinregisters,globals,andthestackthat
ause-after-freeerrorisdetected,DOUBLETAKEreportsthealloca- look like pointers are added to the work queue. Any eight-byte
tionanddeallocationsitesoftheobject,andallinstruction(s)that alignedvaluethatfallswithintherangeofallocatedheapmemory
wrotetotheobjectafteritwasfreed. istreatedasapointer.
Ateachstepinthescan,thedetectortakesthefirstitemoffthe
Evidence-BasedErrorDetection
workqueue. Usingtheheapmetadatalocatedbeforeeachobject,
Figure3illustrateshowwedetectuse-after-freeerrorsusingDOU- the detector finds the bounds of each object. Each object has a
BLETAKE. Our use-after-free detector delays the re-allocation of headercontainingamarkedbitandanallocatedbit. Ifthemarked
freed memory. We adopt the approach used by AddressSanitizer bit is set, this object has already been visited. The detector then
ofmaintainingaFIFOquarantinelist[38].Inourimplementation, removes this object and moves on to the next item in the queue.
objectsarereleasedfromthequarantinelistwhenthetotalsizeof Iftheobjectisallocatedbutnotyetmarked,thedetectormarksit
quarantinedobjectsexceeds16megabytes,orwhentherearemore asreachablebysettingthemarked bitandaddsallpointervalues
than1,024quarantinedobjects. (Notethatallthresholdsusedby withintheobject’sboundstotheworkqueue.Oncetheworkqueue
thedetectorareeasilyconfigurable.) isempty,DOUBLETAKEendsitsscan.
Thedetectoroverwritesthefirst128bytesofallobjectsinthe DOUBLETAKEthentraversestheentireheaptofindanyleaked
quarantinelist(whichhaveallbeenfreedbytheprogram)withca- objects:theseareallocatedbutunmarked(unreachable). Ifitfinds
naryvalues.Thisthresholdstrikesacompromisebetweenerrorde- memoryleaks,re-executionbegins. Notethatusingthisapproach,
tectionandefficiency.Wehavefoundempiricallythatfillinglarger ourdetectorcanalsofindpotentialdanglingpointers(thatis,reach-
objectswithcanaries(i.e.,goingbeyond128bytestothefullsize ablefreedobjects). Thisoptionisdisabledbydefaultbecause,un-
likeotherapplications, potentialdanglingpointerdetectioncould Category Functions
producefalsepositives.
Repeatable getpid,sleep,pause
Re-ExecutionandErrorIsolation Recordable mmap,gettimeofday,time,clone,open
During re-execution, the leak detector checks the results of each Revocable write,read
malloccall. Whentheallocationofaleakedobjectisfound,the Deferrable close,munmap
detectorrecordsthecallstackusingthebacktracefunction. At
Irrevocable fork, exec, exit, lseek, pipe, flock, socket
theendoftheepochre-execution,thedetectorreportsthelastcall related system calls
stackforeachleakedobjectsincethelastsiteisresponsibleforthe
memoryleak. Table2: Systemcallshandledby DOUBLETAKE. Allunlisted
systemcallsareconservativelytreatedasirrevocable,andwill
4. IMPLEMENTATIONDETAILS end the current epoch. Section 4.3 describes how DOUBLE-
DOUBLETAKE is implemented as a library for Linux applica- TAKEhandlescallsineachcategory.
tions.ItcanbelinkeddirectlyoratruntimeusingtheLD_PRELOAD
mechanism. DOUBLETAKEisthusconvenienttouse: thereisno 4.2 EpochStart
needtochangeorrecompileapplications,touseaspecializedhard-
wareplatform,runinsideavirtualmachine,ormodifytheOS. Atthebeginningofeachepoch,DOUBLETAKEtakesasnapshot
At startup, DOUBLETAKE begins the first epoch. This epoch ofprogramstate.DOUBLETAKEsavesallwritablememory(stack,
continuesuntiltheprogramissuesanirrevocablesystemcall(see heap,andglobals)fromthemainprogramandanylinkedlibraries,
Section4.3fordetails). Beforeanirrevocablesystemcall, DOU- and saves the register state of each thread with the getcontext
BLETAKEchecksprogramstateforevidenceoferrors. Thedetails function. To reduce the cost of snapshots, DOUBLETAKE does
arepresentedinSection3. not checkpoint any read-only memory. To identify all writable
If no errors are found, DOUBLETAKE ends the current epoch, mapped memory, DOUBLETAKE processes the /proc/self/map
issues the irrevocable system call, and begins a new epoch. If file, whichonLinuxidentifieseverymappedmemoryregionand
it finds evidence of an error, DOUBLETAKE enters re-execution itsattributes(otheroperatingsystemsimplementsimilarfunction-
mode. DOUBLETAKE will then re-execute with instrumentation ality). DOUBLETAKE also records the file positions of all open
activatedandreportthelinesofcoderesponsiblefortheerror(s). files,whichletsprogramsissuereadandwritesystemcallswith-
The remainder of this section describes the implementation of outendingthecurrentepoch.DOUBLETAKEusesthecombination
DOUBLETAKE’scorefunctionality. ofsavedmemorystate,filepositionsandregisterstorollbackexe-
cutionifitfindsevidenceofanerror.
4.1 StartupandShutdown
4.3 NormalExecution
At program startup, DOUBLETAKE performs initialization and
startsthefirstepoch. DOUBLETAKEneedstogetinearlytointer- Onceasnapshothasbeenwritten, DOUBLETAKE letsthepro-
pose on system calls and install its own heap implementation. It gram execute normally but interposes on heap allocations/deallo-
accomplishes this by marking its own initialization function with cations and system calls in order to set tripwires and support re-
theconstructorattribute. Since DOUBLETAKE mustwraplibrary execution.
functionsthateventuallyinvokewithsystemcalls,asdescribedin
SystemCalls
Section 4.3, it collects the addresses of all intercepted functions
during this initialization phase. DOUBLETAKE acquires memory DOUBLETAKEendseachepochwhentheprogramattemptstois-
fromtheOStoholditsheap,collectsthenamesandrangesofall sue an irrevocable system call. However, most system calls can
globalsbyanalyzing/proc/self/maps,installssignalhandlerfor safelybere-executedorundonetoenablere-execution.
segmentationviolations,andpreparesthedatastructureforrecord- DOUBLETAKE dividessystemcallsintofivecategories, shown
ingandhandlingsystemcalls. inTable2.Systemcallscouldbeinterceptedusingptrace,butthis
Fortechnicalreasons,DOUBLETAKEmustpostponethecheck- would add unacceptable overhead during normal execution. In-
pointingofprogramstate(andthusthebeginningofthefirstepoch) stead, DOUBLETAKE interposesonalllibraryfunctionsthatmay
until just before execution enters the application enters its main issuesystemcalls.
function. Thisdelayisnecessarytoletkeylow-levelstartuptasks
complete.Forexample,C++performsitsinitializationforthestan- • Repeatablesystemcallsdonotmodifysystemstate,andre-
dardstreamaftertheexecutionofconstructorfunctions(including, turnthesameresultduringnormalexecutionandre-execution.
inthiscase, DOUBLETAKEitself). BecauseDOUBLETAKErelies Nospecialhandlingisrequiredforthesecalls.
on streams to report any errors it detects, by definition it cannot
start the first epoch before that point. To make this all possible, • Recordablesystemcallsmayreturndifferentresultsifthey
we interpose on the libc_start_main function, and pass a cus- are re-executed. DOUBLETAKE records the result of these
tommainfunctionimplementedbyDOUBLETAKEthatperformsa systemcallsduringnormalexecution,andreturnsthesaved
snapshotjustbeforeenteringtheapplication’srealmainroutine. result during re-execution. Some recordable system calls,
DOUBLETAKEtreatsprogramterminationastheendofthefinal suchasmmap,changethestateofunderlyingOS.
epoch.Aswithanyotherepoch,ifitfindsevidenceofprogramer-
rors,DOUBLETAKEre-executestheprogramtopinpointtheexact • Revocablesystemcallsmodifysystemstate,butDOUBLE-
causesoferrors.Thislogicisembeddedinafinalizermarkedwith TAKE can save the original state beforehand and restore it
thedeconstructorattributethatDOUBLETAKEinstalls. priortore-execution. MostfileI/Ooperationsfallintothis
category. For example, although write modifies file con-
tents, DOUBLETAKE canwritethesamecontentduringre-
execution. Thewritefunctionalsochangesthecurrentfile
position,butthefilepositioncanberestoredtothesavedone deferredsystemcalls,clearsthelogsforallrecordedsystemcalls,
usinglseekpriortore-execution. andbeginsthenextepoch.
Atthebeginningofeachepoch,DOUBLETAKEsavesallfile 4.5 Rollback
descriptorsofopenedfilesinahashtable. Maintainingthis
hashtablehelpstoidentifywhetherareadandwritecall Ifanerrorisfound,DOUBLETAKErollsbackprogramstateprior
tobeginningre-execution.Thisrollbackmustbehandledwithcare.
is operating on sockets or not, because socket communica-
Forexample, restoringthesavedstackmaycorruptthecurrent
tionsmustbetreatedasirrevocablesystemcalls.Inaddition,
stackifthesizeofthesavedstackislargerthanthatofthecurrent
DOUBLETAKEmustsavestreamcontentsreturnedbyfread
stack.DOUBLETAKEthusswitchestoatemporarystackduringits
inordertosupportre-execution.
rollback phase. When performing rollback, the saved state of all
• Deferrable system calls will irrevocably change program writablememoryiscopiedback,whichalsorecoversthestatusof
state, butcansafelybedelayeduntiltheendofthecurrent itsheap. DOUBLETAKEalsorecoversthefilepositionsofopened
epoch.DOUBLETAKEdelaysallcallstomunmapandclose, filessothatallread/writecallscanbeissuednormallyduringre-
execution.
andexecutesthesesystemcallsbeforestartinganewepoch
whenthereisnoneedtore-executetheprogram. DOUBLETAKEthensetshardwarewatchpointsonallcorrupted
addressesinordertoreporttherootcausesofbufferoverflowsor
• Irrevocablesystemcallschangeinternally-visibleprogram danglingpointers.Sincedebugregistersarenotdirectlyaccessible
state,andcannotberolledbackandre-executed. DOUBLE- in user mode, DOUBLETAKE utilizes the perf_event_open call
TAKEendsthecurrentepochbeforethesesystemcalls. toloadwatchedaddressesintothedebugregisters. DOUBLETAKE
alsosetsaSIGTRAPhandlerforwatchpointssothatitwillgetnoti-
DOUBLETAKE reduces the number of irrevocable system calls fiedwhentheseaddressesareoverwritten(e.g.,duringbufferover-
by observing their arguments; in some cases, they are not neces- flowsorusesoffreedobjects).
sarilyirrevocable. Forexample,whenfcntlinvokedwithF_GET, Onceallwatchpointshavebeenplaced,DOUBLETAKEusesthe
DOUBLETAKEtreatsitasarepeatablesystemcallsinceitissimply setcontextcalltorestoreregisterstateandbeginre-execution.
areadoffilesystemstate.However,ittreatsthiscallasirrevocable
4.6 Re-Execution
ifinvokedwithF_SET,sincethecallthenactuallyupdatesthefile
system. Duringre-execution,DOUBLETAKEreplaysthesavedresultsof
recordable system calls from the log collected during normal ex-
MemoryManagement ecution,whileavoidinginvokingactualsystemcalls;thatis,their
executionissimulated. Alldeferredsystemcallsareconvertedto
AsdescribedinSection2.3,DOUBLETAKEinterceptsmemoryal-
locationsanddeallocationstoimplanttripwires,identifyheapcor- no-ops while the program is re-executing. DOUBLETAKE issues
othertypesofsystemcallsnormally.
ruption,andfacilitatere-execution.DOUBLETAKEreplacesthede-
faultheapwithafixed-sizeBiBOP-styleallocatorwithper-thread DOUBLETAKE’sheapdesignanditsrollbackhasrecoveredthe
memory state to the snapshotted state. To repeat the replayable
subheapsandpower-of-twosizeclasses. Webuiltthisheapusing
theHEAPLAYERSframework[4]. memory uses, DOUBLETAKE simply repeats memory allocations
anddeallocationsfromapplicationsandlibrariesaccordingtothe
DOUBLETAKEimplantstripwiresdifferentlyfordifferentanal-
programorder.Theadditionalmemoryuseshappenedinthereplay
yses.Todetectheap-basedbufferoverflows,DOUBLETAKEplaces
phase,suchasbookkeepingthecallstackofmemoryuses,willbe
canaries along with each heap object. In order to find use-after-
freeerrors,DOUBLETAKEpostponesthereuseoffreedobjectsby satisfiedfromDOUBLETAKE’sinternalheapandwillnotaffectthe
memoryusesofapplications.
puttingthemintoaquarantinelistandfillingthemwithcanaries.
Formemoryleakdetection,thereisnoneedtoimplanttripwires, Duringreplay,DOUBLETAKEenablestrackingofpreciseinfor-
mationinthememoryallocator: allallocationsanddeallocations
becausetheevidenceofaleakcanbefoundwithoutthem.
recordtheircallingcontextsothesecanbereportedlater,ifneeded.
Toidentifyheapcorruption,DOUBLETAKEmaintainsabitmap
Notethatrecordingcallsitesduringordinaryexecutionwouldbe
thatrecordsthelocationsofallheapcanaries. Thebitmaprecords
everywordofheapmemorythatcontainsacanary,whichwillbe prohibitively expensive, imposing 20–30% overhead; DOUBLE-
checkedattheendofeachepoch. Ifanyofthesewordsaremodi- TAKE’sstrategyremovesthisoverheadfromnormalexecution.
fied,DOUBLETAKEnotifiesthedetectiontool. Finally,DOUBLETAKEhandlestrapscausedbyaccessestowatch-
Tospeedre-execution,DOUBLETAKEusesitsheapallocatorto points. Inside the trap handler, DOUBLETAKE first determines
whichwatchpointcausedthecurrenttrapiftherearemultiplewatch-
satisfymemoryrequestsfromtheapplicationandcorrespondingli-
braries,andmaintainsaseparateheapforinternaluseonly.Forex- points. Italsofiltersoutanyaccessesfrom DOUBLETAKE itself.
ample,thememorythatDOUBLETAKEusestorecordsystemcalls DOUBLETAKE prints the callsite stack of the instruction respon-
sibleforabufferoverfloworuse-after-freeerrorsandtheirmem-
results is allocated from its internal heap and there is no need to
replaytheseallocationsduringre-execution. Anyadditionalmem- oryallocation(ordeallocation)sites.Formemoryleaks,DOUBLE-
ory allocations during the replay phase are also satisfied from its TAKEreportstheallocationcallsiteoftheleakedobject.
internalheap.
5. EVALUATION
4.4 EpochEnd
WeevaluateDOUBLETAKEtodemonstrateitsefficiency,interms
Each epoch ends when the program issues an irrevocable sys- ofexecutiontime,memoryoverhead,andeffectivenessatdetecting
temcall.Attheendofeachepoch,DOUBLETAKEchecksprogram errors. All experiments are performed on a quiescent Intel Core
stateforerrors. Theseanalysis-specificerrorchecksaredescribed 2 dual-processor system with 16GB of RAM, running on Linux
in Section 3. If an error is found, DOUBLETAKE rolls back ex- 3.13.0-53-generic with glibc-2.19. Each processor is a 4-core
ecution to the immediately-preceding epoch, and switches to re- 64-bit Intel Xeon, operating at 2.33GHz with a 4MB shared L2
execution mode. If no error is found, DOUBLETAKE issues any cacheanda32KBper-coreL1cache.Allprogramsarebuiltas64-
e+ 2" 2.3"
m1.8" DoubleTake:+OF+
/1.6"
n1.4"
u DoubleTake:+OF+ML+
R1.2"
d+ 1"
e0.8" DoubleTake:+OF+ML+DP+
z
mali00..46"" AddressSani/zer:+OF+DP+
r0.2"
o 0"
N
400.perlben4c0h"1.bzip2"403.gcc4"29.4m4c5f."go4b5m6.k"hm m4e45r6"8.2.sljiebnqgu"an4t6u4.mh"246741.roef"mnetp4p74"38.3a.sxtalara"ncbm4k"33.m4il4c"4.na4m4d"7.de4a5lI0I."so4pl5e3.x"povray4"704.l8b2.ms"phinx3" AVERAGE"
Figure4: Runtimeoverheadof DOUBLETAKE (OF=BufferOverflowDetection, ML=MemoryLeakDetection, DP=Dangling
Pointers Detection) and AddressSanitizer, normalized to each benchmark’s original execution time. With all detections enabled,
DOUBLETAKEonlyintroduces4%performanceoverheadonaverage.
bitexecutablesusingLLVM3.2withtheclangfront-endand-O2 tialamountoftimefillingfreedmemorywithcanariesforapplica-
optimizations. All evaluations on SPEC CPU2006 are exercised tionswithalargenumberofmallocandfreecalls. Thus,DOU-
withthe“ref”(reference)inputset. BLETAKErunsmuchslowerfortheapplicationgccwhenthede-
tectionofuse-after-freeerrorsisenabled.h264refaddssignificant
5.1 RuntimeOverhead
overheadonDOUBLETAKEbecauseofitslargenumberofepochs.
WeevaluateDOUBLETAKE’sruntimeandmemoryoverheadacross Table5presentsdetailedbenchmarkcharacteristics. The“Pro-
all of the C and C++ SPEC CPU2006 benchmarks, 19 in total. cesses”columnshowsthenumberofdifferentprocessinvocations
WecompareDOUBLETAKEwiththepreviousstate-of-the-arttool, (bycallingfork).Thenumberofepochsissignificantlylowerthan
Google’s AddressSanitizer [38]. As mentioned earlier, Address- thenumberofactualsystemcalls,demonstratingDOUBLETAKE’s
Sanitizercandetectbufferoverflowsanduse-after-freeerrors,but effectivenessatreducingepochsviaitslightweightsystemcallhan-
itonlydetectsmemoryleaksattheendofexecution. Bycontrast, dling.Thebenchmarkswiththehighestoverheadsharethefollow-
DOUBLETAKEdetectsalloftheseerrorsattheendofeveryepoch. ingcharacteristics: theyconsistofasubstantialnumberofepochs
In our evaluation, DOUBLETAKE discovered several memory (e.g.,perlbenchandh264ref)orareunusuallymalloc-intensive
leaks, which trigger rollback and error identification. To isolate (e.g.,gcc,omnetpp,andxalancbmk).
normal execution overhead, we disable DOUBLETAKE’s rollback Runtime Overhead Summary: For nearly all of the bench-
inourevaluation. Thatis, ourrunswith DOUBLETAKE incurall marks we examine, DOUBLETAKE substantially outperforms the
oftheoverheadofordinarytracking(includingimplantingoftrip- state of the art. For most benchmarks, DOUBLETAKE’s runtime
wiresandexaminingstate)butdonotmeasurethetimetorollback overheadisunder3%.
andlocateerrors;ingeneral,thiscostislowandinanyeventdoes
5.2 MemoryOverhead
notaffectbug-freeexecution,whichisthecommoncase.Foreach
benchmark,wereporttheaverageruntimeofthreeruns. Wemeasureprogrammemoryusagebyrecordingthepeakphys-
Figure4presentsexecutiontimeoverheadresultsforDOUBLE- icalmemoryusage. Virtualmemoryconsumptionisgenerallynot
TAKEandAddressSanitizer. Onaverage,DOUBLETAKEimposes relevantfor64-bitplatforms,whichhaveenormousvirtualaddress
only 4% overhead with all three error detectors enabled. When ranges. DOUBLETAKE’spre-allocatedheapandinternalheapcon-
use-after-free detection (DP) is disabled, DOUBLETAKE exhibits sume 8GB of virtual memory space. We compute peak physical
noobservableoverhead. AddressSanitizerhasanaverageruntime memoryusagebyperiodicallycollectingprocess-levelinformation
overheadover30%;recallthatAddressSanitizeronlyperformsleak (onLinux,thisisavailableinthe/proc/self/smapspseudo-file),
detection at the end of program execution, while DOUBLETAKE andsummingtheproportionalsetsizesofmemorysegments.
performsiteveryepoch. Figure5presentsmemoryoverheadforDOUBLETAKEandAd-
For17outof19benchmarks, DOUBLETAKE outperformsAd- dressSanitizer (Table 3 has full details). On average, both across
dressSanitizer.For14benchmarks,DOUBLETAKE’sruntimeover- thebenchmarksuiteandwhenbrokendownbyfootprint(large(>
headwithalldetectorsenabledisunder3%. Unsurprisingly,both 100MB)andsmall(<100MB)), DOUBLETAKE imposesconsid-
DOUBLETAKEandAddressSanitizersubstantiallyoutperformVal- erablylowermemoryoverheadthanAddressSanitizer. DOUBLE-
grindonallbenchmarks. TAKE imposeslowermemoryoverheadthanAddressSanitizeron
Fourofthebenchmarkshavehigherthanaverageoverheadfor allbutthreebenchmarks:perlbench,h264,andnamd.
DOUBLETAKE andAddressSanitizer(400.perlbench, 403.gcc, Wedrilldowntoexploretheapplicationandanalysischaracter-
464.h264ref,and483.xalancbmk).BothDOUBLETAKEandAd- isticsthatcontributetoDOUBLETAKE’smemoryoverhead:
dressSanitizer substantially increase these applications’ memory
footprints(seeTable3).Weattributetheirincreasedexecutiontime • Numberofepochs:MuchofDOUBLETAKE’smemoryover-
tothisincreasedmemoryfootprintanditscorrespondingincreased headcomesfromthesnapshotofwritablememorytakenat
cacheandTLBpressure. thebeginningofeachepoch. However,thefirstsnapshotis
DOUBLETAKE’suse-after-freedetectionaddsroughly4%run- oftensmallbecausetheheapisalmostemptybeforethemain
timeoverhead,butonlygccandh264refrunwithmorethan20% routine. For example, the benchmarks bzip2, mcf, sjeng,
overhead. AsdescribedinSection3.2, allfreedobjectsarefilled milc,andlbmruninasingleepoch,andaccordinglyexhibit
withcanaries(upto128bytes). DOUBLETAKEspendsasubstan- verylowmemoryoverhead.
15"
34"
ed++Use+10" Original+
alizry+ DoubleTake+
mo
rm AddressSani:zer+
oe 5"
NM
0"
400.perlben4c0h1".bzip24"03.gc4c"29.44m5c.f"go4b56m.k"hm m44e56r"82..lsijbeqngu"a4nt6u4.hm"427614.roef"mnet4p47p"83.3.axstalaar"ncbm4k3"3.4mil4c4".na4m4d7".de4a5lI0I."so4pl5e3.x"povra4y"704.l8b2.sm"phinx3" AVERAGE"
Figure5: MemoryoverheadofDOUBLETAKEandAddressSanitizer.
• Systemcalllogs:Systemcalllogsintroduceadditionalmem- MemoryUsage(MB)
oryoverheadthatdependsonthenumberofrecordedsystem Benchmark Original Address DOUBLE
Sanitizer TAKE
calls.
largefootprint(>100MB)
400.perlbench 656 1481 1834
• Analysis-specificoverhead:Othersourcesofmemoryover-
401.bzip2 870 1020 989
head are analysis-specific. Buffer overflow detection adds 403.gcc 683 2293 1791
canariesbetweenheapobjects,whichcanincreasememory 429.mcf 1716 1951 2000
usage for programs with many small allocations. For this 458.sjeng 179 220 212
analysis,DOUBLETAKEalsomaintainsabitmapthatmarks 471.omnetpp 172 538 299
473.astar 333 923 479
theplacementofcanaries: foreacheight-byteswordofthe
483.xalancbmk 428 1149 801
heap, DOUBLETAKE adds a bit to mark whether this word 433.milc 695 1008 917
hasthecanariesornot.Finally,use-after-freedetectionadds 447.dealII 514 2496 1724
constant-sizememoryoverheadbydelayingmemoryreuse. 450.soplex 441 1991 1104
Notethatanysimilardynamicanalysesmustimposesimilar 470.lbm 418 496 477
overheads. geometricmean +117% +72%
smallfootprint(<100MB)
445.gobmk 28 137 66
MemoryOverheadSummary:Onaverage,DOUBLETAKEim-
456.hmmer 24 256 82
poses lower memory overhead than AddressSanitizer. For large
462.libquantum 66 144 132
footprintapplications,itincreasesmemoryconsumptionforappli- 464.h264ref 65 179 247
cationsby72%onaverage. 444.namd 46 79 96
453.povray 3 133 37
5.3 Effectiveness 482.sphinx3 45 181 103
geometricmean +395% +208%
We evaluate the effectiveness of DOUBLETAKE on a range of
overallgeometricmean +194% +114%
applications, includingsynthetictestcases, standardbenchmarks,
andreal-worldapplications. Table3: AddressSanitizerand DOUBLETAKE memoryusage,
Synthetictestcasesandbenchmarks: WefirstevaluateDOU- inmegabytes.Thetopsectionlistsmemoryoverheadforlarge-
BLETAKE on 3 synthetic test cases, 26 test cases from the NIST footprintapplications(over100MB),whilethebottomsection
SAMATEReferenceDatasetProject.Thiscorpusincludes14cases presents overhead for small-footprint applications. DOUBLE-
withbufferoverflowsand12caseswithoutoverflows[17].Wealso TAKE’smemoryoverheadisgenerallylessthanAddressSani-
evaluateDOUBLETAKEon19C/C++benchmarksfromtheSPEC tizer’s.
CPU2006benchmarksuite.
Forheapoverflows,DOUBLETAKEdetectsallknownoverflows
in one synthetic test case and 14 test cases of SAMATE suite. 23,43],includingbc,gcc-4.4.7,gzip,libHX,polymorph,and
Forthe 12cases withoutoverflows in SAMATEsuite, DOUBLE- vim-6.3.
TAKEhasnofalsepositives.FortheSPECCPU2006benchmarks, In every case, DOUBLETAKE detected all known or converted
DOUBLETAKE did not find any heap buffer overflows and use- errors. Convertederrorsareexistingglobalorarrayoverflowsthat
after-frees, whichisthesameresultfoundwithAddressSanitizer. DOUBLETAKEcurrentlycannnotdetect;weconvertedthesetoheap
However,DOUBLETAKEdetectedasignificantnumberofmemory overflowstoverifyitseffectiveness. DOUBLETAKEalsoidentified
leaksinperlbenchandgccofSPECCPU2006,whichweverified memory leaks in gcc-4.4.7 and vim-6.3, which we confirmed
usingValgrind’sMemchecktool. with Valgrind. To evaluate the detection of use-after-free errors,
Real applications: We also ran DOUBLETAKE with a variety wemanuallyinjectederrorsonrealapplications,suchasvim-7.3,
ofapplicationswithknownerrorsorimplantederrors,listedinTa- lsandwc. DOUBLETAKEidentifiedallofthesememoryerrors.
ble4. ToverifytheeffectivenessofDOUBLETAKE’sbufferover- Notethattheerrorsobservedintheseapplicationsaretriggered
flowdetection,wecollectedapplicationsfromevaluationsofprior onlybyspecificinputs.Inthecommoncase,theseapplicationsper-
buffer overflow detection tools, Bugzilla, and bugbench [16, 20, formasexpected.ThisisexactlythecaseforwhichDOUBLETAKE
Application Description LOC ErrorType alimitationofasallcanary-baseddetectors.Ifanoverflowtouches
bc basiccalculator 12K KnownOverflow
memory only in adjacent objects and skips over canaries, DOU-
gzip compressorexpandfiles 5K ConvertedOverflow
libHX commonlibrary 7K KnownOverflow BLETAKE’send-of-epochscanwillnotrevealanyevidenceofthe
polymorph filenameconverter 0.4K ConvertedOverflow overflow.Boththebufferoverflowanduse-after-freedetectorscan
vim-6.3 texteditor 282K KnownOverflow detecterrorsonlyonwrites.Toreduceoverhead,theuse-after-free
gcc-4.7 GNUCompilerCollection 5784K Unknownleaks detectoronlyplacescanariesinthefirst128bytesoffreedobjects.
vim-6.3 texteditor 282K Unknownleak Ifawritetofreedmemorygoesbeyondthisthreshold,ourdetector
ls directorylisting 3.5K ImplantedUAF
willnotfindit. Thememoryleakdetectorwillnotproducefalse
wc wordcount 0.6K ImplantedUAF
positives, but non-pointer values that look like pointers to leaked
vim-7.4 texteditor 332K ImplantedUAF
objectscanleadtofalsenegatives. Finally,ifaleakedobjectwas
Table4: Errordetection: DOUBLETAKEdetectsbothknown not allocated in the current epoch, DOUBLETAKE’s re-execution
(injectedandnon-injected)andpreviouslyunknownerrorson will not be able to find the object’s allocation site (a limitation
theaboveapplications(anyreportederrorsarereal,as DOU- sharedbyAddressSanitizer). Inpractice,DOUBLETAKE’sepochs
BLETAKEhasazerofalsepositiverate). arelongenoughtocollectallocationsiteinformationforallleaks
detectedduringourevaluation.
Benchmark Processes Epochs Syscalls #Mallocs Whileevidence-baseddynamicanalysescanrunwithverylow
400.perlbench 3 43 60068 360605640 overhead,theycannotdetecterrorsifthereisnoevidence,oritis
401.bzip2 6 6 968 168 notpracticaltoforceevidenceoftheirexistence. Evidence-based
403.gcc 9 9 155505 28458514 analysis also depends on errors being generally monotonic: once
429.mcf 1 1 24443 5 anerrorhasoccurred,itsevidenceneedstopersistuntiltheendof
445.gobmk 5 5 2248 658034
theepochinordertoensuredetection.
456.hmmer 2 2 46 2474268
458.sjeng 1 1 23 5 Finally,thecurrentprototypeofDOUBLETAKEislimitedtoexe-
462.libquantum 1 1 11 179 cutingsingle-threadedcode. However,webelievethisisprimarily
464.h264ref 3 825 2592 146827 anengineeringquestion.Evidence-basedanalysisdoesnotdepend
471.omnetpp 1 1 19 267168472 onfullydeterministicreplay. Considerthecaseofamemoryerror
473.astar 2 2 102 4799955 arisingduetoarace. Duringreplay,thesamesequenceofwrites
483.xalancbmk 1 1 123706 135155557
may not recur, and thus the hardware watchpoints in those ad-
433.milc 1 1 12 6517
444.namd 1 1 470 1324 dressesmightnotbetriggered.Inthisscenario,DOUBLETAKEcan
447.dealII 1 1 8131 151332314 simply continue execution having successfully masked the error,
450.soplex 2 2 37900 310619 orrepeatedlyre-executetheepochinanefforttoexposethedata
453.povray 1 1 25721 2461141 race. Because of this flexibility, DOUBLETAKE will not need to
trackdetailsabouttheorderingofmemoryaccesses,whichiswhat
Table5:Benchmarkcharacteristics. makes deterministic record-and-replay systems expensive. Sup-
porting multithreaded programs will only require interception of
synchronizationoperationstoallowDOUBLETAKEtopausethreads
isideal,sinceitslowoverheadisdesignedtomakeitfeasibletouse
atepochboundariesandtotracktheorderofsynchronizationoper-
itindeployedsettings.
ations.
Detailedreporting: DOUBLETAKEreportspreciseinformation
aimed at helping programmers identify the exact causes of dif-
7. RELATEDWORK
ferent memory errors, as shown in Figure 6(a). For buffer over-
flows,DOUBLETAKEreportsthecallsitesandlinenumbersofthe DynamicInstrumentation:Numerouserrordetectiontoolsuse
overflowandtheoriginalmemoryallocation. Formemoryleaks, dynamicinstrumentation, includingmanycommercialtools. Val-
DOUBLETAKE reports the last call site of its memory allocation. grind’sMemchecktool, Dr. Memory, Purify, IntelInspector, and
Foruse-after-freeserror,DOUBLETAKEreportsbothallocationand SunDiscoverallfallintothiscategory[8,14,18,28,32]. These
deallocationcallsites, andtheinstruction(s)thatwrotetotheob- toolsusedynamicinstrumentationengines,suchasPin,Valgrind,
ject after it was freed. In general, DOUBLETAKE provides more andDynamiRIO[7,24,28].Thesetoolscandetectmemoryleaks,
detailedinformationthanAddressSanitizer,asseeninFigure6(b). use-after-freeerrors,uninitializedreads,andbufferoverflows.Dy-
In addition, DOUBLETAKE can identify more errors than Ad- namicinstrumentationtoolsaretypicallyeasytousebecausethey
dressSanitizer.DOUBLETAKEcantrackuptofourbufferoverflows donotrequirerecompilation,butthiseaseofusegenerallycomes
oruse-after-freeerrorsduringthesameepochbecauseitsisolation atthecostofhighoverhead.ProgramsrunwithValgrindtake20×
isbasedontheuseofhardwaredebuggingregisters. AddressSani- longerthanusual,andDr. Memoryintroduces10×runtimeover-
tizeralwaysstopsatthedetectionofthefirstsucherror. head. DOUBLETAKEissignificantlymoreefficientthanpriordy-
EffectivenessSummary: Acrosstheapplicationsweexamine, namicinstrumentationtools,withunder5%performanceoverhead.
DOUBLETAKE detects all known or injected errors with no false Severaldynamicanalysistoolsleveragestaticanalysistoreduce
positives. DOUBLETAKE is as effective at finding errors as Ad- the amount and thus the overhead of instrumentation [1, 12, 13,
dressSanitizer,butwithmuchlowerperformanceandmemoryover- 27,33,38]. Whilethesetoolsgenerallyreduceoverheadoverap-
head.Italsoprovidesmoredetailedreportsfortheseerrors. proaches based exclusively on dynamic instrumentation, but can-
notdetecterrorsincodethatwasnotrecompiledwiththisinstru-
mentationinplace(e.g., insidelibraries). Inaddition, ourresults
6. DISCUSSION
show that AddressSanitizer (the previous state-of-the-art, which
The analyses we have built using DOUBLETAKE (heap buffer dependsonstaticanalysis)isconsiderablyslowerthan DOUBLE-
overflows, use-after-freeerrors, andmemoryleaks)havenofalse TAKE, whichcanalsoperformitsanalysisontheentireprogram
positives,buttheycanhavefalsenegatives. Ourheapbufferover- (includinglibraries)withnorecompilation.
flowdetectorcannotidentifyallnon-contiguousbufferoverflows, Interposition: DOUBLETAKE useslibraryinterpositionexclu-
DoubleTake:+Heap+buffer+overflow+at+address+0x100000120+with+value+0x2038313a31312032.++
The+heap+object+has+size+100+and+starts+at+0x100000020.!
Caught!a!heap!overflow!at!0x100000120.!Current!call!stack:! ERROR:%AddressSani/zer:%heap3buffer3overflow%on%address%0x7ffff7f521c7%at%pc%0x40a396%%
tests/SAMATE/overflow/Heap_overflow_15.cpp:16! bp%0x7fffffffe800%sp%0x7fffffffdI0.%WRITE%of%size%1%at%0x7ffff7f521c7%thread%T0%
! %%%%#0%0x40a395%(tests/SAMATE/overflow/Heap_overflow_153pthread+0x40a395)%
Memory!allocaFon!site:!! %%%%#1%0x407ff9%(tests/SAMATE/overflow/Heap_overflow_153pthread+0x407ff9)%
tests/SAMATE/overflow/Heap_overflow_15.cpp:13! %%%%#2%0x7ffff6d7bec4%(/lib/x86_643linux3gnu/libc32.19.so+0x21ec4)%
(a) DOUBLETAKEReport (b)AddressSanitizerReport
Figure6: ExamplereportsofDOUBLETAKEandAddressSanitizerforbufferoverflowidentification.
sivelyduringnormalexecution.Moreexpensiveinstrumentationis 9. ACKNOWLEDGEMENTS
onlyintroducedafteranerrorhasbeendetected. BoundsChecker ThismaterialisbaseduponworksupportedbytheNationalSci-
interposesonWindowsheaplibrarycallstodetectmemoryleaks, enceFoundationunderGrantNo.CNS-1525888.CharlieCurtsinger
use-after-free errors and buffer overflows [26]. Many prior ap- wassupportedbyaGooglePhDResearchFellowship. Wethank
proaches use a mix of library interposition and virtual memory Kostya Serebryany for his assistance with our evaluation of Ad-
techniquestodetectmemoryerrors[3,5,9,15,21,25,30,31,35, dressSanitizer,andQiangZeng,DinghaoWuandPengLiuforpro-
43],thoughtheiroverheadismuchhigherthanDOUBLETAKE’s. vidingthetestcasesusedintheirCruiserpaper.WealsothankScott
Recordandreplay:Severalreplay-basedapproachestargetsoft- Kaplanforhissuggestionsandcommentsduringthedevelopment
waredebuggingand/orfaulttolerance[6,19,36,37,39,40].Flash- of DOUBLETAKE. Finally, wethankJohnVilkforhisinvaluable
backsupportsreplaydebuggingbyemployingashadowprocessto suggestionsthathelpedimprovethispaper.
checkpointthestateofanapplication,andrecordingtheresultsof
systemcallstofacilitatethereplay.Triageusesreplaytoautomate 10. REFERENCES
the failure diagnosis process for crashing bugs [40]. Both Flash- [1] P.Akritidis,M.Costa,M.Castro,andS.Hand.Baggy
backandTriageneedcustomkernelsupport. boundschecking:anefficientandbackwards-compatible
AftersightistherelatedworkthatisclosestinspirittoDOUBLE- defenseagainstout-of-boundserrors.InProceedingsofthe
TAKE [10]. It separates analysis from normal execution by log- 18thconferenceonUSENIXsecuritysymposium,SSYM’09,
ginginputstoavirtualmachineandexportingthemtoaseparate pages51–66,Berkeley,CA,USA,2009.USENIX
virtualmachinefordetailed(slow)analysisthatcanrunofflineor Association.
concurrentlywithapplicationexecution.Aftersightmonitorsappli-
[2] G.AltekarandI.Stoica.ODR:Output-deterministicreplay
cationsrunninginavirtualmachine,whichaddssomeamountof
formulticoredebugging.InProceedingsoftheACM
workload-dependentoverhead.VM-basedrecordingaloneaddsad-
SIGOPS22ndSymposiumonOperatingSystemsPrinciples,
ditionalruntimeoverhead,anaverageof5%ontheSPECCPU2006
SOSP’09,pages193–206,NewYork,NY,USA,2009.
benchmarks.Aftersight’sdynamicanalysesareoffloadedtounused
ACM.
processors,whichmaynotbeavailableinsomedeployments. Un-
[3] H.AyguenandM.Eddington.DUMA-DetectUnintended
likeAftersight,DOUBLETAKEdoesnotrequiretheuseofavirtual
MemoryAccess.http://duma.sourceforge.net/.
machine,doesnotrelyonadditionalprocessorsfordynamicanal-
[4] E.D.Berger,B.G.Zorn,andK.S.McKinley.Composing
yses,andincursloweraverageoverhead.
high-performancememoryallocators.InProceedingsofthe
Speckisanotherreplay-basedsystemfocusedonsecuritycheck-
ACMSIGPLAN2001conferenceonProgramminglanguage
ing,includingtaintanalysisandvirusscanning[29].Securitychecks
designandimplementation,PLDI’01,pages114–124,New
generallyrequireapplicationstohaltimmediatelyupondetecting
York,NY,USA,2001.ACM.
anerror,functionalitythatDOUBLETAKEbydesigndoesnotpro-
[5] M.D.BondandK.S.McKinley.Bell:Bit-encodingonline
vide. Other systems have focused on reducing the performance
memoryleakdetection.InProceedingsofthe12th
overheadofrecording[2,22,34,41].
InternationalConferenceonArchitecturalSupportfor
ProgrammingLanguagesandOperatingSystems,ASPLOS
8. CONCLUSION XII,pages61–72,NewYork,NY,USA,2006.ACM.
Thispaperintroducesevidence-baseddynamicanalysis, anew [6] T.C.BressoudandF.B.Schneider.Hypervisor-basedfault
lightweightdynamicanalysistechnique. Evidence-baseddynamic tolerance.InProceedingsoftheFifteenthACMSymposium
analysisworksforerrorsthatnaturallyleaveevidenceoftheiroc- onOperatingSystemsPrinciples,SOSP’95,pages1–11,
currence,orcanbeforcedtodoso.Theseerrorsincludekeyprob- NewYork,NY,USA,1995.ACM.
lemsforCandC++programs: bufferoverflows,dangling-pointer [7] D.Bruening,T.Garnett,andS.Amarasinghe.An
errors,andmemoryleaks.Evidence-baseddynamicanalysisisfast infrastructureforadaptivedynamicoptimization.In
because it lets the application run at full speed until an error is ProceedingsoftheinternationalsymposiumonCode
detected; execution is then rolled back and replayed with instru- generationandoptimization:feedback-directedandruntime
mentationatthepointwheretheevidencewasfound,pinpointing optimization,CGO’03,pages265–275,Washington,DC,
theerror. WepresentDOUBLETAKE,anevidence-baseddynamic USA,2003.IEEEComputerSociety.
analysis framework, and implement these analyses using it. The [8] D.BrueningandQ.Zhao.Practicalmemorycheckingwith
resultinganalysesarethefastesttodate,imposingonaverageun- dr.memory.InProceedingsofthe9thAnnualIEEE/ACM
der5%overhead. Theseresultsdemonstratetheeffectivenessand InternationalSymposiumonCodeGenerationand
efficiency of this approach, which promises to speed testing and Optimization,CGO’11,pages213–223,Washington,DC,
dramaticallyincreasethereachofdynamicanalysisbyextending USA,2011.IEEEComputerSociety.
ittodeployedsettings.DOUBLETAKEisavailablefordownloadat [9] J.Caballero,G.Grieco,M.Marron,andA.Nappa.
http://github.com/plasma-umass/DoubleTake. Undangle:earlydetectionofdanglingpointersin