Provable defenses against adversarial examples via the convex outer adversarial polytope

21 Pages·2017·0.74 MB·English

Provable defenses against adversarial examples
via the convex outer adversarial polytope
J.ZicoKolter EricWong
ComputerScienceDepartment MachineLearningDepartment
Carnegie-MellonUniversity Carnegie-MellonUniversity
Pittsburgh,PA15213 Pittsburgh,PA15213
[email protected] [email protected]
Abstract
WeproposeamethodtolearndeepReLU-basedclassifiersthatareprovablyrobust
againstnorm-boundedadversarialperturbations(onthetrainingdata;forpreviously
unseenexamples,theapproachwillbeguaranteedtodetectalladversarialexam-
ples,thoughitmayflagsomenon-adversarialexamplesaswell). Thebasicideaof
theapproachistoconsideraconvexouterapproximationofthesetofactivations
reachablethroughanorm-boundedperturbation,andwedeveloparobustoptimiza-
tionprocedurethatminimizestheworstcaselossoverthisouterregion(viaalinear
program). Crucially,weshowthatthedualproblemtothislinearprogramcanbe
representeditselfasadeepnetworksimilartothebackpropagationnetwork,leading
toveryefficientoptimizationapproachesthatproduceguaranteedboundsonthe
robustloss. Theendresultisthatbyexecutingafewmoreforwardandbackward
passesthroughaslightlymodifiedversionoftheoriginalnetwork(thoughpossibly
withmuchlargerbatchsizes),wecanlearnaclassifierthatisprovablyrobustto
anynorm-boundedadversarialattack.Weillustratetheapproachonatoy2Drobust
classificationtask,andonasimpleconvolutionalarchitectureappliedtoMNIST,
whereweproduceaclassifierthatprovablyhaslessthan8.4%testerrorforany
adversarialattackwithbounded(cid:96) normlessthan(cid:15) = 0.1. Thisrepresentsthe
∞
largestverifiednetworkthatweareawareof,andwediscussfuturechallengesin
scalingtheapproachtomuchlargerdomains. Allcodeforexperimentsisavailable
athttp://github.com/locuslab/convex_adversarial.
1 Introduction
Recentworkindeeplearninghasdemonstratedtheprevalenceofadversarialexamples[Goodfellow
etal.,2015],datapointsfedtoamachinelearningalgorithmwhicharevisuallyindistinguishablefrom
“normal”examples,butwhicharespecificallytunedsoastofoolorsomehowmisleadthemachine
learningsystem.Recenthistoryinadversarialclassificationhasfollowedsomethingofavirtual“arms
race”: practionersalternativelydesignnewwaysof“hardening”classifiersagainstexistingattacks,
andthenanewclassofattacksisdevelopedthatcanpenetratethisdefense. Distillation[Papernot
etal.,2016]iseffectiveatpreventingadversarialexamplesuntilitisnot[CarliniandWagner,2017].
Thereisnoneedtoworryaboutadversarialexamplesunder“realistic”settingsofrotationandscaling
[Luetal.,2017]untilthereis[AthalyeandSutskever,2017]. Neitherdoesthefactthattheadversary
lacksfullknowledgeofthemodelappeartobeaproblem: “black-box”attacksarealsoextremely
effective[Papernotetal.,2017]. Giventhepotentiallyhigh-stakesnatureofmanymachinelearning
systems,wefeelthissituationisuntennable: the“cost”ofhavingaclassifierbefooledjustonceis
potentiallyextremelyhigh,andsotheattackersarethedefacto“winners”ofthiscurrentgame.
Webelievethatoneoftheonlywaystomakeprogressinusabledefensesistodevelopnetworksthat
areprovablyrobusttoadversarialattacks. Verificationofneuralnetworksisanactiveresearchtopic
31stConferenceonNeuralInformationProcessingSystems(NIPS2017),LongBeach,CA,USA.
z z
z^ z^
ℓ u ℓ u
BoundedReLUset Convexrelaxation
Figure1: IllustrationoftheconvexReLUrelaxationovertheboundedset[(cid:96),u].
[Huangetal.,2017,Katzetal.,2017,Ehlers,2017],whichtypicallyrequireacombinatorialsearch
overactivationnonlinearities(oftenviaasatisfiabilitysolver)combinedwithcontinuoussearchover
thelinearregions. Butthesetypicallyinvolvehardcombinatorialoptimizationproblemsthatlikely
willneverscaletothesizeofrealisticnetworks.
Inthisworkwetakeadifferentapproach: ratherthantrytoverifya“hard”network,wewanttolearn
anetworkthatisinherentlyeasytoverify. Specifically,weconsidertheadversarialpolytopeofa
ReLU-baseddeepnetwork,thesetofallfinal-layeractivationsachievablethroughaperturbationof
theinput. Althoughthisisahighlynon-convexset,wecanconsideraconvexouterapproximation
of the polytope, and efficiently optimize over this outer bound; if no adversarial example exists
withintheouterbound,nonecanexistwithintheactualadversarialpolytopeeither. Wespecifically
consideraconvexouterboundwhereoptimizationoverthisboundtakestheformofalinearprogram.
Cruciallyhowever(sincerepeatedlysolvinganLPwiththenumberofvariablesequaltothenumber
ofactivationsisnotpracticalforlargenetworks),weshowthatwecanwritethedualproblemof
thisLPintheformofanotherneuralnetwork,similartothebackpropagationnetwork,andwhich
providesguaranteedboundsontheoptimalsolutionofthelinearprogram. Theendresultisthatviaa
fewmoreforwardandbackwardpassesthroughthenetwork(thoughpotentiallywithlargerbatch
sizes,tocomputethenecessaryactivationbounds),wecanmakeguaranteedstatementsaboutthe
errorthatanyadversarialattackcanachieve.
Finally,weintegrateourapproachwithinarobustlearningframework. Robustoptimization[Ben-Tal
etal.,2009]hasbeenrecentlybroughttotheforefrontofresearchinadversarialexamples[Madry
etal.,2017],butthisworkfocusesonsolvingtherobustoptimizationproblemsheuristically,anddoes
notprovideguarantees. Byusingourconvexouterbound,wecanoptimizeaclassifier’sworst-case
loss over the entire outer convex approximation; if we are able to achieve suitably low training
error, this implies that the classifier is robust to any adversarial attack. A similar approach was
proposedconcurrentlytothisworkAnonymous[2018],thoughwithaverydifferentmethod,and
wediscussconnectionsinAppendixA.Weillustratetheresultsonasmalltoyproblemandona
convolutionalnetworkappliedtoMNIST,whereweproduceaclassifierwithaguaranteeoflessthan
8.39%errorforanynorm-boundedperturbationsofsize(cid:15)=0.1;althoughstillsmallscale,thislatter
caserepresentsthelargestverifiednetworkthatweareawareof.
2 Methodology
Tofixnotation,weconsideraklayerfeedforwardReLU-basedneuralnetwork,f : R|x| → R|y|
θ
givenbytheequations
zˆ =W z +b , i=1,...,k−1
i+1 i i i
z =max{zˆ,0},i=2,...,k−2
i i
(1)
z =x
1
f (x)≡zˆ
θ k
whereuseθ = {W ,...,W ,b ,...,b }todenotethesetofallparametersofthenetwork,
1 k−1 1 k−1
andwhereW representalinearoperatorsuchasmatrixmultiplyorconvolution. Weusetheset
i
Z (x)todenotethesetofallfinal-layeractivationsattainablebyperturbingxbysome∆with(cid:96)
(cid:15) ∞
normboundedby(cid:15). Z (x)isanon-convexset(itcanberepresentedviaanintegerprogramasin
(cid:15)
[LomuscioandMaganti,2017]),socannoteasilybeoptimizedover.
Aconvexouterbound Thestartingpointofourapproachistoconsideraconvexouterboundon
theadversarialpolytope. Specifically,givenknownlowerandupperbounds(cid:96),uforthepre-ReLU
activation,wecanreplacetheReLUconstraintswiththeirupperconvexenvelope,asshowninFigure
2
5;thesamerelaxationattheactivationlevelwasusedin[Ehlers,2017]thoughthereitwasusedasa
sub-stepforexact(combinatorial)verificationofnetworks,andtheprocedureforactuallycomputing
thecrucialbounds(cid:96)anduisdifferent. WedenotethisouterboundZ˜ (x).
(cid:15)
Efficientoptimization Theproblemoffindingthe“mostadversarial”examplewithintheouter
convexadversarialpolytopecanbecastasthefollowingoptimizationproblem
minimizecTzˆ, subjecttozˆ ∈Z˜ (x) (2)
k k (cid:15)
zˆk
wherecisavectorimplyingwhichactivationswewanttominimizeandwhichwewanttomaximize;
withtheReLUapproximationabovethisproblembecomesalinearprogram,whichtheoretically
couldbesolvedwithoff-the-shelfsolvers. However,althoughLPsolversare“efficient”,theresulting
problemshaveaoptimizationvariableassociatedwitheachhiddenunitinthenetwork,whichwould
notbepracticallyfeasibletosolve. Instead,wefocusonthedualproblemofthisLP.Asshownin
theAppendix,wecanshowthatthefeasiblesetofthedualproblemcanalsobeexpressedasadeep
network,namelythenetwork
ν =−c
k
νˆ =WTν ,i=k−1,...,1
i i i+1
 0 j ∈I− (3)
 i
ν = νˆ j ∈I+
i,j i,j i
 ui,j [νˆ ] −α [νˆ ] j ∈I
ui,j−(cid:96)i,j i,j + i,j i,j − i
whereαisasetoffreevariables,andwhereI−,I+,andI denotethesetofactivationswherethe
i i i
lowerandupperboundsarebothnegative,bothpositive,orspanzerorespectively; thisinfactis
extremelysimilartothebackpropagationnetworkforReLU-basedactivations,exceptfortheterms
inI . Theobjectivetermofthedualproblemisthengivenby
i
k−1 k−1
(cid:88) (cid:88)(cid:88)
J(ν)=− νT b −xTνˆ −(cid:15)(cid:107)νˆ (cid:107) + (cid:96) [ν ] (4)
i+1 i 1 1 1 i,j i,j +
i=1 i=2j∈Ii
whichtherefore,foranyassignmenttotheα’s,givesaguaranteedlowerboundontheoriginalLP
(despitethefactthatthisformulationofthedualisnotconvex).
Computingactivationbounds Finally,wenotethatwecanusethisdualproblemformulationto
iterativelyfindtheactivationlowerandupperbounds(cid:96) andu bychoosingc=Iorc=−I.This
i,j i,j
canbemadepracticalby1)choosingafixedsolutionα =u /(u −(cid:96) )touseforcomputing
i,j i,j i,j i,j
all bounds; and 2) by caching terms in the dual network computation. The end result is that we
cancomputeboundsforalllayersviaasingleforwardpassthroughthenetwork,albeitatthecost
ofrunningasingleexamplethroughthenetworkforeachinputdimension. Thisisadmittedlythe
poorest-scaling aspect of our approach, and comes from the non-linear terms in (4). However, a
numberofapproachestoscalingthistolarger-sizedinputsispossible,includingbottlenecklayers
earlierinthenetwork,e.g. PCAprocessingoftheimages,probabilisticnormbounds,orothersimilar
constructs.
Robust optimization After all this has been done, we can replace the standard empirical loss
minimizationwithaguaranteedupperbound
N
(cid:88)
minimize L(−J(g (I−e 1T,α )),y ) (5)
θ,α θ,xi,(cid:15) yi i i
i=1
whereg denotesthedualnetworkandLdenotesalossfunction. Essentially,wearereplacing
θ,xi,(cid:15)
thelossintheminimizationproblemwithaguaranteedboundderivedfromthedualproblem. Allthe
networkterms,includingtheupperandlowerboundcomputation,aredifferentiable,sothewhole
optimizationcanbesolvedwithanystandardstochasticgradientvariant,andtheresultisanetwork
that(ifweachievelowloss)isguaranteedrobusttoadversarialexamples.
3
100
Train Robust Error Train Robust Loss
Train Error Train Loss
sification Error10­1 TTeesstt  RErorbourst Error s Entropy Loss100 TTeesstt  RLoosbsust Loss
s s
a o
Cl Cr
10­1
10­2
0 5 10 15 20 0 5 10 15 20
Epoch Epoch
Figure2: TrainingprogressforourmethodappliedtoasimpleConvNetarchitectureonMNIST.
Robusterrorandlossindicateprovableupperboundsontheloss/errorachievablebyanyadversarial
perturbationwith(cid:96) normboundedby(cid:15)=0.1.
∞
3 Experiments
WepresentexperimentalresultsofrobustlytrainingaconvolutionalclassifierontheMNISTdataset.
Codefortheexperimentshereandadditionalexperimentsonasimple2Drobustclassificationtask
isavailableathttp://github.com/locuslab/convex_adversarial. WeuseasimpleLeNet-
stylenetwork,withmaxpoolinglayersreplacedbystride-2convolutionsandReLUsafteralllayers,
andweuseAdamKingmaandBa[2015]tooptimizenetworkparameters. Specifically,ournetwork
isoftheform:
x⇒Conv_16x4x4,s=2⇒ReLU⇒Conv_16x4x4,s=2⇒ReLU⇒FC100⇒ReLU⇒y. (6)
Figure9showsthetrainingprogressusingourprocedurewitharobustsoftmaxlossfunctionand
(cid:15)=0.1. The“robusterror”and“robustloss”hereareourboundsontherobusterrorandloss;thatis,
weknowthatanynorm-boundedadversarialtechniquewillnotbeabletoachievelossorerrorthatis
anyhigher. Thefinalclassifierafter20epochsreachesatesterrorof2.93%witharobusttesterrorof
8.39%. Wealsotestedournetworkagainsttwocommonclassesofattacks: thefastgradientsign
methodGoodfellowetal.[2015],andtheprojectedgradientdescentapproachMadryetal.[2017].
Foratraditionally-trainedclassifier(with1.2%testerror)theFGSMapproachresultsin39.7%error,
while PGD results in 94.0% error. On the classifier trained with our method, however, we only
achieve2.93%testerror,butFGSMandPGDonlyachieveerrorsof5.8%and6.2%respectively
(both,naturally,belowourboundof8.4%). TheseresultsaresummarizedinTable2. Althoughthese
resultsarerelativelysmall-scale,thesomewhatsurprisingabilityhereisthatbyjustconsideringafew
moreforward/backwardpassesinamodifiednetworktocomputeanalternativeloss,wecanderive
guaranteederrorboundsforanyadversarialattack. Thisrepresentsthelargestverifiedclassifierthat
weareawareof,andbyavoidinganycombinatorialoptimization,hasthepotentialtoscaletomuch
largerproblemsstill.
4 Conclusion
This work proposed a simple method for training a verifiable network with guaranteed bounds
onadversarialerror. Thoughwebelieveitrepresentsasignificantstepindealingwithadversarial
examples,futureworkremainsin1)gettingthemethodtoscaletostate-of-the-art-sizednetworks,and
2)incharacterizingadversarialinputsbeyondsimplenorm-bounds. Further,thegenerictechniques
ofboundingoptimizationoveraneuralnetwork,usingadualnetwork,islikelytofindadditional
applicationsindeeplearning,asmanyproblemsinvolveoptimizingoverinputstoanetwork.
Table1: Errorsforastandardandourrobustly-trainedclassifierontheMNISTtestset.
Network Non-adversarial FGSM PGD RobustBound
Standardtraining 1.2% 39.7% 94.0% 100%
Robusttraining 2.9% 5.8% 6.2% 8.4%
4
References
Anonymous. Certifieddefensesagainstadversarialexamples. InUnderreviewatICLR2018,2018.
Anish Athalye and Ilya Sutskever. Synthesizing robust adversarial examples. arXiv preprint
arXiv:1707.07397,2017.
Aharon Ben-Tal, Laurent El Ghaoui, and Arkadi Nemirovski. Robust optimization. Princeton
UniversityPress,2009.
Emmanuel J Candes and Terence Tao. Near-optimal signal recovery from random projections:
Universalencodingstrategies? IEEEtransactionsoninformationtheory,52(12):5406–5425,2006.
Nicholas Carlini and David Wagner. Towards evaluating the robustness of neural networks. In
SecurityandPrivacy(SP),2017IEEESymposiumon,pages39–57.IEEE,2017.
MoustaphaCisse,PiotrBojanowski,EdouardGrave,YannDauphin,andNicolasUsunier. Parseval
networks: Improvingrobustnesstoadversarialexamples. InInternationalConferenceonMachine
Learning,pages854–863,2017.
RuedigerEhlers. Formalverificationofpiece-wiselinearfeed-forwardneuralnetworks. InInterna-
tionalSymposiumonAutomatedTechnologyforVerificationandAnalysis,2017.
IanGoodfellow, JonathonShlens,andChristianSzegedy. Explainingandharnessingadversarial
examples. InInternationalConferenceonLearningRepresentations,2015. URLhttp://arxiv.
org/abs/1412.6572.
XiaoweiHuang,MartaKwiatkowska,SenWang,andMinWu. Safetyverificationofdeepneural
networks. InInternationalConferenceonComputerAidedVerification, pages3–29.Springer,
2017.
GuyKatz,ClarkBarrett,DavidDill,KyleJulian,andMykelKochenderfer. Reluplex: Anefficient
smtsolverforverifyingdeepneuralnetworks. arXivpreprintarXiv:1702.01135,2017.
DiederikKingmaandJimmyBa. Adam: Amethodforstochasticoptimization. InInternational
ConferenceonLearningRepresentations,2015.
AlessioLomuscioandLalitMaganti. Anapproachtoreachabilityanalysisforfeed-forwardrelu
neuralnetworks. arXivpreprintarXiv:1706.07351,2017.
Jiajun Lu, Hussein Sibai, Evan Fabry, and David Forsyth. No need to worry about adversarial
examplesinobjectdetectioninautonomousvehicles. arXivpreprintarXiv:1707.03501,2017.
Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu.
Towardsdeeplearningmodelsresistanttoadversarialattacks. arXivpreprintarXiv:1706.06083,
2017.
NicolasPapernot,PatrickMcDaniel,XiWu,SomeshJha,andAnanthramSwami. Distillationasa
defensetoadversarialperturbationsagainstdeepneuralnetworks. InSecurityandPrivacy(SP),
2016IEEESymposiumon,pages582–597.IEEE,2016.
NicolasPapernot,PatrickMcDaniel,IanGoodfellow,SomeshJha,ZBerkayCelik,andAnanthram
Swami. Practicalblack-boxattacksagainstdeeplearningsystemsusingadversarialexamples. In
Proceedingsofthe2017ACMAsiaConferenceonComputerandCommunicationsSecurity,2017.
HuanXu,ConstantineCaramanis,andShieMannor. Robustnessandregularizationofsupportvector
machines. JournalofMachineLearningResearch,10(Jul):1485–1510,2009.
Appendix
Inthisappendixweprovideamuchmoredetaileddescriptionofthegeneraladversarialpolytope,
ourconvexouterapproximationandalgorithms,andfurtherexperimentalresults.
5
A Relationtoconcurrentwork
Concurrently with this publication, Anonymous [2018] released a paper on similar methods for
learningclassifiersprovablyrobusttoadversarialexamples. Althoughverysimilaratahighlevel
(both techniques employ an outer bound on the adversarial polytope, and use the dual problem
to provide a guaranteed bound on optimization), the actual bound and methodology of the two
approachesisquitedifferent. Inparticular,whilethispaperproposesalinearprogrammingbased
outerboundontheadversarialpolytope,theotherworkproposesasemidefiniteprogrammingbased
bound,andcurrentlyfocusesonlyonthesettingoftwo-layernetworks. WhileSDPrelaxationsin
generalhavetheabilitytobetighterthanLPrelaxations,thenotableadvantageoftheLPrelaxation
hereisthatitallowsforasimpleboundofadualproblemviaadualnetwork;incontrasttheSDP
boundrequiresamorecomplicatedoptimizationschemethatinvolvesoptimizingtheeigenvalueofa
particularmatrixviaaLanczositeration,anditsomewhatuncleartoushowtheapproachextends
tomultilayernetworksand/oroperatorslikeconvolutionsthataretooexpensivetoexpressintheir
matrixform(thoughextensionstotheSDPboundmaybepossibletohandlethesecases). Finallyin
termsofthereportedresultsthemselves,thisotherworkisabletoachieveaprovableboundof35%
erroronMNISTwith(cid:15) = 0.1,whiletheapproachwepresentachievesaprovableboundof8.4%
error.
B Theadversarialpolytope
Webeginbyconsideringacharacterizationoftheexactadversarialpolytopeofadeepnetwork;avery
similarapproachisusedinLomuscioandMaganti[2017],thoughweincludeithereforcompleteness,
as the adversarial polytope plays a foundational role in our subsequent main contribution in the
followingsection.
Tofixnotation,weconsideraklayerfeedforwardReLU-basedneuralnetwork,f : R|x| → R|y|,
θ
wheretheoutputofthenetworkisgivenbythepre-softmaxfinal-layeractivations(sothattheywould
correspondtothelogitsofamulti-classclassificationarchitecture). Thenetworkstructureisgivenby
theequations
zˆ =W z +b , i=1,...,k−1
i+1 i i i
z =max{zˆ,0},i=2,...,k−2
i i
(7)
z =x
1
f (x)≡zˆ
θ k
where we will use θ = {W ,...,W ,b ,...,b } to denote the set of all parameters of the
1 k−1 1 k−1
network. AlthoughwewritethelineartermsinthenetworkasmatrixoperationsW z +b ,wenote
i i i
thatW couldbeanylinearoperatorsuchasaconvolution(andwewillindeedconsiderconvolutions
i
in our experimental results). We also are assuming that the network here only has ReLU non-
linearities. Themethodweproposecanpotentiallybeextendedtootherpolytopicactivationssuchas
max-poolingorleakyReLUs,butweleavethisforfuturework.1 Wealsonotethatwecanintegrate
morecomplexnetworksthathavepassthroughlayersormorecomplexstructure,butagainweuse
thesimplefeedforwardnetworkforsimplicityofnotationthroughoutthepaper.
Nowweconsiderasetofallowableperturbationsx+∆where(cid:107)∆(cid:107) ≤(cid:15)(weusethe(cid:96) normto
∞ ∞
boundadversarialexamples,asthisisoneofthemostcommonsettingsinpractice,thoughother
normsarepossibleaswell). Theadversarialpolytopeisdefinedasthesetofallpossiblez terms
k
achievableunderthisperturbation
Z (x)={zˆ |zˆ =f(x+∆),(cid:107)∆(cid:107) ≤(cid:15)} (8)
(cid:15) k k ∞
TheconceptualideaoftheadversarialpolytopeisillustratedinFigure3. Thissetprovidesuswithall
theinformationweneedaboutadversarialexamples(atleastthosedefinedbythesimpleperturbations
definedabove,whichalreadyencompassapowerfulset). Wewilldiscussthisinmoredetailshortly,
butoptimizingovertheadversarialpolytopecantellus,forinstance,whetherthereisanyexample
within(cid:15)ofxthatcanchangetheclassoftheexample. Italsoformsthebasisfordevelopingarobust
1We can also potentially integrate other non-piecewise-linear activations such as sigmoids, though this
requiresanapproximationifwedowanttousepolyhedralmethods, sowefocushereonpiecewise-linear
activations.
6
Input x and Final layer z^ and
k
allowable perturbations Deep network adversarial polytope
Figure3: Illustrationoftheadversarialpolytope(i.e.,the“reachable”regionforagivenallowable
perturbation).
optimizationapproachthatguaranteesthatnoadversarialexamples(atleastonthetrainingdata)are
possible.
Although it may not be apparent, Z (x) is a connected polytope, a property which follows from
(cid:15)
thefactthattheentirefunctioniscomposedoflinearoperationsandthenon-linearReLUoperator,
whichisstilldefinedbya(non-convex)polytope: fory =ReLU(x),theReLUisdefinedbytheset
(y =0∧x≤0)∨(y =x∧x>0)). Indeed,wecanexpressthetrueadversarialpolytopeinterms
ofthebilinearintegerconstraints
zˆ =W z +b , i=1,...,k−1
i+1 i i i
z =zˆ ·y , i=2,...,k−1
i i i
z1 ≤x+(cid:15) (9)
z ≥x−(cid:15)
1
y ∈{0,1}|zi|
i
wherey isasetofbinaryintegervariablesthatencode“whichside”oftheReLUthepre-ReLU
i
activationszˆ lieson. AlthoughthisisnotinaformthatcanbeeasilyhandledbyexistingIPsolvers
i
becauseofthebilineartermzˆ ·y ,itcanbeputintoanallowableformusingastandardtrickfrom
i i
integerprogrammingknowaslinearization. Specifically,supposingthatwehavesomelowerand
upperbounds(cid:96) ,u onthepre-ReLUactivationszˆ (fortheintegerprogrammingformulations,these
i i i
boundscanbeveryloose, socanbederivedforinstancefromLipschitzboundsofthenetwork),
thentheabovesetofconstraintsisequivalenttothemixedintegerconstraints(wedon’tcoverthis
transformationindetailhere,butitisaverycommonpatternintheIPliterature)
zˆ =W z +b , i=1,...,k−1
i+1 i i i
zˆ ≤u ·y , i=2,...,k−1
i i i
z ≥zˆ, i=2,...,k−1
i i
z ≥0, i=2,...,k =1
i
(10)
z ≤zˆ −(1−y )·(cid:96)
i i i i
z ≤x+(cid:15)
1
z ≥x−(cid:15)
1
y ∈{0,1}|zi|.
i
Itissimpletocheckthatify =0,thisimpliesthecorrespondingz =0(theconstraintssimplify
i,j i,j
toz ≥ 0andz ≤ 0);similarly,fory = 1,theconstraintsimplythatz = zˆ . Thisform,
i,j i,j i,j i,j i,j
whichnowcanbefedtooff-the-shelfsolvers,letsusreasonableabouttheexactnatureofadversarial
examplesforaclassifier.
B.1 Querieswithintheadversarialpolytope
The adversarial polytope allows us to formalize the notion of finding or preventing adversarial
examplesinaclassifier. Consideringsomedatapoint/labelpair(x,y),aswellasatargetclassy
targ
(theclassthatwearegoingtoattempttofooltheclassifierintoproducing),wecandeterminewhether
ornotthereexistsanadversarialexample(within(cid:15)(cid:96) normofx)thatmakesthelabely more
∞ targ
7
likelythatybysolvingtheoptimizationproblem
minimize(z ) −(z ) , subjectto z ∈Z (x). (11)
k y k ytarg k (cid:15)
Iftheoptimalobjectiveofthisoptimizationproblemislessthan0,thentheredoesexistanadversarial
examplethatmakesthelabely morelikelythany;thisfollowsimmediatelyfromthenatureofthe
targ
optimizationproblem: iftheobjectiveisnegative,thenthevalueofthey activationisgreaterthan
targ
theyactivation,whichguaranteesthedesiredproperty. Thisoptimizationproblemisamixedinteger
linearprogram,and(usingthepolytopeformulationprovidedin(10))canbesolvedbyanumberof
off-the-shelfintegerprogrammingsolvers. Otherquestionscanbeansweredaswell: forinstanceif
wewanttofindtheminimum-perturbationadversarialexamplepossible,thenwecansolvetheMILP
minimize (cid:15), subjectto(z ) ≤(z ) , z ∈Z (x). (12)
k y k ytarg k (cid:15)
Besidescreatingadversarialexamples,theadversarialpolytopecanalsobeusedtotestthepotential
adversarial nature of unknown examples. If we are provided an example x ∈ Rn as input to a
classifier,anaturalquestiontoaskis: isthispotentiallyanadversarialexample,e.g.,anexample
thathasbeencraftedtofoolourclassifier. Whileitisdifficulttoanswerthisquestioningeneral,
insomecasesitispossibletoguaranteethataparticularexampleisnotadversarial,atleastunder
thesetofallowableperturbations. Specifically,ifwelety = argmax f (x)bethepredicted
pred i i
classoftheexample,thenwecancheck(usingtheapproachabove)whetherthereareadversarial
examplesforally (cid:54)=y . Ifnosuchexampleexists,thenweknowthatthenetworkclassifies
targ pred
thesamplexasbeingthesameclassfor all(cid:15)perturbations,i.e.,itcouldnotbeaninstanceofa
“good”example(whichwouldbeclassifiedcorrectly)perturbedby(cid:15)tobeclassifiedincorrectly. This
allowsusamethodforguaranteeingthatanexampleisnotadversarial,evenifwecannotdothe
converse,guaranteeanexamplethatdoescrossclasseswithintheadversarialpolytopeisactually
adversarial. However,becauseweknowfromexperiencethatmostdeepclassifiersareeasytofool,it
islikelythatatypically-trainednetworkwillflagmostinputsasbeingpotentiallyadversarial;thus,
aswewilladdressnext,weneedadditionalmethodsfortrainingclassifierssuchthattheyarelikely
toproduceclassifiersnotsensitivetoadversarialperturbations.
B.2 Robustoptimizationandlearningrobustclassifier
Finally,wediscusshowwecanusetheadversarialpolytopetodevelop“hardened”classifiers(more)
robusttoadversarialexamples,baseduponapproachesinrobustoptimization. Robustoptimization
inthecontextoflinearclassifiershasbeenusedforsometimewithinmachinelearning(e.gXuetal.
[2009]). Thebasicideaisthat,givenatrainingsetofx ,y pairs,insteadofsimplyminimizingthe
i i
lossatthesedatapoints,weminimizethelossattheworstlocation(i.e. thatwiththehighestloss)in
an(cid:15)ballaroundeachx . Assumingalinearclassifierh (x) = θTx(andbinaryclassification,for
i θ
simplicity),thisresultsintheoptimizationproblem
N
(cid:88)
minimize max (cid:96)(θT(x+∆)·y ) (13)
i
θ (cid:107)∆(cid:107)≤(cid:15)
i=1
Usingthethefactthat
max θT∆=(cid:15)(cid:107)θ(cid:107) (14)
∗
(cid:107)∆(cid:107)≤(cid:15)
where(cid:107)·(cid:107) denotesthedualnorm,therobustoptimizationproblemcanbesimplifiedas
∗
N
(cid:88)
minimize max (cid:96)(θTx·y +(cid:107)θ(cid:107) ). (15)
i ∗
θ (cid:107)∆(cid:107)≤(cid:15)
i=1
Asmentionedabove,recentworkhashighlightedtheconnectionbetweenadversarialclassification
androbustoptimizationMadryetal.[2017],inparticularattemptingtosolvethenon-convexanalogue
ofthenativemin-maxproblem(13)by(projected)gradientdescentoverθand∆;thisconnectionto
robustoptimizationwasalsodiscussedintheoriginaladversarialexamplepaperGoodfellowetal.
[2015],withthefastgradientsignmethodreducingtothesamesolutionofthismin-maxproblemfor
alinearclassifier.
Toconsiderhowtousetheadversarialpolytopetomoreexactlyformulatetherobustoptimization
problem,weconsidertrainingadeepclassifierwithamulticlasshingeloss;otherlossessuchasthe
8
True adversarial polytope
Convex outer bound
(z^ )
k 2
(z^ )
k 1
Figure4:Simpleconceptualillustrationofthe(non-convex)adversarialpolytope,andanouterconvex
bound.
morecommoncross-entropylosscanbeeasilyapproximated,buttheconnectionismoreobvious
withthisloss. Optimizingsuchadeepnetworkisgivenbytheoptimizationproblem
N
(cid:88)
minimize [1+maxf (x)−f (x)] . (16)
θ i=1 y(cid:54)=yi y yi +
Thislossiszeroifthetrueclassoutputf (x)isatleast1largerthananyalternativeclassoutput
yi
f (x)fory (cid:54)=y ,andscaleslinearlywiththelargestactivationotherthanthetrueclassotherwise.
y i
Because the term f (x)−f (x) is exactly the objective that we considered previously, we can
y yi
encodethemin-maxformulationofrobustoptimizationbyconsidering|y|−1differentclassification
problems(oneforeachalternativelabely (cid:54)=y )overtheadversarialpolytope
i
N
(cid:88)
minimize max [1+maxf (x+∆)−f (x+∆)]
θ i=1(cid:107)∆(cid:107)∞≤(cid:15) y(cid:54)=yi y yi +
(17)
N
(cid:88)
≡minimize [1+maxmax(x )((zˆ ) −(zˆ ) )] .
θ i=1 y(cid:54)=yizˆ∈Z(cid:15) i k y k yi +
Theinnermaximizationispreciselytheoptimizationproblemweconsideredearlieroffindingthe
mostextremepointwithintheadversarialpolytope. Ifthisobjectivecanbeminimizedwithzero
training loss, then we are guaranteed that for every data point in the training set, no adversarial
exampleispossible(allpointswithintheadversarialpolytopearelabeledcorrectlybytheclassifier).
Testpointsnaturallyhavenosuchguarantee,butwecanstillusethemethoddescribedabovetotest
whetherornottheyarepotentiallyadversarial. Andifempiricalgeneralizationpatternsfordeep
networksremaininthissetting,itislikelythatclassifierswhichdon’tadmittoadversarialexamples
onthetrainingsetmaywellhavesimilarpropertiesonthetestset(andwewilldemonstratethis
empiricallyourselvesshortly).
C Convexouterboundsontheadversarialpolytope
Despiteallthepromiseoftheprecedingsection,thereisanotabledisadvantagethatrendersthis
approachmoreaconceptualstrategyratherthananactualtool: thefactthatactuallysolvingamixed
binaryintegerprogramisanNP-hardtask. Althoughitiswell-appreciatedthatmanyinstancesof
reasonably-sizedbinaryintegerprogramsareempiricallytractableviabranchandcutalgorithms,in
thiscasetheoptimizationproblemofinteresthasanumberofbinaryvariablesequaltothenumberof
hiddenunitsinthenetwork,anumberthatcaneasilygrowintothemillionsforlargedeepnetworks.
Thisissimplynottractabletosolveviaexactsolutionstotheintegerprogram.
Instead, the key claim of this section is that we can instead consider a convex relaxation of the
adversarial polytope that is a strict outer bound. This concept is illustrated in Figure 4. That is,
it contains all points (and more) within the adversarial polytope, but is a convex set and so can
beoptimizedoverefficiently(atleastaccordingtosomedefinitionof“efficient”,whichwillbea
majordiscussionpointthroughoutthissection). Thekeypointisthatbecauseofthisouterbound
property,wecanperformthesametestsforpotentialadversarialexamplesordevelopthesamerobust
optimizationtechniques,whichwillprovideidenticalguarantees. Thatis,ifwelookforthe“most
adversarial”exampleoverourouterapproximation,andfindthatitdoesnotchangethepredicted
9
z z
z^ z^
ℓ u ℓ u
Bounded ReLU set Convex relaxation
Figure5: IllustrationoftheconvexReLUrelaxationovertheboundedset[(cid:96),u].
classofanexample,thenweareguaranteedthatnoactualadversarialexamplecanchangetheclass
either. Orwecandeveloprobustclassifiersthatoptimizeovertheworst-caselosswithintheouter
approximation,andthese(assumingzerotrainingloss)alsowillguaranteethattheclassifierdoesnot
allowforadversarialexamples. 2.
Inthissectionwewillfirstdefineourconvexouterapproximationtotheadversarialpolytope,and
discusstheoptimizationproblemthatariseswhenoptimizingoverit. However,althoughthisisa
convexoptimizationproblemandhence“tractable”,itisfarfrompracticallyefficienttosolvethis
optimizationproblem exactly forlargenetworks. Thus, in thenextsection weconsider the dual
problemofoptimizationovertheconvexouterbound. Bystandardresultsinconvexity,anydual
feasiblesolutionrepresentsalowerboundonminimizationovertheconvexadversarialapproximation,
andasweshow,thedualproblemitselfcanbeformulatedasadeepnetworkthatiseffectivelya
typeof“adjointsystem”oftheoriginalnetwork,andwhicheasilyallowsustoattaindualfeasible
points;thus,wecanemploysimplegradientdescentmethodstooptimizethisdualproblem,giving
usguaranteedlowerboundsontheoriginaloptimizationproblem,evenifwedon’tsolvethedual
problem to optimality. Next, we discuss how this dual framework can be fit into the context of
learningarobustclassifier,showingthatwecanindeedsolvetherobustclassifierlearningproblem
usingthedualproblem.Finally,wediscussmethodsforfindingthenecessaryupperandlowerbounds
neededfortheconvexouterapproximation.
C.1 Aconvexouterbound
Notethatthedefinitionofouradversarialpolytopeinvolvedtwotypesofterms: thelinearterms
zˆ =W z +b (whichareconvex),andthenon-convexReLUtermsz =max{0,zˆ}. Thus,to
i+1 i i i i i
formaconvexrelaxationofthisset,weneedtoreplacetheReLUtermsbysomerelaxedversion.
The idea that underlies our convex outer bound is quite simple, and indeed has been previously
consideredinthecontextofsearchingovertheactualadversarialpolytopeEhlers[2017];however,as
mentionedabove,thispreviousworkfocusedontheouterboundasausefulstepinsearchingover
thetrueadversarialpolytopeforagivenclassifier,whileweconsiderithereasanouterboundthatwe
willexplicitlytrainover. However,wedoemphasizethattheformalisminthissectionatthesingle
activationlevelisvirtuallyidenticaltothatpreviouslyproposedinthispriorwork,thoughbecause
ourfullboundwillalsoinvolveiterativecomputationofupperandlowerboundsforintermediate
activations,ourfinalouterboundendsupbeingquitedifferent.
Supposethatforsomepre-ReLUactivationzˆandadversarialpolytopeZ (x)wehaveanupperand
(cid:15)
lowerbound(cid:96)anduovertheallowablevaluesfortheactivation(intheremainderofthisparagraph,
forsimplicity,we’llletzˆandzdenotescalarvaluesforasingleactivation,whichwe’lllaterextend
elementwise to all activations). Note that these upper and lower bounds are guaranteed to exist,
asperturbationof(cid:15)oftheinputcanonlychangetheactivationsofthenetworkbysomebounded
amount(thisiseffectivelythe“adversarial”polytopebutforinneractivationsofthenetwork). Then
wecanreplacethenon-convexReLUconstraintbyit’sconvexhullovertheallowableupperand
lowerbounds. Thisreducestothreecases:
1. If(cid:96)≤u≤0,thenwecanreplacetheconstraintz =max{0,zˆ}withtheconstraintz =0,
astheReLUwillalwayslieinthezeroregion.
2Note,however,thattheouterapproximationislessusefulforactuallyfindadversarialexamples:aswewill
see,becauseelementsinthispolytopemaynotcorrespondtoanyactualpaththroughtheoriginalnetwork,itis
unclearwhatifanythingcanbesaidabouttheactual“examples”foundviathismethod. However,giventhe
relativeeaseoffindingversuspreventingadversarialexamplesinmoderndeeparchitectures,thisisnotoverly
bothersome
10