stochastic and adversarial bandits

23 Pages·2012·0.34 MB·English

JMLR:WorkshopandConferenceProceedingsvol(2012)1–23
The best of both worlds: stochastic and adversarial bandits
Se´bastienBubeck [email protected]
DepartmentofOperationsResearchandFinancialEngineering
PrincetonUniversity
Princeton,NJ,USA
AleksandrsSlivkins [email protected]
MicrosoftResearch
MountainView,USA
Abstract
We present a new bandit algorithm, SAO (Stochastic and Adversarial Optimal) whose regret is
(essentially) optimal both for adversarial rewards and for stochastic rewards. Specifically, SAO
√
combines the O( n) worst-case regret of Exp3 (Auer et al., 2002b) and the (poly)logarithmic
regret of UCB1 (Auer et al., 2002a) for stochastic rewards. Adversarial rewards and stochastic
rewards are the two main settings in the literature on multi-armed bandits (MAB). Prior work on
MABtreatsthemseparately,anddoesnotattempttojointlyoptimizeforboth. Thisresultfallsinto
the general agenda to design algorithms that combine the optimal worst-case performance with
improvedguaranteesfor“nice”probleminstances.
1. Introduction
Multi-armed bandits (henceforth, MAB) is a simple model for sequential decision making under
uncertainty that captures the crucial tradeoff betweenexploration (acquiring new information) and
exploitation (optimizing based on the information that is currently available). Introduced in early
1950-ies (Robbins, 1952), it has been studied intensively since then in Operations Research, Elec-
tricalEngineering,Economics,andComputerScience.
The “basic” MAB framework can be formulated as a game between the player (i.e., the algo-
rithm) and the adversary (i.e., the environment). The player selects actions (“arms”) sequentially
from a fixed, finite set of possible options, and receives rewards that correspond to the selected
actions. For simplicity, it is customary to assume that the rewards are bounded in [0,1]. In the ad-
versarialmodelonemakesnootherrestrictionsonthesequenceofrewards,whileinthestochastic
model we assume that the rewards of a given arm is an i.i.d sequence of random variables. The
performancecriterionistheso-calledregret,whichcomparestherewardsreceivedbytheplayerto
therewardsaccumulatedbyahypotheticalbenchmarkalgorithm. Atypical,standardbenchmarkis
thebestsinglearm. SeeFigure1foraprecisedescriptionofthisframework.
Adversarial rewards and stochastic rewards are the two main reward models in the MAB lit-
erature. Both are now very well understood, in particular thanks to the seminal papers (Lai and
Robbins, 1985; Auer et al., 2002a,b). In particular, the Exp3 algorithm from (Auer et al., 2002b)
√
attains a regret growing as O( n) in the adversarial model, where n is the number of rounds, and
UCB1 algorithm from (Auer et al., 2002a) attains O(logn) in the stochastic model. Both results
(cid:13)c 2012S.Bubeck&A.Slivkins.
BUBECKSLIVKINS
Knownparameters: K arms;nrounds(n≥K ≥2).
Unknownparameters(stochasticmodel):
K probabilitydistributionsν ,...,ν on[0,1]withresp. meansµ ,...,µ .
1 K 1 K
Foreachroundt=1,2,...,n;
(1) algorithmchoosesI ∈{1,...,K},possiblyusingexternalrandomization;
t
(2) adversarysimultaneouslyselectsrewardsg =(g ,...,g )∈[0,1]K.
t 1,t K,t
- inthestochasticmodel,eachrewardg ∼ν isdrawnindependently.
i,t i
(3) theforecasterreceives(andobserves)therewardg .
It,t
Hedoesnotobservetherewardsfromtheotherarms.
Goal: Minimizetheregret,definedintheadversarialmodelby:
n n
(cid:88) (cid:88)
R = max g − g ,
n i∈{1,...,K} i,t It,t
t=1 t=1
andinthestochasticmodelby:
n (cid:18) (cid:19)
(cid:88)
R = max µ −µ .
n i∈{1,...,K} i It
t=1
Figure1: TheMABframework: adversarialrewardsandstochasticrewards.
are essentially optimal. It is worth noting that UCB1 and Exp3 have influenced, and to some extent
inspired,anumberoffollow-uppapersonricherMABsettings.
However,itiseasytoseethatUCB1incursatrivialΩ(n)regretintheadversarialmodel,whereas
√
Exp3 has Ω( n) regret even in the stochastic model.1 This raises a natural question that we aim
toresolveinthispaper: canweachievethebestofbothworlds? Isthereabanditalgorithmwhich
matchestheperformance ofExp3intheadversarial model, andattainstheperformance ofUCB1if
therewardsareinfactstochastic? Amorespecific(andslightlymilder)formulationisasfollows:
√
IsthereabanditalgorithmthathasO˜( n)regretintheadversarialmodelandpolylog(n)
regretinthestochasticmodel?
We are not aware of any prior work on this question. Intuitively, we introduce a new tradeoff:
a bandit algorithm has to balance between attacking the weak adversary (stochastic rewards) and
defending itself from a more devious adversary that targets algorithm’s weaknesses, such as be-
ing too aggressive if the reward sequence is seemingly stochastic. In particular, while the basic
√
exploration-exploitation tradeoff induces O(logn) regret in the stochastic model, and O( n) re-
gret in the adversarial model, it is not clear a priori what are the optimal regret guarantees for this
newattack-defensetradeoff.
1.ThisisclearlytruefortheoriginalversionofExp3withamixingparameter. However,thismixingisunnecessary
againstobliviousadversaries(Stoltz,2005).Theregretoftheresultingalgorithminthestochasticmodelisunknown.
2
THEBESTOFBOTHWORLDS: STOCHASTICANDADVERSARIALBANDITS
We answer the above question affirmatively, with a new algorithm called SAO (Stochastic and
AdversarialOptimal). Toformulateourresult,weneedtointroducesomenotation. Inthestochastic
model, let µ be the expected single-round reward from arm i. A crucial parameter is the minimal
i
gap: ∆ = min µ∗ − µ , where µ∗ = max µ . With this notation, UCB1 attains regret
i:µi<µ∗ i i i
O(K logn)inthestochasticmodel,whereK isthenumberofarms. Wearelookingforthefollow-
∆ √
ing: regretE[R ] = O˜( Kn)intheadversarialmodelandregretE[R ] = O˜(K)inthestochastic
n n ∆
model,whereO˜(·)hidespolylog(n)factors. Ourmainresultisasfollows.
Theorem1 ThereexistsanalgorithmSAOfortheMABproblemsuchthat:
√
(a) intheadversarialmodel,SAOachievesregretE[R ] ≤ O( nK log3/2(n) logK).
n
(b) inthestochasticmodel,SAOachievesregretE[R ] ≤ O(K log2(n)logK).
n ∆
Moreover, with very little extra work we can obtain the corresponding high-probability versions
(seeTheorem15foraprecisestatement).
Itiseasier, andmoreinstructive, toexplainthemainideasonthespecialcaseoftwoarmsand
obliviousadversary.2 Thisspecialcase(withasimplifiedalgorithm)ispresentedinSection3. Due
tothepagelimit,thegeneralcaseisfleshedoutintheAppendix.
Discussion. ThequestionraisedinthispapertouchesuponanimportantthemeinMachineLearn-
ing, and more generally in the design of algorithms with partially known inputs: how to achieve a
goodworst-caseperformanceandalsotakeadvantageof“nice”probleminstances. Inthecontextof
MABitisnaturaltofocusonthedistinctionbetweenstochasticandadversarialrewards,especially
given the prominence of the two models in the MAB literature. Then our “best-of-both-worlds”
question is the first-order specific question that one needs to resolve. Also, we provide the first
analysisofthesameMABalgorithmunderbothadversarialandstochasticrewards.
Once the “best-of-both-worlds” question is settled, several follow-up questions emerge. Most
immediately,itisnotclearwhetherthepolylogfactorscanbeimprovedtomatchtheoptimalguar-
antees for each respective model; a lower bound would indicate that the “attack-defence” tradeoff
is fundamentally different from the familiar explore-exploit tradeoffs. A natural direction for fur-
ther work is rewards that are adversarial on a few short time intervals, but stochastic most of the
time. Moreover,itisdesirabletoadaptnotonlytothebinarydistinctionbetweenthestochasticand
adversarialrewards,butalsotosomeformofcontinuoustradeoffbetweenthetworewardmodels.
Finally, we acknowledge that our solution is no more (and no less) than a theoretical proof of
concept. Morework,theoreticalandexperimental,andperhapsnewideasorevennewalgorithms,
are needed for a practical solution. In particular, a practical algorithm should probably go beyond
whatweaccomplishinthispaper,alongthelinesofthetwopossibleextensionsmentionedabove.
Related work. The general theme of combining worst-case and optimistic performance bounds
havereceivedconsiderableattentioninpriorworkononlinelearning. Anaturalincarnationofthis
themeinthecontextofMABconcernsprovingupperboundsonregretthatcanbewritteninterms
ofsomecomplexitymeasureoftherewards,andmatchtheoptimalworst-casebounds. Tothisend,
a version of Exp3 achieves regret O˜((cid:112)KG∗), where G∗ ≤ n is the maximal cumulative reward
n n
ofasinglearm, andthecorrespondinghighprobabilityresultwasrecentlyprovedinAudibertand
Bubeck (2010). In Hazan and Kale (2009), the authors obtain regret O˜((cid:112)KV∗), where V∗ ≤ n
n n
2.Anobliviousadversaryfixestherewardsg forallroundtwithoutobservingthealgorithm’schoices.
i,t
3
BUBECKSLIVKINS
is the maximal “temporal variation” of the rewards.3 Similar results have been obtained for the
full-feeback (“experts”) version in Cesa-Bianchi et al. (2007); Abernethy et al. (2008). Also, the
regret bound for UCB1 depends on the gap ∆, and matches the optimal worst-case bound for the
stochasticmodel(uptologarithmicfactors). Moreover,adaptivityto“nice”probleminstancesisa
crucial theme in the work on bandits in metric spaces (Kleinberg et al., 2008; Bubeck et al., 2011;
Slivkins, 2011), an MAB settingin which someinformation on similaritybetween arms isa priori
availabletoanalgorithm.
√
Thedistinctionbetweenpolylog(n)andΩ( n)regrethasbeencrucialinotherMABsettings:
bandits with linear rewards (Dani et al., 2008), bandits in metric spaces (Kleinberg and Slivkins,
2010), and an extension of MAB to auctions (Babaioff et al., 2009; Devanur and Kakade, 2009;
Babaioffetal.,2010). Interestingly,herewehavefourdifferentMABsettings(includingtheonein
thispaper)inwhichthisdistinctionoccursforfourdifferentreasons,withnoapparentconnections.
A proper survey of the literature on multi-armed bandits is beyond the scope of this paper; a
reader is encouraged to refer to Cesa-Bianchi and Lugosi (2006) for background. An important
high-leveldistinctionisbetweenBayesianandnon-BayesianMABformulations. Bothhavearich
literature;thispaperfocusesonthelatter. The“basic”MABversiondefinedinthispaperhasbeen
extendedinvariouspaperstoincludeadditionalinformationand/orassumptionsaboutrewards.
Most relevant to this paper are algorithms UCB1 (Auer et al., 2002a) and Exp3 (Auer et al.,
2002b). UCB1 has a slightly more refined regret bound than the one that we cited earlier: R =
n
O((cid:80) logn ) with high probability. A matching lower bound (up to the considerations of
i:µi<µ∗ µ∗−µi
thevarianceandconstantfactors)isprovedinLaiandRobbins(1985). Severalrecentpapers(Auer
and Ortner, 2010; Honda and Takemura, 2010; Audibert et al., 2009; Audibert and Bubeck, 2010;
Maillard and Munos, 2011; Garivier and Cappe´, 2011; Perchet and Rigollet, 2011) improve over
UCB1,obtainingalgorithmswithregretboundsthatareevenclosertothelowerbound.
√
The regret bound for Exp3 is E[R ] = O( nKlogK), and a version of Exp3 achieves this
n √
with high probability (Auer et al., 2002b). There is a nearly matching lower bound of Ω( Kn).
RecentlyAudibertandBubeck(2010)haveshavedoffthelogK factor,achievinganalgorithmwith
√
regretO( Kn)intheadversarialmodelagainstanobliviousadversary.
High-level ideas. For clarity, let us consider the simplified algorithm for the special case of two
armsandobliviousadversary. Thealgorithmstartswiththeassumptionthatthestochasticmodelis
true,andthenproceedsinthreephases: “exploration”,“exploitation”,andthe“adversarialphase”.
In the exploration phase, we alternate the two arms until one of them (say, arm 1) appears signif-
icantly better than the other. When and if that happens, we move to the exploitation phase where
wefocusonarm1,butre-samplearm2withsmallprobability. Aftereachroundwecheckseveral
consistencyconditionswhichshouldholdwithhighprobabilityiftherewardsarestochastic. When
andifoneoftheseconditionsfails,wedeclarethatwearenotinthecaseofstochasticrewards,and
switchtorunningabanditalgorithmfortheadversarialmodel(aversionofExp3).
Here we have an incarnation of the “attack-defense” tradeoff mentioned earlier in this section:
the consistency conditions should be (a) strong enough to justify using the stochastic model as an
operating assumption while the conditions hold, and (b) weak enough so that we can check them
3.TheresultinHazanandKale(2009)doesnotshedlightonthequestioninthepresentpaper,becausethe“temporal
variation”concernsactualrewardsratherthanexpectedrewards.Inparticular,temporalvariationisminimalwhenthe
actualrewardofeacharmisconstantovertime,and(essentially)maximalinthestochasticmodelwith0-1rewards.
4
THEBESTOFBOTHWORLDS: STOCHASTICANDADVERSARIALBANDITS
despite the low sampling probability of arm 2. The fact that (a) and (b) are not mutually exclusive
issurprisingandunexpected.
Moreprecisely,theconsistencyconditionsshouldbestrongenoughtoinsureusfromlosingtoo
muchinthefirsttwophasesevenifweareintheadversarialmodel. Weuseaspecificre-sampling
scheduleforarm1whichisrareenoughsothatwedonotaccumulatemuchregretifthisisindeed
abadarm,andyetsufficienttochecktheconsistencyconditions.
ToextendtotheK-armcase,we“interleave”explorationandexploitation,“deactivating”arms
one by one as they turn out to be suboptimal. The sampling probability of a given arm increases
while the arm stays active, and then decreases after it is deactivated, with a smooth transition be-
tweenthetwophases. Thiscomplicatedbehavior(andthefactthatwehandlegeneraladversaries)
inturnnecessitateamoredelicateanalysis.
2. Preliminaries
Weconsiderrandomizedalgorithms,inthesensethatI (thearmchosenattimet)isdrawnfroma
t
probabilitydistributionp on{1,...,K}. Wedenotebyp theprobabilitythatI = i. Forbrevity,
t i,t t
letI = 1 . Givensucharandomizedalgorithm,itisawell-knowntricktouseg˜ = gi,tIi,t
i,t {It=i} i,t pi,t
asanunbiasedestimateoftherewardg . Nowforarmiandtimetweintroduce:
i,t
• G = (cid:80)t g (fixed-armcumulativereward fromarmiuptotimet),
i,t s=1 i,s
• G(cid:101)i,t = (cid:80)ts=1g˜i,s (estimatedcumulativereward fromarmiuptotimet),
• G(cid:98)i,t = (cid:80)ts=1gi,sIi,s (algorithm’scumulativereward fromarmiuptotimet),
• T (t) = (cid:80)t I (thesamplingtimeofarmiuptotimet).
i s=1 i,s
• Thecorrespondingaverages: Hi,t = 1t Gi,t,H(cid:101)i,t = 1t G(cid:101)i,t,andH(cid:98)i,t = G(cid:98)i,t/Ti(t).
G isthecumulativerewardofa“fixed-armalgorithm”thatalwaysplaysarmi. Recallthatour
i,t
benchmarksaremax G fortheadversarialmodel,andmax E[G ]forthestochasticmodel.
i i,t i i,t
NotethatH(cid:101)i,t,H(cid:98)i,t (andG(cid:101)i,t,G(cid:98)i,t)areobservedbyanalgorithmwhereasHi,t (andGi,t)isnot.
Informally,H(cid:101)i,t andH(cid:98)i,t areestimatesfortheexpectedrewardµi inthestochasticmodel,andH(cid:101)i,t
isanestimateforthebenchmarkrewardH intheadversarialmodel.
i,t
In the stochastic model we define the gap of arm i as ∆ = (max µ ) − µ , and the
i 1≤j≤K j i
minimalgap∆ = min ∆ .
i:∆i>0 i
Followingtheliterature,wemeasurealgorithm’sperformanceintermsofregret R andR as
n n
defined in Figure1. The two notionsof regret are somewhat different, in particular the “stochastic
regret”R isnot exactlyequaltotheexpected“adversarialregret”R . However, inthestochastic
n n
(cid:113)
modeltheyareapproximatelyequal:4 E[R ] ≤ E[R ] ≤ E[R ]+ 1 nlogK.
n n n 2
3. ThecaseofK = 2arms
Wewillderivea(slightlyweakerversionof)themainresultforthespecialcaseofK = 2armsand
obliviousadversary,usingasimplifiedalgorithm. Thisversioncontainsmostoftheideasfromthe
generalcase,butcanbepresentedinamorelucidfashion.
4.Thisfactiswell-knownandeasytoprove,e.g.seeProposition34inAudibertandBubeck(2010).
5
BUBECKSLIVKINS
√
We are looking for the “best-of-both-worlds” feature: O˜( n) regret in the adversarial model,
andO˜(1)regretinthestochasticmodel,where∆ = |µ −µ |isthegap. Ourgoalinthissection
∆ 1 2
istoobtainthisfeatureinthesimplestwaypossible. Inparticular,wewillhidetheconstantsunder
the O() notation, and will not attempt to optimize the polylog(n) factors; also, we will assume
obliviousadversary. Wewillprovethefollowingtheorem:
Theorem2 ConsideraMABproblemwithtwoarms. Thereexistsaalgorithmsuchthat:
√
(a) againstanobliviousadversary,itsexpectedregretisE[R ] ≤ O( n log2n).
n
(b) inthestochasticmodel,itsexpectedregretsatisfiesE[R ] ≤ O(1 log3n).
n ∆
Bothregretboundsalsoholdwithprobabilityatleast1− 1.
n
Notethatinthestochasticmodel, regrettriviallycannotbelargerthan∆n, sopart(b)trivially
√
impliesregretE[R ] ≤ O˜( n).
n
Ouranalysisproceedsviahigh-probabilityargumentsanddirectlyobtainsthehigh-probability
guarantees. Thehigh-probabilityargumentstendtomaketheanalysiscleaner;wesuspectitcannot
bemademuchsimplerifweonlyseekboundsonexpectedregret.
3.1. AsimplifiedSAO(StochasticandAdversarialOptimal)Algorithm
The algorithm proceeds in three phases: exploration, exploitation, and adversarial phase. In the
explorationphase,wealternatethetwoarmsuntiloneofthemappearssignificantlybetterthanthe
other. In exploitation phase, we focus on the better arm, but re-sample the other arm with small
probability. We check several consistency conditions which should hold with high probability if
therewardsarestochastic. Whenandifoneoftheseconditionsfails,wedeclarethatwearenotin
the case of stochastic rewards, and switch to running a bandit algorithm for the adversarial model,
namelyalgorithmExp3.P(Aueretal.,2002b).
The algorithm is parameterized by C = Θ(logn) which we will chose later in Section 3.2.
crn
Theformaldescriptionofthethreephasesisasfollows.
(Explorationphase) In each round t, pick an arm at random: p = p = 1. Go to the next
1,t 2,t 2
phaseassoonast > Ω(C2 )andthefollowingconditionfails:
crn
√
|H(cid:101)1,t−H(cid:101)2,t| < 24Ccrn/ t. (1)
Letτ∗ bethedurationofthisphase. Withoutlossofgenerality,assumeH(cid:101)1,τ∗ > H(cid:101)2,τ∗. This
means,informally,thatarm1isselectedforexploitation.
(Exploitationphase) Ineachroundt > τ ,pickarm2withprobabilityp = τ∗,andarm1with
∗ 2,t 2t
theremainingprobabilityp = 1− τ∗.
1,t 2t
Aftertheround,checkthefollowingconsistencyconditions:
√ √
8Ccrn/ τ∗ ≤ H(cid:101)1,t−H(cid:101)2,t ≤ 40Ccrn/ τ∗ (2)
(cid:40) √
|H(cid:101)1,t−H(cid:98)1,t| ≤ 6Ccrn/ t
√ (3)
|H(cid:101)2,t−H(cid:98)2,t| ≤ 6Ccrn/ τ∗.
Ifoneoftheseconditionsfails,gotothenextphase.
(Adversarialphase) RunalgorithmExp3.PfromAueretal.(2002b).
6
THEBESTOFBOTHWORLDS: STOCHASTICANDADVERSARIALBANDITS
Discussion. The exploration phase is simple: Condition (1) is chosen so that once it fails then
(assumingstochasticrewards)theseeminglybetterarmisindeedthebestarmwithhighprobability.
Intheexploitationphase,wedefinethere-samplingscheduleforarm2andacollectionof“con-
sistency conditions”. The re-sampling schedule should be sufficiently rare to avoid accumulating
muchregretifarm2isindeedabadarm. Theconsistencyconditionsshouldbesufficientlystrong
tojustifyusingthestochasticmodelasanoperatingassumptionwhiletheyhold. Namely,anadver-
sary constrainedby these conditions shouldnot be able to inflicttoo much regret onour algorithm
in the first two phases. Yet, the consistency conditions should be weak enough so that they hold
withhigh-probabilityinthestochasticmodel,despitethelowsamplingprobabilityofarm2.
ItisessentialthatweusebothH(cid:98)i,t andH(cid:101)i,t intheconsistencyconditions: theinterplayofthese
two estimators allows us to bound regret in the adversarial model. Other than that, the conditions
thatweusearefairlynatural(thesurprisingpartisthattheywork). Condition(2)checkswhetherthe
relationbetweenthetwoarmsisconsistentwiththeoutcomeoftheexplorationphase,i.e. whether
arm1stillseemsbetterthanarm2,butnottoomuchbetter. Condition(3)checkswhetherforeach
arm i, the estimate H(cid:101)i,t is close to the average H(cid:98)i,t. In the stochastic model, both estimate the
expected gain µ , so we expect them to be not too far apart. However, our definition of “too far”
i
shouldbeconsistentwithhowoftenagivenarmissampled.
3.2. Concentrationinequalities
The“probabilistic”aspectoftheanalysisisconfinedtoprovingthatseveralpropertiesofestimates
and sampling times hold with high probability. The rest of the analysis can proceed as if these
propertiesholdwithprobability1. Inparticular,wehavemadeourcoreargumentessentiallydeter-
ministic,whichgreatlysimplifiespresentation.
All high-probability results are obtained using an elementary concentration inequality loosely
knownasChernoffBounds. Forthesakeofsimplicity,weuseaslightlyweakerformulationbelow
(seeAppendixAforaproof),whichusesjustoneinequalityforallcases.
Theorem3(ChernoffBounds) LetX ,t ∈ [n]beaindependentrandomvariablessuchthatX ∈
t t
[0,1]foreacht. LetX = (cid:80)n X betheirsum,andletµ = E[X]. Then
t=1 t
√
Pr[|X −µ| > C max(1, µ)] < 2e−C/3, foranyC > 1. (4)
WewilloftenneedtoapplyChernoffBoundstosumswhosesummandsdependonsomeevents
intheexecutionofthealgorithmsandthereforearenotmutuallyindependent. However,inallcases
these issues are but a minor technical obstacle which can be side-stepped using a slightly more
carefulsetup.5 Inparticular,wesometimesfinditusefultoworkintheprobabilityspaceobtainedby
conditioningontheoutcomeoftheexplorationphase. Specifically,thepost-explorationprobability
spaceistheprobabilityspaceobtainedbyconditioningonthefollowingevents: thattheexploration
phaseends,thatithasaspecificdurationτ ,andthatarm1ischosenforexploitation.
∗
Throughouttheanalysis,wewillobtainconcentrationboundsthatholdwithprobabilityatleast
1−2n−4. WewilloftentakeaUnionBoundoverallroundst,whichwillimplysuccessprobability
at least 1−2n−3. To simplify presentation, we will allow a slight abuse of notation: we will say
5.However,theindependenceissuesappearprohibitiveforK > 2armsorifweconsideranon-obliviousadversary.
Soforthegeneralcaseweresortedtoamorecomplicatedanalysisviamartingaleinequalities.
7
BUBECKSLIVKINS
withhighprobability(abbreviatedw.h.p.),whichwillmeanmeanwithprobabilityatleast1−2n−3
oratleast1−2n−4,dependingonthecontext.
To parameterize the algorithm, let us fix some C = 12 ln(n) such that Theorem 3 with
crn
C = C ensuressuccessprobabilityatleast1−2n−4.
crn
3.3. Analysis: adversarialmodel
We need to analyze our algorithm in two different reward models. We start with the adversarial
model,sothatwecanre-usesomeoftheclaimsprovedheretoanalyzethestochasticmodel.
Recall that τ denotes the duration of the exploration phase (which in general is a random
∗
variable). FollowingtheconventionfromSection3.1thatwhenevertheexplorationphaseends,the
armchosenforexploitationisarm1. (Notethatwedonotassumethatarm1isthebestarm.)
We start the analysis by showing that the re-sampling schedule in the exploitation phase does
notresultinplayingarm2toooften.
Claim4 Duringtheexploitationphase,arm2isplayedatmostO(τ logn)timesw.h.p..
∗
Proof. Wewillworkinthepost-explorationprobabilityspaceS. Weneedtoboundfromabovethe
(cid:80)
sum I ,wheretrangesovertheexploitationphase. However,ChernoffBoundsdonotimmedi-
t 2,t
atelyapplysincethenumberofsummandsitselfisarandomvariable. Further,ifweconditionona
specificdurationofexploitationthenwebreakindependencebetweensummands. Wesidestepthis
issue by considering an alternative algorithm in which exploitation lasts indefinitely (i.e., without
thestoppingconditions),andwhichusesthesamerandomnessastheoriginalalgorithm. Itsuffices
toboundfromabovethenumberoftimesthatarm2isplayedduringtheexploitationphaseinthis
alternative algorithm; denote this number by N. Letting J be the arm selected in round t of the
t
alternativealgorithm,wehavethatN = (cid:80)n 1 isasumof0-1randomvariables,andin
t=τ∗+1 {Jt=i}
S thesevariablesareindependent. Moreover,inS itholdsthat
E[N] = (cid:80)n p = τ (cid:80)n 1 = O(τ logn).
t=τ∗+1 2,t ∗ t=τ∗+1 2t ∗
Therefore,theclaimfollowsfromChernoffBounds.
NowweconnecttheestimatedcumulativerewardsG(cid:101)i,t withthebenchmarkGi,t. Morespecifi-
cally,wewillboundfromaboveseveralexpressionsoftheform|H(cid:101)i,t −Hi,t|. Naturally,theupper
boundforarm1willbestrongersincethisarmisplayedmoreoftenduringexploitation. Toensure
that the bound for arm 2 is strong enough we need to play this arm “sufficiently often” during ex-
ploitation. (WhereasClaim4ensuresthatwedonotplayit“toooften”.) Hereandelsewhereinthis
analysis, we find it more elegant to express some of the claims in terms of the average cumulative
rewards(suchasH ,etc.)
i,t
√
Claim5 (a) Withhighprobability,|H(cid:101)i,τ∗ −Hi,τ∗| < 2Ccrn/ τ∗ foreacharmi.
(b) Foranyroundtintheexploitationphase,withhighprobabilityitholdsthat
(cid:40) √
|H(cid:101)1,t−H1,t| < 3Ccrn/ t,
√ (5)
|H(cid:101)2,t−H2,t| < 3Ccrn/ τ∗.
8
THEBESTOFBOTHWORLDS: STOCHASTICANDADVERSARIALBANDITS
(cid:80)
Proof. For part (a), we are interested in the sum g I . As in the proof of Claim 4,
t≤τ∗ i,t i,t
ChernoffBoundsdonotimmediatelyapplysincethenumberofsummandsτ isarandomvariable
∗
(andconditioningonaparticularvalueofτ tamperswithindependencebetweensummands). Solet
∗
us consider an alternative algorithm in which the exploration phase proceeds indefinitely, without
the stopping condition, and uses the same randomness as the original algorithm.6 Let J be the
t
arm selected in round t of this alternative algorithm, and define A = (cid:80)t g 1 . Then
i,t s=1 i,t {Jt=i}
(whenrunonthesameprobleminstance)bothalgorithmscoincideforanyt ≤ τ , soinparticular
∗
G(cid:101)i,t = 2Ai,t. Now, Ai,t is the sum of bounded independent random variables wi√th expectation
G /2. Therefore by Chernoff Bounds w.h.p. it holds that |A −G /2| < C t for each t,
i,t i,t i,t crn
whichimpliestheclaim.
Forpart(b),wewillanalyzetheexploitationphaseseparately. Letusworkinthepost-exploration
probabilityspaceS. WewillconsiderthealternativealgorithmfromtheproofofClaim4(inwhich
exploitation continues indefinitely). This way we do not need worry that we implicitly condition
on the event that a particular round t > τ belongs to the exploitation phase. Clearly, it suf-
∗
fices to prove (5) for this alternative algorithm. To facilitate the notation, define the time interval
(cid:80) (cid:80)
INT = {τ∗+1,...,t},anddenoteGi,INT = s∈INTgi,t andG(cid:101)i,INT = s∈INTg(cid:101)i,t.
To handle arm 1, note that in S, G(cid:101)i,INT is a sum of independent random variables, with expec-
tation G . Since p ≥ 1 for any t ∈ INT, the summands g are bounded by 2. Therefore by
i,INT 1,t 2 (cid:101)1,t
ChernoffBoundswithhighprobabilityitholdsthat
√
|G(cid:101)1,INT−G1,INT| < 2Ccrn t−τ∗.
Fromthisandpart(a)itfollowsthatw.h.p.
√ √ √
|G(cid:101)1,t−G1,t| < 2Ccrn( τ∗+ t−τ∗) < 3Ccrn t,
whichimpliestheclaimforarm1.
Handling arm 2 requires a little more work since the summands g may be large (since they
(cid:101)2,t
haveasmallprobabilityp inthedenominator). Foreacht ∈ INT,
2,t
2s 2t s 2t
(cid:80) (cid:80) (cid:80)
G(cid:101)2,INT = s∈INT τ g2,s I2,s = τ s∈INT t g2,s I2,s = τ s∈INT Xs,
∗ ∗ ∗
where X = s g I ∈ [0,1]. In S, random variables X , s ∈ INT are mutually independent,
s t 2,s 2,s s
andtheexpectationoftheirsumis
(cid:104) (cid:105)
µ (cid:44) E(cid:2)(cid:80)s∈INT Xs(cid:3) = τ2∗t E G(cid:101)2,INT = τ2∗t G2,INT ≤ τ2∗ t−tτ∗.
Noting that G ≤ t−τ and letting α = τ∗, we obtain µ ≤ τ∗(1−α). By Chernoff Bounds
2,INT ∗ t 2
w.h.p. itholdsthat
(cid:80) (cid:112)
| X −µ| < C τ (1−α).
s∈INT s crn ∗
GoingbacktoG(cid:101)2,INT andG2,INT,weobtain:
2t (cid:112) 2t √
|G(cid:101)2,INT−G2,INT| < Ccrn τ∗(1−α) < Ccrn √ 1−α.
τ τ
∗ ∗
6.Notethatthisisnotthesame“alternativealgorithm”astheoneintheproofofClaim4.
9
BUBECKSLIVKINS
Frompart(a),wehavethat|G(cid:101)i,τ∗ −Gi,τ∗| < Ccrn √2τt∗ α. Therefore,
2t √ √ 3t
|G(cid:101)2,t−G2,t| < Ccrn √ ( α+ 1−α) < Ccrn √ .
τ τ
∗ ∗
CombiningClaim5(b)andCondition(2),weobtain:
Corollary6 In the exploitation phase, for any round t (except possibly the very last round in the
phase)itholdsw.h.p. thatG > G .
1,t 2,t
ByCorollary6,regretaccumulatedbyroundtintheexploitationphaseis,withhighprobability,
√
equaltoG1,t−G(cid:98)1,t−G(cid:98)2,t. Thefollowingclaimupper-boundsthisquantitybyO( t log2n). The
proofofthisclaimcontainsourmainregretcomputation.
Claim7 Foranyroundtintheexploitationphaseitholdsw.h.p. that
√
G(cid:98)1,t+G(cid:98)2,t−G1,t ≥ −O( t log2n).
Proof. Throughoutthisproof,letusassumethatthehigh-probabilityeventsinClaim4andClaim5(b)
actuallyhold;wewillomit“withhighprobability”fromhereon.
Lettbesome(butnotthelast)roundintheexploitationphase. First,
(cid:104) (cid:105) (cid:104) (cid:105) √
H(cid:98)1,t−H1,t = H(cid:101)1,t−H1,t + H(cid:98)1,t−H(cid:101)1,t ≥ −O(Ccrn/ t). (6)
We have upper-bounded the two square brackets in (6) using, respectively, Claim 5(b) and Con-
dition (3). We proved that algorithm’s average for arm 1 (H(cid:98)1,t) is not too small compared to the
corresponding benchmark average H1,t, and we used the estimate H(cid:101)1,t as an intermediary in the
proof.
Similarly, using Condition (2), Condition (3), and Claim 5(b) to upper-bound the three square
bracketsinthenextequation,weobtainthat
(cid:104) (cid:105) (cid:104) (cid:105) (cid:104) (cid:105) √
H(cid:98)2,t−H1,t = H(cid:101)2,t−H(cid:101)1,t + H(cid:98)2,t−H(cid:101)2,t + H(cid:101)1,t−H1,t ≥ −O(Ccrn/ τ∗). (7)
Here we have proved that the algorithm did not do too badly playing arm 2, even though this arm
wassupposedtobesuboptimal. Specifically,weestablishthatalgorithm’saverageforarm2(H(cid:98)2,t)
isnottoosmallcomparedtothebenchmarkaverageforarm1(H1,t). Again,theestimatesH(cid:101)1,tand
H(cid:101)2,t servedusasintermediariesintheproof.
Finally,letusgofromboundsonaveragerewardstoboundsoncumulativerewards(andprove
theclaim). Combining(6),(7)andClaim4,wehave:
(cid:104) (cid:105)
(cid:80)
G(cid:98)1,t+G(cid:98)2,t−G1,t = i=1,2 Tj(t) H(cid:98)i,t−H1,t
(cid:104) √ √ (cid:105)
≥ −O(C ) T (t)/ t+T (t)/ τ
crn 1 2 ∗
√ √
≥ −O(C )( t+ τ logn)
crn ∗
√
≥ −O( t log2n).
10