Estimating Quality in Multi-Objective Bandits Optimization

1.1 MB·

by  Audrey Durand

Estimating Quality in Multi-Objective Bandits Optimization
Audrey Durand and Christian Gagn´e
Computer Vision and Systems Laboratory
Universit´e Laval, Qu´ebec (QC), Canada
[email protected], [email protected]
{ }
7
April 4, 2017
1
0
2

r Abstract
p
A Manyreal-worldapplicationsarecharacterizedbyanumberofconflictingperformancemea-
sures. As optimizing in a multi-objective setting leads to a set of non-dominated solutions, a
1
preference function is required for selecting the solution with the appropriate trade-off between

] the objectives. The question is: how good do estimations of these objectives have to be in
G
order for the solution maximizing the preference function to remain unchanged? In this paper,
L we introduce the concept of preference radius to characterize the robustness of the preference
. function and provide guidelines for controlling the quality of estimations in the multi-objective
s
c setting. More specifically, we provide a general formulation of multi-objective optimization un-
[
derthebanditssetting. Weshowhowthepreferenceradiusrelatestotheoptimalgapanduseit

2 toprovideatheoreticalanalysisoftheThompsonsamplingalgorithmfrommultivariatenormal
v priors. We finally present experiments to support the theoretical results and highlight the fact
5 that one cannot simply scalarize multi-objective problems into single-objective problems.
9
0
1
1 Introduction
0
.
1
0 Multi-objective optimization (MOO) [4] is a topic of great importance for real-world applications.
7
Indeed, optimization problems are characterized by a number of conflicting, even contradictory,
1
: performance measures relevant to the task at hand. For example, when deciding on the healthcare
v
treatment to follow for a given sick patient, a trade-off must be made between the efficiency of the
i
X
treatment to heal the sickness, the side effects of the treatment, and the treatment cost. MOO
r
a is often tackled by combining the objective into a single measure (a.k.a. scalarization). Such
approaches are said to be a priori, as the preferences over the objectives is defined before doing the
optimizationitself. Thechallengeliesinthedeterminationoftheappropriatescalarizationfunction
to use and its parameterization. Another way to conduct MOO consists in learning the optimal
trade-offs (the so-called Pareto-optimal set). Once the optimization is completed, techniques from
thefieldofmulti-criteriadecision-makingareappliedtohelptheusertoselectthefinalsolutionfrom
the Pareto-optimal set. These a posteriori techniques may require a huge number of evaluations to
have a reliable estimation of the objective values over all potential solutions. Indeed, the Pareto-
optimal set can be quite large, encompassing a majority, if not all, of the potential solutions. In
this work, we tackle the MOO problem where the scalarization function exists a priori, but might
be unknown, in which case a user can act as a black box for articulating preferences. Integrating
1
the user to the learning loop, she can provide feedback by selecting her preferred choice given a set
of options – the scalarization function lying in her head.
More specifically, we consider problems where outcomes are stochastic and costly to evaluate (e.g.,
involving a human in the loop). The challenge is therefore to identify the best solutions given
random observations sampled from different (unknown) density distributions. We formulate this
problemasmulti-objectivebandits, whereweaimatfindingthesolutionthatmaximizestheprefer-
encefunctionwhilemaximizingtheperformanceofthesolutionsevaluatedduringtheoptimization.
TheThompsonsampling(TS)technique[8]isatypicalapproachforbanditsproblems,whereittries
potential solutions based on a Bayesian posterior over their expected outcome. Here we consider
TS from multivariate normal (MVN) priors for multi-objective bandits.
Let the right choice denote the option that maximize the preference function – the option that the
user would select given that she had knowledge of the Pareto-optimal set. A learning algorithm
for the multi-objective bandits setting aims at learning good-enough estimations of the available
optionstoallowtheusertomaketherightchoicesanditsperformancedependsontherobustnessof
thepreferencefunctiontothequalityofestimations. Wethereforeneedameasureforcharacterizing
the quality of estimations required in order for the option maximizing the preference function to
remain unchanged. For that purpose, we introduce the concept of preference radius providing the
tolerance range over objective value estimations, such that the user preference would remain the
same as if the Pareto-optimal set was known. We use this concept for providing a theoretical
analysis of TS from MVN priors.
Finally,weperformsomeempiricalexperimentstosupportthetheoreticalresultsandalsohighlight
the importance to tackle multi-objective bandits problem as such instead of scalarizing those under
the traditional bandit setting.
The original contributions of the paper consist in:
providingageneralformulationoftheMOOundertheapriorimulti-objectivebanditssetting;
•
proposing the preference radius to characterize the robustness of the preference function to
•
the estimations quality;
proposing a theoretical analysis of the TS algorithm from MVN priors;
•
showing with empirical experiments that multi-objective bandits cannot simply be brought
•
back to single-objective bandits.
2 Multi-Objective Bandits
A multi-objective bandits problem is described by a (finite) set of actions , also referred to
A
as the design space, each of which is associated with a d-dimensional expected outcome µ =
a
(µ ,...,µ ) Rd. For simplicity, we assume that the objective space = [0,1]d. In this
a,1 a,d
∈ X ∈ X
episodic game, an agent interacts with an environment characterized by a preference function f.
The agent iteratively chooses to perform an action a(t) and obtains a noisy observation of µ .
a(t)
Algorithm 1 describes this multi-objective bandits problem.1
1Scalars are written unbolded; vectors are boldfaced. Throughout the paper it is considered that the operators
+, −, ×, and ÷ applied on a vector v =(v ,...,v ) and a scalar s correspond to the operation between each item
1 d
of v and s, e.g., v+s = (v +s,...,v +s), and that the operators applied on two vectors v = (v ,...,v ) and
1 d 1 d
u=(u ,...,u ) correspond to itemwise operations between v and u, e.g., v+u=(v +u ,...,v +u ).
1 d 1 1 d d
2
Algorithm 1 Multi-objective bandits setting
On each episode t 1:
≥
1. The agent selects an action a(t) to play given f.
2. The agent observes the outcome z(t) = µ +ξ(t), where ξ(t) are i.i.d. random variables.
a(t)
3. The agent updates its estimates.
An algorithm for a multi-objective bandits problem is a (possibly randomized) method for choos-
ing which action to play next, given a history of previous choices and obtained outcomes, =
t
a(s),z(s) t−1. Let = argmax f(µ ) and let (cid:63) denote the optimal action. The opHtimal
{ }s=1 O a∈A a ∈ O
gap ∆ = f(µ ) f(µ ) measures the expected loss of playing action a instead of the optimal
a (cid:63) a
−
action. The agent’s goal is to design an algorithm with low expected (cumulative) regret2:
(cid:20) T (cid:21) T
R(T) = E (cid:88)(cid:0)f(µ ) f(z(t))(cid:1) = (cid:88)(cid:88)P[a(t) = a]∆ . (1)
(cid:63) a
−
t=1 a∈At=1
This quantity measures the expected performance of the algorithm compared to the expected
performance of an optimal algorithm given knowledge of the outcome distributions, i.e., always
sampling from the distribution with the expectation maximizing f. The algorithm faces a trade-off
between playing an action that might maximize f, and choosing to gather an additionnal sample
from a relatively unexplored action in order to improve its estimate. Typically, we assume that the
algorithm maintains one estimated θ (t) per action a on time t. Let (t) = argmax f(θ (t))
a a∈A a
O
denote the set of actions with an estimate maximizing f. The algorithm hence plays a(t) (t).
∈ O
Inmanysituations,theenvironmentprovidingthepreferencefunctionisaperson,letuscallherthe
expert user. Unfortunately, people are generally unable to scalarize their choices and preferences.
Thereforetheycannotexplicitelyprovidetheirpreferencefunction. However, givenseveraloptions,
users can tell which one they prefer and thus can be used as a black box to provide feedback in the
learning loop. Algorithm 2 describes the user-guided multi-objective bandits problem.
Algorithm 2 User-guided multi-objective bandits setting
On each episode t 1:
≥
1. The agent selects an action a(t) (t) to play.
∈ O
2. The agent observes the outcome z(t) = µ +ξ(t), where ξ(t) are i.i.d. random variables.
a(t)
3. The agent updates its estimates.
4. The agent shows options θ (t) to the expert user.
a a∈A
{ }
5. The expert user indicates its preference (t) = argmax f(θ (t)).
a∈A a
O
Pareto-optimality: Given two d-dimensional options x = (x ,...,x ) and y = (y ,...,y ). x
1 d 1 d
is said to dominate, or Pareto-dominate, y (denoted x y) if and only if x > y for at least
i i
(cid:23)
one i and x y otherwise. The dominance is strict (denoted x y) if and only if x > y for
i i i i
≥ (cid:31)
all i = 1,...,d. Finally, the two vectors are incomparable (denoted x y) if x (cid:7) y and y (cid:7) x.
(cid:107)
Pareto-optimal options represent the best compromises amongst the objectives and are the only
options that need to be considered in an application. We say that these options constitute the
2Also known as the scalarized regret [5].
3
1
2
e
v
ti
c
e
j
b
O
0
0 1
Objective 1
Figure 1: Example of dominated (black) and non-dominated (white) expected outcomes.
Pareto front = a : (cid:64)µ µ b . Fig. 1 shows an example of dominated and non-
a a a∈A
P { (cid:23) ∀ ∈ A}
dominated expcted outcomes in a d = 2 objectives space. A user facing a multi-criteria decision
making problem must select its preferred non-dominated option. Dominated options are obviously
discarded by default.
Related Works The multi-objective bandits problem as already been addressed in the a posteri-
orisetting,wherethegoalistodiscoverthewholeParetofrontforaposterioridecisionmaking[5,9].
This is different from the a priori optimization problem tackled here. The aim of algorithms in the
posteriori setting is to simultaneously minimize the Pareto-regret and the unfairness metrics. Also
known as the (cid:15)-distance [6], the Pareto-regret associated with playing action a is the minimum
value (cid:15) such that µ + (cid:15) is not dominated by any other actions. In other words, any actions
a a a
standing on the front is considered equally good by the expert user. This is like considering that
= , which corresponds to the following preference function
O P
f(µ ) = 1 and f(µ ) = 1 (cid:15) ,
(cid:63) a a
−
such that ∆ = (cid:15) . Note that any single-objective bandits algorithm optimizing a single objective
a a
could minimize the Pareto-regret regardless of the other objectives. This is addressed by the
unfairness metric, measuring the disparity in the amount of plays of non-dominated actions – the
idea being to force algorithms to explore the whole Pareto front evenly.
InMOOsettings[10],thegoalistoidentifythePareto-optimalset withoutevaluatingallactions.
P
The quality of a solution is typically given by the hypervolume error V( ) V( ), where the
S P − S
V( ) is the volume enclosed between the origin and µ (and similarly for ). However,
a a∈P
P { } S
the hypervolume error does not give information about the quality of the estimation of actions.
Identifying the Pareto front alone does not guarantee that the actions are well estimated and,
therefore, that an expert user choice based on these estimations would lead to the right choice.
4
3 Preference Radius
Let θ (t) denote the estimation associated with action a on episode t. Let (t) = a : (cid:64)θ (t)
a b
P { (cid:31)
θ (t) b denote the estimated Pareto front given these options. By definition, the optimal
a a∈A
∀ ∈ A} (cid:0) (cid:1)
options (t) = argmax f θ (t) (t). Let
a∈A a
O ⊆ P
B(c,r) x : x c < r, i = 1,...,d
i i
⊆ { ∈ X | − | }
denote a ball of center c and radius r. In order to characterize the difficulty of a multi-objective
bandits setting, we define the following problem-dependent quantity.
Definition 1. For each action a , we define the preference radius ρ as any radius such that
a
∈ A
if θ (t) B(µ ,ρ ) for all actions, then
a a a
∈
(cid:63) : (cid:63) (t) and a (t) a ,a .
∃ ∈ O ∈ O (cid:54)∈ O ∀ ∈ A (cid:54)∈ O
The radii correspond to the robustness of the preference function, that is how much simultaneous
variation of each action can happen before the set of optimal options changes. The radius ρ is
a
directly linked to the gap ∆ = f(µ ) f(µ ). For a suboptimal action a , a large ρ indicates
a (cid:63) a a
− (cid:54)∈ O
that action a is far from being optimal. Therefore, f(µ ) must be low and its gap ∆ must be
a a
large. Also, the preference radii of suboptimal actions depend on the preference radius of the
optimal action(s). Larger optimal action radii imply smaller radii for suboptimal actions.
Let α ,...,α [0,1] denote weights such that (cid:80)d α = 1. The weighted L metric f(x) =
1 d ∈ i=1 i p
(cid:0)(cid:80)d p(cid:1)1/p
α x with p 1 is often used to represent decision functions. This function is known as
i=1 i i ≥
the linear scalarization when p = 1 and as the Chebyshev scalarization when p = . The following
∞
examples show the link between the preference radii and the gap for these two common functions.
Example 1 (Linear scalarization). The linear scalarization function is given by
d
(cid:88)
f(x) = α x .
i i
i=1
Consider the optimal action (cid:63) and the suboptimal action a. By definition of the preference radii,
we have that
min f(θ ) > max f(θ )
(cid:63) a
θ(cid:63)∈B(µ(cid:63),ρ(cid:63)) θa∈B(µa,ρa)
d d
(cid:88) (cid:88)
(α µ α ρ ) > (α µ +α ρ )
i (cid:63),i i (cid:63) i a,i i a
−
i=1 i=1
f(µ ) ρ > f(µ )+ρ
(cid:63) (cid:63) a a
−
∆ > ρ +ρ .
a (cid:63) a
Figure 2 shows examples of preference radii with a linear preference function.
Example 2 (Chebyshev scalarization). The Chebyshev scalarization [3] function is given by
f(x) = max α x .
i i
1≤i≤d
5
1.0 1.0
0.8 0.8
2 2
e 0.6 e 0.6
v v
ti ti
c c
je 0.4 je 0.4
b b
O O
0.2 0.2
0.0 0.0
0.0 0.2 0.4 0.6 0.8 1.0 0.0 0.2 0.4 0.6 0.8 1.0
Objective 1 Objective 1
Figure 2: Examples of preference radii around the optimal (white) and suboptimal (black) actions
given the linear preference function f(x) = 0.4x +0.6x .
1 2
Consider the optimal and suboptimal actions (cid:63) and a, and let
i = argmaxα (µ ρ ), i = argmaxα (µ ρ ).
(cid:63) i (cid:63),i (cid:63) a i a,i a
− −
1≤i≤d 1≤i≤d
By definition of the preference radii, we have that
min f(θ ) > max f(θ )
(cid:63) a
θ(cid:63)∈B(µ(cid:63),ρ(cid:63)) θa∈B(µa,ρa)
max α (µ +ρ ) > max α (µ ρ )
i (cid:63),i (cid:63) i a,i a
1≤i≤d 1≤i≤d −
α µ α ρ > α µ +α ρ
i(cid:63) (cid:63),i− i(cid:63) (cid:63) ia a,i ia a
f(µ ) α ρ > f(µ )+α ρ
(cid:63) − i(cid:63) (cid:63) a ia a
∆ > α ρ +α ρ .
a i(cid:63) (cid:63) ia a
The difficulty here is that i and i respectively depend on ρ and ρ . Consider a 2-objectives
(cid:63) a (cid:63) a
setting, we can define
α µ α µ α µ α µ
2 (cid:63),2 1 (cid:63),1 1 a,1 2 a,2
τ = − , τ = −
(cid:63) a
α α α α
2 1 2 1
− −
as thresholds such that
(cid:26) (cid:26)
1 if ρ > τ 1 if ρ < τ
(cid:63) (cid:63) a a
i = , i =
(cid:63) 2 otherwise a 2 otherwise.
Figure 3 shows examples of preference radii with a Chebyshev preference function.
Outside L metrics, other scalarization functions are often based on constraints. For example,
p
using the (cid:15)-constraint scalarization technique, a user assigns a constraint to every objectives except
a target objective (cid:96). All options that fail to respect one of the contraints get a value of 0, while
the options that respect all constraints get a value of x . The following example shows the relation
(cid:96)
between the preference radius and the gap given a preference function that is articulated as an
(cid:15)-constraint scalarization technique.
6
1.0 1.0
0.8 0.8
2 2
e 0.6 e 0.6
v v
ti ti
c c
je 0.4 je 0.4
b b
O O
0.2 0.2
0.0 0.0
0.0 0.2 0.4 0.6 0.8 1.0 0.0 0.2 0.4 0.6 0.8 1.0
Objective 1 Objective 1
Figure 3: Examples of preference radii around the optimal action (white) and suboptimal actions
(black) given a Chebyshev function with α = 0.4 and α = 0.6.
1 2
Example 3 (Epsilon-constraint). The (cid:15)-constraint function is given by
(cid:26)
x if x (cid:15) i 1,...,d ,i = (cid:96)
f(x) = (cid:96) i ≥ i ∀ ∈ { } (cid:54)
0 otherwise.
Consider the optimal and suboptimal actions (cid:63) and a. By definition of the preference radii, we have
that
ρ min µ (cid:15) .
(cid:63) (cid:63),i i
≤ 1≤i≤d,i(cid:54)=(cid:96) −
We decompose ρ = ρ +ρ¯ such that
a a a
¯
ρ = min 0, max (cid:15) µ
a i a,i
¯ { 1≤i≤d,i(cid:54)=(cid:96) − }
denotes the radius required in order for action a to respect the constraints, that is to get f(µ ) > 0,
a
and ρ¯ denotes the leftover leading to a gap reduction. Finally, we have that
a
µ ρ > µ +ρ +ρ¯ and ∆ > ρ +ρ .
(cid:63),(cid:96) (cid:63) a,(cid:96) a a a (cid:63) a
− ¯
Figure 4 shows examples of preference radii with an (cid:15)-constraint preference functions.
4 Thompson Sampling
In this section we use the preference radius to analyze the Thompson sampling (TS) [8] algorithm
usingmultivariatenormal(MVN)priors. RecallthatTSmaintainsaposteriordistributionπ (t)on
a
the mean µ given a prior π and the history of observations . On episode t, one option θ (t) is
a 0 t a
H
sampledfromeachposteriordistributionπ (t). Thealgorithmchoosestosamplea(t) (t). Recall
a
∈ O
that (t) = argmax f(θ (t)). ThereforeP[a(t) = a]isproportionnaltotheposteriorprobability
a∈A a
thataOmaximizesthepreferencefunctiongiventhehistory . LetN (t) = (cid:80)t−1 I[a(s) = a]denote
Ht a s=1
the number of times action a has been played up to episode t. Also let
µˆ = (cid:80)st−=11:a(s)=az(s) and Σˆ = 1 (cid:88)t−1 (cid:0)z(s) µˆ (t)(cid:1)(cid:0)z(s) µˆ (t)(cid:1)(cid:62)
a,t a,t a a
N (t) N (t) 1 − −
a a
− s=1:a(s)=a
7
1.0 1.0
0.8 0.8
2 2
e 0.6 e 0.6
v v
ti ti
c c
je 0.4 je 0.4
b b
O O
0.2 0.2
0.0 0.0
0.0 0.2 0.4 0.6 0.8 1.0 0.0 0.2 0.4 0.6 0.8 1.0
Objective 1 Objective 1
(a) (cid:96)=1, (cid:15) =0.1 (b) (cid:96)=2, (cid:15) =0.3
2 1
Figure 4: Examples of preference radii around the optimal (white) and suboptimal (black) actions
given two different configurations of (cid:15)-contraints.
respectively denote the empirical mean and covariance, and let Σ and µ denote priors. For MVN
0 0
priors, the posterior over µ is given by a MVN distribution (µ˜ (t),Σ˜ (t)), where
a d a a
N
Σ˜ (t) = (cid:0)Σ−1+N (t)Σ−1(cid:1)−1 and
a 0 a a
µ˜ (t) = Σ˜ (t)(cid:0)Σ−1µ +N (t)Σ−1µˆ (t)(cid:1)
a a 0 0 a a a
for the known covariance matrix Σ . Since assuming that Σ is known might be unrealistic in
a a
practice, one can consider the non-informative covariance Σ = I . With non-informative priors
a d
µ = 0 and Σ = I ,3 the algorithm corresponds to a direct extension of the one-dimensional
0 d×1 0 d
TS from Gaussian priors by Agrawal and Goyal [2]. Algorithm 3 shows the resulting TS procedure
from MVN priors.
Algorithm 3 Thompson sampling from MVN priors
1: for all episode t 1 do
≥
2: for all action a do
3: sample θa(t) =∈ Ad(cid:0)(Id+Na(t)Id)−1Na(T)µˆa(t),(Id+Na(t)Id)−1(cid:1)
N
4: end for
5: (t) = argmaxa∈Af(θa(t))
O
6: play a(t) (t) and observe z(t)
∈ O
7: end for
The following proposition provide general regret bounds for TS from MVN priors, which are inde-
pendent from the preference function. The next theorem specializes these regret bounds for three
well known preference function families.
Proposition1. Assumingσ-sub-Gaussiannoisewithσ2 1/(4d), theexpectedregretofThompson
≤
30 indicates a column vector of d elements and I indicates a d×d identity matrix.
d×1 d
8
sampling with multivariate normal priors (Algorithm 3) is bounded by
(cid:88) (cid:20) ln(dT∆2) 4 ln(dT∆2) ln(dT∆2)(cid:21)
R(T) (C(d)+4d)(1+σ)∆ a + +2∆ a +2σ2∆ a ,
≤ a ρ2 ∆ a(ρ r )2 a r2
(cid:63) a a a a
a∈A,a(cid:54)=(cid:63) −
where C(d) is such that for i C(d), e−(cid:113)(4πlini)d d.
≥ ≤ i2
Theorem 1. Assume either a linear (Example 1), Chebyshev (Example 2), or (cid:15)-constraint (Exam-
ple 3) preference function. Assuming σ-sub-Gaussian noise with σ2 1/(4d), the expected regret
≤
of Thompson sampling from multivariate normal priors (Algorithm 3) is bounded by
(cid:88) (cid:20) ln(dT∆2) 4 (cid:21)
R(T) (8C(d)+24d+18+72σ2)(1+σ)2 a + ,
≤ ∆ ∆
a a
a∈A,a(cid:54)=(cid:63)
where C(d) is such that for i C(d), e−(cid:113)(4πlini)d d, that is of order (√dNT lnd+√dNT lnN),
≥ ≤ i2 O
where N = . More specifically, for d lnN, the regret bound is of order (√dNT lnN).
|A| ≤ O
Remark 1. For d = 1, we can take C(d) = e11 as in Agrawal and Goyal [2]. For d = 2 we can
take C(d) = e18, for d = 3 we can take C(d) = e26, and so on for any d N.
∈
For d = 1, the order of the regret bounds given by Theorem 1 match the order of the regret bounds
provided by Agrawal and Goyal [2] for TS from Gaussian priors in the single-objective bandits
setting assuming [0,1]-bounded outcomes. However we observe that the noise tolerance decreases
linearly with the dimension d of the objective space. This means that the more dimensions we
have, the less noise we can bear in order for these bounds to hold, given the provided analysis.
5 Theoretical Analysis
In this section we start by proving Proposition 1 that provides a regret bound for TS with MVN
priors that is independent from the preference function. Then we use the relations between the gap
and the preference radius in three preference function families to obtain Theorem 1.
5.1 Proof of Proposition 1
The following analysis extends the work of Agrawal and Goyal [2] for the 1-dimensional setting to
the d-dimensional setting. We rewrite Equation 1 as
(cid:88)
R(T) = ∆ E[N (T +1)],
a a
a∈A
where we control E[N (T)]. The proof relies on several facts (see Appendix A) that extend Cher-
a
noff’s inequalities and concentration/anti-concentration bounds from the 1-dimensional setting to
thed-dimensionalsettingusingtheconceptsofPareto-dominationandpreferenceradius. Weintro-
duce the following quantities and events to control the quality of mean estimations and the quality
of samples.
9
Definition 2 (Quantitiesr ). For each suboptimal action a, we choose a quantity r < ρ , where ρ
a a a a
is a preference radius. By definition of the preference radii, we have µ µ +r µ +ρ µ
a a a a a (cid:63)
≺ ≺ ≺ −
ρ . Recall that f(x) < f(y) if x y. Hence we have f(µ ) < f(µ +r ) < f(µ +ρ ) < f(µ ρ )
(cid:63) a a a a a (cid:63) (cid:63)
≺ −
for each suboptimal action a.
Definition 3 (Events Eµ(t), Eθ(t)). For each suboptimal action a, define Eµ(t) as the event that
a a a
µˆ (t) µ +r , and define Eθ(t) as the event that θ (t) µ +ρ . More specifically, they are the
a a a a a a a
≺ ≺
event that suboptimal action a is well estimated and well sampled, respectively.
Definition 4 (Filtration ). Define filtration = a(s),z(s),s = 1,...,t 1 .
t t
F F { − }
We decompose
T
(cid:88)
E[N (T)] = P[a(t) = a]
a
t=1
T T T
= (cid:88)P[a(t) = a,Eµ(t),Eθ(t)]+(cid:88)P[a(t) = a,Eµ(t),Eθ(t)]+(cid:88)P[a(t) = a,Eµ(t)]
a a a a a
t=1 t=1 t=1
(cid:124) (cid:123)(cid:122) (cid:125) (cid:124) (cid:123)(cid:122) (cid:125) (cid:124) (cid:123)(cid:122) (cid:125)
(A) (B) (C)
and control each part separately. Part (A) deals with the situation where a suboptimal action is
played while being well estimated and well sampled. We control this by bounding bad estimation
and bad samples for the optimal action. Part (B) deals with the situation where a suboptimal
action is played while being well estimated but badly sampled. We control this using Gaussian
concentration inequalities. Finally, part (C) deals with the situation where a suboptimal action
is played while being badly estimated. We control this using Chernoff inequalities. Putting the
following results together, we obtain
ln(dT∆2) 4 ln(dT∆2) ln(dT∆2)
∆ E[N (T)] ∆ (2C(d)+8d)(1+σ)2 a + +∆ 2 a +∆ 2σ2 a
a a ≤ a ρ2 ∆ a (ρ r )2 a r2
(cid:63) a a a a
−
for σ2 1/(4d), where C(d) is such that for i C(d), e−(cid:113)(4πlini)d d. Summing over all
≤ ≥ ≤ i2
suboptimal actions, we obtain Proposition 1.
5.1.1 Bounding (A)
By definition of TS, for suboptimal a to be played on episode t, we must (at least) have f(θ (t)) >
a
f(θ (t)). By definition of event Eθ(t) and the preference radii, we have f(θ (t)) < f(θ (t)) if
(cid:63) a a (cid:63)
θ (t) µ ρ . Let τ denote the time step at which action (cid:63) is selected for the kth time for k 1,
(cid:63) (cid:63) (cid:63) k
(cid:31) − ≥
10