Online Optimization : Competing with Dynamic Comparators

0.23 MB·English

by  Ali Jadbabaie

1
Online Optimization : Competing with
Dynamic Comparators
Ali Jadbabaie1, Alexander Rakhlin2, Shahin Shahrampour1 and Karthik Sridharan3
5
1
0 Abstract
2
n Recent literature on online learning has focused on developing adaptive algorithms that take advantage of a
a regularityofthesequence ofobservations, yetretainworst-caseperformance guarantees. Acomplementarydirection
J
istodevelop predictionmethodsthat performwellagainstcomplex benchmarks. Inthispaper, weaddress thesetwo
6
2 directions together. We present a fully adaptive method that competes with dynamic benchmarks in which regret

guarantee scaleswithregularityof thesequence of cost functions andcomparators. Notably, theregretbound adapts
]
G
to the smaller complexity measure in the problem environment. Finally, we apply our results to drifting zero-sum,
L
two-player games where both players achieve no regret guarantees against best sequences of actions in hindsight.
.
s
c
[
I. INTRODUCTION

1
v The focus of this paper is an online optimization problem in which a learner plays against an adversary or
5
2 nature. At each round t 1,...,T , the learner chooses an action xt from some convex feasible set Rd.
∈ { } X ⊆
2
Then, nature reveals a convexfunction f to the learner. As a result, the learner incurs the correspondingloss
6 t
∈F
0
f (x ). A learner aims to minimize his regret, a comparison to a single best action in hindsight:
. t t
1
0 T T
5 RegsT , ft(xt)−mx in ft(x). (1)
1 t=1 ∈X t=1
X X
v: Let us refer to this as static regret in the sense that the comparator is time-invariant. In the literature, there are
i
X numerous algorithms that guarantee a static regret rate of (√T) (see e.g. [1]–[3]). Moreover, when the loss
O
r
functions are strongly convex, a rate of (logT) could be achieved [4]. Furthermore, minimax optimality of
a
O
algorithms with respect to the worst-case adversary has been established (see e.g. [5]).
There are two major directions in which the above-mentioned results can be strengthened: (1) by exhibiting
algorithms that compete with non-static comparator sequences (that is, making the benchmark harder), and (2)
[1]AliJadbabaie and Shahin Shahrampour arewith theDepartment ofElectrical andSystems Engineering atthe University ofPennsylvania,
Philadelphia, PA19104USA.(e-mail:[email protected]; [email protected]).
[2] Alexander Rakhlin is with the Department of Statistics at the University of Pennsylvania, Philadelphia, PA 19104 USA. (e-mail:
[email protected]).
[3] Karthik Sridharan is with the Department of Computer Science at Cornell University, Ithaca, NY 14850 USA. (e-mail: sridha-
[email protected]).
January27,2015 DRAFT
2
by proving regret guarantees that take advantage of niceness of nature’s sequence (that is, exploiting some non-
adversarial quality of nature’s moves). Both of these distinct directions are important avenues of investigation. In
the present paper,we attempt to address these two aspects by developinga single, adaptive algorithm with a regret
bound that shows the interplay between the difficulty of the comparison sequence and niceness of the sequence of
nature’s moves.
With respect to the first aspect, a more stringent benchmark is a time-varying comparator, a notion that can be
termed dynamic regret [3], [6]–[8]:
T T
Regd , f (x ) f (x ), (2)
T t t − t ∗t
t=1 t=1
X X
where x ,argmin f (x). More generally, dynamic regret against a comparator sequence u T is
∗t x∈X t { t}t=1
T T
Regd(u ,...,u ), f (x ) f (u ).
T 1 T t t − t t
t=1 t=1
X X
Itiswell-knownthatinthe worstcase, obtaininga boundondynamicregretis notpossible.However,itispossible
to achieve worst-case bounds in terms of
T
C (u ,...,u ), u u , (3)
T 1 T t t 1
− −
t=1
X(cid:13) (cid:13)
i.e., the regularity of the comparator sequence, interpolating b(cid:13)etween the(cid:13)static and dynamic regret notions. Fur-
thermore, the authors in [9] introduce an algorithm which proposes a variant of C involving a dynamical model.
T
Intermsoftheseconddirection,thereareseveralwaysofincorporatingpotentialregularityofnature’ssequence.
The authors in [10], [11] bring forward the idea of predictable sequences – a generic way to incorporate some
external knowledge about the gradients of the loss functions. Let M T be a predictable sequence computable
{ t}t=1
by the learner at the beginning of round t. This sequence can then be used by an algorithm in order to achieve
regret in terms of
T
D , f (x ) M 2. (4)
T t t t
∇ −
t=1 ∗
X(cid:13) (cid:13)
Theframeworkofpredictablesequencescapturesvari(cid:13)ationandpath-l(cid:13)engthtyperegretbounds(see e.g.[12],[13]).
Yet another way in which niceness of the adversarial sequence can be captured is through a notion of temporal
variability studied in [14]:
T
V , sup f (x) f (x) . (5)
T t t 1
x − −
Xt=1 ∈X(cid:12) (cid:12)
What is interesting—and intuitive—dynamic regret (cid:12)against the optim(cid:12)al sequence x T becomes a feasible
{ ∗t}t=1
objective when V is small. When only noisy versions of gradients are revealed to the algorithm, Besbes et al.
T
in [14] show that using a restarted Online Gradient Descent (OGD) [3] algorithm, one can get a bound of form
T2/3(V +1)1/3 on the expected regret. However,the regretboundsattained in [14] are only valid when an upper
T
bound on V is known to the learner before the game begins. For the full information online convex optimization
T
January27,2015 DRAFT
3
setting, when one receives exact gradients instead of noisy gradients, a bound of order V is trivially obtained by
T
simply playing (at each round) the minimum of the previous round.
The three quantities we just introduced — C ,D ,V — measure distinct aspects of the online optimization
T T T
problem, and their interplay is an interesting object of study. Our first contribution is to develop a fully adaptive
method (without prior knowledge of these quantities) whose dynamic regret is given in terms of these three
complexity measures. This is done for the full information online convex optimization setting, and augments the
existing regret bounds in the literature which focus on only one of the three notions — C ,D ,V — (and not
T T T
all the three together). To establish a sub-linearbound on the dynamic regret, we utilize a variantof the Optimistic
Mirror Descent (OMD) algorithm [10].
When noiseless gradients are available and we can calculate variations at each round, we not only establish a
regret bound in terms of V and T (withouta prioriknowledge of a bound on V ), but also show how the bound
T T
can in fact be improvedwhen deviation D is o(T). We further also show how the bound can automatically adapt
T
to C the length of sequence of comparators. Importantly,this avoids suboptimal bounds derived only in terms of
T
one of the quantities — C ,V — in an environment where the other one is small.
T T
Thesecondcontributionofthispaperisthetechnicalanalysisofthealgorithm.Theboundonthedynamicregret
is derived by applying the doubling trick to a non-monotone quantity which results in a non-monotone step size
sequence (which has not been investigated to the best of authors’ knowledge).
We provide uncoupled strategies for two players playing a sequence of drifting zero sum games. We show how
when the two players play the provided strategies, their pay offs converge to the average minimax value of the
sequence of games (provided the games drift slowly). In this case, both players simultaneously enjoy no regret
guaranteesagainst best sequencesof actionsin hindsightthat vary slowly. This is a generalizationof the results by
Daskalakis et al. [15], and Rakhlin et al. [11], both of which are for fixed games played repeatedly.
II. PRELIMINARIES AND PROBLEMFORMULATION
A. Notation
Throughoutthe paper, we assume that for any action x Rd at any time t, it holds that
∈X ⊂
f (x) G. (6)
t
| |≤
We denote by the dualnorm of , by [T] the set of naturalnumbers 1,...,T , and by f the shorthand
1:t
k·k∗ k·k { }
of f ,...,f , respectively. Whenever C is written without arguments, it will refer to regularity C (x ,...,x ) of
1 t T T ∗1 ∗T
the sequence of minimizers of the loss functions. We point out that our initial statements hold for the regularity
of any sequence of comparators. However, for upper bounds involving √C , one needs to choose a computable
T
quantity to tune the step size, and hence our main results are stated for C (x ,...,x ).
T ∗1 ∗T
The quantity D is defined with respect to an arbitrary predictable sequence M T , but this dependence is
T { t}t=1
omitted for brevity.
January27,2015 DRAFT
4
B. Comparing with existing regret bounds in the dynamic setting
We state and discuss relevant results from the literature on online learning in dynamic environments. For any
comparator sequence u T and the specific minima sequence x T the following results are established in
{ t}t=1 { ∗t}t=1
the literature:
Reference Regret Notion
Regret Rate
[3], [9] T f (x ) f (u )
t=1 t t − t t
√PT(1+CT(u1,...,uT))
O
[14] (cid:16) T E[f (x )] f (x ) (cid:17)
t=1 t t − t ∗t
P T2/3(1+VT)1/3
O
[11] (cid:0)T f (x ) f (u)(cid:1)
t=1 t t − t
P √DT
O
Our work Tt=1ft(cid:0)(xt)−(cid:1)ft(x∗t)
O˜ √DT +1+min P (DT +1)CT,(DT +1)1/3T1/3VT1/3
(cid:16) np o(cid:17)
where ˜() hides the logT factor. Lemma 1 below also yields a rate of √DT +1(1+CT(u1,...,uT)) for any
O · O
comparator sequence u T . A detailed explanation of the bounds will b(cid:0)e done after Theorem 3. (cid:1)
{ t}t=1
We remark that the authors in [14] consider a setting in which a variation budget (an upper bound on V )
T
is known to the learner, but he/she only has noisy gradients available. Then, the restarted OGD guarantees the
mentioned rate for convex functions; the rate is modified to (V +1)T for strongly convex functions.
T
For the case of noiseless gradients, we first aim to showpthat our algorithm is adaptive in the sense that the
learner needs not know an upper bound on V in advance when he/she can calculate variations observed so far.
T
Furthermore,we shallestablish thatourmethodrecoversthe knownboundsforstationarysettings(aswellas cases
where V does not change gradually along the time horizon)
T
C. Comparison of Regularity and Variability
We now show that V and C are not comparable in general. To this end, we consider the classical problem
T T
of prediction with expert advice. In this setting, the learner deals with the linear loss f (x) = f ,x on the
t t
h i
d-dimensional probability simplex. Assume that for any t 1, we have the vector sequence
≥
( 1,0,0,...,0) , if t even
f = −T .
t  (0, 1,0,...,0) , if t odd
 −T
Setting u , the comparator of round t, to be the minimizer of f , i.e. u =x , we have
t  t t ∗t
T T
C = x x =Θ(T) V = f f = (1),
T k ∗t − ∗t−1k1 T k t− t−1k∞ O
t=1 t=1
X X
according to (3) and (5), respectively. We see that V is considerably smaller than C in this scenario. On the
T T
other hand, consider prediction with expert advice with two experts. Let f = ( 1/2,0) on even rounds and
t
−
January27,2015 DRAFT
5
f = (0,1/2) on odd rounds. Expert 1 remains to be the best throughout the game, and thus C = (1),
t T
O
while variation V = Θ(T). Therefore, one can see that taking into account only one measure might lead us to
T
suboptimalregretbounds.We show that both measures play a key role in our regretbound.Finally, we note that if
M = f (x ), the notion of D can be related to V in certain cases, yet we keep the predictable sequence
t t 1 t 1 T T
∇ − −
arbitrary and thus as playing a role separate from V and C .
T T
III. MAIN RESULTS
A. Optimistic Mirror Descent and Relation to Regularity
We now outline the OMD algorithm previously proposed in [10]. Let be a 1-strongly convex function with
R
respect to a norm , and (, ) represent the Bregman divergence with respect to . Also, let be the set
t
k·k DR · · R H
containing all available information to the learner at the beginning of time t. Then, the learner can compute the
vector M : Rd, which we call the predictable process. Supposing that the learner has access to the side
t t
H →
informationM Rd fromthe outsetofroundt, the OMD algorithmis characterizedvia the followinginterleaved
t
∈
sequence,
x =argmin η x,M + (x,xˆ ) (7)
t x∈X t t DR t−1
(cid:26) (cid:27)
(cid:10) (cid:11)
xˆ =argmin η x, + (x,xˆ ) , (8)
t x∈X t ∇t DR t−1
(cid:26) (cid:27)
(cid:10) (cid:11)
where , f (x ), and η is the step size that can be chosen adaptively to attain low regret. One could observe
t t t t
∇ ∇
thatforM =0,theOMDalgorithmamountstothewell-knownMirror Descentalgorithm[16],[17].Ontheother
t
hand,thespecialcase ofM = recoverstheschemeproposedin [13]. Itisshownin[10]thatthe static regret
t t 1
∇−
satisfies
Regs 4R D +1 ,
T ≤ max T
(cid:16)p (cid:17)
using the step size
1
η =R min D + D − ,1 ,
t max t 1 t 2
− −
(cid:26)(cid:16)p p (cid:17) (cid:27)
where R2 , sup (x,y). The following lemma extends the result to arbitrary sequence of comparators
max x,y∈XDR
u T . Throughout, we assume that M 2 =1 by convention.
{ t}t=1 k∇0− 0k
∗
Lemma 1. Let be a convexset in a Banachspace . Let : R be a 1-stronglyconvexfunctionon with
X B R B 7→ X
respect to a norm , and let denote the dual norm. For any L>0, employing the time-varying step size
k·k k·k∗
L
η = ,
t
ts−=10k∇s−Msk2+ ts−=20k∇s−Msk2
∗ ∗
q q
and running the Optimistic Mirror DesPcent algorithm for any cPomparator sequence u T , yields
{ t}t=1
γC (u ,...,u )+4R2
Regd(u ,...,u ) 2 1+D L+2 1+D T 1 T max,
T 1 T ≤ T T L
p p
January27,2015 DRAFT
6
so long as (x,z) (y,z) γ x y , x,y,z .
DR −DR ≤ k − k ∀ ∈X
Lemma 1 underscores the fact that one can get a tighter bound for regret once the learner advances a sequence
of conjectures M T well-aligned with the gradients. Moreover, if the learner has prior knowledge of C (or
{ t}t=1 T
an upper bound on it), then the regret bound would be (D +1)C by tuning L.
T T
O
Note that when the function is Lipschitz on , t(cid:16)hpe Lipschitz con(cid:17)dition on the Bregman divergence is
R X
automatically satisfied. For the particular case of KL divergence this can be achieved via mixing a uniform
distribution to stay away from boundaries (see e.g. section 4.2 of the paper in this regard). In this case, the
constant γ is of (logT).
O
B. The Adaptive Optimistic Mirror Descent Algorithm
The main objective of the paper is to develop the Adaptive Optimistic Mirror Descent (AOMD) algorithm. The
AOMDalgorithmincorporatesallnotionsofvariationD ,C andV toderiveacomprehensiveregretbound.The
T T T
proposedmethod builds on the OMD algorithm with adaptive step size, combined with a doublingtrick applied to
athresholdgrowingnon-monotonically(seee.g.[1],[10]forapplicationofdoublingtrickonmonotonequantities).
The scheme is adaptive in the sense that no prior knowledge of D , C or V is necessary.
T T T
Observethatthepriorknowledgeofa variationbudget(an upperboundonV ) doesnottellus howthe changes
T
betweencostfunctionsaredistributedthroughoutthegame.Forinstance,thevariationcanincreasegraduallyalong
the time horizon, while it can also take place in the form of discrete switches. The learner does not have any
informationaboutthe variationpattern.Therefore,she mustadopta flexible strategythatachieveslow regretin the
benign case of finite switches or shocks, while it is simultaneously able to compete with the worst-case of gradual
change. Before describing the algorithm, let us first use Lemma 1 to bound the general dynamic regret in terms of
D , C and V .
T T T
Lemma 2. Let be a convex set in a Banach space . Let : R be a 1-strongly convex function on
X B R B 7→ X
with respect to a norm . Run the Optimistic Mirror Descent algorithm with the step size given in the statement
k·k
of Lemma 1. Letting the comparator sequence be u T , for any L>2R we have
{ t}t=1 max
4γR TV
Regd(u ,...,u ) 4 1+D L+1 γC (u ,...,u )>L2 4R2 max T,
T 1 T ≤ T T 1 T − max L2 4R2
− max
p (cid:8) (cid:9)
so long as (x,z) (y,z) γ x y , x,y,z .
DR −DR ≤ k − k ∀ ∈X
We now describe AOMD algorithmshown in table 1, and provethat it automatically adaptsto V , D and C .
T T T
Thealgorithmcanbecastasa repeatedOMD usingdifferentstepsizes. ThelearnersetstheparameterL=3R
max
inLemma1,andrunstheOMDalgorithm.Alongtheprocess,thelearnercollectsdeviation,variationandregularity
observed so far, and checks the doubling condition in table 1 after each round. Once the condition is satisfied, the
learner doubles L, discards the accumulated deviation, variation and regularity, and runs a new OMD algorithm.
Note importantly that the doubling condition results in a non-monotone sequence of step size during the learning
process.
January27,2015 DRAFT
7
Algorithm 1 Adaptive Optimistic Mirror Descent Algorithm
Parameter : R , some arbitrary x
max 0
∈X
Initialize N =1, C =V =0, D =1, x =x , L =3R , ∆ =0 and k =1.
(1) (1) (1) 1 0 1 max 1 1
for t=1 to T do
% check doubling condition
if L2N <γmin C(N) , V(2N/)3∆2N/3D(−N1)/3 +4Rm2ax then
% incremennt N and double Lo
N
N =N +1
L =3R 2N 1, C =V =0, D =1 and ∆ =0
N max − (N) (N) (N) N
k =t
N
end if
Play x and suffer loss f (x )
t t t
Calculate M (predictable sequence) and gradient = f (x )
t+1 t t t
∇ ∇
% update D ,C ,V and ∆
(N) (N) (N) N
D =D + M 2
(N) (N) t t
k∇ − k
∗
C =C + x x
(N) (N) ∗t − ∗t−1
V =V +s(cid:13)up f (x(cid:13)) f (x)
(N) (N) (cid:13) x∈X | t (cid:13) − t−1 |
∆ =∆ +1
N N
% set step-size and perform optimistic mirror descent update
1
η =L D + D M 2 −
t+1 N (N) (N) t t
(cid:18) q −k∇ − k∗(cid:19)
p
xˆ =argmin η x, + (x,xˆ )
t t t t 1
x∈X (cid:26) ∇ DR − (cid:27)
(cid:10) (cid:11)
x =argmin η x,M + (x,xˆ )
t+1 t+1 t+1 t
end for x∈X (cid:26) (cid:10) (cid:11) DR (cid:27)
Noticethatoncewehavecompletedrunningthealgorithm,N isthenumberofdoublingepochs,∆ isthenumber
i
of instances in epoch i, k and k 1 are the start and end points of epoch i, ∆ =T , N C =C ,
i i+1− i=1 i i=1 (i) T
Ni=1D(i) =DT +N and Ni=1V(i) =VT. Also, there is a technical reason foPr initialization chPoice of L which
PshallbecomeclearintheproPofofLemma2.Theorem3showstheboundenjoyedbytheproposedAOMDalgorithm.
Theorem 3. Assume that (x,z) (y,z) γ x y , x,y,z , and let C = T x x . The
DR −DR ≤ k − k ∀ ∈X T t=1 ∗t − ∗t−1
AOMD algorithm enjoys the following bound on dynamic regret : P (cid:13) (cid:13)
(cid:13) (cid:13)
Regd ˜ D +1 + ˜ min (D +1)C ,(D +1)1/3T1/3V1/3 ,
T ≤O T O T T T T
(cid:16)p (cid:17) (cid:16) np o(cid:17)
where ˜() hides a logT factor.
O ·
Based on Theorem 3 we can obtain the following table that summarizes bounds on Regd for various cases
T
January27,2015 DRAFT
8
(disregarding the first term ˜ √D +1 in the bound above):
T
O
(cid:0) (cid:1)
Regime Rate
C T2/3(D +1) 1/3V2/3 ˜ C (D +1)
T ≤ T − T O T T
V D +1 ˜ (cid:16)(Dp +1)2/3T1/3(cid:17)
T T T
≤ O
D V 1 (cid:0) ˜ V2/3T1/3 (cid:1)
T ≤ T − O T
D = (T) ˜(cid:0)T2/3V1/3(cid:1)
T O O T
The following remarks are in order : (cid:0) (cid:1)
In all cases, given the condition V = o(T), the regret is sub-linear. When the gradients are bounded, the
T
•
regime D = (T) always holds, guaranteeing the worst-case bound of ˜ T2/3V1/3 .
T O O T
Theorem3 allowsus to recover ˜(1) regretfor certain cases whereVT = (cid:0)(1). Letna(cid:1)ture dividethe horizon
• O O
into B batches, and play a smooth convex function f (x) on each batch i [B], that is for some H > 0 it
i i
∈
holds that
f (x) f (y) H x y , (9)
i i i
k∇ −∇ k ≤ k − k
∗
i [B] and x,y . Set M = f (xˆ ) and note that the gradients are Lipschitz continuous. In this
t i t 1
∀ ∈ ∀ ∈X ∇ −
case, the OMD correspondingto each batch can be recognizedas the Mirror Prox method [18], which results
in ˜(1) regret during each period. Also, since C = (1) the bound in Theorem 3 is of (logT).
T
O O O
IV. APPLICATIONS
A. Competing with Strategies
So far, we mainly considered dynamic regret Regd defined in Equation 2. However, in many scenarios one
T
might want to consider regret against a more specific set of strategies, defined as follows :
T T
RegΠ , f (x ) inf f (π (f )),
T t t −π Π t t 1:t−1
Xt=1 ∈ Xt=1
where each π Π is a sequence of mappings π = (π ,...,π ) and π : t 1 . Notice that if Π is the set
1 T t −
∈ F → X
of all mappings then RegΠ corresponds to dynamic regret Regd and if Π corresponds to set of constant history
T T
independent mappings, that is, each π Π is indexed by some x and πx() = ... =πx() = x, then RegΠ
∈ ∈X 1 · T · T
corresponds to the static regret Regs. We now define
T
T
CΠ = π (f ) π (f ) ,
T t∗ 1:t−1 − t∗−1 1:t−2
t=1
X(cid:13) (cid:13)
whereπ =arginf t f (π (f )). As(cid:13)sume thatthereexists seque(cid:13)nceof mappingsC˜ ,...,C˜ whereC˜
t∗ π Π s=1 s s 1:s 1 1 T t
∈ −
maps any f1,...,ft toPreals and is such that for any t and any f1,...,ft 1,
−
C˜ (f ) C˜ (f ),
t 1 1:t 1 t 1:t
− − ≤
January27,2015 DRAFT
9
and further, for any T and any f ,...,f ,
1 T
T
π (f ) π (f ) C˜ (f ).
t∗ 1:t−1 − t∗−1 1:t−2 ≤ T 1:T
t=1
X(cid:13) (cid:13)
In this case a simple modification of A(cid:13)OMD algorithm where C(cid:13) ’s are replaced by C˜ (f ) leads to
(N) ∆N kN:kN+1−1
the following corollary of Theorem 3.
Corollary 4. Assume that (x,z) (y,z) γ x y , x,y,z . The AOMD algorithm with the
DR − DR ≤ k − k ∀ ∈ X
modification mentioned above achieves the following bound on regret
RegΠ ˜ D +1 + ˜ min (D +1)C˜ (f ),(D +1)1/3T1/3V1/3 .
T ≤O T O T T 1:T T T
(cid:16)p (cid:17) (cid:18) (cid:26)q (cid:27)(cid:19)
The corollary naturally interpolatesbetween the static and dynamic regret. In other words, letting C˜ (f )=0
T 1:T
(whichholdsforconstantmappings),we recovertheresultof[11](uptologarithmicfactors),whereasC˜ (f )=
T 1:T
C simply recovers the regret bound in Theorem 3 corresponding to dynamic regret. The extra log factor is the
T
cost of adaptivity of the algorithm as we assume no prior knowledge about the environment.
B. Switching Zero-sum Games with Uncoupled Dynamics
ConsidertwoplayersplayingT zerosumgamesdefinedbymatricesA [ 1,1]m nforeacht [T].Wewould
t ×
∈ − ∈
like to providestrategies for the two playerssuch that, if both playershonestly follow the prescribedstrategies, the
average payoffs of the players approach the average minimax value for the sequence of games at some fast rate.
Furthermore,wewouldalsoliketoguaranteethatifoneoftheplayers(saythesecond)deviatesfromtheprescribed
strategy, then the first player still has small regret against sequence of actions that do not change drastically. To
this end, one can use a simple modification of the AOMD algorithm for both players that uses KL divergence as
, and mixes in a bit of uniform distribution on each round, producing an algorithm similar to the one in [11]
DR
for unchanging uncoupled dynamic games. The following theorem provides bounds for when both players follow
the strategy and bound on regret for player I when player II deviates from the strategy.
January27,2015 DRAFT
10
On round t, Player I performs
Play x and observe f A
t t⊤ t
Update
xˆ (i) xˆ (i)exp η [fTA ]
t ∝ ′t−1 {− t t t i}
xˆ′t =(1−β)xˆt+(β/n)1n
x (i) xˆ (i)exp η [fTA ]
t+1 ∝ ′t {− t+1 t t i}
and simultaneously Player II performs
Play f and observe A x
t t t
Update
fˆ(i) fˆ (i)exp η [A x ]
t ∝ t′−1 {− t′ t t i}
fˆ =(1 β)fˆ +(β/m)1
t′ − t m
ft+1(i)∝fˆt′(i)exp{−ηt′+1[Atxt]i}
Note that in the description of the algorithm as well as the following proposition and its proof, any letter with
the prime symbol refers to Player II, and it is used to differentiate the letter from its counterpart for player I.
Proposition 5. Define F , t f A f A 2 , and let
t i=1 i⊤ i− i⊤−1 i−1 ∞
P (cid:13) (cid:13) L 1
η =(cid:13) min log(T2n) (cid:13) , .
t ( Ft 1+ Ft 2 32L)
− −
Also define A , t A x A x 2 , and lept p
t i=1k i i− i−1 i−1k∞
P L 1
η =min log(T2m) , .
t′ ( At 1+ At 2 32L)
− −
Let β =1/T2, Mt =ft⊤1At 1, and Mt′ =At 1xt 1.pWhen Plapyer I uses the prescribed strategy, irrespective of
− − − −
the actions of player II, the regret of Player I w.r.t. any sequence of actions u ,...,u is bounded as :
1 T
T 2√F L
f A x f A u 2log(T2n)(C (u ,...,u )+2) 32L+ T +log(T2n) F .
t⊤ t t− t⊤ t t ≤ T 1 T log(T2n)L 2 T
t=1 (cid:18) (cid:19)
X(cid:0) (cid:1) p
Further if both players follow the prescribed strategies then, as long as
2L2 >max C ,C +3, (10)
{ T T′ }
we get,
T T T
t=1fts∈u∆pmft⊤Atxt ≤ t=1xti∈n∆fnfts∈u∆pmft⊤Atxt+ 25T6L + 21L +4t=1kAt−1−Atk∞
X X X
+32L log(T2n)CT +log(T2m)CT′ +2log(T4nm)
+ CT(cid:0)+CT′ +4 20+4q Tt=1LkAt−1−Atk2∞. (cid:1)
P
(cid:0) (cid:1)
January27,2015 DRAFT