Online Learning: Stochastic and Constrained Adversaries

38 Pages·2011·0.44 MB·English

Online Learning: Stochastic and Constrained Adversaries
Alexander Rakhlin Karthik Sridharan Ambuj Tewari
Department of Statistics TTIC Computer Science Department
University of Pennsylvania Chicago, IL University of Texas at Austin
June 16, 2011
Abstract
Learningtheoryhaslargelyfocusedontwomainlearningscenarios. Thefirstistheclassicalstatistical
setting where instances are drawn i.i.d. from a fixed distribution and the second scenario is the online
learning, completely adversarial scenario where adversary at every time step picks the worst instance to
providethelearnerwith. Itcanbearguedthatintherealworldneitheroftheseassumptionsarereason-
able. It is therefore important to study problems with a range of assumptions on data. Unfortunately,
theoretical results in this area are scarce, possibly due to absence of general tools for analysis. Focusing
ontheregretformulation,wedefinetheminimaxvalueofagamewheretheadversaryisrestrictedinhis
moves. Theframeworkcapturesstochasticandnon-stochasticassumptionsondata. Buildingonthese-
quentialsymmetrizationapproach,wedefineanotionofdistribution-dependentRademachercomplexity
for the spectrum of problems ranging from i.i.d. to worst-case. The bounds let us immediately deduce
variation-type bounds. We then consider the i.i.d. adversary and show equivalence of online and batch
learnability. In the supervised setting, we consider various hybrid assumptions on the way that x and
y variables are chosen. Finally, we consider smoothed learning problems and show that half-spaces are
onlinelearnableinthesmoothedmodel. Infact,exponentiallysmallnoiseaddedtoadversary’sdecisions
turns this problem with infinite Littlestone’s dimension into a learnable problem.
1 Introduction
We continue the line of work on the minimax analysis of online learning, initiated in [1, 14, 13]. In these
papers, an array of tools has been developed to study the minimax value of diverse sequential problems
under the worst-case assumption on Nature. In [14], many analogues of the classical notions from statistical
learning theory have been developed, and these have been extended in [13] for performance measures well
beyondtheadditiveregret. Theprocessofsequential symmetrization emergedasakeytechniquefordealing
withcomplicatednestedminimaxexpressions. Intheworst-casemodel,thedevelopedtoolsappeartogivea
unified treatment to such sequential problems as regret minimization, calibration of forecasters, Blackwell’s
approachability, Phi-regret, and more.
Learning theory has been so far focused predominantly on the i.i.d. and the worst-case learning scenarios.
Muchlessisknownaboutlearnabilityin-betweenthesetwoextremes. Inthepresentpaper,wemakeprogress
towards fillingthis gap. Insteadof examiningvariousperformance measures, asin [13], wefocus onexternal
regret and make assumptions on the behavior of Nature. By restricting Nature to play i.i.d. sequences, the
results boil down to the classical notions of statistical learning in the supervised learning scenario. By not
placing any restrictions on Nature, we recover the worst-case results of [14]. Between these two endpoints
of the spectrum, particular assumptions on the adversary yield interesting bounds on the minimax value of
the associated problem.
By inertia, we continue to use the name “online learning” to describe the sequential interaction between
the player (learner) and Nature (adversary). We realize that the name can be misleading for a number of
1
reasons. First, the techniques developed in [14, 13] apply far beyond the problems that would traditionally
becalled“learning”. Second, inthispaperwedealwithnon-worst-caseadversaries, whiletheword“online”
often (though, not always) refers to worst-case. Still, we decided to keep the misnomer “online learning”
whenever the problem is sequential.
Adapting the game-theoretic language, we will think of the learner and the adversary as the two players
of a zero-sum repeated game. Adversary’s moves will be associated with “data”, while the moves of the
learner – with a function or a parameter. This point of view is not new: game-theoretic minimax analysis
has been at the heart of statistical decision theory for more than half a century (see [3]). In fact, there is a
well-developed theory of minimax estimation when restrictions are put on either the choice of the adversary
or the allowed estimators by the player. We are not aware of a similar theory for sequential problems with
non-i.i.d. data.
In particular, minimax analysis is central to nonparametric estimation, where one aims to prove optimal
rates of convergence of the proposed estimator. Lower bounds are proved by exhibiting a “bad enough”
distribution of the data that can be chosen by the adversary. The form of the minimax value is often
inf supE(cid:107)fˆ−f(cid:107)2 (1)
fˆ f
∈F
where the infimum is over all estimators and the supremum is over all functions f from some class F. It is
often assumed that Y = f(X )+(cid:15) , with (cid:15) being zero-mean noise. An estimator can be thought of as a
t t t t
strategy, mapping the data {(X ,Y )}T to the space of functions on X. This description is, of course, only
t t t=1
a rough sketch that does not capture the vast array of problems considered in nonparametric estimation.
In statistical learning theory, the data are i.i.d. from an unknown distribution P and the associated
X Y
minimax problem in the supervised setting with square loss is ×
(cid:26) (cid:27)
Vbatch,sup =inf sup E(Y −fˆ(X))2− inf E(Y −f(X))2 (2)
T
fˆ PX×Y f∈F
wheretheinfimumisoverallestimators(orlearningalgorithms)andthesupremumisoveralldistributions.
Unlike nonparametric regression which makes an assumption on the “regression function” f ∈F, statistical
learning theory often aims at distribution-free results. Because of this, the goal is more modest: to predict
as well as the best function in F rather than recover the true model. In particular, (2) sidesteps the issue of
approximation error (model misspecification).
What is known about the asymptotic behavior of (2)? The well-developed statistical learning theory tells
us that (2) converges to zero if and only if the combinatorial dimensions of F (that is, the VC dimension for
binary-valued, or scale-sensitive for real-valued functions) are finite. The convergence is intimately related
to the uniform Glivenko-Cantelli property. If indeed the value in (2) converges to zero, an algorithm that
achieves this is Empirical Risk Minimization. For unsupervised learning problems, however, ERM does not
necessarily drive the quantity Efˆ(X)−inf Ef(X) to zero.
f
∈F
The formulation (2) no longer makes sense if the data generating process is non-stationary. Consider the
opposite from i.i.d. end of the spectrum: the data are chosen in a worst-case manner. First, consider an
oblivious adversarywhofixestheindividualsequencex ,...,x aheadofthegameandrevealsitone-by-one.
1 T
A frequently studied notion of performance is regret, and the minimax value can be written as
(cid:34) (cid:35)
1 (cid:88)T 1 (cid:88)T
Voblivious = inf sup E f (x )− inf f(x ) (3)
T {fˆt}Tt=1(x1,...,xT) f1,...,fT T t=1 t t f∈F T t=1 t
where the randomized strategy for round t is fˆ :Xt 1 (cid:55)→Q, with Q being the set of all distributions on F.
t −
That is, the player furnishes his best randomized strategy for each round, and the adversary picks the worst
sequence.
2
A non-oblivious (adaptive) adversary is, of course, more interesting. The protocol for the online interaction
is the following: on round t the player chooses a distribution q on F, the adversary chooses the next move
t
x ∈X, the player draws f from q , and the game proceeds to the next round. All the moves are observed
t t t
by both players. Instead of writing the value in terms of strategies, we can write it in an extended form as
(cid:34) (cid:35)
1 (cid:88)T 1 (cid:88)T
V = inf sup E ··· inf sup E f (x )− inf f(x ) (4)
T t t t
q1∈Qx1∈Xf1∼q1 qT∈QxT∈XfT∼qT T t=1 f∈F T t=1
This is precisely the quantity considered in [14]. The minimax value for notions other than regret has been
studied in [13]. In this paper, we are interested in restricting the ways in which the sequences (x ,...,x )
1 T
are produced. These restrictions can be imposed through a smaller set of mixed strategies that is available
totheadversaryateachround,orasanon-stochasticconstraintateachround. Theformulationwepropose
captures both types of assumptions.
The main contribution of this paper is the development of tools for the analysis of online scenarios where
the adversary’s moves are restricted in various ways. Further, we consider a number of interesting scenarios
(such as smoothed learning) which can be captured by our framework. The present paper only scratches
the surface of what is possible with sequential minimax analysis. Many questions are to be answered: For
instance, one can ask whether a certain adversary is more powerful than another adversary by studying the
value of the associated game.
The paper is organized as follows. In Section 2 we define the value of the game and appeal to minimax
duality. Distribution-dependent sequential Rademacher complexity is defined in Section 3 and can be seen
to generalize the classical notion as well as the worst-case notion from [14]. This section contains the main
symmetrization result which relies on a careful consideration of original and tangent sequences. Section 4
is devoted to analysis of the distribution-dependent Rademacher complexity. In Section 5 we consider
non-stochastic constraints on the behavior of the adversary. From these results, variation-type results are
seamlessly deduced. Section 6 is devoted to the i.i.d. adversary. We show equivalence between batch and
online learnability. Hybrid adversarial-stochastic supervised learning is considered in Section 7. We show
that it is the way in which the x variable is chosen that governs the complexity of the problem, irrespective
of the way the y variable is picked. In Section 8 we introduce the notion of smoothed analysis in the online
learning scenario and show that a simple problem with infinite Littlestone’s dimension becomes learnable
once a small amount of noise is added to adversary’s moves. Throughout the paper, we use the notation
introduced in [14, 13], and, in particular, we extensively use the “tree” notation.
2 Value of the Game
Consider sets F and X, where F is a closed subset of a complete separable metric space. Let Q be the set
of probability distributions on F and assume that Q is weakly compact. We consider randomized learners
who predict a distribution q ∈Q on every round.
t
Let P be the set of probability distributions on X. We would like to capture the fact that sequences
(x ,...,x ) cannot be arbitrary. This is achieved by defining restrictions on the adversary, that is, subsets
1 T
of “allowed” distributions for each round. These restrictions limit the scope of available mixed strategies for
the adversary.
Definition 1. A restriction P on the adversary is a sequence P ,...,P of mappings P : Xt 1 (cid:55)→ 2
1:T 1 T t − P
such that P (x ) is a convex subset of P for any x ∈Xt 1.
t 1:t 1 1:t 1 −
− −
Note that the restrictions depend on the past moves of the adversary, but not on those of the player. We
will write P instead of P (x ) when x is clearly defined.
t t 1:t 1 1:t 1
− −
3
Using the notion of restrictions, we can give names to several types of adversaries that we will study in this
paper.
• A worst-case adversary is defined by vacuous restrictions P (x )=P. That is, any mixed strategy
t 1:t 1
is available to the adversary, including any deterministic point d−istributions.
• Aconstrained adversary isdefinedbyP (x )beingthesetofalldistributionssupportedontheset
t 1:xt−1
{x∈X :C (x ,...,x ,x)=1}forsomedeterministicbinary-valuedconstraintC . Thedeterministic
t 1 t 1 t
constraint can, for ins−tance, ensure that the length of the path determined by the moves x ,...,x
1 t
stays below the allowed budget.
• A smoothed adversary picks the worst-case sequence which gets corrupted by an i.i.d. noise. Equiva-
lently, we can view this as restrictions on the adversary who chooses the “center” (or a parameter) of
the noise distribution. For a given family G of noise distributions (e.g. zero-mean Gaussian noise), the
restrictions are obtained by all possible shifts P ={g(x−c ):g ∈G,c ∈X}.
t t t
• Ahybrid adversary inthesupervisedlearninggamepickstheworst-caselabely , butisforcedtodraw
t
the x -variable from a fixed distribution [9].
t
• Finally, an i.i.d. adversary is defined by a time-invariant restriction P (x ) = {p} for every t and
t 1:t 1
some p∈P. −
For the given restrictions P , we define the value of the game as
1:T
(cid:34) (cid:35)
T T
(cid:88) (cid:88)
V (P ) =(cid:52) inf sup E inf sup E ··· inf sup E f (x )− inf f(x ) (5)
T 1:T t t t
q1∈Qp1∈P1 f1,x1 q2∈Qp2∈P2 f2,x2 qT∈QpT∈PT fT,xT t=1 f∈F t=1
where f has distribution q and x has distribution p . As in [14], the adversary is adaptive, that is, chooses
t t t t
p based on the history of moves f and x .
t 1:t 1 1:t 1
− −
At this point, the only difference from the setup of [14] is in the restrictions P on the adversary. Because
t
these restrictions might not allow point distributions, the suprema over p ’s in (5) cannot be equivalently
t
written as the suprema over x ’s.
t
The value of the game can also be written in terms of strategies π ={π }T and τ ={τ }T for the player
t t=1 t t=1
and the adversary, respectively, where π : (F ×X ×P)t 1 → Q and τ : (F ×X ×Q)t 1 → P. Crucially,
t − t −
the strategies also depend on the mappings P . The value of the game can equivalently be written in the
1:T
strategic form as
(cid:34) (cid:35)
T T
(cid:88) (cid:88)
V (P )=infsup E ... E f (x )− inf f(x ) (6)
T 1:T t t t
π τ fx11∼∼πτ11 fxTT∼∼πτTT t=1 f∈F t=1
A word about the notation. In [14], the value of the game is written as V (F), signifying that the main
T
object of study is F. In [13], it is written as V ((cid:96),Φ ) since the focus is on the complexity of the set
T T
of transformations Φ and the payoff mapping (cid:96). In the present paper, the main focus is indeed on the
T
restrictions on the adversary, justifying our choice V (P ) for the notation.
T 1:T
The first step is to apply the minimax theorem. To this end, we verify the necessary conditions. Our
assumption that F is a closed subset of a complete separable metric space implies that Q is tight and
Prokhorov’s theorem states that compactness of Q under weak topology is equivalent to tightness [18].
Compactnessunderweaktopologyallowsustoproceedasin[14]. Additionally,werequirethattherestriction
sets are compact and convex.
4
Theorem 1. Let F and X be the sets of moves for the two players, satisfying the necessary conditions
for the minimax theorem to hold. Let P be the restrictions, and assume that for any x , P (x )
1:T 1:t 1 t 1:t 1
satisfies the necessary conditions for the minimax theorem to hold. Then − −
(cid:34) (cid:35)
T T
(cid:88) (cid:88)
V (P )= sup E ... sup E inf E [f (x )]− inf f(x ) . (7)
T 1:T p1∈P1 x1∼p1 pT∈PT xT∼pT t=1ft∈F xt∼pt t t f∈F t=1 t
The nested sequence of suprema and expected values in Theorem 1 can be re-written succinctly as
(cid:34) (cid:35)
T T
(cid:88) (cid:88)
V (P )= supE E ...E inf E [f (x )]− inf f(x ) (8)
T 1:T p∈P x1∼p1 x2∼p2(·|x1) xT∼pT(·|x1:T−1) t=1ft∈F xt∼pt t t f∈F t=1 t
(cid:34) (cid:35)
T T
(cid:88) (cid:88)
= supE inf E [f (x )]− inf f(x )
p∈P t=1ft∈F xt∼pt t t f∈F t=1 t
where the supremum is over all joint distributions p over sequences, such that p satisfies the restrictions
as described below. Given a joint distribution p on sequences (x ,...,x ) ∈ XT, we denote the associated
1 T
conditional distributions by p (·|x ). We can think of the choice p as a sequence of oblivious strategies
t 1:t 1
{p : Xt 1 (cid:55)→ P}T , mapping the p−refix x to a conditional distribution p (·|x ) ∈ P (x ). We
t − t=1 1:t 1 t 1:t 1 t 1:t 1
will indeed call p a “joint distribution” or a−n “oblivious strategy” interchangeably.−We say that−a joint
distribution p satisfies restrictions if for any t and any x ∈ Xt 1, p (·|x ) ∈ P (x ). The set of
1:t 1 − t 1:t 1 t 1:t 1
alljointdistributionssatisfyingtherestrictionsisdenotedby−P. WenotethatT−heorem1can−notbededuced
immediately from the analogous result in [14], as it is not clear how the restrictions on the adversary per
each round come into play after applying the minimax theorem. Nevertheless, it is comforting that the
restrictions directly translate into the set P of oblivious strategies satisfying the restrictions.
Beforecontinuingwithourgoalofupper-boundingthevalueofthegame,letusanswerthefollowingquestion:
Is there an oblivious minimax strategy for the adversary? Even though Theorem 1 shows equality to some
quantity with a supremum over oblivious strategies p, it is not immediate that the answer to our question
is affirmative, and a proof is required. To this end, for any oblivious strategy p, define the regret the player
would get playing optimally against p:
(cid:34) (cid:35)
T T
(cid:88) (cid:88)
Vp =(cid:52) inf E inf E ··· inf E f (x )− inf f(x ) . (9)
T f1∈F x1∼p1f2∈F x2∼p2(·|x1) fT∈F xT∼pT(·|x1:T−1) t=1 t t f∈F t=1 t
The next proposition shows that there is an oblivious minimax strategy for the adversary and a minimax
optimal strategy for the player that does not depend on its own randomizations. The latter statement for
worst-case learning is folklore, yet we have not seen a proof of it in the literature.
Proposition 2. For any oblivious strategy p,
(cid:34) (cid:35)
T T
(cid:88) (cid:88)
V (P ) ≥ Vp = infE E E f (x )− inf f(x ) (10)
T 1:T T π ft∼πt(·|x1:t−1) xt∼pt t t f t
t=1 ∈F t=1
withequalityholdingforp whichachievesthesupremum1 in (8). Importantly, theinfimumisoverstrategies
∗
π = {π }T of the player that do not depend on player’s previous moves, that is π : Xt 1 (cid:55)→ Q. Hence,
t t=1 t −
there as an oblivious minimax optimal strategy for the adversary, and there is a corresponding minimax
optimal strategy for the player that does not depend on its own moves.
Proposition 2 holds for all online learning settings with legal restrictions P , encompassing also the no-
1:T
restrictionssettingofworst-caseonlinelearning[14]. Theresultcruciallyreliesonthefactthattheobjective
is external regret.
1Here,andintherestofthepaper,ifasupremumisnotachieved,aslightlymodifiedanalysiscanbecarriedout.
5
3 Symmetrization and Random Averages
Theorem 1 is a useful representation of the value of the game. As the next step, we upper bound it with
an expression which is easier to study. Such an expression is obtained by introducing Rademacher random
variables. This process can be termed sequential symmetrization and has been exploited in [1, 14, 13].
The restrictions P , however, make sequential symmetrization a bit more involved than in the previous
t
papers. The main difficulty arises from the fact that the set P (x ) depends on the sequence x , and
t 1:t 1 1:t 1
symmetrization (that is, replacement of x with x ) has to be done−with care as it affects this depe−ndence.
s (cid:48)s
Roughly speaking, in the process of symmetrization, a tangent sequence x ,x ,... is introduced such that
(cid:48)1 (cid:48)2
x and x are independent and identically distributed given “the past”. However, “the past” is itself an
t (cid:48)t
interleaving choice of the original sequence and the tangent sequence.
Define the “selector function” χ:X ×X ×{±1}(cid:55)→X by
(cid:26)
x if (cid:15)=1
χ(x,x,(cid:15))= (cid:48)
(cid:48) x if (cid:15)=−1
When x and x are understood from the context, we will use the shorthand χ ((cid:15)) := χ(x ,x ,(cid:15)). In other
t (cid:48)t t t (cid:48)t
words, χ selects between x and x depending on the sign of (cid:15).
t t (cid:48)t
Throughout the paper, we deal with binary trees, which arise from symmetrization [14]. Given some set
Z, an Z-valued tree of depth T is a sequence (z ,...,z ) of T mappings z : {±1}i 1 (cid:55)→ Z. The T-tuple
1 T i −
(cid:15)=((cid:15) ,...,(cid:15) )∈{±1}T defines a path. For brevity, we write z ((cid:15)) instead of z ((cid:15) ).
1 T t t 1:t 1
−
Given a joint distribution p, consider the “(X ×X)T−1 (cid:55)→ P(X × X)”- valued probability tree ρ =
(ρ ,...,ρ ) defined by
1 T
(cid:0) (cid:1)
ρ ((cid:15) ) (x ,x ),...,(x ,x ) =(p (·|χ ((cid:15) ),...,χ ((cid:15) )),p (·|χ ((cid:15) ),...,χ ((cid:15) ))). (11)
t 1:t 1 1 (cid:48)1 T 1 (cid:48)T 1 t 1 1 t 1 t 1 t 1 1 t 1 t 1
− − − − − − −
Inotherwords,thevaluesofthemappingsρ ((cid:15))areproductsofconditionaldistributions,whereconditioning
t
is done with respect to a sequence made from x and x depending on the sign of (cid:15) . We note that
s (cid:48)s s
the difficulty in intermixing the x and x sequences does not arise in i.i.d. or worst-case symmetrization.
(cid:48)
However, in-between these extremes the notational complexity seems to be unavoidable if we are to employ
symmetrization and obtain a version of Rademacher complexity.
As an example, consider the “left-most” path (cid:15) = −1 in a binary tree of depth T, where 1 = (1,...,1)
is a T-dimensional vector of ones. Then all the selectors χ(x ,x ,(cid:15) ) in the definition (11) select the se-
t (cid:48)t t
quence x ,...,x . The probability tree ρ on the “left-most” path is, therefore, defined by the conditional
1 T
distributions p (·|x ). Analogously, on the path (cid:15)=1, the conditional distributions are p (·|x ).
t 1:t 1 t (cid:48)1:t 1
− −
(cid:0) (cid:1)
Slightly abusing the notation, we will write ρ ((cid:15)) (x ,x ),...,(x ,x ) for the probability tree since ρ
t 1 (cid:48)1 t 1 (cid:48)t 1 t
clearlydependsonlyontheprefixuptotimet−1. Throughoutthe−pape−r,itwillbeunderstoodthatthetree
ρ is obtained from p as described above. Since all the conditional distributions of p satisfy the restrictions,
so do the corresponding distributions of the probability tree ρ. By saying that ρ satisfies restrictions we
then mean that p∈P.
Sampling of a pair of X-valued trees from ρ, written as (x,x) ∼ ρ, is defined as the following recursive
(cid:48)
process: for any (cid:15)∈{±1}T,
(x ((cid:15)),x ((cid:15)))∼ρ ((cid:15))
1 (cid:48)1 1
(x ((cid:15)),x ((cid:15)))∼ρ ((cid:15))((x ((cid:15)),x ((cid:15))),...,(x ((cid:15)),x ((cid:15)))) for 2≤t≤T (12)
t (cid:48)t t 1 (cid:48)1 t 1 (cid:48)t 1
− −
To gain a better understanding of the sampling process, consider the first few levels of the tree. The roots
x ,x of the trees x,x are sampled from p , the conditional distribution for t = 1 given by p. Next, say,
1 (cid:48)1 (cid:48) 1
(cid:15) = +1. Then the “right” children of x and x are sampled via x (+1),x (+1) ∼ p (·|x ) since χ (+1)
1 1 (cid:48)1 2 (cid:48)2 2 (cid:48)1 1
6
selectsx . Ontheotherhand, the“left”childrenx (−1),x (−1)arebothdistributedaccordingtop (·|x ).
(cid:48)1 2 (cid:48)2 2 1
Now, suppose (cid:15) =+1 and (cid:15) =−1. Then, x (+1,−1),x (+1,−1) are both sampled from p (·|x ,x (+1)).
1 2 3 (cid:48)3 3 (cid:48)1 2
TheproofofTheorem3revealswhysuchintricateconditionalstructurearises,andSection4showsthatthis
structure greatly simplifies for i.i.d. and worst-case situations. Nevertheless, the process described above
allows us to define a unified notion of Rademacher complexity for the spectrum of assumptions between the
two extremes.
Definition 2. The distribution-dependent sequential Rademacher complexity of a function class F ⊆R is
X
defined as
(cid:34) (cid:35)
T
(cid:88)
R (F,p) =(cid:52) E E sup (cid:15) f(x ((cid:15)))
T (x,x(cid:48)) ρ (cid:15) t t
∼ f
∈F t=1
where (cid:15) = ((cid:15) ,...,(cid:15) ) is a sequence of i.i.d. Rademacher random variables and ρ is the probability tree
1 T
associated with p.
We now prove an upper bound on the value V (P ) of the game in terms of this distribution-dependent
T 1:T
sequential Rademacher complexity. This provides an extension of the analogous result in [14] to adversaries
more benign than worst-case.
Theorem 3. The minimax value is bounded as
V (P )≤2supR (F,p). (13)
T 1:T T
p P
∈
A more general statement also holds:
(cid:34) (cid:40) (cid:41)(cid:35)
T
(cid:88)
V (P )≤ supE sup f(x )−f(x )
T 1:T (cid:48)t t
p P f
∈ ∈F t=1
(cid:34) (cid:35)
T
(cid:88)
≤2supE E sup (cid:15) (f(x ((cid:15)))−M (p,f,x,x,(cid:15)))
(x,x(cid:48)) ρ (cid:15) t t t (cid:48)
p P ∼ f
∈ ∈F t=1
for any measurable function M with the property M (p,f,x,x,(cid:15)) = M (p,f,x,x,−(cid:15)). In particular, (13)
t t (cid:48) t (cid:48)
is obtained by choosing M =0.
t
The following corollary provides a natural “centered” version of the distribution-dependent Rademacher
complexity. That is, the complexity can be measured by relative shifts in the adversarial moves.
Corollary 4. For the game with restrictions P ,
1:T
(cid:34) (cid:35)
(cid:88)T (cid:16) (cid:17)
V (P )≤2supE E sup (cid:15) f(x ((cid:15)))−E f(x ((cid:15)))
T 1:T (x,x(cid:48)) ρ (cid:15) t t t 1 t
p P ∼ f −
∈ ∈F t=1
where E denotes the conditional expectation of x ((cid:15)).
t 1 t
−
Example 1. Suppose F is a unit ball in a Banach space and f(x)=(cid:104)f,x(cid:105). Then
(cid:13) (cid:13)
(cid:13)(cid:88)T (cid:16) (cid:17)(cid:13)
V (P )≤2supE E (cid:13) (cid:15) x ((cid:15))−E x ((cid:15)) (cid:13)
T 1:T (x,x(cid:48)) ρ (cid:15)(cid:13) t t t 1 t (cid:13)
p P ∼ (cid:13) − (cid:13)
∈ t=1
Suppose the adversary plays a simple random walk (e.g., p (x|x ,...,x ) = p (x|x ) is uniform on a
t 1 t 1 t t 1
unit sphere). For simplicity, suppose this is the only strategy allowed by t−he set P. Th−en x ((cid:15))−E x ((cid:15))
t t 1 t
−
7
are independent increments when conditioned on the history. Further, the increments do not depend on (cid:15) .
t
Thus,
(cid:13) (cid:13)
(cid:13)(cid:88)T (cid:13)
V (P )≤2E(cid:13) Y (cid:13)
T 1:T (cid:13) t(cid:13)
(cid:13) (cid:13)
t=1
where {Y } is the corresponding random walk.
t
4 Analyzing Rademacher Complexity
The aim of this section is to provide a better understanding of the distribution-dependent sequential
Rademacher complexity, as well as ways of upper-bounding it. We first show that the classical Rademacher
complexity is equal to the distribution-dependent sequential Rademacher complexity for i.i.d. data. We
further show that the distribution-dependent sequential Rademacher complexity is always upper bounded
by the worst-case sequential Rademacher complexity defined in [14].
It is already apparent to the reader that the sequential nature of the minimax formulation yields long
mathematical expressions, which are not necessarily complicated yet unwieldy. The functional notation and
the tree notation alleviate much of these difficulties. However, it takes some time to become familiar and
comfortable with these representations. The next few results hopefully provide the reader with a better feel
for the distribution-dependent sequential Rademacher complexity.
Proposition 5. Consider the i.i.d. restrictions P ={p} for all t, where p is some fixed distribution on X.
t
Let ρ be the process associated with the joint distribution p=pT. Then
R (F,p)=R (F,p)
T T
where
(cid:34) (cid:35)
T
(cid:88)
R (F,p)=(cid:52) E E sup (cid:15) f(x ) . (14)
T x1,...,xT∼p (cid:15) f t t
∈F t=1
is the classical Rademacher complexity.
Proof. By definition, we have,
(cid:34) (cid:35)
T
(cid:88)
R (F,p)=E E sup (cid:15) f(x ((cid:15))) (15)
T (x,x(cid:48)) ρ (cid:15) t t
∼ f
∈F t=1
In the i.i.d. case, however, the tree generation according to the ρ process simplifies: for any (cid:15)∈{±1}T,t∈
[T],
(x ((cid:15)),x ((cid:15)))∼p×p .
t (cid:48)t
Thus, the 2·(2T −1) random variables x ((cid:15)),x ((cid:15)) are all i.i.d. drawn from p. Writing the expectation (15)
t (cid:48)t
explicitly as an average over paths, we get
(cid:34) (cid:35)
1 (cid:88) (cid:88)T
R (F,p)= E sup (cid:15) f(x ((cid:15)))
T 2T (x,x(cid:48))∼ρ f t t
(cid:15) 1 T ∈F t=1
∈{± }
(cid:34) (cid:35)
1 (cid:88) (cid:88)T
= E sup (cid:15) f(x )
2T x1,...,xT∼p f t t
(cid:15) 1 T ∈F t=1
∈{± }
(cid:34) (cid:35)
T
(cid:88)
=E E sup (cid:15) f(x ) .
(cid:15) x1,...,xT∼p f t t
∈F t=1
8
The second equality holds because, for any fixed path (cid:15), the T random variables {x ((cid:15))} have joint
t t [T]
distribution pT. ∈
Proposition 6. For any joint distribution p,
R (F,p)≤R (F)
T T
where
(cid:34) (cid:35)
T
(cid:88)
R (F)=(cid:52) supE sup (cid:15) f(x ((cid:15))) . (16)
T (cid:15) t t
x f
∈F t=1
is the sequential Rademacher complexity defined in [14].
Proof. To make the ρ process associated with p more explicit, we use the expanded definition:
(cid:34) (cid:35)
T
(cid:88)
RT(F,p)=Ex1,x(cid:48)1∼p1E(cid:15)1Ex2,x(cid:48)2∼p2(·|χ1((cid:15)1))E(cid:15)2 ... ExT,x(cid:48)T∼pT(·|χ1((cid:15)1),...,χT−1((cid:15)T−1))E(cid:15)T fsup (cid:15)tf(xt)
∈F t=1
(cid:34) (cid:35)
T
(cid:88)
≤ sup E sup E ... sup E sup (cid:15) f(x ) (17)
(cid:15)1 (cid:15)2 (cid:15)T t t
x1,x(cid:48)1 x2,x(cid:48)2 xT,x(cid:48)T f∈F t=1
(cid:34) (cid:35)
T
(cid:88)
=supE supE ... supE sup (cid:15) f(x )
(cid:15)1 (cid:15)2 (cid:15)T t t
x1 x2 xT f∈F t=1
=R (F) .
T
The inequality holds by replacing expectation over x ,x by a supremum over the same. We then get rid of
t (cid:48)t
x ’s since they do not appear anywhere.
t
An interesting case of hybrid i.i.d.-adversarial data is considered in Lemma 17, and we refer to its proof as
another example of an analysis of the distribution-dependent sequential Rademacher complexity.
We now turn to general properties of Rademacher complexity. The proof of next Proposition follows along
the lines of the analogous result in [14].
Proposition 7. Distribution-dependent sequential Rademacher complexity satisfies the following properties.
1. If F ⊂G, then R(F,p)≤R(G,p).
2. R(F,p)=R(conv(F),p).
3. R(cF,p)=|c|R(F,p) for all c∈R.
4. For any h, R(F +h,p)=R(F,p) where F +h={f +h:f ∈F}
Next, we consider upper bounds on R(F,p) via covering numbers. Recall the definition of a (sequential)
cover, given in [14]. This notion captures sequential complexity of a function class on a given X-valued tree
x.
Definition 3. A set V of R-valued trees of depth T is an α-cover (with respect to (cid:96) -norm) of F ⊆R on
p X
a tree x of depth T if
(cid:32) (cid:33)1/p
1 (cid:88)T
∀f ∈F, ∀(cid:15)∈{±1}T ∃v∈V s.t. |v ((cid:15))−f(x ((cid:15)))|p ≤α
t t
T
t=1
The covering number of a function class F on a given tree x is defined as
N (α,F,x)=min{|V|:V is an α−cover w.r.t. (cid:96) -norm of F on x}.
p p
9
Using the notion of the covering number, the following result holds.
Theorem 8. For any function class F ⊆[−1,1] ,
X
(cid:26) (cid:90) 1 (cid:27)
(cid:112)
R (F,p)≤E inf 4Tα+12 T log N (δ,F,x) dδ .
T (x,x(cid:48)) ρ 2
∼ α α
The analogous result in [14] is stated for the worst-case adversary, and, hence, it is phrased in terms of the
maximalcoveringnumbersup N (δ,F,x). Theproof,however,holdsforanyfixedx,andthusimmediately
x 2
impliesTheorem8. Iftheexpectationover(x,x)inTheorem8canbeexchangedwiththeintegral, wepass
(cid:48)
to an upper bound in terms of the expected covering number E N (δ,F,x).
(x,x(cid:48)) ρ 2
∼
The following simple corollary of the above theorem shows that the distribution-dependent Rademacher
complexity of a function class F composed with a Lipschitz mapping φ can be controlled in terms of the
Dudley integral for the function class F itself.
Corollary 9. Fix a class F ⊆ [−1,1] and a function φ : [−1,1]×Z (cid:55)→ R. Assume, for all z ∈ Z, φ(·,z)
Z
is a Lipschitz function with a constant L. Then,
(cid:26) (cid:90) 1 (cid:27)
(cid:112)
R (φ(F),p)≤L E inf 4Tα+12 T log N (δ,F,z) dδ .
T (z,z(cid:48)) ρ 2
∼ α α
where φ(F)={z (cid:55)→φ(f(z),z):f ∈F}.
The statement can be seen as a covering-number version of the Lipschitz composition lemma.
5 Constrained Adversaries
In this section we consider adversaries who are constrained in the sequences of actions they can play. It is
often useful to consider scenarios where the adversary is worst case, yet has some budget or constraint to
satisfywhilepickingtheactions. Examplesofsuchscenariosinclude,forinstance,gameswheretheadversary
isconstrainedtomakemovesthatarecloseinsomefashiontothepreviousmove,lineargameswithbounded
variance, and so on. Below we formulate such games quite generally through arbitrary constraints that the
adversary has to satisfy on each round.
Specifically, for a T round game consider an adversary who is only allowed to play sequences x ,...,x
1 T
such that at round t the constraint C (x ,...,x ) = 1 is satisfied, where C : Xt (cid:55)→ {0,1} represents the
t 1 t t
constraintonthesequenceplayedsofar. Theconstrainedadversarycanbeviewedasastochasticadversary
with restrictions on the conditional distribution at time t given by the set of all Borel distributions on the
set
X (x ) =(cid:52) {x∈X :C (x ,...,x ,x)=1}.
t 1:t 1 t 1 t 1
− −
Sincesetincludesallpointdistributionsoneachx∈X ,thesequentialcomplexitysimplifiesinawaysimilar
t
to worst-case adversaries. We write V (C ) for the value of the game with the given constraints. Now,
T 1:T
assume that for any x , the set of all distributions on X (x ) is weakly compact in a way similar to
1:t 1 t 1:t 1
compactnessofP. That−is, P (x )satisfythenecessarycondit−ionsfortheminimaxtheoremtohold. We
t 1:t 1
have the following corollaries of Th−eorems 1 and 3.
Corollary 10. Let F and X be the sets of moves for the two players, satisfying the necessary conditions for
the minimax theorem to hold. Let {C :Xt 1 (cid:55)→{0,1}}T be the constraints. Then
t − t=1
(cid:34) (cid:35)
T T
(cid:88) (cid:88)
V (C ) = supE inf E [f (x )]− inf f(x ) (18)
T 1:T p∈P t=1ft∈F xt∼pt t t f∈F t=1 t
where p ranges over all distributions over sequences (x ,...,x ) such that C (x )=1 for all t.
1 T t 1:t 1
−
10