Distributionally Robust Counterpart in Markov Decision Processes

1.3 MB·English

by  Pengqian Yu

1
Distributionally Robust Counterpart in Markov Decision Processes
Pengqian Yu, Huan Xu
Abstract
This paper studies Markov Decision Processes under parameter uncertainty. We adapt the distributionally
robust optimization framework, and assume that the uncertain parameters are random variables following an
unknowndistribution,andseeksthestrategywhichmaximizestheexpectedperformanceunderthemostadversarial
distribution.Inparticular,wegeneralizepreviousstudy[1]whichconcentratesondistributionsetswithveryspecial
structure to much more generic class of distribution sets, and show that the optimal strategy can be obtained
efficiently under mild technical condition. This significantly extends the applicability of distributionally robust
5 MDP to incorporate probabilistic information of uncertainty in a more flexible way.
1
0
2 Index Terms

y Markov processes, robust control, parameter uncertainty, distributional robustness.
a
M
I. INTRODUCTION
3
1 Markov Decision Processes (MDPs) are widely used tools to model stochastic sequential decision mak-

ing problems (e.g., [2]–[4]). A strategy that achieves maximal expected accumulated reward is considered
]
Y optimal. However, in practice, the transition probabilities and reward parameters are typically estimated
S from finite and possibly noisy data, which often deviate from their true values. Such deviation, called
.
s “parameter uncertainty”, can cause the performance of the optimal policies to degrade significantly (see
c
[ experiments in [5]).

Inspired by the “robust optimization” framework in mathematical programming (e.g., [6]–[9]), many
2
v efforts have been made to alleviate the effect of parameter uncertainty in MDPs (e.g., [1], [10]–[15]).
8 Most previous study (e.g., [10], [11], [14]–[16]) focuses on the “robust MDP” which treats the uncertain
1
parameter as a fixed yet unknown element of a given “uncertainty set”, and aims to find the strategy that
4
7 achieves best performance under the worst parameter. This set-inclusive formulation of uncertainty can
0
be conservative as it cannot incorporate probabilistic information of the uncertainty that is often available
.
1 in practice (e.g., [12], [17]). To overcome this, [1] proposed the distributionally robust MDP approach,
0
which can incorporates certain kind of probabilistic information of the uncertainty. More specifically, this
5
1 approach treats the uncertain parameters as a random variable following an unknown distribution, while
:
v the distribution is known to belong to a set of distributions, called the “ambiguity set”, and the goal is to
Xi seek a strategy that archives the maximum expected performance under the most adversarial distribution
of the uncertain parameters. Indeed, this approach is the multi-stage counter-part of the distributionally
r
a
robust optimization (e.g., [1], [18]) which considers the following: Given a utility function u(x,ξ) where
x ∈ X is the optimizing variable and ξ is the unknown parameter, distributionally robust optimization
solves max [inf E u(x,ξ)], where C is a-priori known set of distributions.
x∈X µ∈C ξ∼µ
We highlight our contributions by comparing with [1]. In [1], the state-wise ambiguity set is restricted
to the following form: C˜ = {µ |µ (Oi) ≥ αi ∀i = 1,...,n }, where αi ≤ αj and Oi is a proper set
s s s s s s s s s
of uncertain parameters with a “nested-set” structure, i.e., satisfying Oi ⊆ Oj, for all i < j (see Fig.
s s
1a). This setup can effectively model distributions with a single mode (such as a Gaussian distribution),
but less so when modeling multi-mode distributions such as a mixture Gaussian distribution. Moreover,
other probabilistic information such as mean, variance etc can not be incorporated. Thus, in this paper, we
This work is partially supported by the Ministry of Education of Singapore through AcRF Tier Two grant R-265-000-443-112.
P. Yu is with the Department of Mechanical Engineering, National University of Singapore, 9 Engineering Drive 1, Singapore 117575,
Singapore (e-mail: [email protected]).
H. Xu is with the Department of Mechanical Engineering, National University of Singapore, 9 Engineering Drive 1, Singapore 117575,
Singapore (e-mail: [email protected], Tel: [phone]).
2
extend the distributionally robust MDP approach to handle ambiguity sets with more general structures.
In particular, we consider a class of ambiguity sets first proposed in [18] as a unifying framework
for modeling and solving distributionally robust single-stage optimization problems, and embed them
into distributionally robust MDPs setup. These ambiguity sets are considerably more general: they are
characterized by a class of Oi which can either be nested or disjoint (as shown in Fig. 1b), and moreover,
s
additional linear constraints are allowed to define the ambiguity set, which can be used to incorporate
probabilistic information such as mean, covariance or other variation measures. We show that, under this
more general class of ambiguity set, the resulting distributionally robust MDPs remain tractable under
mild technical conditions, and often outperforms previous methods partly due to the fact that it can model
uncertainty in a more flexible way.
The rest of the paper is organized as follows. Section II provides some background of uncertain MDPs
and presents our problem setup and necessary assumptions. Finite horizon and discounted reward infinite
horizon distributionally ambiguous MDPs are discussed in Section III and Section IV, respectively. We are
particularly interested in the solution approach, i.e., how to solve the distributionally ambiguous MDPs.
We present simulation results on a machine replacement problem and a path planning problem, both with
parameter uncertainty in Section V, which shows that the proposed approach outperforms the nominal,
the robust, and the distributionally robust approach proposed by [1]. We conclude the paper in Section
VI. As this work generalizes [1], we reuse some of its results, which will be indicated when we encounter
in the corresponding sections.
II. PRELIMINARIES
Throughout the paper, we use capital letters to denote matrices, and bold face letters to denote column
vectors. Row vectors are represented as the transpose of column vectors. We use 1 to denote the indicator
function, and use e (m) to denote the ith elementary vector of length m. Rn denotes the set of nonnegative
i +
realn-vectors,andtr(A)denotesthetraceofsquarematrixA.IfC isthesetofjointprobabilitydistribution
(cid:81)
of three random vectors a,b and c, then C denotes the set of marginal distribution of (a,b). We
(a,b)
use ⊕ to represent mixture distribution: given two probability distribution F , F and a Bernoulli random
1 2
variable x which takes value 1 w.p. p, xF ⊕(1−x)F is a random variable such that it follows distribution
1 2
F w.p. p, and follows F w.p. 1−p. We use N(m,σ2) to represent a Gaussian distribution with mean
1 2
m and variance σ2.
A (finite) Markov Decision Process (MDP) is defined as a 6-tuple (cid:104)T,γ,S,A,p,r(cid:105). Here, T is the
possibly infinite decision horizon; γ ∈ (0,1] is the discount factor; S is the state set and A is the
s
action set of state s ∈ S, both assumed to be finite. The parameter p and r are the transition probability
and the expected reward, respectively. That is, for s ∈ S and a ∈ A ,r(s,a) is the expected reward
s
and p(s(cid:48)|s,a) is the probability that the next state is s(cid:48). Following [2], we denote the set of all history-
dependent randomized strategies by ΠHR, and the set of all Markovian randomized strategies by ΠMR.
We use subscript s to denote the value associated with the state s, e.g., r denotes the vector form of
s
the rewards associated with the state s, and π is the (randomized) action chosen at state s for strategy
s
π. The elements in the vector p are listed in the following way: the transition probabilities of the same
s
action are arranged in the same block, and inside each block they are listed according to the order of the
next state. We use s to denote the (random) state following s, and ∆(s) to denote the probability simplex
(cid:78) (cid:78)
on A . We use to represent Cartesian product, e.g., p = p .
s s∈S s
For a given strategy π ∈ ΠHR, we denote the expected (discounted) total-reward under parameters pair
(p,r) as
(cid:40) (cid:41)
T
(cid:88)
u(π,p,r) (cid:44) E(p,r) γi−1r(s ,a ) .
π i i
i=1
˜
ADistributionallyAmbiguousMDP(DAMDP)isdefinedasatuple(cid:104)T,γ,S,A,C (cid:105),wherethetransition
S
probability p and the expected reward r are unknown. Instead, they are assumed to obey a joint distribution
3
µ (also unknown) that belongs to a known ambiguity set C (cid:44) (cid:81) C˜ , where (cid:81) means taking the
0 S (p,r) S (p,r)
marginal distribution of (p,r).
˜
While the DAMDP framework can be very general, most C results in formulations computationally
S
˜
intractable (e.g., [1], [19]). Hence, our first requirement of C is that the parameters among different states
S
are independent. That is,
(cid:79)
C˜ (cid:44) {µ|µ = µ ,µ ∈ C˜,∀s ∈ S},
S s s s
s∈S
˜
where “state-wise ambiguity set” C is a set of distributions of parameters of state s. By the definition of
s
C , the state-wise property applies to C as well. This property is same as the concept of “s-rectangularity”
S S
in[16],andisessentialforreducingDAMDPtorobustMDPinLemma1.Inaddition,[20]showedthatthe
robust MDP with coupled uncertainty sets is computationally challenging, which implies solving DAMDP
with nonrectangular ambiguity sets is even harder.
We now discuss the admissible state-wise ambiguity set. Our formulation of the state-wise ambiguity
set follows the unifying framework of [18]. In specific, given s ∈ S, the state-wise ambiguity set is
representable with the following standard form
   
p
 s (cid:12)E [F p +G r +H u˜ ] = c ,
C˜s (cid:44) µsrs(cid:12)(cid:12)(cid:12)µ(p(cid:0)sO,rsi,(cid:1)u˜s∈)∼[µαsi,αs i]s, ∀is∈s I s s s . (1)
 u˜ s s s s s 
s
Here, F ∈ Rk×(|As|×|s|),G ∈ Rk×|As|,H ∈ Rk×Q,c ∈ Rk; I = {1,2,...,n } is an index set and
s s s s s s
Oi ⊆ R|As|×|s|×R|As|×RQ is a set of possible values of the parameters (p ,r ,u˜ ), termed “confidence
s s s s
set”; and αi,αi ∈ [0,1], αi ≤ αi for all i ∈ I , is the lower bound and upper bound of the probability
s s s s s
that the parameters belong to the confidence set. Thus, each confidence set Oi provides an estimation
s
of the uncertain parameters pair (p ,r ,u˜ ) subject to a different confidence level. Ambiguity sets C˜
s s s s
contain prescribed conic representable confidence sets and mean values residing on an affine manifold,
which turns out to be rich enough to encompass and extend several ambiguity sets considered in the recent
literature. We abuse the notation here to use C˜ to denote the set of joint distributions of (p ,r ,u˜ ), and
s s s s
hence the set of joint distribution of (p ,r ) is C (cid:44) (cid:81) C˜.
s s s (ps,rs) s
Notice that here we use a classical technique called “lifting”: we introduce an auxiliary random vector u˜,
so that some non-linear relationship can be modeled linearly. For example, as we see below, a constraint on
the variance can be modeled using this standard form, which is otherwise impossible if auxiliary variable
is not introduced. This lifting technique thus allows us to model a rich variety of structural information
about the marginal distribution of (p,r) in a unified manner.
Note that, when n = 1 and the distribution set involving solo probability bound is of the form
s
C˜ = {µ (p ,r ,u˜ )|µ (O1) = 1,∀i ∈ I ,s ∈ S} (i.e., the distribution set only contains the support of
s s s s s s s s
random variables), the DAMDP reduces to the classical robust MDP formulation ( [10], [11]), where the
a-priori information of unknown parameters is that they belong to an uncertainty set.
Assumption 1 to 3 are standard requirements for the confidence sets, proposed in [18]. The first one
asserts the relationship between different confidence sets.
Assumption 1 (Nesting condition): For any s ∈ S, all i,i(cid:48) ∈ I and i (cid:54)= i(cid:48), we have either Oi (cid:98) Oi(cid:48),
s s s
Oi(cid:48) (cid:98) Oi or Oi ∩Oi(cid:48) = ∅.
s s s s
Here Oi (cid:98) Oi(cid:48) means that a set Oi is strictly included in a set Oi(cid:48), i.e., Oi is contained in the interior
s s s s s
of Oi(cid:48). The nesting condition is illustrated in Fig. 1b. The nesting condition implies a strict partial order
s
on the confidence sets Oi with respect to the (cid:98)-relation, with additional requirement that incomparable
s
sets must be disjoint. Nested condition is a fundamental assumption needed for the tractability of the
˜
distributionally robust optimization problems (cf. [18]). We remark here that when C is of form (1), it
s
trivially satisfies the nesting condition. Nesting condition can be verified analytically. Interested readers
may refer to [18] Section 3.
4
(a) (b)
Fig. 1. Illustration of the confidence sets.
We highlight the contributions of the paper by comparing with [1] who also propose a distributionally
robust MDP approach. In [1], the state-wise ambiguity set is restricted to the following form:
C˜ = {µ |µ (cid:0)Oi(cid:1) ≥ αi ∀i = 1,...,n };
s s s s s s
where αi ≤ αj, Oi ⊆ Oj, ∀i < j.
s s s s
Observe that our formulation is more general as we allow additional linear constraints (in terms of
expectation). This allows us to apply more statistical methods to estimate the uncertain parameters in
MDPs. Moreover, their formulation requires a more restrictive nesting set condition where the confidence
sets must have an incremental structure O1 ⊆ O2 ⊆ ··· ⊆ Ons, which is illustrated in Fig. 1a. While, as
s s s
shown in Fig. 1b, the confidence sets we use can be disjoint and thus provide significantly more powerful
modeling ability. This new structure is obviously more flexible, and can model more general distributions.
For example, it can better characterize multimodal distributions such as mixed Gaussian distributions: We
may establish confidence sets centered around corresponding peaks of different Gaussian components, as
long as they do not intersect with each other.
˜
Next, for any s ∈ S we require that C satisfy the following regularity condition.
s
Assumption 2 (Regularity conditions for C˜): 1) The confidence set Ons is bounded and has prob-
s s
ability one, that is, αns = αns = 1.
s s
2) There is a probability distribution µ (p ,r ,u˜ ) ∈ C˜ such that µ (Oi) ∈ (αi,αi) whenever
s s s s s s s s s
αi < αi, i ∈ I .
s s s
The condition 1 of Assumption 2 ensures that the confidence set with largest index, Ons, contains the
s
support of the joint unknown parameters pair (p ,r ,u˜ ). The second condition stipulates that there is a
s s s
probability distribution µ (p ,r ,u˜ ) ∈ C˜ that satisfies the probability bounds in (1) as strict inequalities
s s s s s
whenever the corresponding probability interval [αi,αi] is non-degenerate. This assumption allows the
s s
strong duality to hold for distributionally robust counterpart, which we will define later.
For each individual Oi, we make the following assumption for tractability.
s
Assumption 3: For any s ∈ S and i ∈ I , Oi is nonempty and convex. Each confidence set Oi is
s s s
defined as
  
p
 s (cid:12) 
 
(cid:12)
Osi = rs ∈ R|As|×|s| ×R|As| ×RQ (cid:12)(cid:12)Bsips +Dsirs +Esiu˜s (cid:22)Ksi bis ,
 
 u˜ 
s
where Bi ∈ RLi×(|As|×|s|), Di ∈ RLi×|As|, Ei ∈ RLi×Q, bi ∈ RLi, Ki are proper cones (i.e., a closed,
s s s s s
convex and pointed cone with nonempty interior).
Similarly to [10], we define the non-stationary model as: when a state is visited multiple times, each
time it can take a different parameter realization. We define the stationary model as: uncertain parameters
are chosen by nature depending on the decision maker’s strategy once for all, and remain fixed thereafter.
Each model leads to a game between the decision maker and nature, where the decision maker seeks to
5
maximize the minimum expected reward, with the nature being the minimizing player. The second model
is attractive for statistical reasons. Unfortunately, it turns out to be hard to solve. Therefore in this paper,
for finite horizon case, we assume it shares the same property of non-stationary model: multiple visits to
a state can be treated as visiting different states. This can easily be done by introducing dummy states.
For finite horizon case, as [1], we make the following assumption, which will simplify our derivations.
Assumption 4: 1) Each state belongs to only one stage.
2) The terminal reward equals zero.
3) The first stage only contains one state sini.
Using the condition 1 of Assumption 4, we partition S according to the stage each state belongs to.
That is, we let S be the set of states belong to tth stage.
t
III. FINITE HORIZON DISTRIBUTIONALLY ROBUST MDPS
This section focuses on uncertain MDPs with a finite number of decision stages. We show that a strategy
defined through backward induction, which we called S-robust strategy, is a distributionally robust strategy.
We further show that such strategy is solvable in polynomial time under mild technical conditions. This
thus generalizes similar results in [1] to a significant more general class of ambiguity sets.
For π ∈ ΠHR and µ ∈ C , we denote the expected performance of a DAMDP as
S
(cid:90)
w(π,µ,(sini)) (cid:44) E {u(π,p,r)} = u(π,p,r)dµ(p,r).
(p,r)∼µ
Definition 1: A strategy π∗ ∈ ΠHR is distributionally robust with respect to C if it satisfies that for
S
all π ∈ ΠHR,
inf w(π,µ,(sini)) ≤ inf w(π∗,µ(cid:48),(sini)).
µ∈CS µ(cid:48)∈CS
In words, each strategy is evaluated by its expected performance under the (respective) most adversarial
distribution of the uncertain parameters, and a distributionally robust strategy is the optimal strategy
according to this measure.
The main focus of this section is deriving approaches to solve the distributionally robust strategy. To
this end, we need the following definition.
˜
Definition 2: Given a DAMDP (cid:104)T,γ,S,A,C (cid:105) with T < ∞, we define the S-robust problem as
S
following
1) For s ∈ S , the S-robust value v˜ (s) (cid:44) 0.
T T
2) For s ∈ S , where t < T, the S-robust value v˜(s) and S-robust action π˜ are defined as
t t s
v˜(s) (cid:44) max {min E {E(ps,rs)[r(s,a)+γv˜ (s)]}},
t πs∈∆(s) µs∈Cs (ps,rs)∼µs πs t+1 (2)
π˜ ∈ arg max {min E {E(ps,rs)[r(s,a)+γv˜ (s)]}}.
s πs∈∆(s) µs∈Cs (ps,rs)∼µs πs t+1
3) A strategy π˜∗ is a S-robust strategy if ∀s ∈ S, and every history h ends at s, we have π˜∗, conditioned
s
on history h, is a S-robust action.
Note that the definition essentially requires that the strategy must be robust with respect to each sub-
problem, and hence the name “S-robust”. The following theorem shows any S-robust strategy π∗ is
distributionally robust, and is the main result of this paper.
Theorem 1: Let T < ∞. Under Assumptions 1, 3 and 4, if π∗ is a S-robust strategy, then
1) π∗ is distributionally robust strategy with respect to C .
S
2) There exists µ∗ ∈ C such that (π∗,µ∗) is a saddle point. That is,
s
sup w(π,µ∗,(sini)) = w(π∗,µ∗,(sini)) = inf w(π∗,µ,(sini)).
π∈ΠHR µ∈CS
6
Proof: The proof follows a similar structure as that of Theorem 3.1 in [1]: We first state a lemma
from [1], which shows that for a given strategy, the expected performance under admissible µ depends
only on the expected value of the parameters. Thus we are able to reduce the distributionally robust MDPs
to the classical robust MDPs. Then we show that the set of expected value of the uncertain parameters is
convex and compact. Finally, by applying the results of classical robust MDPs we prove the theorem.
ThefollowingLemmaisindeedLemma3.2of[1].Hencewestatetheresultwithouttheproof.Interested
readers may refer to [1] for the proof.
Lemma 1: Under the state-wise property of the ambiguity set C˜ , fix π ∈ ΠHR and µ ∈ C , denote
S S
p = E (p) and r = E (r). We have
µ µ
w(π,µ,(sini)) = u(π,p,r).
Lemma 1 essentially means that for any strategy, the expected performance under an admissible
distribution µ only depends on the expected value of the parameters under µ. Thus, the distributionally
robustMDPsreducestotherobustMDPs.Nextwecharacterizethesetofexpectedvalueoftheparameters.
Lemma 2: For s ∈ S and π ∈ ∆(s), we define the set
s
Z = {E (p ,r )|µ ∈ C }.
s µs s s s s
Then set Z is convex and compact.
s
Proof:First,weshowthat,fors ∈ S andπ ∈ ∆(s),thesetdefinedasZ˜ = {E (p ,r ,u˜ )|µ ∈ C˜}
s s µs s s s s s
is convex and compact.
Fix s ∈ S, and two distributions µ1,µ2 ∈ C˜, and λ ∈ [0,1], we have
s s s
λE (p ,r ,u˜ )+(1−λ)E (p ,r ,u˜ ) = E (p ,r ,u˜ ),
µ1 s s s µ2 s s s λµ1+(1−λ)µ2 s s s
s s s s
which holds due to the linearity of the expectation operation. Next, we show λµ1+(1−λ)µ2 ∈ C˜. Since
s s s
µ1(p1,r1,u˜1),µ2(p2,r2,u˜2) ∈ C˜, for s ∈ S, we have,
s s s s s s s s s
(cid:90) (cid:90)
λ (F p1 +G r1 +H u˜1)dµ1(p1,r1,u˜1)+(1−λ) (F p2 +G r2 +H u˜2)dµ2(p2,r2,u˜2)
s s s s s s s s s s s s s s s s s s s s
On On
s s
= λc +(1−λ)c = c ,
s s s
(cid:90) (cid:90)
λ 1 dµ1(p1,r1,u˜1)+(1−λ) 1 dµ2(p2,r2,u˜2)
[(p1s,r1s,u˜1s)∈Osi] s s s s [(p2s,r2s,u˜2s)∈Osi] s s s s
On On
s s
≤ λαi +(1−λ)αi = αi, ∀i ∈ I ,
s s s s
(cid:90) (cid:90)
λ 1 dµ1(p1,r1,u˜1)+(1−λ) 1 dµ2(p2,r2,u˜2)
[(p1s,r1s,u˜1s)∈Osi] s s s s [(p2s,r2s,u˜2s)∈Osi] s s s s
On On
s s
≥ λαi +(1−λ)αi = αi, ∀i ∈ I .
s s s s
˜
Hence the convexity follows. To show that compactness, notice that C is weakly closed (i.e., closed
s
w.r.t. to the weak topology) since the feasible set of each of constraint is weakly closed and so does
˜ ˜
their intersection. Thus, Z is closed since it is the image of C under expectation (which is a continuous
s s
function). This implies Z˜ is compact since Ons is bounded and hence Z˜ is bounded.
s s s
˜
Finally, since Z is the projection onto the first two coordinates of set Z , its convexity and compactness
s s
are straightforward at this stage.
Lemma 2 implies that, for s ∈ S and π ∈ ∆(s), there exists (p∗,r∗) ∈ Z that satisfies
s s s s
inf u(π ,p ,r ) = u(π ,p∗,r∗).
s s s s s s
(ps,rs)∈Zs
Furthermore, we can construct (cid:78) Z = {E (p,r)|µ ∈ C } by state-wise decomposability of C .
s∈S s µ S S
We complete the proof of Theorem 1 by using the equivalence of distributionally robust MDPs and
(cid:78)
robust MDPs where the uncertainty set is Z . It is well known that for robust MDPs , saddle point
s∈S s
7
of the minimax objective exists ( [10], [11]). More precisely, there exist π∗ ∈ ΠHR,(p∗,r∗) ∈ (cid:78) Z ,
s∈S s
such that
inf u(π∗,p,r) = u(π∗,p∗,r∗) = sup u(π,p∗,r∗)
(p,r)∈(cid:78)s∈SZs π∈ΠHR
holds.Moreover,wecanconstructπ∗ and(p∗,r∗)state-wiseasπ∗ = (cid:78) π∗ and(p∗,r∗) = (cid:78) (p∗,r∗).
s∈S s s∈S s s
For each s ∈ S , π∗ and (p∗,r∗) solves the zero-sum game
t s s s
max min E(ps,rs)[r(s,a)+γv˜ (s)].
πs∈∆(s)(ps,rs)∈Zs πs t+1
Thus π∗ is any S-robust action, and hence π∗ can be any S-robust strategy. From Lemma 2, there exists
s
µ∗(p∗,r∗) ∈ C satisfying E (p ,r ) = (p∗,r∗). Let µ∗ = (cid:78) µ∗. By applying Lemma 1, we have
s s s s µ∗s s s s s s∈S s
sup w(π,µ∗,(sini)) = sup u(π,p∗,r∗),
π∈ΠHR π∈ΠHR
w(π∗,µ∗,(sini)) = u(π∗,p∗,r∗),
inf w(π,µ,(sini)) = inf u(π∗,p,r).
µ∈CS (p,r)∈(cid:78)s∈SZs
Above leads to
sup w(π,µ∗,(sini)) = w(π∗,µ∗,(sini))
π∈ΠHR
= inf w(π∗,µ,(sini)).
µ∈CS
Thus, part (2) of Theorem 1 holds. Part (1) follows immediately.
Define v˜ as the vector form of v˜ (s(cid:48)) for all s(cid:48) ∈ S , and
t+1 t+1 t+1
 (cid:124) 
v˜ e (|A |)
t+1 1 s
V˜ (cid:44)  ... .
s  
(cid:124)
v˜ e (|A |)
t+1 |As| s
Therefore, the expected reward under fixed parameter realization (p ,r ) and any policy π ∈ ∆(s) is
s s s
E(ps,rs)[r(s,a)+v˜ (s)]
πs t+1
(cid:88) (cid:88) (cid:88)
= π (a)r(s,a)+ π (a)p(s(cid:48)|s,a)v˜ (s(cid:48))
s s t+1
a∈As a∈Ass(cid:48)∈St+1
(cid:124) (cid:124)˜
=r π +p V π .
s s s s s
We now investigate the computational aspect of the S-robust action.
Theorem 2: For s ∈ S where t < T, the S-robust action is the optimal solution of the following
t
optimization problem.
minimize w
w,πs,β,κ,λ,νi
(cid:88)
subject to c(cid:124)β + [αiκ −αiλ ] ≤ w
s s i s i
i∈Is
ν(cid:124)bi − (cid:88) [κ −λ ] ≤ 0, i ∈ I
i s i(cid:48) i(cid:48) s
i(cid:48)∈A(i)
Bi(cid:124)ν +V˜ π +F(cid:124)β = 0, i ∈ I
s i s s s s
Di(cid:124)ν +π +G(cid:124)β = 0, i ∈ I
s i s s s
Ei(cid:124)ν +H(cid:124)β = 0, i ∈ I
s i s s
π ∈ ∆(s),
s
β ∈ Rk, κ,λ ∈ Rns, ν ∈ Ki∗.
+ i s
8
Here, Ki∗ represents the cone dual to Ki, and set A(i) (cid:44) {i}∪{i(cid:48) ∈ I : Oi (cid:98) Oi(cid:48)}.
s s s s s
Proof: First, for s ∈ S , since the expectation term in (2) is independent of the auxiliary random
t
vector u˜ , the S-robust problem can be written as
s
minimize w
w,πs
subject to max−E {E(ps,rs)[r(s,a)+γv˜ (s)]} ≤ w
µs∈C˜s (ps,rs)∼µs πs t+1
π ∈ ∆(s).
s
Now we consider the constraint, which we call distributionally robust counterpart:
max−E {E(ps,rs)[r(s,a)+γv˜ (s)]} ≤ w.
µs∈C˜s (ps,rs)∼µs πs t+1
The distributionally robust counterpart can be equivalently expressed as below
(cid:90) 
maxµismize (cid:90)− Osns E(πpss,rs)[r(s,a)+γv˜t+1(s)]dµs(ps,rs,u˜s)

subject to (F p +G r +H u˜ )dµ (p ,r ,u˜ ) = c 
s s s s s s s s s s s 
Osns ≤ w.
(cid:90)
1 dµ (p ,r ,u˜ ) ≥ αi,∀i ∈ I 
(cid:90)Osns [(ps,rs,u˜s)∈Osi] s s s s s s 

Osns 1[(ps,rs,u˜s)∈Osi]dµs(ps,rs,u˜s) ≤ αis,∀i ∈ Is 
Theorem 2 of [18] states that for the above maximization problem, the strong duality holds under
Assumption 1 and 2. The dual problem is of the following form:
(cid:88) 
minimize c(cid:124)sβ + [αisκi −αisλi] 
β,κ,λ 
i∈Is 
subject to [F p +G r +H u˜ ](cid:124)β + (cid:88) [κ −λ ] ≥ −E(ps,rs)[r(s,a)+γv˜ (s)],
s s s s s s i(cid:48) i(cid:48) πs t+1 ≤ w, (3)
i(cid:48)∈A(i) 

∀(ps,rs,u˜s) ∈ Osi,∀i ∈ Is

β ∈ Rk, κ,λ ∈ Rns 
+
where A(i) (cid:44) {i}∪{i(cid:48) ∈ I : Oi (cid:98) Oi(cid:48)}.
s s s
The ith semi-infinite constraint in the above problem can be written as, for each i ∈ I ,
s
(cid:88) 
maximize −{[Fsps +Gsrs +Hsu˜s](cid:124)β + [κi(cid:48) −λi(cid:48)]+r(cid:124)sπs +p(cid:124)sV˜sπs}
ps,rs,u˜s i(cid:48)∈A(i) ≤ 0.
subject to Bip +Dir +Eiu˜ (cid:22) bi. 
s s s s s s Ksi s
The above problem is a convex optimization problem. By applying the duality theory of convex
optimization [21], we get its dual below. For each i ∈ I ,
s
minimize ν(cid:124)bi − (cid:88) [κ −λ ] 
i s i(cid:48) i(cid:48) 
νi i(cid:48)∈A(i) 

subject to Bi(cid:124)ν +V˜ π +F(cid:124)β = 0
s i s s s
Di(cid:124)ν +π +G(cid:124)β = 0 ≤ 0.
s i s s 
Ei(cid:124)ν +H(cid:124)β = 0 
s i s 

ν ∈ Ki∗. 
i s
9
Thus, the ith semi-infinite constraint can be equivalently reformulated as follows: there exists ν ∈ Ki∗,
C(cid:124)ν +V˜ π +A(cid:124)β = 0, D(cid:124)ν +π +B(cid:124)β = 0 and Ei(cid:124)ν +H(cid:124)β = 0 such that ν(cid:124)bi−(cid:80) [κ −iλ ] ≤s0.
i i s s i i s s i s i s i(cid:48)∈A(i) i(cid:48) i(cid:48)
Finally, by combining constraints for all i ∈ I , we obtain the optimization formulation stated in the
s
theorem.
Thus, since for s ∈ S , ∆(s) is compact, we can solve the S-robust action in polynomial time if all Ki
t s
are “easy” cones such as linear, conic quadratic or semidefinite cones. Moreover, using Theorem 1, by
backward induction, we can obtain the S-robust strategy efficiently.
By virtue of the lifting technique [18, Thm 5], we show below that several widely used ambiguity sets
˜
are indeed special cases of C defined in (1). We further list their corresponding S-robust problems.
s
Example 1 (Mean): Assume that we only know a noisy empirical estimator of the exact mean of p .
s
That is, given G ∈ RM×(|As|×|s|), f ∈ RM and p ∼ µ (p ), GE [p ] (cid:22) f, where K is a proper
s s s ps∼µs(ps) s K
cone. [18] shows that C˜, which involves the auxiliary random vector u˜ ∈ RM, can be expressed as
s s
(cid:26) (cid:12)E [u˜ ] = f, (cid:27)
C˜ = µ (p ,u˜ )(cid:12)(cid:12) u˜s∼µs s .
s s s s
(cid:12)µ (Gp (cid:22) u˜ ) = 1
s s K s
(cid:81) ˜ (cid:81) ˜
Note that µ (p ) ∈ C , where C denotes the marginal distribution of p under the joint
s s ps s ps s s
probability distribution of two random vectors p and u˜ . The problem in Theorem 2 now takes the form
s s
minimize w
w,πs,κ,ν
(cid:124)
subject to κ+f ν ≤ w
(cid:124)
κ+r π ≥ 0
s s
˜ (cid:124)
V π +G ν = 0
s s
π ∈ ∆(s), ν ∈ K∗.
s
The same argument can be developed for mean of r . This example can also be treated via “classical”
s
robust optimization models by virtue of Lemma 1.
Example 2 (Variance): This example imposes conditions on both mean and covariance of the distribu-
tion. First, we assume that the mean of the random reward r ∼ µ (r ) is given by E [r ] = m,
s s s rs∼µs(rs) s
and E [(r −m)(r −m)(cid:124)] (cid:22) Σ for Σ ∈ S|As|. We denote the ambiguity set under fixed mean m
rs∼µs(rs) s s +
as C˜(m). As discussed in [18], C˜(m), which involves a random matrix U˜ ∈ R|As|×|As|, can be expressed
s s s
as
 
(cid:32) (cid:33) E [r ] = m, E [U˜ ] = Σ,
C˜(m) = µ rs (cid:12)(cid:12)(cid:12) rs(cid:18)∼(cid:20)µs s 1 (r −U˜s∼mµs)(cid:124)(cid:21)s (cid:19) .
s s U˜ (cid:12)µ s (cid:23) 0 = 1
 s s (r −m) U˜ 
s s
(cid:81) ˜ (cid:81) ˜
We have µ (r ) ∈ C (m), where C (m) denotes the marginal distribution of r under the joint
s s rs s rs s s
˜
probability distribution of random variables r and U . Next, we consider the case where given G ∈
s s
RM×|As|, f ∈ RM, the mean of random reward r is restricted by Gm ≤ f. In this case, the ambiguity
s
˜ (cid:83) ˜
set can be written as C = C (m). By Theorem 2, the S-robust action can be solved using the
s Gm≤f s
following formulation
minimize w
w,πs,κ,ν,γ,Y
(cid:124) (cid:124)
subject to κ+tr(ΣY )−f ν ≤ w
22
(cid:124)˜ (cid:124)
κ+p V π −Y −f γ ≥ 0
s s s 11
(cid:124)
π −Y −Y −G ν = 0
s 12 21
(cid:124)
Y +Y +G γ = 0
12 21
(cid:20) (cid:124)(cid:21)
Y Y
Y − 11 12 = 0
Y Y
21 22
π ∈ ∆(s), ν,γ ≥ 0, Y ∈ S|As|+1.
s +
10
The same argument can be developed for variance of p .
s
Example 3 (Mean Absolute Deviation): Assume that E [|r − m|] ≤ f for m,f ∈ R|As|. [18]
rs∼µs(rs) s
shows that C˜, which involves the auxiliary random vector u˜ ∈ R|As|, can be expressed as
s s
 
E [u˜ ] = f,
 (cid:12) u˜s∼µs s 
C˜s = µs(rs,u˜s)(cid:12)(cid:12) (cid:18)u˜s ≥ rs −m(cid:19) .
(cid:12)µ = 1
 s u˜ ≥ m−r 
s s
(cid:81) ˜ (cid:81) ˜
Note that µ (r ) ∈ C , where C denotes the marginal distribution of r under the joint
s s rs s rs s s
probability distribution of random variables r and u˜ . The problem in Theorem 2 in this case can
s s
be represented as
minimize w
w,πs,κ,ν
(cid:124)
subject to κ−f ν ≤ w
(cid:124)˜ (cid:124)
κ+p V π +m π ≥ 0
s s s s
π ∈ ∆(s), ν ≥ 0.
s
The corresponding S-robust problem can be obtained for mean absolute deviation of p .
s
Example 4 (Expected Huber Loss Function): For a scalar z ∈ R the Huber Loss function is defined as
(cid:26)
1z2 if |z| ≤ δ,
H (z) = 2
δ δ(|z|− 1δ) otherwise,
2
where δ > 0 is a prescribed robustness parameter.
Assume that E [H (f(cid:124)r )] ≤ g for f ∈ R|As| and g ∈ R . [18] shows that C˜, which involves
rs∼µs(rs) δ s + s
the auxiliary random variables u˜ ,v˜ ,w˜ ,s˜ ,t˜ ∈ R , can be expressed as
s s s s s +
   E [w˜ ] = g, 
 r ,u˜ , w˜s∼µs s 
C˜s = µssv˜˜sss,,tw˜˜sss,(cid:12)(cid:12)(cid:12)(cid:12)µsδu˜(u˜≥s −s˜s˜,s)+v˜ s˜≥22s +t˜,δ(v˜fs(cid:124)−r t˜=s)u+˜ t−˜22s v˜≤ w˜s, = 1.
s s s s s s s
(cid:81) ˜ (cid:81) ˜
Note that µ (r ) ∈ C , where C denotes the marginal distribution of r under the joint
s s rs s rs s s
probability distribution of six random variables. The S-robust problem in Theorem 2 is
minimize w
w,πs,κ,ν,γ,η,θ
subject to κ−gν ≤ w
3γ2 3η2
κ+p(cid:124)V˜ π +3δγ +3δη −3δ2ν − − ≥ 0
s s s 2ν 2ν
π +θf = 0
s
γ −δν −θ = 0
η −δν +θ = 0
π ∈ ∆(s), ν,γ,η ≥ 0.
s
The corresponding S-robust problem for the case of transition probability can be obtained in a similar
way.
Before concluding this section, we briefly discuss the computational complexity. We denote M as the
maximal computational effort in calculating possibly randomized S-robust action π for each s ∈ S . Thus,
s t
the total time-complexity is O(T|S|M).
Remark 1 (Stationary & non-stationary model): The S-robust value we derived in this section is for
non-stationary model. It provides a lower bound for stationary model, which is generally hard to solve.
Thus, one can use the non-stationary model to approximate the stationary model, when the latter is
intractable.