Update or Wait: How to Keep Your Data Fresh

0.34 MB·

by  Yin Sun

1
Update or Wait: How to Keep Your Data Fresh
Yin Sun†, Elif Uysal-Biyikoglu‡, Roy D. Yates⋆, C. Emre Koksal†, and Ness B. Shroff†§
†Dept. of ECE, §Dept. of CSE, The Ohio State University, Columbus, OH
‡Dept. of EEE, Middle East Technical University, Ankara, Turkey
⋆Dept. of ECE, Rutgers University, North Brunswick, NJ
Abstract—In thiswork westudyhow to manage thefreshness idle/busy
6 of status updates sent from a source to a destination via a
1 communication server. A proper metric of data freshness at the
0 destination is the age-of-information, which is defined as how
2 old the freshest received updateis since themoment thisupdate
was generated at the source. A logical policy is the zero-wait
p Source Destination
policy, i.e., the source submits a fresh update once the server is
e
S free,whichachievesthemaximumthroughputandtheminimum Server
averagedelay.Surprisingly,thiszero-waitpolicydoesnotalways

6 minimize the average age. This motivates us to study how to
2 optimally control the status updates to keep data fresh and to Fig.1. Systemmodel.
understandwhenthezero-waitpolicyisoptimal.Weintroducea
] penaltyfunctiontocharacterizethelevelofdissatisfactionondata Most existing research on the age-of-information focuses
T staleness and formulate the average age penalty minimization on an “enqueue-and-forward” model [4]–[10], where status
I problemasaconstrainedsemi-Markovdecisionprocess(SMDP) updates are randomly generated or arrive at a source node.
.
s with an uncountable state space. Despite the difficulty of this Thesourceenqueuestheseupdatesandforwardsthemlaterto
c problem,wedevelopefficientalgorithmstofindtheoptimalstatus
[ update policy and provide sufficient and necessary conditions aremotedestinationthroughanetwork.Itisworthnotingthat
to characterize the optimality of the zero-wait policy. We show the goal of age minimizationdiffersfrom those of throughput
8
that the zero-wait policy can be far from the optimum if (i) maximization and delay minimization: A high update fre-
v
the penalty function grows quickly with respect to the age, or quency increases the system throughput, but may also induce
4
(ii) the update service times are positively correlated and follow
8 a large waiting time in the queue which in turn increases the
a heavy-tail distribution. To the best of our knowledge, this is
2 age;ontheotherhand,alowupdatefrequencycanreducethe
the first optimal control policy which is proven to minimize the
2 age-of-information in status update systems. queueing delay, but the destination may end up having stale
0
status informationdue to notenoughupdates[4], [8], [10]. In
.
1 [4], [9], it was found that a good policy is to discard the old
0 I. INTRODUCTION
packets waiting in the queue if a new sample arrives, which
6 In recent years, the proliferation of mobile devices and
can greatly reduce the impact of queueing delay.
1
applications has significantly boosted the need for real-time
: In this paper, we study a “generate-at-will”model depicted
v informationupdates,suchasnews,weatherreports,emailnoti-
in Fig. 1. In this model, the source monitors the server’s
i fications,stockquotes,socialupdates,mobileads,etc.Timely
X idle/busy state, and in contrast to [4]–[10], is able to generate
status updates are also critical in network-based monitoring
r status updates at any time by its own will. Hence, no updates
a and control systems, including sensor networks used in tem- need to be generated when the server is busy, which com-
peratureandairpollutionmonitoring[1],surroundmonitoring
pletely eliminates the waiting time in the queue and hence
in autonomousvehicles[2], and phasordataupdatesin power
the queue in Fig. 1 is always empty. A simple zero-wait
grid stabilization systems [3].
policy,also knownas the work-conservingpolicy in queueing
A common need in these applications is to maximize the
theory, that submits a fresh update once the server becomes
freshnessofthedataatthedestinationofinformationupdates.
idle, achieves the maximum throughput and the minimum
In light of this, a metric called the age-of-information, or
average delay. Surprisingly, this zero-wait policy does not
simplyage,wasdefinedin[4].Attimet,ifthefreshestupdate alwaysminimizetheaverageageof theinformation[11]. The
at the destination has a time stamp U(t), the age is followingexamplerevealsthereasonbehindthisphenomenon:
Example: Suppose that the source submits a stream of
∆(t)=t−U(t). (1)
update packets to a remote destination. The service times of
Hence, the age is the time elapsed since the freshest received these updates form a periodic sequence
packet was generated.
0,0,2,2,0,0,2,2,0,0,2,2,...
Thispaperwaspresented inpartatIEEEINFOCOM2016.
ThisworkwassupportedinpartbyDTRAgrantHDTRA1-14-1-0058,NSF Supposethatupdate1isgeneratedandsubmittedattime0and
grants CNS-1446582, CNS-1409336, CNS-1518829, CNS-1514260, CNS- deliveredattime0.Underthezero-waitpolicy,update2isalso
1422988,andCNS-1054738,AROgrantW911NF-14-1-0368,andONRgrant
generatedat time 0 and delivered at time 0. However, despite
N00014-15-1-2166. E.Uysal-Biyikoglu wassupported inpartbyTUBITAK
andinpartbyaScience AcademyBAGEPaward. its negligible service time, update 2 has not brought any new
2
∆(t) Markov chain with an uncountable state space, which
generalizesthei.i.d.serviceprocessesstudiedinprevious
work [4]–[12].
• We formulate the average age penalty minimization
2
t problem as a constrained semi-Markov decision process
(cid:127)
(SMDP) with an uncountable state space. Despite the
0 ǫ 2ǫ 2+2ǫ 4+2ǫ
difficulty of this problem, we manage to solve it by a
divide-and-conquer approach: We first prove that there
Fig.2. Evolutionoftheage∆(t)intheǫ-waitpolicyintheexample.
exists a stationary randomized policy that is optimal for
information to the destination after update 1 was delivered, this problem (Theorem 1). Further, we prove that there
because both updates were sampled at time 0. Therefore, exists a stationary deterministic policy that is optimal
the potential benefit of the zero service time of update 2 is for this problem (Theorem 2). Finally, we develop a
wasted! This issue occurs periodically over time: Whenever low-complexity algorithm to find the optimal stationary
two consecutive updates have zero service time, the second deterministic policy (Theorem 3). To the best of our
update of the two is wasted. Therefore, a 1/4 of the updates knowledge, this is the first optimal control policy which
in this sequence are wasted in the zero-wait policy! is proven to minimize the age-of-information(i.e., max-
imize data freshness) in status update systems.
For comparison, consider the ǫ-wait policy that waits for ǫ
seconds after each update with a zero service time, and does • We further investigate when the zero-wait policy is opti-
not wait after each update with a service time of 2 seconds. mal.Forthespecialcaseofproportionalpenaltyfunction
and i.i.d. service times, we devise a simpler solution to
Thetime-evolutionoftheage∆(t)inthenon-zero-waitpolicy
isshowninFig.2.Update1isgeneratedanddeliveredattime minimize the average age (Theorem 4), and obtain a
0. Update 2 is generated and delivered at time ǫ. Update 3 is sufficient and necessary condition to characterize when
the zero-wait policy is optimal (Theorem 5). For general
generated at time 2ǫ and is delivered at time 2+2ǫ. Because
the service time of update 3 is 2 seconds, the latest delivered age penalty functions and correlated service processes,
severalsufficientconditionsfortheoptimalityofthezero-
update at time 2+2ǫ is of the age 2 seconds. Hence, the age
∆(t)dropsto2secondsattime 2+2ǫ.Update4isgenerated wait policy are provided (Lemma 3).
at time 2+2ǫ and is delivered at time 4+2ǫ. At time 4+2ǫ, • Our theoretical and simulation results demonstrate that,
in many scenarios, the optimal policy is to wait for a
the age drops to zero because update 5 is generated at this
time and is delivered immediately. certainamountoftimebeforesubmittinganewupdate.In
particular,thezero-waitpolicycanbefarfromoptimality
It can be shown that the time-average age of the ǫ-wait
if (i) the penalty function grows quickly with respect to
policy is
the age, or (ii) the update service times are positively
(ǫ2/2+ǫ2/2+2ǫ+42/2)/(4+2ǫ) correlated and follow a heavy-tail distribution.
=(ǫ2+2ǫ+8)/(4+2ǫ) seconds.
II. MODELAND FORMULATION
If the waiting time is ǫ = 0.5, the time-average age of the
ǫ-wait policy is 1.85 seconds. If the waiting time is ǫ = 0, it A. System Model
reduces to the zero-wait policy, whose time-average age is 2
We consider a status update system depicted in Fig. 1,
seconds. Hence, the zero-wait policy is not optimal!
where a source generates update packets and sends them to
This example point out a key difference between data
a remote destination through a communication server. This
communicationsystemsandstatusupdatesystems:Incommu-
serverprovidesFirst-ComeFirst-Served(FCFS)servicetothe
nication systems, all packets are equally important; however,
submitted update packets. The service of an update packet
in status update systems, an update packet is useful only if it
is considered complete, when it is successfully received by
carries some fresh information to the destination. While the
the destination. After that, the server becomes available for
theory of data communications is quite mature, the optimal
sending the next packet.
control of status updates remains open.
The source generates and submits updates at successive
For a source that can insert waiting times between updates, times S ,S ,... Update i, submitted at time S , is delivered
0 1 i
theaimofthispaperistoanswerthefollowingquestions:How at time D = S + Y , where Y ≥ 0 is the service time
i i i i
tooptimallysubmitupdatepacketstomaximizedatafreshness
of update i. Suppose that update 0 is submitted to an idle
at the destination? When is the zero-wait policy optimal? To server at time S = −Y and delivered at D = 0, as shown
0 0 0
thatend,the followingarethe keycontributionsof thispaper: in Fig. 3. The source has access to the idle/busy state of
• We generalize existing status update studies by introduc- the server and is able to generate updates at any time by
ingtwonewfeatures:agepenaltyfunctionsandcorrelated its own will. Hence, the source should not generate update
(non-i.i.d.) service processes. We define an age penalty i+1 when the server is busy processing update i, because
function g(∆) to characterize the application-specific thiswillincuranunnecessarywaitingtimeinthequeue.After
level of dissatisfaction for data staleness, where g(·) update i is delivered at time D , the source may introduce a
i
is measurable, non-negative, and non-decreasing. The waiting time Z ∈ [0,M] before submitting update i+1 at
i
updateserviceprocessismodeledasa stationaryergodic time S = D +Z , where M is defined as a maximum
i+1 i i
3
∆(t) g (∆(t))=e0.2∆(t)−1
1
Q0 Q1 Qn−1
Y0 Y2 Yn
D0=0S1D1 S2 D2 Sn−1 Dn−1 SnDn t 0 S1D1 S2 D2 Sn−1 Dn−1 SnDn t
Z0Y1 Z1 Y2 Yn−1 Zn−1 Yn Z0 Y1 Z1 Y2 Yn−1 Zn−1 Yn
(a) Anexponential agepenalty function g1(∆)=e0.2∆−1.
Fig.3. Evolutionoftheage-of-information ∆(t).
g (∆(t))=⌊∆(t)⌋
2
waiting time allowed by the system. The sourcecan switch to
a low-power sleep mode during [Di,Si+1). We assume that Q0 Q1 Qn−1
the service process (Y ,Y ,...) is a stationary and ergodic
0 1
Markov chain with a possibly uncountable state space and
a positive mean 0 < E[Y ] < ∞, which generalizes the
i
i.i.d. service processes in previous studies [4]–[12].1 This
Markovian service process model is introduced to study the 0 S1D1 S2 D2 Sn−1 Dn−1 SnDn t
impact of temporal-correlation on the optimality of the zero-
waitpolicy.InSectionIV,wewillseethatthezero-waitpolicy Z0 Y1 Z1 Y2 Yn−1 Zn−1 Yn
isclosetotheoptimumwhentheserviceprocessisnegatively
(b) Astair-stepagepenaltyfunction g2(∆)=⌊∆⌋.
correlated;but can be far from the optimum when the service
process is positively correlated. Fig.4. Evolutions oftwoagepenalty functions.
At any time t, the most recently received update packet is Consider the time interval [0,D ], where D = n−1(Z +
time-stamped with n n i=0 i
Y ). In this interval, the area under g(∆(t)) can be seen as
i+1
P
U(t)=max{Si :Di ≤t}. (2) the concatenation of the areas Qi, 0≤i≤n−1, such that
The age-of-information ∆(t) is defined as [4] Dn n−1
g(∆(t))dt = Q ,
i
∆(t)=t−U(t), (3) Z0 i=0
X
where
which is also referred to as age. As shown in Fig. 3, the
age ∆(t) is a stochastic process that increases linearly with Di+Zi+Yi+1 Yi+Zi+Yi+1
Q = g(∆(t))dt= g(τ)dτ. (4)
i
t between updates, with downward jumps occurring when
ZDi ZYi
updates are delivered. Specifically, when update i is sent at
In the second equation of (4), we have used the fact that
time t = S , it is delivered at time D = S + Y with
i i i i ∆(t)=t−S =t−(D −Y ) for t∈[D ,D +Z +Y ].
i i i i i i i+1
age ∆(D ) = D − S = Y . After that, the age increases
i i i i Let us define
linearly and reaches ∆(D− ) = Y +Z +Y just before
i+1 i i i+1 y+z+y′
update i+1 is delivered. Then, at time Di+1, the age drops q(y,z,y′)= g(τ)dτ. (5)
to ∆(D )=Y .
i+1 i+1 Zy
Then, Q can be expressed as Q = q(Y ,Z ,Y ). Since
i i i i i+1
B. Problem Formulation g(∆)isnon-negative,thefunctionq(y,z,y′)isnon-decreasing
We introduce an age penalty function g(∆) to represent in z. We assume
the level of dissatisfaction for data staleness or the need for E[q(Y ,M,Y )]<∞, (6)
i i+1
new information update. The function g :[0,∞)→[0,∞) is
assumedtobemeasurable,non-negative,andnon-decreasing. implying E[q(Y ,Z ,Y )]<∞ for all Z ∈[0,M].
i i i+1 i
Someexamplesofg(·)arethepowerfunctiong(∆)=∆a,the Our goal is to minimize the average age penalty by con-
exponential function g(∆) = ea∆, and the stair-step function trolling the sequence of waiting times (Z ,Z ,...). Let π ,
0 1
g(∆) = ⌊a∆⌋, where a ≥ 0 and ⌊x⌋ is the largest integer (Z ,Z ,...) denote a status update policy. We consider the
0 1
no greater than x. Two age penalty functions are depicted in class of causally feasible policies, in which control decisions
Figure 4. Note thatthis age penalty modelis quite general,as are made based on history and current information of the
it allows g(·) to be discontinuous and non-convex. system, as well as the distribution of the service process
To analyzethe averageage penalty,we decomposethe area (Y ,Y ,...). Specifically, Z is determined based on the past
0 1 i
under the curve g(∆(t)) into a sum of disjoint components: realizations of (Y0,Y1,...,Yi), without using the realizations
of future service times (Y ,Y ,...); but the conditional
i+1 i+2
1The results in this paper can be readily extended to a more general distributionof(Y ,Y ,...)given(Y ,Y ,...,Y )isavail-
i+1 i+2 0 1 i
Markovianserviceprocess(Y0,Y1,...)withalongermemory,inwhichWi able. Let Π denote the set of all causally feasible policies
isdefinedasWi=(Yi,Yi+1,...,Yi+k)forsomefinitek,andthesequence
(W0,W1,...)formsaMarkovchain. satisfying Zi ∈[0,M] for all i.
4
The average age penalty per unit time is defined by2 III. OPTIMAL STATUSUPDATEPOLICY
E Dng(∆(t))dt In this section, we develop a chain of theoretical results to
0 solve Problem (8) by exploiting its structure: First, we prove
limsup . (7)
n→∞ hR E[Dn] i thatthereexistsastationaryrandomizedpolicythatisoptimal
forProblem(8).Further,weprovethatthereexistsastationary
Because the service process (Y ,Y ,...) is stationary and
0 1
ergodic, we can obtain E[Y ] = E[Y ] and hence E[D ] = deterministic policy that is optimal for Problem (8). Finally,
i i+1 n
E[ n−1(Z +Y )] = E[ n−1(Y +Z )]. Using this, the we develop a low-complexity algorithm to find the optimal
i=0 i i+1 i=0 i i stationary deterministic policy that solves Problem (8).
dynamic control problem for minimizing the average age
P P
penalty can be formulated as
A. Optimality of Stationary Randomized Policies
E n−1q(Y ,Z ,Y ) A policy π ∈ Π is said to be a stationary randomized
i=0 i i i+1
gopt =mπ∈iΠn linm→s∞up hPE[ in=−01(Yi+Zi)] i (8) pZoli∈cy[,0,ifMit] boabsseedrvoenslyYionanthdethoebnsercvheodosveasluae owfaYitin.gIntitmhies
i i
n−1
s.t. liminf 1E P(Y +Z ) ≥ 1 , (9) case, Zi is determined according to a conditional probability
n→∞ n " i i # fmax measure p(y,A) , Pr[Zi ∈ A|Yi = y] that is invariant for
i=0
X all i = 0,1,... We use Π (Π ⊆ Π) to denote the set of
SR SR
where gopt is the optimum objective value of Problem (8), stationary randomized policies such that
the expectation E is taken over the stochastic service process
(Y0,Y1,...) for given policy π, and fmax is the maximum ΠSR={π∈Π:Given the observation Yi =yi, Zi is chosen
allowed average sampling rate due to a long-term average according to the probability measure p(y ,A) for all i}.
i
resourceconstraint(i.e.,onthepowerresourceorCPUcycles)
Note that (Y ,Z ,Y ) is stationary and ergodic for all
spent on generating status updates. We assume M > 1/f i i i+1
max
stationary randomized policies. In the sequel, when we refer
such that Problem (8) is always feasible and g < ∞. In
opt tothestationarydistributionofastationaryrandomizedpolicy
this paper, we will study Problem (8) both with and without
π ∈Π ,wewillremovesubscripti.Inparticular,therandom
the constraint (9). In Section IV, sufficient and necessary SR
variables (Y ,Z ,Y ) are replaced by (Y,Z,Y′), where
conditions will be provided to characterize when the zero- i i i+1
Z is chosen based on the conditional probability measure
wait policy is optimal for solving Problem (8) without the
Pr[Z ∈ A|Y = y] = p(y,A) after observing Y = y, and
constraint (9).
(Y,Y′) have the same joint distribution as (Y ,Y ). The first
0 1
In Problem (8), Yi is the state of an embedded Markov key result of this paper is stated as follows:
chain,Zi isthecontrolactiontakenafterobservingYi,Yi+Zi Theorem 1: (Optimality of Stationary Randomized Poli-
is the period of stage i, and q(Yi,Zi,Yi+1) is the reward cies) If g : [0,∞) → [0,∞) is measurable and non-negative,
related to both stage i and i + 1. Therefore, Problem (8) (Y ,Y ,...)is a stationaryergodicMarkovchainwith Y ≥0
0 1 i
belongs to the class of constrained semi-Markov decision and 0 < E[Y ] < ∞, and condition (6) is satisfied, then
i
processes (SMDP) with a possibly uncountable state space, there exists a stationary randomizedpolicy that is optimal for
which is generally known to be quite difficult. The class Problem (8).
of SMDPs include Markov decision problems (MDPs) [17], ProofsketchofTheorem1: Foranypolicyπ ∈Π,define
[18] and optimization problems of renewal processes [19] the finite time-horizon average occupations
as special cases. Most existing studies on SMDPs deal with 1 n−1 g n−1
(i) unconstrained SMDPs, e.g., [13], [17], [20], [21], or (ii) an,π,nE q(Yi,Zi,Yi+1) − noptE (Yi+Zi) , (10)
constrained SMDPs with a countable state space, e.g., [16], (cid:20)i=0 (cid:21) (cid:20)i=0 (cid:21)
X X
[22]–[24]. However, the optimality equations (e.g., Bellman’s 1 n−1
equation) for solving unconstrained SMDPs are not applied bn,π , nE (Yi+Zi) . (11)
" #
i=0
to constrained SMDPs [25], and the studies on constrained X
Let Γ be the set of limit points of sequences ((a ,b ),
SMDPswithacountablestatespacecannotbedirectlyapplied SR n,π n,π
n = 1,2,...) associated with all stationary randomized
to Problem (8) which has an uncountable state space.
policiesinΠ .WefirstprovethatΓ isconvexandcompact.
SR SR
Then,weshowthatthereexistsanoptimalpolicyπ ofProb-
opt
lem(8), suchthatthesequence((a ,b ),n=1,2,...)
n,πopt n,πopt
associatedwithpolicyπ hasalimitpoint(a∗,b∗)satisfying
opt
2There are two widely used definitions of average cost per unit time in (a∗,b∗)∈Γ , a∗ ≤0, and b∗ ≥ 1 . Since (a∗,b∗)∈Γ ,
infinite-horizonundiscountedsemi-Markovdecisionprocesses(SMDP)[13]– there exists SaRstationary randomizefdmapxolicy π∗ achieving thSRis
[17]: In one definition, the average cost is the limit of expected total cost
over a finite deterministic horizon divided by the length of the horizon, limitpoint(a∗,b∗).Finally,weshowthatpolicyπ∗ isoptimal
i.e.,limsupT→∞ T1E(cid:2)R0Tg(∆(t))dt(cid:3);intheseconddefinition,theaverage for Problem (8), which completes the proof. The details are
cost is the limit of the expected total cost over a finite number of stages
available in Appendix A.
divided by the expected cumulative time of these stages, as in (7). These
two definitions are both reasonable [13], [16], [17]. They are equal under The convexity and compactness properties of the set of
stationary randomized policies in which the generated semi-Markov chain average occupation measures are essential in the study of
hasoneergodicclass;see[13]–[15]forfiniteandcountable statemodels.In
constrained MDPs [26, Sec. 1.5], which dates back to Der-
general,however,thesecriteriaaredifferent.Inourstudy,theseconddefinition
turnsouttobeanalytically convenient. man’s monograph in 1970 [27]. Recently, it was used in
5
Algorithm 1 Two-layer bisection method for Problem (13) Using Jensen’s inequality, we can show that for any feasible
given l=0, sufficiently large u>gopt, tolerance ǫ1. stationary randomized policy π1 ∈ ΠSR, there is a feasible
repeat stationary deterministic policy that is no worse than policy
c:=(l+u)/2. π . The details are provided in Appendix B.
1
given ζl =0, sufficiently large ζu >0, tolerance ǫ2. LetµY be the probabilitymeasureof Yi, then anybounded
ζ :=ζl, ν :=ζ+c. measurable function z : [0,∞) → [0,M] belongs to the
Compute zν(·) in (17). Lebesgue space L2(µY) [29, Section 3], because
if E[Y +z (Y)]< 1 then
ν fmax ∞ ∞
repeat |z(y)|2dµ (y)≤ M2dµ (y)=M2 <∞.
Y Y
ζ :=(ζl+ζu)/2, ν :=ζ+c. Z0 Z0
Compute z (·) in (17). By Theorems 1 and 2, we only need to consider the class
ν
if E[Y +z (Y)]≥ 1 , ζ :=ζ; else, ζ :=ζ. of stationary deterministic policies Π and Problem (8) is
ν fmax u l SD
until ζ −ζ ≤ǫ . simplified to the following functional optimization problem:
u l 2
end if E[q(Y,z(Y),Y′)]
if f(c)≤0, u:=c; else, l :=c. min (13)
until u−l ≤ǫ . z(·)∈L2(µY) E[Y +z(Y)]
1
1
return z(·):=zν(·). s.t. E[Y +z(Y)]≥ (14)
f
max
0≤z(y)≤M, ∀ y ≥0,
stochasticoptimizationfordiscrete-timequeueingsystemsand where z(·) is the function associated with a stationary deter-
renewal processes, e.g., [19], [28]. The techniques in these ministic policy π ∈ Π , and (Y,Y′) have the same distri-
SD
studies cannot directly handle constrained SMDPs with an bution as (Y ,Y ). The optimum objective value of Problem
0 1
uncountable state space, like Problem (8). One crucial novel (13) is equal to g .
opt
ideainourproofistointroduceg inthedefinitionofaverage
opt
occupationin(10),whichturnsouttobeessentialinlatersteps
forshowingtheoptimalityofthestationaryrandomizedpolicy C. A Low Complexity Solution to Problem (13)
π∗. In addition, we have also used one property of Problem
Lemma 1: If g : [0,∞) → [0,∞) is measurable,
(8) in the proof: the observation Y depends only on the
i+1
immediately preceding state Y and not on the history state non-negative, and non-decreasing, then the functional h :
i
L2(µ )→[0,∞) defined by
and control Y ,...,Y , Z ,...,Z . Y
0 i−1 0 i−1
By Theorem 1, we only need to consider the class of E[q(Y,z(Y),Y′)]
h(z)=
stationary randomized policies ΠSR. Therefore, as shown in E[Y +z(Y)]
Appendix A, Problem (8) can be simplified to the following
is quasi-convex.
functional optimization problem:
Proof: See Appendix C.
E[q(Y,Z,Y′)]
min (12) Therefore, Problem (13) is a functional quasi-convex opti-
p(y,A) E[Y +Z] mizationproblem.InordertosolveProblem(13),weconsider
s.t. E[Y +Z]≥ 1 the following functional convex optimization problem with a
fmax parameter c:
0≤Z ≤M,
f(c)= min E[q(Y,z(Y),Y′)]−cE[Y +z(Y)] (15)
where p(y,A) = Pr[Z ∈ A|Y = y] is the conditional z(·)∈L2(µY)
1
probabilitymeasureofsomestationaryrandomizedpolicy,and s.t. E[Y +z(Y)]≥ (16)
(Y,Y′) have the same distribution as (Y0,Y1). fmax
0≤z(y)≤M, ∀ y ≥0.
B. Optimality of Stationary Deterministic Policies
It is easy to show that g ≤ c if and only if f(c) ≤ 0 [30].
opt
A policy π ∈ Π is said to be a stationary deterministic Therefore, we can solve Problem (13) by a two-layer nested
SR
policy if Z = z(Y ) for all i = 0,1,..., where z :[0,∞)→ algorithm, such as Algorithm 1. In the inner layer, we use
i i
[0,M] is a deterministic function. We use Π (Π ⊆ Π ) bisection to solve Problem (15) for any given parameter c;
SD SD SR
to denote the set of stationary deterministic policies such that in the outer layer, we employ bisection again to search for
a c∗ such that f(c∗) = 0 and thus g = c∗. Algorithm 1
Π ={π ∈Π :Z =z(Y ) for all i}. opt
SD SR i i has low complexity. It requires at most ⌈log ((u−l)/ǫ )⌉×
2 1
Theorem 2: (Optimality of Stationary Deterministic Poli- ⌈log2((ζu−ζl)/ǫ2)⌉ iterationsto terminate and each iteration
cies)Ifg :[0,∞)→[0,∞)ismeasurableandnon-decreasing, involves computing E[zν(Y)] based on (17). The optimality
then there exists a stationary deterministic policy that is of Algorithm 1 is guaranteed by the following theorem:
optimal for Problem (12). Theorem 3: If g : [0,∞) → [0,∞) is measurable, non-
Proof sketch of Theorem 2: Since g(∆) is non- negative,and non-decreasing,then an optimalsolution π to
opt
decreasing, q(y,z,y′) is convex in z for any fixed y and y′. Problem (13) is obtained by Algorithm 1, where the function
6
Algorithm 2 Bisection method for solving Problem (19)
Π
given l =0, sufficiently large u, tolerance ǫ.
Π repeat
SR
Π β :=(l+u)/2.
SD
πopt o := E[(Y +z(Y))] − max 1 ,E[(Y+z(Y))2] ,
fmax 2β
(cid:18) (cid:19)
where z(·) is given by (21).
if o≥0, u:=β; else, l:=β.
until u−l ≤ǫ.
Compute z(·) by (21).
Fig. 5. Illustration of the set of causally feasible policies Π, the set of
return z(·).
stationaryrandomizedpoliciesΠSR,thesetofstationarydeterministicpolicies
ΠSD,andtheobtained optimalpolicy πopt.
z (·) is determined by
ν
Because the Y ’s are i.i.d., Y and Y′ in Problem (13) are also
i
zν(y)=sup{z ∈[0,M]:E[g(y+z+Y′)|Y =y]≤ν}, (17) i.i.d. Using this, we can obtain
and (Y,Y′) follow the same distribution as (Y0,Y1). E[q(Y,z(Y),Y′)]
Proof Sketch of Theorem 3: We use Lagrangian duality
1 1
theory to solve Problem (13). Different from traditional finite =E (Y +z(Y)+Y′)2− Y′2 (18)
2 2
dimensional optimization problems [30], Problem (13) is an (cid:20) (cid:21)
1
infinite dimensional functional optimization problem. There- = E (Y +z(Y))2 +E[Y +z(Y)]E[Y′],
2
fore, the Karush-Kuhn-Tucker (KKT) theorem for infinite
dimensional space [31], [32] and the calculus of variations where in (18)(cid:2)we have use(cid:3)d that E[Y2] = E[Y′2]. Hence,
arerequiredintheanalysis.Inparticular,sincetheLagrangian Problem (13) can be reformulated as
may not be strictly convex for some penalty functions, one- E[(Y +z(Y))2]
sdiidmeednsGioaˆnteaalusxpadceer)ivisatiuvseed(stiomisloarlvetothseubK-gKrTadiceonntdiitnionfisniitne z∈Lm2i(nµY) 2E[Y +z(Y)] +E[Y] (19)
1
Lebesgue space L2(µY). The proof details are provided in s.t. E[Y +z(Y)]≥ (20)
f
Appendix D. max
The policy spaces Π, Π , Π , and the obtained optimal 0≤z(y)≤M, ∀ y ≥0.
SR SD
policy π are depicted in Fig. 5.
opt ThefollowinglemmatellsusthatProblem(19)isafunctional
convex optimization problem.
IV. WHEN IS ITBETTERTO WAIT THAN TOUPDATE? Lemma 2: The functional h :L2(µ )→R defined by
1 Y
When E[Y] ≥ 1/f , the constraint (16) is always sat-
max E[(Y +z(Y))2]
isfied. In this case, a logical policy is the zero-wait policy: h (z)=
1 E[Y +z(Y)]
the source submits a fresh update once the prior update
completes service, i.e., π = (0,0,...). According to is convex on the domain
zero-wait
the example in the introduction, this zero-wait policy is not
dom h = z ∈L2(µ ):z(y)∈[0,M], ∀y ≥0 .
always optimal. When E[Y] < 1/f , due to the con- 1 Y
max
straint (16), the minimum possible average waiting time is Proof: See(cid:8)Appendix E. (cid:9)
E[z(Y)] = 1/fmax −E[Y]. However, even in this case, the UsingtheKKTtheoremforinfinitedimensionalspaceandthe
optimal policy may have additional waiting time such that calculus of variations, we can obtain
E[Y +z(Y)] > 1/fmax. In this section, we will study when Theorem 4: If E[Y] > 0, the optimal solution to Problem
it is optimal to submit updates with the minimum average
(19) is
waiting time and when it is not. z(y)=(β−y)M, (21)
0
where (x)M ,min{max{x,0},M} and β >0 satisfies
A. A Special Case of g(∆)=∆ with i.i.d. Service Times 0
Consider the case that g(∆) = ∆ and the Yi’s are i.i.d. E[Y +z(Y)]=max 1 ,E[(Y +z(Y))2] . (22)
with 0<E[Y]<∞. In this case, Problem (13) has a simpler (cid:18)fmax 2β (cid:19)
solution than that providedby Algorithm 1. Interestingly,this
Proof: See Appendix F.
solution explicitly characterizes whether the optimal control
Equation(21)hastheformofawater-fillingsolution,where
z(·) can have the minimum average waiting time such that
the water-level β is given by the root of equation (22). One
E[Y +z(Y)]=1/f .
max canobservethat(17)reducesto(21)ifg(∆)=∆,theY ’sare
Asshownin Fig.3, Qn = 12 (Yn+Zn+Yn+1)2−Yn2 is i.i.d., and ν is replaced by β+E[Y]. The root β of equiation
the area of a trapezoid. This corresponds to
(cid:2) (cid:3) (22) can be simply solved by the bisection search method in
1 Algorithm 2. We note that Algorithm 2 has lower complexity
q(y,z,y′)= (y+z+y′)2−y2 .
2 than Algorithm 1 in the special case of g(∆) = ∆ and i.i.d.
(cid:2) (cid:3)
7
2.3
service process, while Algorithm 1 can obtain the optimal
Optimal policy
policy in more general scenarios. Constant wait
2.2 Minimum wait
Theorem4 providesa closed-formcriterion on whether the
optimal z(·) satisfies E[Y + z(Y)] = 1/fmax. Specifically, e age2.1
(o2p1ti)maanldco(2n2tr)otlezll(·u)sstahtiastfiiefs1/fmax ≥ E[(Y+2zβ(Y))2], then the Averag 2
1.9
1 E[(Y +z(Y))2]
E[Y +z(Y)]= ≥ , (23)
f 2β 1.8
max 1 1.2 1.4 1.6 1.8 2
1/f
suchthattheoptimalpolicyhastheminimumpossibleaverage max
waitingtime;otherwise,if1/fmax < E[(Y+2zβ(Y))2],theoptimal Fig.6. Averageagevs.1/fmax withi.i.d.discrete servicetimes.
control z(·) satisfies
E[(Y +z(Y))2] 1 6
E[Y +z(Y)]= > , (24)
2β f Optimal policy
max 5.5 Constant wait
Minimum wait
such that the optimal policy can not achieve the minimum
poFssuirbtlheeramveorraeg,eifwEai[tYin]g≥tim1/ef.max (for example fmax = ∞), Average age4.55
the zero-wait policy is feasible. Meanwhile, the constraint
(20) is always satisfied and can be removed. In this case, 4
the optimality of the zero-wait policy is characterized by the
3.5
following theorem. 1 2 3 4 5 6 7 8
1/f
Theorem 5: If E[Y] > 0 and the zero-wait policy is max
feasible,thenthezero-waitpolicyisoptimalforProblem(19) Fig. 7. Average age vs. 1/fmax with i.i.d. log-normal distributed service
if, and only if, times.
E[Y2]
≤y , (25) process models are considered: The first is a discrete Markov
2E[Y] inf chain with a probability mass function Pr[Y =0]=Pr[Y =
i i
where y =inf{y ∈[0,∞):Pr[Y ≤y]>0}. 2]=0.5 and a transition matrix
inf
Proof: See Appendix G.
p 1−p
Informally speaking, y is the smallest possible value of P = .
inf 1−p p
the random service time Y. From Theorem 5, it immediately (cid:20) (cid:21)
follows that: Hence, the Yi’s are i.i.d. when p = 0.5, and the corre-
Corollary 1: If E[Y] > 0 and the zero-wait policy is lation coefficient between Yi and Yi+1 is ρ = 2p − 1.
The second is a log-normal distributed Markov chain, where
feasible, then the following assertions are true:
Y = eσXi/E[eσXi] and (X ,X ,...) is a Gaussian Markov
(a). If the service times are positive and constant such that i 0 1
process satisfying the first-order AR equation
Y = const > 0, the zero-wait policy is optimal for
Problem (19). X =ηX + 1−η2W ,
i+1 i i
(b). Iftheservicetimedistributionsatisfiesy =0,thezero-
inf
wait policy is not optimal for Problem (19). whereσ >0isthescaleparametperoflog-normaldistribution,
η ∈ [−1,1] is the parameter of the AR model, and the
Remark:AsonecanreadilyseefromCorollary1(b),thezero-
W ’s are i.i.d. Gaussian random variables with zero mean
wait policy is not optimal for many commonly used service i
andunitvariance.Thelog-normaldistributedMarkovchainis
time distributions in communication and queueing theory,
normalized such that E[Y ] = 1. According to the properties
suchasexponentialdistribution,geometricdistribution,Erlang i
of log-normaldistribution, the correlation coefficient between
distribution, hyperexponentialdistribution, etc.
Y and Y is ρ=(eη−1)/(e−1). Then, the Y ’s are i.i.d.
1) Simulation Results: We use “Optimal policy” to refer i i+1 i
when η =0.
to the policy provided in Theorem 3 (or its special case in
Theorem 4), and compare it with two reference policies: Figures 6 and 7 illustrate the average age vs. 1/fmax
for i.i.d. discrete and log-normal distributed service times,
• “Constant wait”: Each update is followed by a constant
waiting time Z = 1/f −E[Y] before submitting the respectively.Inbothfigures,onecanobservethattheconstant
max
wait policy always incurs a larger average age than the
next update.
optimal policy. In addition, as expected from (23) and (24),
• “Minimumwait”: The updatewaitingtime is determined
by Z =z(Y), where z(·) is given by (21) and β in (22) as 1/fmax exceeds a certain threshold, the optimal policy
is chosen to satisfy E[z(Y)]=1/f −E[Y].3 meets the constraint (20) with equality. For smaller values
max
When E[Y] = 1/f , both the constant wait and minimum of 1/fmax, the constraint (20) is not active in the optimal
max
solution. Consequently, the minimum wait policy deviates
wait policies reduce to the zero-wait policy. Two service
from the optimal policy for small values of 1/f , which
max
3Thispolicywascalled “β-minimum”in[11]. is in accordance to Corollary 1(b).
8
3 45
Optimal policy 40 Optimal policy
Zero wait Zero wait
2.5 35
Average age 2 Average age12235050
1.5
10
5
1 0
-1 -0.5 0 0.5 1 0 0.5 1 1.5 2
Correlation Coefficient ρ σ
Fig.8. Average agevs.thecorrelation coefficient ρbetween Yi andYi+1 Fig. 10. Average age vs. the distribution parameter σ ofi.i.d. log-normal
fordiscreteservice times,whereE[Y]≥1/fmax. distributed servicetimes,whereE[Y]≥1/fmax.
14
Optimal policy 2
12 Zero wait Optimal policy
Zero wait
1.5
Average age1068 Average age penalty0.15
4
2 0
-0.5 0 0.5 1
Correlation Coefficient ρ 0 0.2 0.4 0.6 0.8 1
α
Fig.9. Average agevs.thecorrelation coefficient ρbetween Yi andYi+1
forlog-normaldistributed servicetimes,whereE[Y]≥1/fmax. Ffuign.cti1o1n.swiAthvedriascgreeteagseervpiecneatlitmyevss,.wthheerepEar[aYm]e≥ter1/αfmoafxs,tgai(r∆-st)ep=p⌊eαn∆alt⌋y,
andρ=0.4.
B. General Age Penalties and Correlated Service Processes
For general age penalties and correlated service processes,
zero-wait policy, leading to a larger gap from the optimum.
it is essentially difficult to find closed-form characterization
In Fig. 9, the range of ρ is [(e−1 −1)/(e−1),1) and the
on whether the optimal control z(·) can have the minimum
average waiting time such that E[Y + z(Y)] = 1/f . sub-optimalitygap of the zero-waitpolicy also increases with
max
Therefore, we focus on the case of E[Y] ≥ 1/f (this is ρ. The point ρ=1 is not plotted in these figures because the
max
corresponding Markov chains are not ergodic.
equivalent to removing the update frequency constraint (16))
andstudywhenthezero-waitpolicyminimizestheaverageage Figure 10 considers the average age vs. the parameter σ
penalty. Sufficient conditions for the optimality of the zero- of log-normal distributed service times, where ρ = (e0.5 −
wait policy are provided as follows: 1)/(e−1). We observe that the zero-wait policy is optimal
Lemma 3: Suppose that E[Y] ≥ 1/f , g(·) is measur- for small σ and is not optimal for large σ. When σ = 0, the
max
able, non-negative, and non-decreasing. The zero-wait policy service times are constant, i.e., Yi = 1 for all i, and hence
is optimalforProblem (13) if one of the followingconditions by Lemma 3, the zero-waitpolicy is optimal. For large σ, the
is satisfied: time-averageageofthe zero-waitpolicyissignificantlylarger
than the optimum.Thisimplies that the sub-optimalitygap of
1). The correlation coefficient between Y and Y is −1;
i i+1
the zero-wait policy can be quite large for heavy-tail service
2). The Y ’s are equal to a constant value;
i
time distributions.
3). g(·) is a constant function.
Figures 11-16 show the average age penalty vs. the param-
Proof: See Appendix H.
eterαofthreetypesofagepenaltyfunctions,wherethestair-
1) Simulation Results: We now provide some simulation
step function g(∆) = ⌊α∆⌋ is considered in Fig. 11 and 12,
results for general age penalties and/or correlated service
the power function g(∆) = ∆α is considered in Fig. 13 and
processes. Figures 8 and 9 depict the average age vs. the
14,andtheexponentialfunctiong(∆)=eα∆−1isconsidered
correlationcoefficientρ between Y and Y for discrete and
i i+1
in Fig. 15 and 16. The correlation coefficient is ρ = 0.4 for
log-normal distributed service times, respectively. In Fig. 8,
discrete service times, and is ρ = (e0.5 − 1)/(e − 1) for
therangeofρis[−1,1).We observethatthezero-waitpolicy
log-normal distributed service times. We find that the zero-
is optimal when ρ ∈ [−1,−0.5], and the performance gap
wait policy is optimal if α = 0, in which case g(∆) is a
between the optimal policy and the zero-wait policy grows
constant function. When α>0, the zero-wait policy may not
with ρ when ρ ≥ −0.5. This is in accordance with the
be optimal.
example in the introduction: As ρ grows, the occurrence of
two consecutive zero service times (i.e., (Y ,Y ) = (0,0)) These simulation results suggest that the conditions in
i i+1
increases.Therefore,moreandmoreupdatesarewastedinthe Lemma 3 are sufficient but not necessary.
9
8 600
7 OZeprtoim waal iptolicy 500 OZeprtoim waal iptolicy
Average age penalty23456 Average age penalty234000000
100
1
0 0
0 0.2 0.4 0.6 0.8 1 0 0.5 1 1.5 2
α α
Fig. 12. Average age penalty vs. the parameter α of stair-step penalty Fig.14. Averageagepenaltyvs.theparameterαofpowerpenaltyfunctions
functions withlog-normal distributed servicetimes,whereE[Y]≥1/fmax, with log-normal distributed service times, where 1/fmax ≤E[Y], g(∆)=
g(∆)=⌊α∆⌋,andρ=(e0.5−1)/(e−1). ∆α,andρ=(e0.5−1)/(e−1).
7
Optimal policy 16 Optimal policy
6 Zero wait 14 Zero wait
Average age penalty345 Average age penalty1146802
2
2
1 0
0 0.5 1 1.5 2 0 0.2 0.4 0.6 0.8 1
α α
Fig.13. Averageagepenaltyvs.theparameterαofpowerpenaltyfunctions Fig. 15. Average age penalty vs. the parameter α of exponential penalty
with discrete service times, where E[Y]≥1/fmax, g(∆)=∆α, andρ= functionswithdiscreteservicetimes,where1/fmax≤E[Y],g(∆)=eα∆−
0.4. 1,andρ=0.4.
V. CONCLUSION where g is the optimum objective value of Problem (26).
SR
Since Π ⊆Π, we can obtain
SR
We studied the optimal control of status updates from a
source to a remote destination via an FCFS communication gSR ≥gopt. (27)
server. We generalized the existing studies on the age-of-
It is easy to show that the (Y ,Z ,Y )’s are stationary
i i i+1
information to cover general age penalty functions and non-
and ergodic for all stationary randomized policies. This, to-
i.i.d. service processes. We developed efficient algorithms
gether with the condition that g(·) is measurable, tells us
to find the optimal status update policy for minimizing the
that q(Y ,Z ,Y ) is stationary and ergodic [33, Theorems
i i i+1
average age penalty. We showed that, surprisingly, in many
7.1.1 and 7.1.3]. For any stationary randomized policy π =
scenarios,theoptimalpolicyistowaitforacertainamountof
(Z ,Z ,...)∈Π , we obtain
0 1 SR
timebeforesubmittinganewupdate.Inparticular,thewidely-
adopted zero-wait policy can be far from the optimum if (i) 1 n−1
E q(Y ,Z ,Y ) =E[q(Y ,Z ,Y )], (28)
the penalty function grows quickly with respect to the age, i i i+1 0 0 1
n
" #
and (ii) the update service times are positively correlated and Xi=0
n−1
have heavy tail distributions. 1
E (Y +Z ) =E[Y +Z ]. (29)
i i 0 0
n
" #
i=0
X
APPENDIX A Hence, Problem (26) can be reformulated as Problem (12).
PROOF OF THEOREM1
B. The Upper Bound of g is Tight, i.e., g =g
A. An Upper Bound of g opt SR opt
opt
We will show g = g in 4 steps. The following
SR opt
By restricting Π in Problem (8) to Π , we obtain the
SR definitions are needed: Since g is finite, for each causally
opt
following problem:
feasible policy π =(Z ,Z ,...)∈Π we can define a and
0 1 n,π
E n−1q(Y ,Z ,Y ) bn,π as in (10) and (11), respectively.
g = min limsup i=0 i i i+1 (26) Further, define ΓSR as the set of limit points of sequences
SR π∈ΠSR n→∞ hPE[ in=−01(Yi+Zi)] i (an,π,bn,π),n=1,2,... associatedwithallstationaryran-
s.t. linm→i∞nf n1E"nXi=−01P(Yi+Zi)#≥1/fmax, (cid:0)daranondmdiionzmteedirzvpeaodllYipcoii+eliscZiπeis∈arπeΠ∈sStRaΠt.(cid:1)iSBoRne,actrahyuesaesnedtqheueregrneocwdeiac(radfonqr,(πaY,libl,nsZ,tπait),iYohnia+as1rya)
10
tain that q(Y,Z ,Y′) converges weakly to q(Y,Z ,Y′).
120 Optimal policy (jk) (∞)
Zero wait Then, using the condition (6), together with the dominated
100
alty convergence theorem (Theorem 1.6.7 of [33]) and Theorem
e pen 80 3.2.2 of [33], we can obtain limk→∞(djk,ejk) = (d∞,e∞).
g
e a 60 Hence, ((dj,ej),j =1,2,···) has a convergentsubsequence.
g
Avera 40 bFyurtuhseirn,gwethceancgoenndeitriaotneaalsptartoiboanbairlyitryanpdom(yi,zAed)pcoolircryesπp∞on,SdR-
∞
20
ing to µ . Then, (d ,e ) is the limit point generated
∞ ∞ ∞
0 by the stationary randomized policy π , which implies
0 0.01 0.02 0.03 0.04 0.05 ∞,SR
α (d ,e ) ∈ Γ . In summary, any sequence (d ,e ) in Γ
∞ ∞ SR j j SR
hasaconvergentsubsequence(d ,e )whoselimit(d ,e )
Fig. 16. Average age penalty vs. the parameter α of exponential penalty jk jk ∞ ∞
functions withlog-normal distributed servicetimes,where1/fmax≤E[Y], is also in ΓSR. Therefore, ΓSR is a compact set.
g(∆)=eα∆−1,andρ=(e0.5−1)/(e−1).
Step 2: We will show that there exists an optimal policy
π ∈ΠofProblem(8)suchthatthesequence(a ,b )
opt n,πopt n,πopt
associatedwith policy π has atleast onelimit pointin Γ .
unique limit point in the form of opt SR
Since the sequence (Y ,Y ,...) is a Markov chain, the
E[q(Y,Z,Y′)]−g E[Y +Z],E[Y +Z] . (30) 0 1
opt observation Yi+1 depends only on the immediately preceding
Hence,Γ(cid:0)SR isthesetofallpoints(E[q(Y,Z,Y′)]−(cid:1)goptE[Y + state Yi and noton the history state and controlY0,...,Yi−1,
Z],E[Y +Z]), where each point is associated with a condi- Z0,...,Zi−1. Therefore, Yi is the sufficient statistic [17, p.
tional probability measure p(y,A) = Pr[Z ∈ A|Y = y], and 252] for solving Problem (8). This tells us that there exists
the measure of (Y,Y′) is the same as that of (Y0,Y1). Note an optimal policy πopt = (Z0,Z1,...)∈Π of Problem (8) in
that E[Y]=E[Y′]. whichthecontrolactionZiisdeterminedbasedononlyYi,but
not the history state and control Y ,...,Y , Z ,...,Z
Step 1: We will show thatΓ is a convexand compactset. 0 i−1 0 i−1
SR
[17]. We will show that the sequence (a ,b ) associ-
It is easy to show that ΓSR is convex by considering a sta- n,πopt n,πopt
ated with this policy π has at least one limit point in Γ .
tionary randomized policy that is a mixture of two stationary opt SR
randomized policies. It is known that Zi takes values in the standard Borel
For compactness, let ((d ,e ),j = 1,2,···) be any se- space (R,R), where R is the Borel σ-field. According to
j j
quence of points in Γ , we need to show that there is a [33, Thoerem 5.1.9], for each i there exists a conditional
SR
convergentsubsequence (djk,ejk) whose limit is also in ΓSR. probability measure p′i(y,A) such that p′i(y,A) = Pr(Zi ∈
Since (dj,ej) ∈ ΓSR, there must exist (Y,Z(j),Y′) with A|Yi = y) for almost all y. That is, the control action Zi
conditional probability pj(y,A)=Pr[Z(j) ∈ A|Y = y], such is determined based on Yi and the conditional probability
thatdj =E[q(Y,Z(j),Y′)]−goptE[Y+Z(j)],ej =E[Y+Z(j)]. measure p′i(y,A) = Pr(Zi ∈ A|Yi = y). One can use this
Let µj be the joint probability measure of (Y,Z(j),Y′), then conditional probability p′i(y,A) to generate a stationary ran-
(dj,ej) is uniquely determined by µj. For any L satisfying domized policy πi′,SR ∈ΠSR. Then, the one-stage expectation
L≥M, we can obtain (E[q(Yi,Zi,Yi+1)] − goptE[Yi + Zi],E[Yi + Zi]) is exactly
the limit point generated by the stationary randomized policy
µj(Y ≤L,Z(j) ≤L,Y′ ≤L) πi′,SR.Thus,(E[q(Yi,Zi,Yi+1)]−goptE[Yi+Zi],E[Yi+Zi])∈
=Pr(Y ≤L,Y′ ≤L) Γ for all i = 0,1,2,... Using (10), (11), and the fact that
SR
≥Pr(Y +Y′ ≤L) ΓSR is convex, we can obtain (an,πopt,bn,πopt) ∈ ΓSR for all
E[Y +Y′] n = 1,2,3... In other words, the sequence (an,πopt,bn,πopt)
≥1− , ∀ j, is within Γ . Since Γ is a compact set, the sequence
L SR SR
(a ,b ) must have a convergent subsequence, whose
where the equality is due to the fact that Z ≤M ≤L and n,πopt n,πopt
(j) limit is in Γ .
SR
the last inequality is due to Markov’s inequality. Therefore,
Step3:Let(a∗,b∗)∈Γ beonelimitpointofthesequence
for any ǫ, there is an L such that SR
(a ,b ) associated with policy π . We will show that
n,πopt n,πopt opt
liminfµj(|Y|≤L,|Z(j)|≤L,|Y′|≤L)≥1−ǫ. a∗ ≤0 and b∗ ≥1/fmax.
j→∞
Policy π is feasible for Problem (8) and meanwhile
Hence, the sequence of measures µ is tight. By Helly’s opt
j achieves the optimum objective value g . Hence,
selectiontheorem[33, Theorem3.9.2],thereisa subsequence opt
ofmeasuresµjk thatconvergesweaklytoalimitmeasureµ∞. limsupcn,πopt =g , (31)
Let (Y,Z(∞),Y′) and p∞(y,A) = Pr[Z∞ ∈ A|Y = y] n→∞ bn,πopt opt
denote the random vector and conditional probability cor- liminfb ≥1/f , (32)
responding to the limit measure µ , respectively. We can n→∞ n,πopt max
∞
define d = E[q(Y,Z ,Y′)] − g E[Y + Z ], e = where
∞ (∞) opt (∞) ∞
E[Y +Z ]. Since the function q(y,z,y′) is in the form of n−1
(∞) 1
an integral, it is continuous and thus measurable. Using the cn,πopt , nE q(Yi,Zi,Yi+1) .
continuousmappingtheorem[33,Theorem3.2.4],we canob- (cid:20)i=0 (cid:21)
X