Table Of ContentAPPROXIMATION OF TARGET PROBLEMS IN
BLACKWELL SPACES
By Giacomo Aletti∗ Diane Saada†
9
0 University of Milan∗ and The Hebrew University of Jerusalem†
0
2 On a weakly Blackwell space we show how to define a Markov
chain approximating problem, for the target problem. The approxi-
n
matingproblemisprovedtoconvergetotheoptimalreducedproblem
a
J underdifferentpseudometrics.Acomputationalexampleofcompres-
3 sion of information is discussed.
1
] Let X be an homogeneous Markov chain. Suppose the process stops
n
R
once it reaches an absorbing class, called the target, according to a given
P
stopping rule: the resulting problem is called target problem (TP). Theidea
.
h
is to reduce the available information in order to only use the necessary
t
a information which is relevant with respect to the target. A new Markov
m
chain, associated with a new equivalent but reduced matrix is defined. In
[
the (large) finite case, the problem has been solved for TPs: in [1–3], it
1 has been proved that any TP on a finite set of states has its “best target”
v
equivalent Markov chain. Moreover, this chain is unique and there exists a
1
7 polynomial time algorithm to reach this optimum.
8 The question is now to find, in generality, an ǫ–approximation of the
1
Markov problem when the state space is measurable. The idea is to merge
.
1
into one group the points that ǫ-behave the same with respect to the ob-
0
9 jective, but also in order to keep an almost equivalent Markov chain, with
0 respect to the other “groups”. The construction of these groups is done
:
v through equivalence relations and hence each group corresponds to a class
i of equivalence. In fact, there are many other mathematical fields where ap-
X
proximation problemsare faced by equivalences. For instance, in integration
r
a
theory, we use simple functions, in functional analysis, we use the density of
countable generated subspaces and in numerical analysis, we use the finite
elements method.
In this paper, the approximation is made by means of discrete equiva-
lences, which will be defined in the following. The purpose of any approxi-
mationistoreachtheexactsolutionwhenǫ → 0.Weprovethatthesequence
of approximations tends to the optimal exact equivalence relation definedin
AMS 2000 subject classifications: Primary 60J10; secondary 60J05, 05C20
1
G. ALETTI, D.SAADA/TARGETPROBLEMS IN BLACKWELL SPACES 2
[1–3],whenwerefinethegroups.Finerequivalence willimplybetterapprox-
imation, and accordingly the limit will be defined as a countably generated
equivalence.
Under a very general Blackwell type hypothesis on the measurable space,
weshowthatitisequivalenttospeakoncountablygeneratedequivalencere-
lationshipsoronmeasurablerealfunctionsonthemeasurablespaceofstates.
Ifwe do notwork underthis framework of Blackwell spaces, we can befaced
toparadoxes,asitisexplainedby[6],ofenlargingσ–algebras,whiledecreas-
ing the information available to a decision-maker. The ǫ–approximation of
the Markov chain depends always upon the kind of objective. In [9], Jer-
rum deals with ergodic Markov chains. His objective is to approximate the
stationary distribution by means of a discrete approximating Markov chain,
whose limit distribution is close in a certain sense to the original one. How-
ever, unlike our following work, his purpose is not the explicit and unified
construction of the approximating process. In this paper, we focus on the
target problem. We solve extensively the TP, where the objective is con-
nected with the conditional probability of reaching the target T, namely
P(X ∈ T|X = x), for any n,x. This part extends the work in [1–3], since
n 0
TPs’ approximation may help to understand the behavior of those TPs
where the best equivalent Markov chain is also very large. The setting of an
approximating problem can be extended to a general form, but we will not
develop it in this paper.
1. Main results. Let (X,X) be a measurable space. We equip it with
an Assumption (A0) that will be explained when required. Let P be any
transition probability on (X,X). An homogeneous Markov process (X ) is
n n
naturally associated to (X,X,P). In the target problem, we are interested
in the probabilities of reaching the target class T within n steps, namely in
P {X ∈ T} X = x for any n and x.
n 0
(cid:0) (cid:12) (cid:1)
The set T is a priori given an(cid:12)d does not change through the computations.
Definition 1. Let (X,X) be a measurable space and let T ∈ X. Let
F ⊆ X be a sub σ-algebra of X such that T ∈ F. A function P : X ×F →
[0,1] is a transition probability on (X,F) if
• P(x,·) is a probability measure on F, for any x∈ X;
• P(·,F) is F-measurable, for any F ∈ F.
Given a transition probability P on (X,F), we denote by Pn the transition
G. ALETTI, D. SAADA/TARGETPROBLEMS IN BLACKWELL SPACES 3
probability on (X,F) given inductively by
P1 = P; Pn+1(x,F) = P(x,dy)Pn(y,F)
ZX
We denoteby TrP(X,X,F) thesetof thetransition probabilities on (X,F).
We denote by TP = ∪ TrP(X,X,F) the set of all transition probabil-
X F⊆X
ities on X, that we equip with a suitable pseudometric d
d(P ,P ) = sup βn Pn(x,T)−Pn(x,T) .
1 2 1 2
x
n
X (cid:12) (cid:12)
It is such that (cid:12) (cid:12)
d(P ,P )= 0 ⇐⇒ Pn(x,T) = Pn(x,T), for any n and x.
1 2 1 2
This pseudo-metric is obviously compatible with the target T and appears
as a loss function of using P instead of P , whatever is the initial point x
2 1
(β ∈ (0,1) is a discount rate). The point will be, with no surprise, to let this
distance go to 0.
AtargetproblemisdefinedthroughatransitionprobabilityP ∈ (TP ,d).
X
Definition 2. A target problem is a quadruple(X,F,T,P), where P ∈
TrP(X,X,F) andT ∈ F.Asimple target problem is atarget problemwhere
F is generated by an at most countable partition of X.
The main purpose of this paper is to approximate any target problem
with a sequence of simple target problems in the spirit of the construction
the Lebesgue integral, wherethe integral of a function f is approximated by
the integral of simple functions f = c I . The Lebesgue approximation
n i i Ci
requires at each step n ∈ N two choices: the choice of the subdivision (C )
P i i
and the choice of the function values (c ) on each subdivision.
i i
Definition 3. We call strategy Stra sequence of maps (Str ) from the
n n
set of the target problems to the set of the simple target problems.
In the “Lebesgue example” given above, the strategy is related to the
“objective” of the problem (the integral) and the pseudometric d(f,f ) =
n
|f −f |dx is required to go to 0 as n goes to infinity. Here also, a strategy
n
is meaningful if d(P,Str (P)) tends to 0 as n goes to infinity. Moreover,
R n
for what concerns applications, given a target problem (X,X,T,P) a good
strategy should not need the computation of Pn, n > 1. The first main
result of this paper is theexistence of a class of good strategies, called target
algorithms.
G. ALETTI, D.SAADA/TARGETPROBLEMS IN BLACKWELL SPACES 4
Theorem 1. For any target problem (X,F,T,P) and any target algo-
rithm Str,
lim d(P,P )= 0,
n
n→∞
where (X,F ,T,P )= Str (X,F,T,P).
n n n
Two questions immediately arise: does the sequence (Str (P)) have a
n n
limit (and in which sense)? Moreover, since d is defined as a pseudometric,
does this limit depend on the choice of Str?
Theextensionoftheconceptofcompatibleprojectiongivenin[1–3]toour
frameworkwillenableustounderstandbettertheanswertothesequestions.
A measurable set A 6= ∅ of a measurable space (X,X) is an X-atom if it has
no non-empty measurable proper subset. No two distinct atoms intersect. If
the σ-field is countably generated, say by the sequence {A } then the atoms
n
of X are of the form ∩ C where each C is either A or X \A .
n n n n n
Definition 4. Anequivalencerelationshipπonameasurableset(X,X)
is measurable (discrete) if there exists a (discrete) random variable f :
(X,X) → (R,B ) (B denotes the Borel σ-algebra), such that
R R
xπy ⇐⇒ f(x)= f(y),
and we denote it by π = π . Let (X,F,T,P) be a target problem. A com-
f
patible projection is a measurable equivalency π such that
f
(1) P(x,F) = P(y,F), ∀xπ y,∀F ∈ σ(f).
f
A compatible projection π is said to be optimal if π ⊇ π′, for any other
compatible projection π′.
Remark 1. This definition is well posed if
π = π ⇐⇒ σ(f) = σ(g).
f g
Assumption (A0) ensures that the definition of measurable equivalency is
indeed well posed. This assumption will be stated and discussed in Section
4.
Theorem 2. If π = π is a compatible projection for the target problem
f
(X,F,T,P), then there exists a target problem (X,σ(f),T,P ). such that
π
P (x,F) = P(x,F) for any F ∈ σ(f).
π
G. ALETTI, D. SAADA/TARGETPROBLEMS IN BLACKWELL SPACES 5
It is not said “a priori” that an optimal compatible projection must exist.
If it is the case, then this equivalence is obviously unique.
Theorem3. Foranytargetproblem(X,F,T,P), there existsa(unique)
optimal compatible projection π.
To conclude the main results, let us first come back to the Lebesgue ex-
ample.Thesimplefunctionf = c I arechosen sothatσ(C )increases
n i i Ci n
toσ(f)andf (x) → f(x).Thefollowing theoremguarantees thesetwofacts
n P
by showing the “convergence” of any strategy to the optimal problem.
Theorem4. LetStr (X,F,T,P) = (X,F ,T,P ), withStrtarget algo-
n n n
rithm and let π be the optimal compatible projection associated to the target
problem (X,F,T,P). Then
• F ⊆ F for any n, and ∨ F = F ;
n n+1 n n π
• lim P (x,F) = P (x,F), for any F ∈ ∪ F .
n n π m m
Remark 2 (The topology Top). In Theorems 1–4, we have proved the
convergence of (P ) to P with respect to the pseudometric d. The pseu-
n n π
dometric topology Top is the topology induced by the open balls B (P) =
r
{Q ∈ TP : d(P,Q) < r}, which form a basis for the topology. Accordingly,
X
the previous theorems may be reread in terms of convergence of P to P on
n
the topological space (TP ,Top).
X
1.1. Connectionwithweakconvergence. Givenastrategy(X,F ,T,P ) ,
n n n
if we want to show a sort of weak convergence of P (x,·) to P(x,·), for any
n
x, we face the two following problems
• each P (x,·) is defined on a different domain (namely, on F );
n n
• we did not have required a topology on X.
First,wewanttointroduceanewdefinitionofprobabilityconvergencewhich
takes into account the first restriction. The idea is given in the following
example.
Example 1. Let F = σ({(i2−n,(i + 1)2−n],i = 0,...,2n − 1}) be
n
the σ-algebra on (0,1] generated by the dyadic subdivision. Suppose we
know that ν : F → [0,1] is the unique probability on F s.t. for any i,
n n n
ν ((i2−n,(i+1)2−n]) = 2−n. Even if ν is not defined on the Borel sets of
n n
(0,1], itisclear thatin“some”sense,itmusthappenthatν → ν ,whereν
n ∗ ∗
is theLebesgue measureon the Borel sets of (0,1]. Note that thecumulative
G. ALETTI, D.SAADA/TARGETPROBLEMS IN BLACKWELL SPACES 6
function of ν is not defined, and therefore a standard weak convergence
n
cannot be verified.
In fact, we know that
i i i
(2) ν −∞, =ν 0, = ,
n 2n n 2n 2n
(cid:16)(cid:0) (cid:3)(cid:17) (cid:16)(cid:0) (cid:3)(cid:17)
i.e., in this case, as n → ∞, we can determine the cumulative function in a
dense subset. This fact allows to hope that ν → ν in a particular sense.
n ∗
Definition 5. Let (X,X,(X ) ) be a filtered space, and set X =
n n ∞
∨ X . Let ν : X → [0,1], n ≥ 1 and ν : X → [0,1] be probability
n n n n ∞ ∞
measures. We say that ν converges totally to ν on the topological space
n ∞
w
(X,τ)asntendstoinfinity,ifν¯ −→ν (converges inweaksenseon(X,τ)),
n ∞
τ
tot
for any ν¯ :X → [0,1], such that ν¯ = ν . We write ν −→ν .
n ∞ n|Xn n n τ ∞
tot
Going back to the example, it is simple to check that ν −→ ν , where
n ∗
τ(0,1]
ν ,ν are given in Example 1 and τ(0,1] is the standard topology on (0,1].
n ∗
In fact, let (ν¯ ) be any extension of (ν ) to the Borel sets of (0,1]. For
n n n n
any t ∈ (0,1), we have by (2) that
1 1
t− ≤ F (t)≤ t+ ,
2n ν¯n 2n
where F is the cumulative function of ν¯ , which implies the weak conver-
ν¯n n
tot
gence of ν¯ to ν and, therefore, ν −→ ν .
n ∗ n ∗
τ(0,1]
For whatconcerns thetopology on X, wewill definethetopological space
(X,̺ ) induced by the pseudometric d associated to the target problem
P P
(X,F,T,P), andthepseudometricd.Inthisway̺ isdefinedonlywiththe
P
data of the problem. One may ask: is this topology too poor? The answer is
no,since it is definedby the interesting pseudometricd .In fact, d (x,y) <
P P
ǫ means that x and y play “almost the same role” with respect to T. A
direct algorithm which takes d into account needs the computation of Pn
P
at each step. In any case, even if d may not be computable, it defines a
P
nontrivial interesting topology ̺ on X. As expected, we have the following
P
theorem.
Theorem 5. Let Str (X,F,T,P) = (X,F ,T,P ), with Str target al-
n n n
gorithm. Then
tot
P −→P.
n
̺P
G. ALETTI, D. SAADA/TARGETPROBLEMS IN BLACKWELL SPACES 7
2. The target algorithm. Inthis section, weintroducethecoreof the
approximating target problem, namely a set of strategies Str which solves
the target problem.
Given a measurable space (X,X) and a target problem (X,F,T,P), the
target algorithm is built in the spirit of the exact one given in [1, 2], which
starts from the largest classes T and X \T and then reaches the optimal
classes according to a backward construction.
The target algorithm defines a strategy Str = (Str ) , where
n n
Str (X,F,T,P) = (X,F ,T,P ),
n n n
and it consists of three steps:
1. the choice of a sequence (∼ ) of equivalences on the simplex on the
ǫn n
unit ball of ℓ with ǫ → 0;
1 n
2. the definition of a filtration (F ) based on (∼ ) where each F is
n n ǫn n n
generated by a countable partition of X;
3. the choice of a suitable measure µ and the definition of (P ) .
n n
2.1. Preliminary results on measurability and equivalency, and the choice
of (∼ ) . Associated to each countably generated sub σ-algebra A ⊆ X,
ǫn n
we define the equivalence relationship π induced by the atoms of A:
A
xπ y ⇐⇒ [x] := ∩{A ∈ A: x∈ A} = ∩{A∈ A: y ∈A} =: [y] .
A A A
Thus, if (A ) is a sequence of countably generated σ-algebras, then
n n
(3) π = ∩ π .
∨nAn n An
Now,theatomsoftheσ-algebraF ofeachsimpletargetproblem(X,F,T,Q)
areatmostcountable, bydefinition.ThenQ mayberepresentedas atransi-
tion matrix onthestate setN.Each row ofQisadistributionprobability on
N (i.e. a sequence (p ) in the simplex S of ℓ ). The first step of the target
n n 1
algorithm istoequipS withtheℓ –normandthentodefineanǫ-equivalence
1
on S.
Wewillalternatively useboththediscreteequivalenciesandthecountable
measurable partitions, as a consequence of the following result, whose proof
is left to appendix.
Lemma 6. Given a measurable space (X,X), there exists a natural bi-
jection between the set of discrete equivalencies on X and the set of the
countable measurable partitions of it.
G. ALETTI, D.SAADA/TARGETPROBLEMS IN BLACKWELL SPACES 8
Let B (0,1) be the unit ball in ℓ and S = {x ≥ 0}∩B (0,1) be the
ℓ1 1 ℓ1
simplex on ℓ . Let Ω = [0,1], for any n, and τ be the standard topology on
1 n
[0,1]. Denote by B the Borel σ-algebra on [0,1] generated by τ. We look
[0,1]
at S as a subset of Π∞ Ω so that the Borel σ-algebra B induced on S is
n=1 n S
∞ B ∩S.
n=1 [0,1]
N
Definition 6. ∼ is an ǫ-equivalence on S if it is a discrete equivalence
ǫ
on (S,B ) and kp−qk < ǫ whenever p ∼ q.
S 1 ǫ
Remark 3. The choice of ℓ –norm on S is linked to the total variation
1
distancebetweenprobabilitymeasures.Thetotalvariation distancebetween
twoprobabilitymeasuresP andQisdefinedbyd (P,Q) = sup |P(A)−
TV A∈Ω
Q(A)|. Now the total variation of a measure µ is kµk(Ω) = sup |µ(A )|,
i i
where the supremum is taken over all the possible partitions of Ω. As (P −
P
Q)(Ω) = 0, we have that d (P,Q) = 1kP − Qk, see [4]. To each p ∈ S
TV 2
correspondsthe probability measure P on N with P(i) = p (and viceversa).
i
Therefore, since kp−qk = kP −Qk= 2d (P,Q), we have
1 TV
p ∼ q =⇒d (P,Q) < ǫ/2
ǫ TV
Example 2. Define the ǫ-cut as follows. p ∼ q ⇐⇒ pn =
ǫ ǫ2−n
qn ,∀n,where⌊x⌋denotestheentirepartofx.Then∼ isanǫ-ejquivaklence
ǫ2−n ǫ
ojn S.kIndeed,
• S/ ∼ is at most countable (since we divide each [0,1] into classes of
ǫ
length ǫ2−n).
• For any p ∈ S
2npn ǫ 2npn +1 ǫ
ǫ ǫ
[p]= {q ∈ S: π (q) = π (p)} = , S
∼ǫ ∼ǫ j 2n k (cid:16)j 2kn (cid:17)
n (cid:20) (cid:19)
Y \
is measurable with respect to B .
S
• ∀p∼ q,
ǫ
kp−qk ≤ ǫ2−n = ǫ.
1
n
X
2.2. The choice of (F ) . Given a sequence (∼ ) of ǫ-equivalences
n n ǫn n∈N
on S, we define the choice of (F ) inductively. This algorithm is a good
n n
candidate to be a strategy for the approximating problem we are facing and
it is based on this idea: consider the equivalence classes given by F and
n−1
divide them again according to the following rule. Starting from any two
G. ALETTI, D. SAADA/TARGETPROBLEMS IN BLACKWELL SPACES 9
points in the same class, we check whether the probabilities to attain any
other F -classes are ǫ-the same. Mathematically speaking:
n−1
Step 0 : F = σ(T) = {∅,T,X \T,X}
0
Step n : F is based on the equivalence F and on ∼ , inductively.
n n−1 ǫn
(n−1)
F is generated by a countable partition of X, say (A ) . We define,
n−1 i i
for any couple (x,y) ∈ X2,
(n−1) (n−1)
(4) (xπ y) ⇐⇒ (xπ y)∧ P(x,A ) ∼ P(y,A ) .
n n−1 i i ǫn i i
(cid:16)(cid:0) (cid:1) (cid:0) (cid:1)(cid:17)
The following Lemma 7 shows that π is a discrete equivalency on (X,X),
n
andthereforeitdefinesF = σ(X/π )asgeneratedbyacountablepartitions
n n
of X.
Remark 4. In applications, (F ) must be finitely generated. This is
n n
notabigrestriction.Infact onecan proveinductively thatthisis always the
case if the projection ∼ divides each component of S into a finite number
ǫn
of subsets, as in Example 2. The choice of the “optimal” sequence (∼ )
ǫn n
is not the scope of this work. We only note that the definition of ∼ can
ǫ
be relaxed and the choice of the sequence (∼ ) may be done interactively,
ǫn n
(n)
obtaining a fewer number of classes (A ) at each step.
i i
Lemma 7. (F ) is a filtration on (X,F). Moreover, for any n ∈ N, π
n n n
is a discrete equivalency on (X,X).
Proof. The monotonicity of (F ) is a simple consequence of (4).
n n
The statement is true for n = 0, since T ∈ X. For the induction step, let
(n−1) (n−1)
{A ,A ,...} ∈ X be the measurable countable partition of X given
1 2
(n−1)
by X/π . The map h : (X,X) → (S,B(S)) given by x 7→ (P(x,A ))
n−1 i i
is thereforemeasurable.As ∼ isadiscreteequivalency on (S,B ),themap
ǫn S
π∼ǫn ◦h : (X,X) → (S/∼ǫn,2S/∼ǫn) is also measurable, where π∼ǫn is the
natural projection associated with ∼ . Thus, two points x,y ∈ X are such
ǫn
that
(n−1) (n−1)
P(x,A ) ∼ P(y,A )
i i ǫn i i
(cid:16)(cid:0) (cid:1) (cid:0) (cid:1)(cid:17)
if and only if their image by π ◦ h is the same point of S/∼ . The
∼ǫn ǫn
new partition of X built by π is thus obtained as an intersection of the
n
(n−1)
sets A ,i ≥ 1 —which formed the π -partition— with the counter-
i n−1
imagesofS/∼ byπ ◦h.Intersectionsbetweentwomeasurablecountable
ǫn ∼ǫn
partitions of X being a measurable countable partition of X, we are done.
G. ALETTI, D.SAADA/TARGETPROBLEMS IN BLACKWELL SPACES 10
2.3. The choice of µ and the definition of (P ) . Before defining (P ) ,
n n n n
we need the following result, which will be proved in Section 5.
Theorem 8. Let (π ) be defined as in the previous section and let
n n
π = ∩ π . Then π is a compatible projection.
∞ n n ∞
As a consequence of Theorem 2 and of Theorem 8, a target problem
(X,∨ F ,T,P ) is well defined. We intend to define P as the µ–weighted
n n ∞ n
mean average of P given the information carried by F .
∞ n
More precisely, let µ be a probability measure on (X,∨ F ) such that
n n
µ(F)> 0, for any F ∈ F ,F 6= ∅ (the existence of such a measure is shown
n
in Example 3).
For any F ∈F ,let YF bethe ∨ F -random variable such that YF(ω) =
n n n
P (ω,F). Define
∞
(5) P (x,F) = E [YF|F ](x), ∀x∈ X,∀F ∈ F .
n µ n n
P is uniquely defined on (X × F ), the only µ-null set of F being the
n n n
empty set. Then we can ensure that P (x,·) is a probability measure, for
n
any x ∈X.
We give in the following an example of the measure µ that has been used
in Equation (5) which justifies its existence.
Example 3. Let (Y ) be a sequence of independent and identically
n n≥0
distributed geometric random variables, with P (j) = 1/2j,j ∈ N. Let
Yi
A = σ(Y ,··· ,Y ) and set A = ∨A . There exists a probability measure
n 0 n n
P on A such that
1
P(∩n {Y = y }) = P (y )⊗···⊗P (y ) = ,
i=0 li i Yl1 1 Yln n 2 ni=0yi
P
and thus, P(A)> 0, ∀A∈ A ,A 6=∅. Moreover, it follows that for any n,
n
(6) A ∈ A ,A ∈ σ(Y ),A 6= ∅,A 6= ∅, =⇒ P(A ∩A )> 0.
1 n 2 n+1 1 2 1 2
We check by induction that we can embed F into A , for any n ≥ 0. The
n n
searched measure µ will be the trace of P on the embedded σ-field ∨ F .
n n
For n =0,defineT 7→ {Y = 1},X\T 7→ {Y ≥ 2}.Theembeddingforms
0 0
a nontrivial partition, and therefore the restriction of P to the embedding
of F defines a probability measure on F with µ (F) > 0 if F 6= ∅.
0 0 0
