Table Of ContentAN ABS ALGORITHM FOR A CLASS OF SYSTEMS OF
STOCHASTIC LINEAR EQUATIONS
Hai-Shan Han1, Antonino Del Popolo2 and Zun-Quan Xia3
9
0
0
2
Abstract. This paper is to explore a model of the ABS Algorithms dealing with
n
a
the solution of a class of systems of linear stochastic equations Aξ = η when η
J
6 is a m-dimensional normal distribution. It is shown that the stepsize α is dis-
i
2
tributed as N(u ,σ ) (being u the expected value of α and σ its variance) and
i i i i i
]
M the approximation to the solutions ξ is distributed as N (U ,Σ ) (being U the
i n i i i
I expected value of ξ and Σ its variance) , for this algorithm model.
. i i
h
p
- Key words: ABS algorithm, stochastic linearly system of equations, distribu-
o
r tion, probability.
t
s
a AMS Subject Classification (2000): 60H35 65H10 65F10.
[
1
v
6 1 Introduction
3
0
4
Since the early fifties, the theory of stochastic equations, and in particular stochastic
.
1
0 integral equations, has been the stage for intense activity. This has been stimulated
9
0 in part by an immense wealth of applications beginning with the Langevin equation
:
v
in statistical mechanics and Wiener’s emphasis on the role of randomness in the
i
X
problems of cybernetics. Applications of random equations are now found in many
r
a
areas of engineering, physical and biological sciences and systems theory.
Historically the study of random equations (stochastic integral equations, stochastic
operator equations, stochastic difference equations, etc.) started with that of integral
equations. The theory of the last has grown along two parallel branches. In one,
1CORA, Department of Applied Mathematics, Dalian University of Technology Dalian 116024,
China; Department of Mathematics,Inner Mongolia University of Nationalities,Tongliao 028000
China
2Dept. of Mathematics, University of Bergamo, via dei Caniana 2, 24127 Bergamo,Italy;
Istanbul Technical University, Ayazaga Campus, Faculty of Science and Letters, 34469
Maslak/Istanbul, Turkey
3CORA, Department of Applied Mathematics, Dalian University of Technology Dalian 116024,
China, [email protected]
initiated by K. Ito in 1951, the source of randomness is a white noise term leading
to an important class of stochastic integral equations which has developed hand in
hand with the theory of Markov processes. In the other, classical linear and nonlinear
integralequationswithrandomright-handsides,randomkernelsordefinedonrandom
domains are studied.
The application of stochastic equations is more extensive than deterministic equa-
tions, because random factors are usually included in many natural phenomena.
The study of random linear equations can at least be traced back to the works of
[Bellman 54], [Rosenbloom 55], [Kalman 62]. Random linear difference equations
arises naturally in many areas of engineering sciences including automatic control
and signal processing, and have drawn much attention from researchers in the past
several decades [Brandt 86].
Attempts were made by many scientists and mathematicians to develop and unify the
theory of random equations employing concepts and methods of probability theory
and functional analysis. The Prague School of probabilists under Spacek initiated a
systematic studyusing probabilisticoperatorequations asmodels forvarioussystems.
This development was further energized by the survey article by [BhRe 72] on various
treatments of random equations under the framework of functional analysis.
Several techniques have been proposed to solve stochastic linear equations (see for
instance [Brandt 1986], [Verma 1996]). Stability problems of random linear equations
have also been studied in [Guo 90], [Guo 93], [GuoLj 95].
An approach that has never been tried to solve stochastic linear equations is through
the use of ABS methods, introduced by Abaffy, Broyden and Spedicato in 1984 in
the form of a large class of methods for solving linear determined or underdetermined
systems. The original formulation of 1984 has been later generalized giving a larger
class of methods that essentialy contains all possible methods building a sequence
of approximations x to the solution having the following property: for x arbitrary,
i 1
the vector x solves the given system. ABS methods have been applied the least
m+1
squares problem, to non-linear equations, to linear and non-linear programming and
eigenvalue problems.
Before showing how the ABS algorithm can be used to solve a system of stochastic
2
equations of the form
Aξ(w) = η(w) (1.1)
where ξ(w), η(w) are random vectors, we recall the ABS algorithm.
Consider the general linear system,
Ax = b or aTx = b , i = 1,···,m (1.2)
i i
where x ∈ IRm, b ∈ IRn, m ≤ n and A ∈ IRm×n, A = (a ,a ,···,a )T.
1 2 m
The class of ABS algorithms to solve Eq. (1.2) was originally introduced by Abaffy,
Broyden and Spedicato (1984), see [AbBs84] and [AbSp89]. The iterate scheme of
the basic ABS class of algorithm is defined as follows:
Basic (unscaled) ABS Class of Algorithms: [AbBs84], [AbBs89]:
(A) Initialization.
Give an arbitrary vector x ∈ IRn, and an arbitrarily nonsingular matrix H ∈
1 1
Rn,n. Set i = 1.
(B) Compute s = H a and τ = τTe = aTx −bTe
i i i i i i i i
(C) Check the compitability of the system of linear equations.
If s 6= 0 then goto (D).
i
If s = 0 and τ = 0 then set x = x , H = H and goto (F), the i-th
i i i+1 i i+1 i
equation is a linear combination of the previous equations. Otherwise stop, the
system has no solution.
(D) Compute the search vector p ∈ IRn by p = HTz where z , the parameter of
i i i i i
Broyden, is arbitrary satisfying the condition zTH a 6= 0
i i i
(E) Update the approximation of the solution x by x = x − α p , where the
i i+1 i i i
stepsize α is computed by α = τ /aTp If i = m stop; x solves the system.
i i i i i m+1
(F) Update the (Abaffian) matrix H . Compute H = H − H a wTH /wTH a
i i+1 i i i i i i i i
where w ∈ IRn, the parameter of Abaffy, is arbitrary satisfying the condition
i
wTH a = 1 or 6= 0
i i i
3
(G) Increment the index i by one and goto (B).
We define n by i matrices A , W and P by
i i i
A = (a , ···, a )T, W = (w , ···, w ), P = (p , ···, p )
i 1 i i 1 i i 1 i
Somepropertiesoftheabove recursion(see forinstance, AbaffyandSpedicato (1989),
[AbSp 89]), that are the basic formulae to be used later on, are listed below .
a. Implicit factorization property ATP = L with L nonsingular lower triangular.
i i i i
b. Null space characterizations
N(H ) = R(AT), N(HT ) = R(W ),
i+1 i i+1 i
N(A ) = R(HT )
i i+1
where N= Null and R=Range.
c. The linear variety containing all solutions to Ax = b consists of the vectors of
the form x = x +HT q where q ∈ IRn is arbitrary.
i+1 i+1
As we shall show in the rest of the paper, this method can be used to solve a linear
stochastic system in the following way. For a linear system
Aξ = η
with a m × n matrix of arbitrarary rank with m ≤ n, the so-called ABS algorithm
produces a solution
ξ = AIη,
with AI some generalised inverse. Now it is quite clear that if η is a normally dis-
tributedvector, soisξ,orinotherwordsintheoriginalsystem therearealsonormally
distributed vector so is ξ as solutions. The mean of ξ can obviuosly be computed from
the mean of η in the usual deterministic way (save for a component in the kernel of
A). So the only interesting question left is how to compute the covariance of ξ from
that of η. Formally it is AIV(AI)T if the covariance of η is V . The paper gives a re-
cursive formula through the ABS algorithm for the special case that the components
of η are uncorrelated, i. e. V = I.
4
This paper is organized as follows. In Section 2, some basic definitions and opera-
tions that will be used below are presented. A basic ABS algorithm, called ABS-S for
stochastic linear equations is given in Section 3. Some properties of the approxima-
tions to solutions and α , are given in section 4. In section 5, the ABS-S algorithm
i
is illustrated by an example. In section 6, α is discussed and same results are also
i
given.
2 Preliminaries
Consider a class of systems of stochastic linear equations being of the form
Aξ(w) = η(w) (2.1)
In this paper, we present an ABS algorithm under the condition that η(w) has a
m-dimensional normal distribution. It will be shown that the iteration step length
α ∼ N(u ,σ ) (being u the expected value of α and σ its variance) and the iteration
i i i i i i
solution ξ is N (U ,Σ ), i=1,2,...m+1.
i n i i
We show that (u ,σ ) and (U ,Σ ) are determined by four iterative formulae, that ξ
i i i i i+1
is a solution of the first i equations and we discuss the step length α .
i
Definition 2.1[GoMi84] Let (Ω,A,P) be a probability space and let ξ: Ω → R be
a real-valued function on Ω. We say that ξ is a random variable on Ω iff for every
x ∈ R, {ω : ξ(ω) ≤ x} ∈ A.
Definition 2.2 If a random variable ξ has a normal distribution with expectation µ
and variance σ2, then it is denoted by ξ ∼ N(µ,σ2).
Definition 2.3[GoMi84] η = (η ,η ,....,η ) has a joint N(µ,Σ) density iff there ex-
1 2 m
ists an m×n matrix A(m ≤ n) of rank m such that η = Aξ+µ and ξ = (ξ ,ξ ,....,ξ )
1 2 n
is a vector of independent N(0,1) random variables. Equivalently ξ is a N(0,I ) mul-
n
tiple normal density.
5
Definition 2.4 A random variable of random distribution ξ is said to be equal to the
random variable of random distribution η, denoted by ξ = η, if the distributions of ξ
and η are the same and Eξ=Eη and Dξ=Dη, where E and D stands for expectation
value and variance, respectively.
Definition 2.5 Consider a system of random linear equations Aξ=η. Suppose that
η has a certain distribution. If the distribution of ξ, denoted by ξ , is given, then ξ
0 0
is a solution of equations Aξ=η.
Property 2.1[GoMi84] If ξ = η +a, a ∈ R, then Eξ=Eη+a, Dξ=Dη.
Property 2.2[Ande 84] If (ξ,η) is a 2-dimensional normal distribution, then aξ+bη
is also a normal distribution, where a and b are constants.
3 ABS-S Algorithm to solve System of Stochastic
Linear Equations
Consider a system of stochastic linear equations
Aξ = η (3.1)
whereη = (η ,η ,···,η )T isam-dimensionalstochasticvectorandA = (a ,a ,···,a )T ∈
1 2 m 1 2 m
Rm,n
The ABS-S algorithm, based on the basic ABS algorithm, is defined, as follows.
ABS-S Algorithm to solve systems of stochastic linear equations
(A1) Initialization.
Give an arbitrary stochastic vector ξ ∈ IRn, and an arbitrarily nonsingular
1
matrix H ∈ Rn,n. Set i = 1
1
(B1) Compute the two quantities:
6
s = H a ,τ = aTξ −η
i i i i i i i
(C1) Check the compatibility of the system of linear equations.
If s 6= 0 then goto (D).
i
If s = 0 and Prob.(τ = 0) = 1 then set
i i
ξ = ξ
i+1 i
H = H
i+1 i
andgoto(F),thei-thequationisalinearcombinationofthepreviousequations.
Otherwise stop, the system has no solution.
(D1) Compute the search vector p ∈ IRn by p = HTz where z , the parameter of
i i i i i
Broyden, is arbitrary satisfying the condition
zTH a 6= 0
i i i
(E1) Update the random approximation ξ of a solution by
i
ξ = ξ −α p
i+1 i i i
where the stepsize α is computed by
i
τ
i
α =
i aTp
i i
If i = m stop; ξ solves the system(3.1).
m+1
(F1) Update the ( Abaffian ) matrix H . Compute
i
H aTw H
H = H − i i i i
i+1 i wTH a
i i i
where w ∈ IRn, the parameter of Abaffy, is arbitrary satisfying the condition
i
wTH a = 1 or 6= 0
i i i
(G1) Increment the index i by one and go to (B1).
7
4 The properties of α and ξ
i i
Let η ∼ N (v,I ) where η = (η ,η ,.....,η )T and v = (v ,v ,....,v )T
m m 1 2 m 1 2 m
Proposition 4.1
τ = aTξ −η ∼ N(aTξ −v ,1)
1 1 1 1 1 1 1
τ aTξ −v 1
1 1 1 1
α = ∼ N( , )
1 aTp aTp (aTp )2
1 1 1 1 1 1
Proof. From η ∼ N (v,I ), we know that η ∼ N(v ,1),i = 1,2,...,m. By
m m i i
property 2.2, τ has a normal distribution and η ∼ N(v,I ), η ∼ N(v ,1),i =
1 m i i
1,2,...,m
Eτ = E(aTξ −η ) = aTξ −v ;
1 1 1 1 1 1 1
Dτ = D(aTξ −η ) = 1;
1 1 1 1
u = Eα = E((aTp )−1τ ) = (aTp )−1(aTξ −v );
1 1 1 1 1 1 1 1 1 1
σ = Dα = D((aTp )−1τ ) = (aTp )−2.
1 1 1 1 1 1 1
Therefore
α = (aTp )−1τ ∼ N(u ,σ )
1 1 1 1 1 1
τ = aTξ −η ∼ N(Eτ ,Dτ )
1 1 1 1 1 1
The proof is completed.
Proposition 4.2 If ξ is generated by the ABS-S algorithm, then
2
aTξ −v p pT
1 1 1 1 1
ξ ∼ N (ξ − p , )
2 n 1 aTp 1 (aTp )2
1 1 1 1
Proof. By using Proposition 4.1, we have that α ∼ N(Eα ,Dα ). By property
1 1 1
2.2, since ξ = ξ −α p has a normal distribution and
2 1 1 1
U = Eξ = E(ξ −α p ) = E(ξ )−E(α p ) = ξ −(aTp )−1(aTξ −v )p
2 2 1 1 1 1 1 1 1 1 1 1 1 1 1
Σ = Dξ = D(ξ −α p ) = (aTp )−2p pT
2 2 1 1 1 1 1 1 1
8
it follows that
ξ ∼ N(U ,Σ )
2 2 2
The proof is completed.
Proposition 4.3 a p = 0, i < j
i j
Proof. see [AbBs 84], [AbSp 89]
Proposition 4.4 If τ is generated by the ABS-S algorithm, then τ has a normal
i i
distribution, and
τ ∼ N(aTξ −aTΣi−1(Eα )p −v ,1+aTΣi−1 cov(α p ,α p )a ),i ≥ 2
i i 1 i j=1 j j i i j,k=1 j j k k i
where
cov(α p ,α p ) = ρ pTp pTp D(α )D(α )
j j k k jk j j k k j k
where ρ is the correlation coefficient of α p and α p
jk j j k k
Proof. By the ABS-S algorithm, we have that τ = aTξ − η , ξ = ξ − α p
i i i i i+1 i i i
therefore
τi = aTi ξi −ηi = aTi (ξi−1 −αi−1pi−1)−ηi = ... = aTi (ξ1 −Σij−=11(αjpj))−ηi
Because α , j < i has a normal distribution, property 2.2, η ∼ N(v ,1) hence τ has
j i i i
a normal distribution, and
Eτ = E(aT(ξ −Σi−1(α p ))−η ) = aT(ξ −Σi−1(Eα )p )−v
i i 1 j=1 j j i i 1 j=1 j j i
Dτ = D(aT(ξ −Σi−1(α p ))−η ) = aT[Σi−1 cov(α p ,α p )]a +1.
i i 1 j=1 j j i i j,k=1 j j k k i
The proof is completed.
Proposition 4.5 If α is generated by the ABS-S algorithm, then
i
aTξ −aTΣi−1(Eα )p −v aT[Σi−1 cov(α p ,α p )]a +1
i 1 i j=1 j j i i j,k=1 j j k k i
α ∼ N( , ),i ≥ 2
i aTp (aTp )2
i i i i
Proof. Since α = (aTp )−1τ and property 2.2, we have that the proposition holds.
i i i i
The proof is completed.
9
Theorem 4.1 If ξ is generated by the ABS-S algorithm, then
i+1
ξ ∼ N (ξ −Σj=i(Eα )p ,Σi cov(α p ,α p ))
i+1 n 1 j=1 j j j,k=1 j j k k
or
ξ ∼ N (Eξ −(Eα )p ,Dξ +p D(α )pT −2cov(ξ ,α p )),i ≥ 2
i+1 n i i i i i i i i i i
j=i−1
Proof. Since ξi = ξi−1−αi−1pi−1 = ξi−2−αi−2pi−2−αi−1pi−1 = ... = ξ1−Σj=1 αjpj,
and α , j = 1,2,...,i−1 have a normal distribution, we have that ξ has a normal
j i
distribution and
Eξ = E(ξ −Σi−1(α p ) = ξ −Σi−1E(α p )
i 1 j=1 j j 1 j=1 j j
Dξ = D(ξ −Σj=i−1(α )p ) = Σi−1 cov(α p ,α p )
i 1 j=1 j j j,k=1 j j k k
in terms of property 2.2. The proof is completed.
Theorem 4.2 ξ is the solution of the first i equations
i+1
Proof. Since ξ = ξ − α p , one has that aTξ = aT(ξ − α p ) = η . If l < i,
i+1 i i i i i+1 i i i i i
then one has that aTξ = aT(ξ −α p ) = aTξ . Therefore, E(aTξ ) = E(aTξ ) and
l i+1 l i i i l i l i+1 l i
D(aTξ ) = D(aTξ ), one has
l i+1 l i
E(aTξ ) = E(aTξ ) = ... = E(aTξ ) = v ,
l i+1 l i l l l
D(aTξ ) = D(aTξ ) = ... = D(aTξ ) = ... = D(aTξ )
l i+1 l i l l 1 1
The proof is completed.
Proposition 4.6 The ABS-S algorithm terminates in finite steps.
Proof. The assertion comes directly from Definition (2.5), i.e. according to the
definition of a solution of the system of the stochastic linear equations Aξ = η. The
proof is completed.
10
