Approximate Message Passing Algorithms for Compressed Sensing

308 Pages·2011·3.55 MB·English

APPROXIMATE MESSAGE PASSING ALGORITHMS FOR
COMPRESSED SENSING
A DISSERTATION
SUBMITTED TO THE DEPARTMENT OF ELECTRICAL
ENGINEERING
AND THE COMMITTEE ON GRADUATE STUDIES
OF STANFORD UNIVERSITY
IN PARTIAL FULFILLMENT OF THE REQUIREMENTS
FOR THE DEGREE OF
DOCTOR OF PHILOSOPHY
Arian Maleki
September 2011
(cid:13)c Copyright by Arian Maleki 2011
All Rights Reserved
ii
I certify that I have read this dissertation and that, in my opinion, it
is fully adequate in scope and quality as a dissertation for the degree
of Doctor of Philosophy.
(Prof. David L. Donoho) Principal Adviser
I certify that I have read this dissertation and that, in my opinion, it
is fully adequate in scope and quality as a dissertation for the degree
of Doctor of Philosophy.
(Prof. Robert M. Gray)
I certify that I have read this dissertation and that, in my opinion, it
is fully adequate in scope and quality as a dissertation for the degree
of Doctor of Philosophy.
(Prof. Andrea Montanari)
Approved for the University Committee on Graduate Studies.
iii
iv
Preface
Compressed sensing refers to a growing body of techniques that ‘undersample’ high-
dimensional signals and yet recover them accurately. Such techniques make fewer
measurements than traditional sampling theory demands: rather than sampling pro-
portional to frequency bandwidth, they make only as many measurements as the
underlying ‘information content’ of those signals. However, as compared with tra-
ditional sampling theory, which can recover signals by applying simple linear recon-
struction formulas, the task of signal recovery from reduced measurements requires
nonlinear, and so far, relatively expensive reconstruction schemes. One popular class
of reconstruction schemes uses linear programming (LP) methods; there is an ele-
gant theory for such schemes promising large improvements over ordinary sampling
rules in recovering sparse signals. However, solving the required LPs is substantially
more expensive in applications than the linear reconstruction schemes that are now
standard. In certain imaging problems, the signal to be acquired may be an image
with 106 pixels and the required LP would involve tens of thousands of constraints
and millions of variables. Despite advances in the speed of LP, such methods are still
dramatically more expensive to solve than we would like. In this thesis we focus on
a class of low computational complexity algorithms known as iterative thresholding.
We study them both theoretically and empirically. We will also introduce a new
class of algorithms called approximate message passing or AMP. These schemes have
several advantages over the classical thresholding approaches. First, they take advan-
tage of the statistical properties of the problem to improve the convergence rate and
predictability of the algorithm. Second, the nice properties of these algorithms enable
us to make very accurate theoretical predictions on the asymptotic performance of
v
LPs as well. It will be shown that more traditional techniques such as coherence and
restricted isometry property are not able to make such precise predictions.
vi
Notational Conventions
Notation Description
R real numbers
RN vector space of real valued N dimensional vectors
x ith element of vector x
i
|·| if applied to a number, absolute value
|·| if applied to a set, cardinality of the set
(cid:80)
(cid:104)x,y(cid:105) x y
i i i
(cid:104)x(cid:105) 1 (cid:80)N x
N i=1 i
(cid:107)x(cid:107)p for a vector (cid:96)p-norm or (cid:96)p seminorm defined as ((cid:80)i|xi|p)p1
(cid:107)x(cid:107) (cid:96) -norm of a vector. Number of nonzero elements in x
0 0
(cid:96)∞ Banach space of all vectors in R∞ with bounded (cid:96) -norm
p p
BN(R) ball of radius R in (cid:96) space
p p
SN−1 unit (cid:96) sphere in RN
2
I indicator function
(a) a if a > 0 and zero otherwise
+
sgn(·) sign of a number
η(a;λ) soft thresholding function, i.e. sgn(a)(|a|−λ)
+
ηH(a;λ) hard thresholding function, i.e. aI(|a| > λ)
ΥN The set of all k-sparse, N-dimensional vectors
k
N (T) minimum number of elements in an (cid:15)-covering of T
(cid:15)
vii
Notation Description
n number of measurements
N dimension of original vector
k sparsity level or the number of nonzeros of the original vector
(cid:113)
(cid:80)
(cid:107)A(cid:107) Frobenius norm or Hilbert Schmidt norm of a matrix: A2
F i,j i,j
(cid:107)A(cid:107) operator norm of a matrix from (cid:96) to (cid:96)
p→q p q
A∗ transpose or adjoint or hermitian of matrix A
A ith column of matrix A
i
A ijth element of matrix A
ij
A submatrix of A with columns restricted to set J
J
σ (A) maximum singular value of matrix A
max
σ (A) minimum singular value of matrix A
min
I identity matrix of size n
n
P(·) probability of an event
E(·) expected value of a random variable
(cid:107)X(cid:107)p for a random variable, [E(Xp)]p1
N(µ,Σ) Gaussian with mean µ and covariance Σ
φ(z) e−√z2/2
2π
Φ(z) (cid:82)z e−√t2/2dt
−∞ 2π
logit(π) log( π )
1−π
F family of distributions with F(0+)−F(0−) > 1−(cid:15), E (µ2) ≤ (cid:15)γ2
(cid:15),γ F
viii
Acknowledgements
During my PhD I benefited significantly from interaction with the people at Stanford.
I would like to take this opportunity to thank them.
First, I wish to express my appreciation and gratitude to my advisor, Professor David
L. Donoho. Dave’s thoughtful discussions and careful criticisms have greatly influ-
enced my thought in the last several years. His vision and creativeness has inspired
this research. His high scientific standards and his enthusiasm for solving seemingly
unsolvable problems were the main deriving forces that made this thesis possible. I
am also indebted to Dave for his endless patience, generous support, and constant
guidance.
I have been fortunate to have enjoyed the support and collaboration of Professor
Gunnar Carlsson. Gunnar was the first person who introduced me to the scientific
world and gave me strong motivation, support, and guidance. He spent much time
to teach me valuable lessons in both research and life. I owe great thanks to him.
This work could not be possible without the collaboration with Professor Andrea
Montanari. Discussing a problem with Andrea has been always a humbling but eye-
opening experience. His modesty has given me the courage to knock the door of his
office whenever I need his expertise. Thanks Andrea!
I would also like to thank my associate advisor Professor Robert M. Gray. I doubt
that anyone who interacts with Professor Gray can fail to be impressed by his tremen-
dous intellectual energy. His big pictures and his vast knowledge of the literature in
information theory, compression, image, and audio processing has always helped me
in choosing research directions. I would also like to thank Professor Gray for his
valuable comments and feedback on this thesis.
ix
My appreciation also goes to Professor Iain Johnstone for both chairing on my com-
mittee and teaching me statistics. I have taken two courses, ‘Theory of statistics’ and
‘Function estimation in noise’, with Iain and they were two of the best-taught courses
I have taken at Stanford.
I wish to express my appreciation to Professor Bala Rajaratnam. Bala’s high mo-
tivation and energy for doing research and exploring new topics have been always
inspiring to me. I have also benefited from his vast knowledge in statistics.
Heather, Helen, Angie, and Nora have always been there to help me with the admin-
istrative tasks. I am grateful to them.
I was very fortunate to have many good friends around me at Stanford: Morteza,
Nima, Rostam, Navid, Amir, Bita, Vahbod, Shadi, Ali, Maryam, Rahul, Satish,
Behrad, Neda, Maryam, Ian, Kjartan, Marie, Mortan, Yaniv, Sewoong, Mohammad,
Tahereh, Paul, Sam, Shahriar, Hossein, Mohsen. Thank you all!
I would also like to take this opportunity to thank my parents for their unconditional
love and support. They have always believed in me, more than I myself do and have
been fully supportive of all my decisions. I would like to dedicate this thesis to them
for their endless love, support, and self-sacrifices.
If it was not for the experiences my sister Atoosa and my brother Arash passed on to
me, I could not be here. I would like to thank them for their help, love, and support.
Finally my deepest gratitude goes to my wife, Shirin. She is not only the love of
my life, but also my favorite classmate, my best friend, and my best collaborator.
Every time I have an idea, Shirin is the first person who reviews it and gives valu-
able feedback and suggestions. Without her endless encouragement and indispensable
support, I could never be even close to the end of my PhD journey. I hope I can be
as encouraging and as supportive for her as she has been for me. Stanford University
will always be one of my most favorite places in the world not only because I spent
several years here, but also because every place here reminds me of a good memory
Shirin and I have had together. I hope the nonstop construction work at Stanford
leaves some of these places untouched so that we can come back and refresh our sweet
memories in the future.
x