Compressed sensing, sparse approximation, and low-rank matrix estimation

171 Pages·2011·1.13 MB·English

Compressed sensing, sparse approximation, and low-rank
matrix estimation
Thesis by
Yaniv Plan
In Partial Fulfillment of the Requirements
for the Degree of
Doctor of Philosophy
California Institute of Technology
Pasadena, California
2011
(Defended February 1, 2011)
ii
' 2011
Yaniv Plan
All Rights Reserved
iii
Dedicated to my wife, Jasmin
iv
Abstract
Theimportanceofsparsesignalstructureshasbeenrecognizedinaplethoraofapplicationsranging
from medical imaging to group disease testing to radar technology. It has been shown in practice
that various signals of interest may be (approximately) sparsely modeled, and that sparse modeling
is often beneficial, or even indispensable to signal recovery. Alongside an increase in applications, a
rich theory of sparse and compressible signal recovery has recently been developed under the names
compressed sensing (CS) and sparse approximation (SA). This revolutionary research has demon-
strated that many signals can be recovered from severely undersampled measurements by taking
advantage of their inherent low-dimensional structure. More recently, an offshoot of CS and SA has
been a focus of research on other low-dimensional signal structures such as matrices of low rank.
Low-rank matrix recovery (LRMR) is demonstrating a rapidly growing array of important appli-
cations such as quantum state tomography, triangulation from incomplete distance measurements,
recommender systems (e.g., the Netflix problem), and system identification and control.
Inthisdissertation,weexamineCS,SA,andLRMRfromatheoreticalperspective. Weconsider
a variety of different measurement and signal models, both random and deterministic, and mainly
ask two questions.
How many measurements are necessary? How large is the recovery error?
We give theoretical lower bounds for both of these questions, including oracle and minimax lower
bounds for the error. However, the main emphasis of the thesis is to demonstrate the efficacy of
convex optimization—in particular (cid:96) and nuclear-norm minimization based programs—in CS, SA,
1
andLRMR.Wederiveupperboundsforthenumberofmeasurementsrequiredandtheerrorderived
by convex optimization, which in many cases match the lower bounds up to constant or logarithmic
factors. The majority of these results do not require the restricted isometry property (RIP), a
ubiquitous condition in the literature.
v
Acknowledgements
Iamverygratefulfortheintelligentwayinwhichmyadvisor,EmmanuelCand`es,stokedmyinterest
first in the classes that I took from him and later through years of research. His intuitive view of
the material and his emphasis on tackling important (and difficult) problems was ideal for me. His
ability to identify such research problems and ask substantial questions about them made it much
easier for me to produce as a graduate student. I also appreciate the many mathematical concepts
he has taught me, and exposed me to, and the resources he has shown me to help me learn material
on my own.
Aside from my advisor, I would like to thank several fellow graduate students and postdocs for
their help and support, both in working through math problems and also for giving me advice on
my writing. In particular, thanks to Stephen Becker, Alex Gittens, Ewout Vandenberg, Deanna
Needell, and Mark Davenport.
Throughout my stay at Caltech, Sheila Schull and Sydney Garstang did a wonderful job in
providing information and help. It is amazing how much easier daily work is with them around.
I would like to thank Joel Tropp, Houman Owhadi, and Babak Hassibi for attending my thesis
defense. Also, thanks to Joel and Houman for the material you have taught me in class—it is quite
clearthatyoucareaboutteachingwell,andthebenefitsofthiscareareevidentintheclassroom. In
fact,asIhopetolectureclassesmyselfinthenearfuture,Iplantoincorporatesomeoftheteaching
techniques that I learned from watching you two (and from Emmanuel as well).
Finally, Iwouldliketothankmyfamilyforthemanyimplicitrolestheyhaveplayed. Thanksto
my parents for never pushing me and always encouraging my mathematical development. I believe
this is why I am self-motivated to continue work in applied math. Thanks to my brother for all of
the useful advice. And of course, thanks to my wife Jasmin for always being there, supporting my
work, moving up and down California with me, and energizing me.
vi
Contents
Abstract iv
Acknowledgements v
List of Tables x
List of Figures xi
1 Introduction 1
1.0.1 Compressed sensing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1
1.0.2 Sparse approximation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2
1.0.3 Low-rank matrix recovery . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2
1.0.4 Peek at the results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3
1.0.5 The restricted isometry property . . . . . . . . . . . . . . . . . . . . . . . . . . . 3
1.0.6 Organization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5
2 A general model for CS 7
2.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7
2.1.1 A RIP-less theory? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7
2.1.2 A general theory . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8
2.1.3 Examples of incoherent measurements . . . . . . . . . . . . . . . . . . . . . . . . 11
2.1.4 Matrix notation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13
2.1.5 Incoherent sampling theorem . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13
2.1.6 Main results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14
2.1.7 Our contribution. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16
2.1.8 Organization of the chapter . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17
2.1.9 Notation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18
2.2 Background CS literature . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18
2.2.1 Asymptotic results and phase transitions . . . . . . . . . . . . . . . . . . . . . . 18
2.2.2 Nonasymptotic results and the RIP . . . . . . . . . . . . . . . . . . . . . . . . . 19
vii
2.2.3 Null space conditions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21
2.2.4 Other algorithms for CS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23
2.3 Fundamental Estimates . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23
2.3.1 Local isometry . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23
2.3.2 Off-support incoherence . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25
2.3.3 Weak RIP. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27
2.3.4 Implications . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28
2.4 Noiseless and Sparse Recovery . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28
2.4.1 Dual certificates . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28
2.4.2 Proof of Lemma 2.4.3 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30
2.5 General Signal Recovery from Noisy Data . . . . . . . . . . . . . . . . . . . . . . . . . . 32
2.5.1 Proof of Theorem 2.1.2 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33
2.5.2 Proof of Lemma 2.5.2 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 36
2.5.3 Proof of Theorem 2.1.3 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 39
2.6 Proof of Theorem 2.3.7 (the weak RIP) . . . . . . . . . . . . . . . . . . . . . . . . . . . 40
2.6.1 Proof of Lemma 2.6.3 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 43
2.6.2 Fine scale: k≥k1 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 45
2.6.3 Coarse scale: k≤0. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 46
2.6.4 Concentration around the mean . . . . . . . . . . . . . . . . . . . . . . . . . . . 49
2.7 Stochastic Incoherence . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 49
2.8 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 52
3 Sparse approximation and model selection 53
3.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 53
3.1.1 The coherence property . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 54
3.1.2 Background literature. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 55
3.1.3 Sparse model selection . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 59
3.1.4 Exact model recovery . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 62
3.1.5 General model selection . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 63
3.1.6 Implications for signal estimation. . . . . . . . . . . . . . . . . . . . . . . . . . . 67
3.1.7 Organization of the chapter . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 69
3.2 Optimality . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 69
3.2.1 For almost all sparse models. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 69
3.2.2 For sufficiently incoherent matrices. . . . . . . . . . . . . . . . . . . . . . . . . . 72
3.3 Proofs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 74
3.3.1 Preliminaries . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 74
viii
3.3.2 Proof of Theorem 3.1.4 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 75
3.3.3 Norms of random submatrices. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 78
3.3.4 Proof of Theorem 3.1.6 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 81
3.3.5 Proof of Theorem 3.1.5 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 83
3.3.6 Proof of (3.3.23) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 85
3.4 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 88
3.4.1 Comparison to related theoretical results . . . . . . . . . . . . . . . . . . . . . . 88
4 Low-rank matrix estimation with the RIP 90
4.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 90
4.1.1 A few applications. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 91
4.1.2 Related literature . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 92
4.1.3 Problem setup . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 94
4.1.4 Algorithms . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 95
4.1.5 Organization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 96
4.1.6 Notation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 97
4.2 Main Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 97
4.2.1 Matrix RIP . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 97
4.2.2 The matrix Dantzig selector and the matrix LASSO are nearly minimax . . . 99
4.2.3 Oracle inequalities. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 101
4.2.4 Extension to full-rank matrices . . . . . . . . . . . . . . . . . . . . . . . . . . . . 105
4.3 Proofs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 106
4.3.1 Proof of Lemma 4.1.1 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 107
4.3.2 Proof of Theorem 4.2.3 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 107
4.3.3 Proof of Theorem 4.2.4 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 110
4.3.4 Proof of Theorem 4.2.4 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 113
4.3.5 Proof of Theorem 4.2.7 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 113
4.3.6 Proof of Theorem 4.2.8 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 115
4.3.7 Extension of proofs to the solution to the LASSO (4.1.5) . . . . . . . . . . . . 122
4.3.8 Proof of Theorem 4.2.5 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 123
4.3.9 Proof of Theorem 4.2.6 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 125
4.4 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 126
5 Matrix completion with noise 127
5.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 127
5.2 Exact Matrix Completion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 129
5.2.1 Geometry and dual certificates . . . . . . . . . . . . . . . . . . . . . . . . . . . . 134
ix
5.3 Stable Matrix Completion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 137
5.3.1 Proof of Theorem 5.3.1 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 139
5.3.2 Comparison with an oracle . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 141
5.4 Numerical Experiments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 142
5.5 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 146
6 Conclusion 147
Bibliography 149
x
List of Tables
5.1 RMSerror(∥Mˆ −M∥F/n)asafunctionofnwhensubsampling20%ofann×nmatrix
of rank two. Each RMS error is averaged over 20 experiments. . . . . . . . . . . . . . . 143