Singular Value Decomposition (SVD) - cse.iitb.ac.in

29 Pages·2015·1.47 MB·English

by  Ajit Rajwade

Singular Value Decomposition
(SVD)
CS 663
Ajit Rajwade
1
Singular value Decomposition
• For any m x n matrix A, the following
decomposition always exists:
T mn
A  USV , A  R ,
T T mm
U U  UU  I ,U R ,
m
T T nn Diagonal matrix with non-
V V  VV  I , V  R ,
n negative entries on the
diagonal – called singular
mn
S  R
values.
Columns of U are the eigenvectors of AAT (called the left singular vectors).
Columns of V are the eigenvectors of ATA (called the right singular vectors).
The non-zero singular values are the positive square roots of
2
non-zero eigenvalues of AAT or ATA.
Singular value Decomposition
• For any m x n real matrix A, the SVD consists of
matrices U,S,V which are always real – this is unlike
eigenvectors and eigenvalues of A which may be
complex even if A is real.
• The singular values are always non-negative, even
though the eigenvalues may be negative.
• While writing the SVD, the following convention is
assumed, and the left and right singular vectors are
also arranged accordingly:
   ...  
1 2 m1 m
3
Singular value Decomposition
• If only r < min(m,n) singular values are non-
zero, the SVD can be represented in reduced
form as follows:
T mn
A  USV , A R ,
mr
U  R ,
nr
V  R ,
rr
S  R
4
Singular value Decomposition
r
T  t
A  USV  S u v
ii i i
i1
This m by n matrix u vT is the product of a column vector u and
i i i
the transpose of column vector v . It has rank 1. Thus A is a
i
weighted summation of r rank-1 matrices.
Note: u and v are the i-th column of matrix U and V
i i
respectively.
5
Singular value decomposition
A  USVT
AAT  (USVT )(USVT )T  USVTVSUT  US2UT
Thus, the left singular vectors of A (i.e. columns of U) are the
eigenvectors of AAT.
The singular values of A (i.e. diagonal elements of S) are
square- roots of the eigenvalues of AAT.
ATA  (USVT )T (USVT )  VSUTUSVT  VS2VT
Thus, the right singular vectors of A (i.e. columns of V) are the
eigenvectors of ATA.
The singular values of A (i.e. diagonal elements of S) are
square- roots of the eigenvalues of ATA.
6
Application: SVD of Natural Images
• An image is a 2D array – each entry contains a
grayscale value. The image can be treated as a
matrix.
• It has been observed that for many image
matrices, the singular values undergo rapid
decay (note: they are always non-negative).
• An image can be approximated with the k
largest singular values and their corresponding
singular vectors:
k
 t
A  S u v ,k  min(m,n)
ii i i
7
i1
Singular values of the Mandrill Image: notice the rapid exponential decay of the
values! This is characteristic of MOST natural images.
Left to right, top to bottom:
Reconstructed image using the first i=
1,2,3,5,10,25,50,100,150 singular values and
singular vectors.
Last image: original
9
Left to right, top to bottom, we display:
Note: the spatial
u vT frequencies increase as
i i
the singular values
where i = 1,2,3,5,10,25,50,100,150.
decrease
Note each image is independently re-
scaled to the 0-1 range for display
purpose. 10