Table Of ContentLecture Notes on Linear Algebra
T
F
Arbind K Lal ASukant Pati
R
D
July 10, 2016
2
T
F
A
R
D
Contents
1 Introduction to Matrices 7
1.1 Definition of a Matrix . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7
1.1.A Special Matrices . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8
1.2 Operations on Matrices . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8
1.2.A Multiplication of Matrices . . . . . . . . . . . . . . . . . . . . . . . . . . . 10
1.2.B Inverse of a Matrix . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14
1.3 Some More Special Matrices . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16
T
1.3.A sub-matrix of a Matrix . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17
F
1.4 Summary . . . . . . . . . . . . . . . . . . .A . . . . . . . . . . . . . . . . . . . . . 21
R
D
2 System of Linear Equations over R 23
2.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23
2.1.A Elementary Row Operations. . . . . . . . . . . . . . . . . . . . . . . . . . 26
2.2 System of Linear Equations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28
2.2.A Elementary Matrices and the Row-Reduced Echelon Form (RREF) . . . . 28
2.2.B Rank of a Matrix . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32
2.2.C Gauss-Jordan Elimination and System of Linear Equations . . . . . . . . 34
2.3 Square Matrices and System of Linear Equations . . . . . . . . . . . . . . . . . . 41
2.3.A Determinant . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 44
2.3.B Adjugate (classically Adjoint) of a Matrix . . . . . . . . . . . . . . . . . . 48
2.3.C Cramer’s Rule . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 51
2.4 Miscellaneous Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 52
2.5 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 54
3 Vector Spaces 55
3.1 Vector Spaces: Definition and Examples . . . . . . . . . . . . . . . . . . . . . . . 55
3.1.A Subspaces . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 60
3.1.B Linear Span . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 62
3.1.C Fundamental Subspaces Associated with a Matrix . . . . . . . . . . . . . 66
3.2 Linear Independence . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 68
3.2.A Basic Results related to Linear Independence . . . . . . . . . . . . . . . . 69
3.2.B Application to Matrices . . . . . . . . . . . . . . . . . . . . . . . . . . . . 70
3
4 CONTENTS
3.2.C Linear Independence and Uniqueness of Linear Combination . . . . . . . 71
3.3 Basis of a Vector Space . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 73
3.3.A Main Results associated with Bases . . . . . . . . . . . . . . . . . . . . . 75
3.3.B Constructing a Basis of a Finite Dimensional Vector Space . . . . . . . . 76
3.4 Application to the subspaces of Cn . . . . . . . . . . . . . . . . . . . . . . . . . . 77
3.5 Ordered Bases . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 82
3.6 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 84
4 Linear Transformations 87
4.1 Definitions and Basic Properties . . . . . . . . . . . . . . . . . . . . . . . . . . . 87
4.2 Rank-Nullity Theorem . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 92
4.2.A Algebra of Linear Transformation . . . . . . . . . . . . . . . . . . . . . . . 96
4.3 Matrix of a linear transformation . . . . . . . . . . . . . . . . . . . . . . . . . . . 99
4.3.A Dual Space*. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 102
4.4 Similarity of Matrices . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 105
4.5 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 107
T
F
5 Inner Product Spaces A 109
R
5.1 Definition and Basic Properties . . . . . . . . . . . . . . . . . . . . . . . . . . . . 109
D
5.1.A Cauchy Schwartz Inequality . . . . . . . . . . . . . . . . . . . . . . . . . . 111
5.1.B Angle between two Vectors . . . . . . . . . . . . . . . . . . . . . . . . . . 112
5.1.C Normed Linear Space . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 116
5.1.D Application to Fundamental Spaces . . . . . . . . . . . . . . . . . . . . . 117
5.1.E Properties of Orthonormal Vectors . . . . . . . . . . . . . . . . . . . . . . 119
5.2 Gram-Schmidt Orthogonalization Process . . . . . . . . . . . . . . . . . . . . . . 121
5.3 Orthogonal Operator and Rigid Motion . . . . . . . . . . . . . . . . . . . . . . . 125
5.4 Orthogonal Projections and Applications. . . . . . . . . . . . . . . . . . . . . . . 127
5.4.A Orthogonal Projections as Self-Adjoint Operators* . . . . . . . . . . . . . 131
5.5 QR Decomposition . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 134
∗
5.6 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 137
6 Eigenvalues, Eigenvectors and Diagonalization 139
6.1 Introduction and Definitions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 139
6.1.A Spectrum of an eigenvalue . . . . . . . . . . . . . . . . . . . . . . . . . . . 144
6.2 Diagonalization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 147
6.2.A Schur’s Unitary Triangularization . . . . . . . . . . . . . . . . . . . . . . . 150
6.2.B Diagonalizability of some Special Matrices . . . . . . . . . . . . . . . . . . 152
6.2.C Cayley Hamilton Theorem . . . . . . . . . . . . . . . . . . . . . . . . . . . 154
6.3 Quadratic Forms . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 156
6.3.A Sylvester’s law of inertia . . . . . . . . . . . . . . . . . . . . . . . . . . . . 158
CONTENTS 5
7 Appendix 165
7.1 Permutation/Symmetric Groups . . . . . . . . . . . . . . . . . . . . . . . . . . . 165
7.2 Properties of Determinant . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 170
7.3 Uniqueness of RREF . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 174
7.4 Dimension of W +W . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 175
1 2
7.5 When does Norm imply Inner Product . . . . . . . . . . . . . . . . . . . . . . . . 176
Index 178
T
F
A
R
D
6 CONTENTS
T
F
A
R
D
Chapter 1
Introduction to Matrices
1.1 Definition of a Matrix
Definition 1.1.1.1. Matrix A rectangular array of numbers is called a matrix.
The horizontal arrays of a matrix are called its rows and the vertical arrays are called its
columns. Let A be a matrix having m rows and n coluTmns. Then, A is said to have order
m n or is called a matrix of size m n and can be repFresented in either of the following forms:
× × A
a a a R a a a
11 12 1n 11 12 1n
··· ···
D
a21 a22 a2n a21 a22 a2n
A = ... ... ·..·.· ... or A= ... ... ·.·..· ... ,
am1 am2 amn am1 am2 amn
··· ···
th th
where a is the entry at the intersection of the i row and j column. To be concise, one
ij
writes A = [a ] or, in short, A = [a ]. We will also use A[i,:] to denote the i-th row of
m n ij ij
×
A, A[:,j] to denote the j-th column of A. We shall mostly be concerned with matrices having
complex numbers as entries.
1 3+i 7 7
For example, if A = then A[1,:] = [1 3 + i 7], A[:,3] = and
"4 5 6 5i# "6 5i#
− −
a = 5. In general, in row vector commas are inserted to differentiate between entries. Thus,
22
A[1,:] = [1, 3+i, 7]. A matrix having only onecolumn is called a column vector and a matrix
with only one row is called a row vector. All our vectors will be column vectors and will be
represented by bold letters. Thus, A[1,:] is a row vector and A[:,3] is a column vector.
Definition 1.1.1.2. Equality of two Matrices Two matrices A = [a ] and B = [b ] having
ij ij
the same order m n are equal if a = b , for each i = 1,2,...,m and j = 1,2,...,n.
ij ij
×
In other words, two matrices are said to be equal if they have the same order and their
corresponding entries are equal.
Example 1.1.1.3. Thelinear system of equations 2x+3y = 5and 3x+2y = 5 can beidentified
2 3 5
with the matrix A = . Note that x and y are unknowns with the understanding
" 3 2 5 #
that x is associated with A[:,1] and y is associated with A[:,2].
7
8 CHAPTER 1. INTRODUCTION TO MATRICES
1.1.A Special Matrices
Definition 1.1.1.4. 1. A matrix in which each entry is zero is called a zero-matrix, denoted
0. For example,
0 0 0 0 0
0 = and 0 = .
2 2 2 3
× "0 0# × "0 0 0#
2. A matrix that has the same number of rows as the number of columns, is called a square
matrix. A square matrix is said to have order n if it is an n n matrix.
×
3. Let A = [a ] be an n n square matrix.
ij
×
(a) Then the entries a ,a ,...,a are called the diagonal entries the principal di-
11 22 nn
agonal of A.
(b) ThenAissaidtobeadiagonalmatrixif a = 0fori =j,denoteddiag[a ,...,a ].
ij 11 nn
6
4 0
For example, the zero matrix 0 and are a few diagonal matrices.
n
"0 1#
(c) If A = diag[a ,...,a ] and a = d for all i = 1,...,n then the diagonal matrix A
11 nn ii
T
is called a scalar matrix.
F
A
(d) Then A= diag[1,...,1] is called the identity matrix, denoted I , or in short I.
n
R
D1 0 0
1 0
For example, I2 = and I3 = 0 1 0.
"0 1#
0 0 1
(e) Then A is said to be an upper triangular matrix if a = 0 for i > j.
ij
(f) Then A is said to be a lower triangular matrix if a = 0 for i < j.
ij
(g) Then A is said to be triangular if it is an upper or a lower triangular matrix.
0 1 4 0 0 0
For example, 0 3 1 is upper triangular, 1 0 0 is lower triangular.
−
0 0 2 0 1 1
−
4. An m n matrix A = [a ] is said to have an upper triangular form if a = 0 for all
ij ij
×
a11 a12 a1n 0 0 1
···
0 a22 a2n 0 1 0 1 2 0 0 1
i > j. For example, the matrices ... ... ·.·..· ... , 0 0 2 and "0 0 0 1 1#
0 0 ··· ann 0 0 0
have upper triangular forms.
1.2 Operations on Matrices
Definition 1.1.2.1. Transpose of a Matrix Let A = [a ] be an m n matrix with real
ij
×
entries. Then the transpose of A, denoted AT = [b ], is an n m matrix with b = a , for all
ij ij ji
×
i,j.
1.2. OPERATIONS ON MATRICES 9
Definition 1.1.2.2. Conjugate Transpose of a Matrix Let A = [a ] be an m n matrix
ij
×
with complex entries. Then the conjugate transpose of A, denoted A , is an n m matrix
∗
×
with (A ) = a , for all i,j, where for a C, a denotes the complex-conjugate of a.
∗ ij ji
∈
Thus, if x is a column vector then xT and x are row vectors and vice-versa. For example, if
∗
1 0
1 4 5 1 4+i 1 0
A = then A∗ = AT = 4 1, whereas if A = then A∗ =
"0 1 2# "0 1 i# "4 i 1+i#
5 2 − −
and note that A = AT.
∗
6
Theorem 1.1.2.3. For any matrix A, (A ) = A. Thus, (AT)T = A.
∗ ∗
Proof. Let A= [a ], A = [b ] and (A ) = [c ]. Clearly, the order of A and (A ) is the same.
ij ∗ ij ∗ ∗ ij ∗ ∗
Also, by definition c = b =a = a for all i,j and hence the result follows.
ij ji ij ij
Remark 1.1.2.4. Note that transpose is studied whenever the entries of the matrix are real.
Since, we are allowing the matrix entries to be complex numbers, we will state and prove the
results for complex-conjugate. The readers should separately very that similar results hold for
transpose whenever the matrix has real entries. T
F
Definition 1.1.2.5. Addition of Matrices LetAA = [a ] and B = [b ] be two m n
ij ij
R ×
matrices. Then, the sum of A and B, denoted A+B, is defined to be the matrix C = [c ] with
D ij
c = a +b .
ij ij ij
Definition 1.1.2.6. Multiplying a Scalar to a Matrix Let A = [a ] be an m n matrix.
ij
×
Then the product of k C with A, denoted kA, is defined as kA = [ka ].
ij
∈
1 4 5 5 20 25 2+i 8+4i 10+5i
Forexample,ifA = then5A = and(2+i)A = .
"0 1 2# "0 5 10# " 0 2+i 4+2i#
Theorem 1.1.2.7. Let A,B and C be matrices of order m n, and let k,ℓ C. Then
× ∈
1. A+B = B +A (commutativity).
2. (A+B)+C = A+(B +C) (associativity).
3. k(ℓA) = (kℓ)A.
4. (k+ℓ)A = kA+ℓA.
Proof. Part 1.
Let A= [a ] and B = [b ]. Then, by definition
ij ij
A+B = [a ]+[b ] = [a +b ] = [b +a ] = [b ]+[a ]= B +A
ij ij ij ij ij ij ij ij
as complex numbers commute. The reader is required to prove the other parts as all the results
follow from the properties of complex numbers.
Definition 1.1.2.8. Additive Inverse Let A be an m n matrix.
×
10 CHAPTER 1. INTRODUCTION TO MATRICES
1. Then there exists a matrix B with A + B = 0. This matrix B is called the additive
inverse of A, and is denoted by A= ( 1)A.
− −
2. Also, for the matrix 0 , A+0 = 0+A = A. Hence, the matrix 0 is called the
m n m n
× ×
additive identity.
Exercise 1.1.2.9. 1. Find a 3 3 non-zero matrix A with real entries satisfying
×
(a) AT = A.
(b) AT = A.
−
2. Find a 3 3 non-zero matrix A with complex entries satisfying
×
(a) A = A.
∗
(b) A = A.
∗
−
3. Find the 3 3 matrix A =[a ] satisfying a = 1 if i = j and 2 otherwise.
ij ij
× 6
4. Find the 3 3 matrix A =[a ] satisfying a = 1 if i j 1 and 0 otherwise.
ij ij
× T| − | ≤
F
5. Find the 4 4 matrix A =[a ] satisfying a = i+j.
× ij ij A
R
6. Find the 4 4 matrix A =[a ] satisfying a = 2i+j.
ij D ij
×
7. Suppose A+B = A. Then show that B = 0.
8. Suppose A+B = 0. Then show that B = ( 1)A = [ a ].
ij
− −
1+i 1
− 2 3 1
9. Let A = 2 3 and B = − . Compute A+B∗ and B +A∗.
"1 1 i 2 #
i 1 −
1.2.A Multiplication of Matrices
Definition 1.1.2.10. Matrix Multiplication / Product Let A= [a ] be an m n matrix
ij
×
and B = [b ] be an n r matrix. The product of A and B, denoted AB, is a matrix C = [c ]
ij ij
×
of order m r with
×
n
c = a b = a b +a b + +a b , 1 i m, 1 j r.
ij ik kj i1 1j i2 2j in nj
··· ≤ ≤ ≤ ≤
k=1
X
Thus, AB is defined if and only if number of columns of A = number of rows of B.
α β γ δ
a b c
For example, if A= and B = x y z t then
"d e f#
u v w s
aα+bx+cu aβ +by+cv aγ +bz+cw aδ+bt+cs
AB = . (1.1.2.1)
"dα+ex+fu dβ +ey+fv dγ +ez+fw dδ+et+fs#
