Matrix Differential Calculus With Applications in Statistics and Econometrics, Third Edition (Wiley Series in Probability and Statistics)

470 Pages·2007·2.21 MB·English

by  Jan R. Magnus

Matrix Differential Calculus
with Applications in Statistics
and Econometrics
WILEY SERIES IN PROBABILITY AND STATISTICS
Established by WALTER E. SHEWHART AND SAMUEL S. WILKS
Editors: Vic Barnett, Noel A. C. Cressie, Nicholas, I. Fisher,
Iain M. Johnstone, J. B. Kadane, David, G. Kendall, David W. Scott,
Bernard W. Silverman, Adrian F. M. Smith, Jozef L. Teugels
Editors Emeritus: Ralph A. Bradley, J. Stuart Hunter
A complete list of the titles in this series appears at the end of this volume
Matrix
Differential
Calculus
with Applications
in Statistics
and Econometrics
Third Edition
JAN R. MAGNUS
CentER, Tilburg University
and
HEINZ NEUDECKER
Cesaro, Schagen
JOHN WILEY & SONS
Chichester • New York • Weinheim • Brisbane • Singapore • Toronto
Copyright(cid:13)c1988,1999JohnWiley&SonsLtd,
BaffinsLane,Chichester,
WestSussexPO191UD,England
National01243779777
International ([phone]
Copyright(cid:13)c1999oftheEnglishandRussianLATEXfileCentER,TilburgUniversity,
P.O.Box90153,5000LETilburg,TheNetherlands
Copyright(cid:13)c2007oftheThirdEditionJanMagnusandHeinzNeudecker.Allrightsreserved.
Publication dataforthe second (revised) edition
LibraryofCongressCataloging in PublicationData
Magnus,JanR.
Matrixdifferentialcalculuswithapplicationsinstatisticsand
econometrics/J.R.MagnusandH.Neudecker—Rev.ed. p. cm.
Includesbibliographicalreferencesandindex.
ISBN0-471-98632-1(alk.paper);ISBN0-471-98633-X(pbk:alk.paper)
1.Matrices. 2.DifferentialCalculus. 3.Statistics.
4.Econometrics. I.Neudecker,Heinz. II.Title.
QA188.M345 1999
512.9′434—dc21 98-53556
CIP
British LibraryCataloguing inPublicationData
AcataloguerecordforthisbookisavailablefromtheBritishLibrary
ISBN0-471-98632-1; 0-471-98633-X(pbk)
Publication dataforthe third edition
Thisisversion07/01.
Lastupdate:16January2007.
Contents
Preface . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xiii
Part One — Matrices
1 Basic properties of vectors and matrices 3
1 Introduction. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3
2 Sets . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3
3 Matrices: addition and multiplication . . . . . . . . . . . . . . . 4
4 The transpose of a matrix . . . . . . . . . . . . . . . . . . . . . 6
5 Square matrices . . . . . . . . . . . . . . . . . . . . . . . . . . . 6
6 Linear forms and quadratic forms . . . . . . . . . . . . . . . . . 7
7 The rank of a matrix . . . . . . . . . . . . . . . . . . . . . . . . 8
8 The inverse . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9
9 The determinant . . . . . . . . . . . . . . . . . . . . . . . . . . 10
10 The trace . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11
11 Partitioned matrices . . . . . . . . . . . . . . . . . . . . . . . . 11
12 Complex matrices . . . . . . . . . . . . . . . . . . . . . . . . . 13
13 Eigenvalues and eigenvectors . . . . . . . . . . . . . . . . . . . 14
14 Schur’s decomposition theorem . . . . . . . . . . . . . . . . . . 17
15 The Jordan decomposition . . . . . . . . . . . . . . . . . . . . . 18
16 The singular-value decomposition . . . . . . . . . . . . . . . . . 19
17 Further results concerning eigenvalues . . . . . . . . . . . . . . 20
18 Positive (semi)definite matrices . . . . . . . . . . . . . . . . . . 23
19 Three further results for positive definite matrices . . . . . . . 25
20 A useful result . . . . . . . . . . . . . . . . . . . . . . . . . . . 27
Miscellaneous exercises. . . . . . . . . . . . . . . . . . . . . . . . . . 27
Bibliographical notes . . . . . . . . . . . . . . . . . . . . . . . . . . . 29
2 Kronecker products, the vec operator and the Moore-Penrose inverse 31
1 Introduction. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31
2 The Kronecker product . . . . . . . . . . . . . . . . . . . . . . 31
3 Eigenvalues of a Kronecker product . . . . . . . . . . . . . . . . 33
4 The vec operator . . . . . . . . . . . . . . . . . . . . . . . . . . 34
5 The Moore-Penrose(MP) inverse . . . . . . . . . . . . . . . . . 36
6 Existence and uniqueness of the MP inverse . . . . . . . . . . . 37
v
vi Contents
7 Some properties of the MP inverse . . . . . . . . . . . . . . . . 38
8 Further properties . . . . . . . . . . . . . . . . . . . . . . . . . 39
9 The solution of linear equation systems . . . . . . . . . . . . . 41
Miscellaneous exercises. . . . . . . . . . . . . . . . . . . . . . . . . . 43
Bibliographical notes . . . . . . . . . . . . . . . . . . . . . . . . . . . 45
3 Miscellaneous matrix results 47
1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 47
2 The adjoint matrix . . . . . . . . . . . . . . . . . . . . . . . . . 47
3 Proof of Theorem 1. . . . . . . . . . . . . . . . . . . . . . . . . 49
4 Bordered determinants . . . . . . . . . . . . . . . . . . . . . . . 51
5 The matrix equation AX =0 . . . . . . . . . . . . . . . . . . . 51
6 The Hadamard product . . . . . . . . . . . . . . . . . . . . . . 53
7 The commutation matrix K . . . . . . . . . . . . . . . . . . 54
mn
8 The duplication matrix D . . . . . . . . . . . . . . . . . . . . 56
n
9 Relationship between D and D , I . . . . . . . . . . . . . . 58
n+1 n
10 Relationship between D and D , II . . . . . . . . . . . . . . 60
n+1 n
11 Conditions for a quadratic form to be positive (negative) sub-
ject to linear constraints . . . . . . . . . . . . . . . . . . . . . . 61
12 Necessary and sufficient conditions for r(A:B)=r(A)+r(B) 64
13 The bordered Gramian matrix . . . . . . . . . . . . . . . . . . 66
14 The equations X A+X B′ =G ,X B =G . . . . . . . . . . 68
1 2 1 1 2
Miscellaneous exercises. . . . . . . . . . . . . . . . . . . . . . . . . . 71
Bibliographical notes . . . . . . . . . . . . . . . . . . . . . . . . . . . 71
Part Two — Differentials: the theory
4 Mathematical preliminaries 75
1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 75
2 Interior points and accumulation points . . . . . . . . . . . . . 75
3 Open and closed sets . . . . . . . . . . . . . . . . . . . . . . . . 76
4 The Bolzano-Weierstrasstheorem . . . . . . . . . . . . . . . . . 79
5 Functions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 80
6 The limit of a function . . . . . . . . . . . . . . . . . . . . . . . 81
7 Continuous functions and compactness . . . . . . . . . . . . . . 82
8 Convex sets . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 83
9 Convex and concave functions . . . . . . . . . . . . . . . . . . . 85
Bibliographical notes . . . . . . . . . . . . . . . . . . . . . . . . . . . 88
5 Differentials and differentiability 89
1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 89
2 Continuity . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 89
3 Differentiability and linear approximation . . . . . . . . . . . . 91
4 The differential of a vector function. . . . . . . . . . . . . . . . 93
5 Uniqueness of the differential . . . . . . . . . . . . . . . . . . . 95
6 Continuity of differentiable functions . . . . . . . . . . . . . . . 96
7 Partial derivatives . . . . . . . . . . . . . . . . . . . . . . . . . 97
Contents vii
8 The first identification theorem . . . . . . . . . . . . . . . . . . 98
9 Existence of the differential, I . . . . . . . . . . . . . . . . . . . 99
10 Existence of the differential, II . . . . . . . . . . . . . . . . . . 101
11 Continuous differentiability . . . . . . . . . . . . . . . . . . . . 103
12 The chain rule . . . . . . . . . . . . . . . . . . . . . . . . . . . 103
13 Cauchy invariance . . . . . . . . . . . . . . . . . . . . . . . . . 105
14 The mean-value theorem for real-valued functions . . . . . . . . 106
15 Matrix functions . . . . . . . . . . . . . . . . . . . . . . . . . . 107
16 Some remarks on notation . . . . . . . . . . . . . . . . . . . . . 109
Miscellaneous exercises. . . . . . . . . . . . . . . . . . . . . . . . . . 110
Bibliographical notes . . . . . . . . . . . . . . . . . . . . . . . . . . . 111
6 The second differential 113
1 Introduction. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 113
2 Second-order partial derivatives . . . . . . . . . . . . . . . . . . 113
3 The Hessian matrix. . . . . . . . . . . . . . . . . . . . . . . . . 114
4 Twice differentiability and second-order approximation,I . . . 115
5 Definition of twice differentiability . . . . . . . . . . . . . . . . 116
6 The second differential . . . . . . . . . . . . . . . . . . . . . . . 118
7 (Column) symmetry of the Hessian matrix . . . . . . . . . . . . 120
8 The second identification theorem . . . . . . . . . . . . . . . . 122
9 Twice differentiability and second-order approximation,II . . . 123
10 Chain rule for Hessian matrices . . . . . . . . . . . . . . . . . . 125
11 The analogue for second differentials . . . . . . . . . . . . . . . 126
12 Taylor’s theorem for real-valued functions . . . . . . . . . . . . 128
13 Higher-order differentials. . . . . . . . . . . . . . . . . . . . . . 129
14 Matrix functions . . . . . . . . . . . . . . . . . . . . . . . . . . 129
Bibliographical notes . . . . . . . . . . . . . . . . . . . . . . . . . . . 131
7 Static optimization 133
1 Introduction. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 133
2 Unconstrained optimization . . . . . . . . . . . . . . . . . . . . 134
3 The existence of absolute extrema . . . . . . . . . . . . . . . . 135
4 Necessary conditions for a local minimum . . . . . . . . . . . . 137
5 Sufficient conditions for a local minimum: first-derivative test . 138
6 Sufficient conditions for a local minimum: second-derivative test140
7 Characterizationof differentiable convex functions . . . . . . . 142
8 Characterizationof twice differentiable convex functions . . . . 145
9 Sufficient conditions for an absolute minimum . . . . . . . . . . 147
10 Monotonic transformations . . . . . . . . . . . . . . . . . . . . 147
11 Optimization subject to constraints . . . . . . . . . . . . . . . . 148
12 Necessary conditions for a local minimum under constraints . . 149
13 Sufficient conditions for a local minimum under constraints . . 154
14 Sufficient conditions for an absolute minimum under constraints158
15 A note on constraints in matrix form . . . . . . . . . . . . . . . 159
16 Economic interpretation of Lagrange multipliers. . . . . . . . . 160
Appendix: the implicit function theorem . . . . . . . . . . . . . . . . 162
viii Contents
Bibliographical notes . . . . . . . . . . . . . . . . . . . . . . . . . . . 163
Part Three — Differentials: the practice
8 Some important differentials 167
1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 167
2 Fundamental rules of differential calculus . . . . . . . . . . . . 167
3 The differential of a determinant . . . . . . . . . . . . . . . . . 169
4 The differential of an inverse . . . . . . . . . . . . . . . . . . . 171
5 Differential of the Moore-Penroseinverse . . . . . . . . . . . . . 172
6 The differential of the adjoint matrix . . . . . . . . . . . . . . . 175
7 On differentiating eigenvalues and eigenvectors . . . . . . . . . 177
8 The differential of eigenvalues and eigenvectors:symmetric case 179
9 The differential of eigenvalues and eigenvectors:complex case . 182
10 Two alternative expressions for dλ . . . . . . . . . . . . . . . . 185
11 Second differential of the eigenvalue function . . . . . . . . . . 188
12 Multiple eigenvalues . . . . . . . . . . . . . . . . . . . . . . . . 189
Miscellaneous exercises. . . . . . . . . . . . . . . . . . . . . . . . . . 189
Bibliographical notes . . . . . . . . . . . . . . . . . . . . . . . . . . . 192
9 First-order differentials and Jacobian matrices 193
1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 193
2 Classification . . . . . . . . . . . . . . . . . . . . . . . . . . . . 193
3 Bad notation . . . . . . . . . . . . . . . . . . . . . . . . . . . . 194
4 Good notation . . . . . . . . . . . . . . . . . . . . . . . . . . . 196
5 Identification of Jacobian matrices . . . . . . . . . . . . . . . . 198
6 The first identification table . . . . . . . . . . . . . . . . . . . . 198
7 Partitioning of the derivative . . . . . . . . . . . . . . . . . . . 199
8 Scalar functions of a vector . . . . . . . . . . . . . . . . . . . . 200
9 Scalar functions of a matrix, I: trace . . . . . . . . . . . . . . . 200
10 Scalar functions of a matrix, II: determinant. . . . . . . . . . . 202
11 Scalar functions of a matrix, III: eigenvalue . . . . . . . . . . . 204
12 Two examples of vector functions . . . . . . . . . . . . . . . . . 204
13 Matrix functions . . . . . . . . . . . . . . . . . . . . . . . . . . 205
14 Kronecker products . . . . . . . . . . . . . . . . . . . . . . . . . 208
15 Some other problems . . . . . . . . . . . . . . . . . . . . . . . . 210
Bibliographical notes . . . . . . . . . . . . . . . . . . . . . . . . . . . 211
10 Second-order differentials and Hessian matrices 213
1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 213
2 The Hessian matrix of a matrix function . . . . . . . . . . . . . 213
3 Identification of Hessian matrices . . . . . . . . . . . . . . . . . 214
4 The second identification table . . . . . . . . . . . . . . . . . . 215
5 An explicit formula for the Hessian matrix . . . . . . . . . . . . 217
6 Scalar functions . . . . . . . . . . . . . . . . . . . . . . . . . . . 217
7 Vector functions . . . . . . . . . . . . . . . . . . . . . . . . . . 219
8 Matrix functions, I . . . . . . . . . . . . . . . . . . . . . . . . . 220