Table Of Content...................................................................................................................................................................................................................................................................................................................................................................................................................................................................
NUMERICAL METHODS FOR LARGE
EIGENVALUE PROBLEMS
(cid:7) (cid:4)
Second edition
(cid:6) (cid:5)
Yousef Saad
Copyright c2011bytheSocietyforIndustrialandAppliedMathematics
(cid:13)
Contents
PrefacetoClassicsEdition xiii
Preface xv
1 BackgroundinMatrixTheoryandLinearAlgebra 1
1.1 Matrices . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1
1.2 SquareMatricesandEigenvalues . . . . . . . . . . . . . . . 2
1.3 TypesofMatrices . . . . . . . . . . . . . . . . . . . . . . . 4
1.3.1 MatriceswithSpecialSrtuctures . . . . . . . . 4
1.3.2 SpecialMatrices . . . . . . . . . . . . . . . . . 5
1.4 VectorInnerProductsandNorms . . . . . . . . . . . . . . . 6
1.5 MatrixNorms . . . . . . . . . . . . . . . . . . . . . . . . . 8
1.6 Subspaces . . . . . . . . . . . . . . . . . . . . . . . . . . . 9
1.7 OrthogonalVectorsandSubspaces . . . . . . . . . . . . . . 11
1.8 CanonicalFormsofMatrices . . . . . . . . . . . . . . . . . 12
1.8.1 ReductiontotheDiagonalForm . . . . . . . . . 14
1.8.2 TheJordanCanonicalForm . . . . . . . . . . . 14
1.8.3 TheSchurCanonicalForm . . . . . . . . . . . 18
1.9 NormalandHermitianMatrices . . . . . . . . . . . . . . . . 21
1.9.1 NormalMatrices . . . . . . . . . . . . . . . . . 21
1.9.2 HermitianMatrices . . . . . . . . . . . . . . . 23
1.10 NonnegativeMatrices . . . . . . . . . . . . . . . . . . . . . 25
2 SparseMatrices 29
2.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . 29
2.2 StorageSchemes . . . . . . . . . . . . . . . . . . . . . . . . 30
2.3 BasicSparseMatrixOperations . . . . . . . . . . . . . . . . 34
2.4 SparseDirectSolutionMethods . . . . . . . . . . . . . . . 35
2.5 TestProblems . . . . . . . . . . . . . . . . . . . . . . . . . 36
2.5.1 RandomWalkProblem . . . . . . . . . . . . . 36
2.5.2 ChemicalReactions . . . . . . . . . . . . . . . 38
2.5.3 TheHarwell-BoeingCollection . . . . . . . . . 40
2.6 SPARSKIT . . . . . . . . . . . . . . . . . . . . . . . . . . . 40
2.7 TheNewSparseMatrixRepositories . . . . . . . . . . . . . 43
ix
x CONTENTS
2.8 SparseMatricesinMatlab . . . . . . . . . . . . . . . . . . . 43
3 PerturbationTheoryandErrorAnalysis 47
3.1 ProjectorsandtheirProperties . . . . . . . . . . . . . . . . . 47
3.1.1 OrthogonalProjectors . . . . . . . . . . . . . . 48
3.1.2 ObliqueProjectors . . . . . . . . . . . . . . . . 50
3.1.3 ResolventandSpectralProjector . . . . . . . . 51
3.1.4 RelationswiththeJordanform . . . . . . . . . 53
3.1.5 LinearPerturbationsofA . . . . . . . . . . . . 55
3.2 A-PosterioriErrorBounds . . . . . . . . . . . . . . . . . . . 59
3.2.1 GeneralErrorBounds . . . . . . . . . . . . . . 59
3.2.2 TheHermitianCase . . . . . . . . . . . . . . . 61
3.2.3 TheKahan-Parlett-JiangTheorem . . . . . . . . 66
3.3 ConditioningofEigen-problems . . . . . . . . . . . . . . . 70
3.3.1 ConditioningofEigenvalues . . . . . . . . . . . 70
3.3.2 ConditioningofEigenvectors . . . . . . . . . . 72
3.3.3 ConditioningofInvariantSubspaces . . . . . . 75
3.4 LocalizationTheorems . . . . . . . . . . . . . . . . . . . . 77
3.5 Pseudo-eigenvalues . . . . . . . . . . . . . . . . . . . . . . 79
4 TheToolsofSpectralApproximation 85
4.1 SingleVectorIterations . . . . . . . . . . . . . . . . . . . . 85
4.1.1 ThePowerMethod . . . . . . . . . . . . . . . . 85
4.1.2 TheShiftedPowerMethod . . . . . . . . . . . 88
4.1.3 InverseIteration . . . . . . . . . . . . . . . . . 88
4.2 DeflationTechniques . . . . . . . . . . . . . . . . . . . . . 90
4.2.1 WielandtDeflationwithOneVector . . . . . . . 91
4.2.2 OptimalityinWieldant’sDeflation . . . . . . . 92
4.2.3 DeflationwithSeveralVectors. . . . . . . . . . 94
4.2.4 PartialSchurDecomposition. . . . . . . . . . . 95
4.2.5 PracticalDeflationProcedures . . . . . . . . . . 96
4.3 GeneralProjectionMethods . . . . . . . . . . . . . . . . . . 96
4.3.1 OrthogonalProjectionMethods . . . . . . . . . 97
4.3.2 TheHermitianCase . . . . . . . . . . . . . . . 100
4.3.3 ObliqueProjectionMethods . . . . . . . . . . . 106
4.4 ChebyshevPolynomials . . . . . . . . . . . . . . . . . . . . 108
4.4.1 RealChebyshevPolynomials . . . . . . . . . . 108
4.4.2 ComplexChebyshevPolynomials . . . . . . . . 109
5 SubspaceIteration 115
5.1 SimpleSubspaceIteration . . . . . . . . . . . . . . . . . . . 115
5.2 SubspaceIterationwithProjection . . . . . . . . . . . . . . 118
5.3 PracticalImplementations . . . . . . . . . . . . . . . . . . . 121
5.3.1 Locking . . . . . . . . . . . . . . . . . . . . . 121
5.3.2 LinearShifts . . . . . . . . . . . . . . . . . . . 123
CONTENTS xi
5.3.3 Preconditioning . . . . . . . . . . . . . . . . . 123
6 KrylovSubspaceMethods 125
6.1 KrylovSubspaces . . . . . . . . . . . . . . . . . . . . . . . 125
6.2 Arnoldi’sMethod . . . . . . . . . . . . . . . . . . . . . . . 128
6.2.1 TheBasicAlgorithm . . . . . . . . . . . . . . . 128
6.2.2 PracticalImplementations . . . . . . . . . . . . 131
6.2.3 IncorporationofImplicitDeflation . . . . . . . 134
6.3 TheHermitianLanczosAlgorithm . . . . . . . . . . . . . . 136
6.3.1 TheAlgorithm . . . . . . . . . . . . . . . . . . 137
6.3.2 RelationwithOrthogonalPolynomials . . . . . 138
6.4 Non-HermitianLanczosAlgorithm . . . . . . . . . . . . . . 138
6.4.1 TheAlgorithm . . . . . . . . . . . . . . . . . . 139
6.4.2 PracticalImplementations . . . . . . . . . . . . 143
6.5 BlockKrylovMethods . . . . . . . . . . . . . . . . . . . . 145
6.6 ConvergenceoftheLanczosProcess . . . . . . . . . . . . . 147
6.6.1 Distancebetween andanEigenvector . . . 147
m
K
6.6.2 ConvergenceoftheEigenvalues . . . . . . . . . 149
6.6.3 ConvergenceoftheEigenvectors . . . . . . . . 150
6.7 ConvergenceoftheArnoldiProcess. . . . . . . . . . . . . . 151
7 FilteringandRestartingTechniques 163
7.1 PolynomialFiltering . . . . . . . . . . . . . . . . . . . . . . 163
7.2 ExplicitlyRestartedArnoldi’sMethod . . . . . . . . . . . . 165
7.3 ImplicitlyRestartedArnoldi’sMethod . . . . . . . . . . . . 166
7.3.1 WhichFilterPolynomials? . . . . . . . . . . . 169
7.4 ChebyshevIteration . . . . . . . . . . . . . . . . . . . . . . 169
7.4.1 ConvergenceProperties. . . . . . . . . . . . . . 173
7.4.2 ComputinganOptimalEllipse . . . . . . . . . 174
7.5 ChebyshevSubspaceIteration . . . . . . . . . . . . . . . . . 177
7.5.1 GettingtheBestEllipse. . . . . . . . . . . . . . 178
7.5.2 Parameterskandm. . . . . . . . . . . . . . . . 178
7.5.3 Deflation . . . . . . . . . . . . . . . . . . . . . 178
7.6 LeastSquares-Arnoldi . . . . . . . . . . . . . . . . . . . . 179
7.6.1 TheLeastSquaresPolynomial. . . . . . . . . . 179
7.6.2 UseofChebyshevBases . . . . . . . . . . . . . 181
7.6.3 TheGramMatrix . . . . . . . . . . . . . . . . 182
7.6.4 ComputingtheBestPolynomial . . . . . . . . . 184
7.6.5 LeastSquaresArnoldiAlgorithms. . . . . . . . 188
8 PreconditioningTechniques 193
8.1 Shift-and-invertPreconditioning . . . . . . . . . . . . . . . 193
8.1.1 GeneralConcepts . . . . . . . . . . . . . . . . 194
8.1.2 DealingwithComplexArithmetic . . . . . . . . 195
8.1.3 Shift-and-InvertArnoldi . . . . . . . . . . . . . 197
xii CONTENTS
8.2 PolynomialPreconditioning . . . . . . . . . . . . . . . . . . 200
8.3 Davidson’sMethod . . . . . . . . . . . . . . . . . . . . . . 203
8.4 TheJacobi-Davidsonapproach . . . . . . . . . . . . . . . . 206
8.4.1 Olsen’sMethod . . . . . . . . . . . . . . . . . 206
8.4.2 ConnectionwithNewton’sMethod . . . . . . . 207
8.4.3 TheJacobi-DavidsonApproach . . . . . . . . . 208
8.5 TheCMS–AMLSconnection . . . . . . . . . . . . . . . . 209
8.5.1 AMLSandtheCorrectionEquation . . . . . . . 212
8.5.2 SpectralSchurComplements . . . . . . . . . . 213
8.5.3 TheProjectionViewpoint . . . . . . . . . . . . 215
9 Non-StandardEigenvalueProblems 219
9.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . 219
9.2 GeneralizedEigenvalueProblems . . . . . . . . . . . . . . . 220
9.2.1 GeneralResults . . . . . . . . . . . . . . . . . 220
9.2.2 ReductiontoStandardForm . . . . . . . . . . . 225
9.2.3 Deflation . . . . . . . . . . . . . . . . . . . . . 226
9.2.4 Shift-and-Invert . . . . . . . . . . . . . . . . . 227
9.2.5 ProjectionMethods . . . . . . . . . . . . . . . 228
9.2.6 TheHermitianDefiniteCase . . . . . . . . . . 229
9.3 QuadraticProblems . . . . . . . . . . . . . . . . . . . . . . 231
9.3.1 FromQuadratictoGeneralizedProblems . . . . 232
10 OriginsofMatrixEigenvalueProblems 235
10.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . 235
10.2 MechanicalVibrations . . . . . . . . . . . . . . . . . . . . . 236
10.3 ElectricalNetworks. . . . . . . . . . . . . . . . . . . . . . . 241
10.4 ElectronicStructureCalculations . . . . . . . . . . . . . . . 242
10.4.1 Quantumdescriptionsofmatter . . . . . . . . . 242
10.4.2 TheHartreeapproximation . . . . . . . . . . . 244
10.4.3 TheHartree-Fockapproximation . . . . . . . . 246
10.4.4 DensityFunctionalTheory . . . . . . . . . . . 248
10.4.5 TheKohn-Shamequation . . . . . . . . . . . . 250
10.4.6 Pseudopotentials . . . . . . . . . . . . . . . . . 250
10.5 StabilityofDynamicalSystems . . . . . . . . . . . . . . . . 251
10.6 BifurcationAnalysis . . . . . . . . . . . . . . . . . . . . . . 252
10.7 ChemicalReactions . . . . . . . . . . . . . . . . . . . . . . 253
10.8 Macro-economics . . . . . . . . . . . . . . . . . . . . . . . 254
10.9 MarkovChainModels . . . . . . . . . . . . . . . . . . . . . 255
References 259
Index 271
Preface to the Classics Edition
This is a revised edition of a book which appeared close to two decades ago.
Someone scrutinizing how the field has evolved in these two decades will make
twointerestingobservations. Ontheonehandtheobserverwillbestruckbythe
staggering number of new developments in numerical linear algebra during this
period. Thefieldhasevolvedinalldirections: theory, algorithms, software, and
novelapplications. Twodecadesagotherewasessentiallynopublicallyavailable
software for large eigenvalue problems. Today one has a flurry to choose from
andtheactivityinsoftwaredevelopmentdoesnotseemtobeabating. Anumber
ofnewalgorithmsappearedinthisperiodaswell. Icanmentionattheoutsetthe
Jacobi-Davidsonalgorithmandtheideaofimplicitrestarts,bothdiscussedinthis
book,butthereareafewothers. Themostinterestingdevelopmenttothenumeri-
calanalystmaybetheexpansionoftherealmofeigenvaluetechniquesintonewer
and more challenging applications. Or perhaps, the more correct observation is
thattheseapplicationswerealwaysthere,buttheywerenotaswidelyappreciated
or understood by numerical analysts, or were not fully developed due to lack of
software.
Thesecondobservationtobemadewhencomparingthestateofthefieldnow
andtwodecadesagoisthatatthesametimethebasictoolsusedtocomputespec-
tra have essentially not changed much: Krylov subspaces are still omnipresent.
On the whole, the new methods that have been developed consist of enhance-
mentstothesebasicmethods,sometimesmajor,intheformofpreconditioners,or
othervariations. Onemightsaythatthefieldhasevolvedevenmorefromgaining
maturitythanfromthefewimportantdevelopments whichtookplace. Thisma-
turityhasbeenbroughtaboutbythedevelopmentofpracticalalgorithmsandby
software. Therefore, synergetic forces played a major role: new algorithms, en-
hancements, and software packages were developed which enabled new interest
frompractitioners,whichinturnsparkleddemandandadditionalinterestfromthe
algorithmdevelopers.
Inlightofthisobservation,Ihavegroupedthe10chaptersofthefirstedition
into three categories. In the first group are those chapters that are of a theoreti-
cal nature (Chapters 1, 3, and 9). These have undergone small changes such as
correctingerrors,improvingthestyle,andaddingreferences.
The second group includes a few chapters that describe basic algorithms or
concepts – for example subspace iteration (Chapter 5) or the tools of spectral
xiii
xiv PREFACETOTHECLASSICSEDITION
approximation (Chapter 4). These have been left unchanged or have received
small updates. Chapters 2 and 10 are also in this group which then consists of
Chapters2,4,5,and10.
Chaptersinthethirdgroup(chapters6to8)underwentthebiggestchanges.
These describe algorithms and their implementations. Chapters 7 and 8 of the
first edition contained a mix of topics some of which are less important today,
andsosomereorganizationwasneeded. Ipreferredtoshortenorreorganizethe
discussion of some of these topics rather than remove them altogether, because
mostarenotcoveredinotherbooks. Atthesametimeitwasnecessarytoadda
fewsectionsontopicsofmorerecentinterest. Theseincludetheimplicitrestart
techniques(inlcudedinChapter7)andtheJacobi-Davidsonmethod(includedas
part of Chapter 7 on preconditioning techniques). A section on AMLS (Auto-
matic Multi-Level Substructuring) which has had excellent success in Structural
Engineeringhasalsobeenincludedwithagoaltolinkittootherknownmethods.
Problemswereleftunchangedfromtheearlieredition,buttheNotesandref-
erencessectionsendingeachchapterweresystematicallyupdated. Somenotation
hasalsobeenalteredfromthepreviouseditiontoreflectmorecommonusage.For
example,theterm“nullspace”hasbeensubstitutedtolesscommonterm“kernel.”
Anon-lineversionofthisbook,alongwithafewresourcessuchastutorials,
andMATLABscripts,ispostedonmywebsite;see:
http://www.siam.org/books/cl66
Finally,IamindebtedtotheNationalScienceFoundationandtotheDepart-
mentofEnergyfortheirsupportofmyresearchthroughouttheyears.
YousefSaad
Minneapolis,January6,2011
Preface
Matrixeigenvalueproblemsariseinalargenumberofdisciplinesofsciencesand
engineering. Theyconstitutethebasictoolusedindesigningbuildings, bridges,
and turbines, that areresistenttovibrations. They allow tomodel queueing net-
works,andtoanalyzestabilityofelectricalnetworksorfluidflow.Theyalsoallow
thescientisttounderstandlocalphysicalphenonemaortostudybifurcationpat-
ternsindynamicalsystems. Infactthewritingofthisbookwasmotivatedmostly
bythesecondclassofproblems.
Several books dealing with numerical methods for solving eigenvalue prob-
lems involving symmetric (or Hermitian) matrices have been written and there
are a few software packages both public and commercial available. The book
by Parlett [148] is an excellent treatise of the problem. Despite a rather strong
demandbyengineersandscientiststhereislittlewrittenonnonsymmetricprob-
lemsandevenlessisavailableintermsofsoftware. The1965bookbyWilkinson
[222]stillconstitutesanimportantreference. Certainly,sciencehasevolvedsince
the writing of Wilkinson’s book and so has the computational environment and
the demand for solving large matrix problems. Problems are becoming larger
andmorecomplicatedwhileatthesametimecomputersareabletodeliverever
higherperformances. Thismeansinparticularthatmethodsthatweredeemedtoo
demandingyesterdayarenowintherealmoftheachievable. Ihopethatthisbook
will be a small step in bridging the gap between the literature on what is avail-
able in the symmetric case and the nonsymmetric case. Both the Hermitian and
thenon-Hermitiancasearecovered,althoughnon-Hermitianproblemsaregiven
moreemphasis.
Thisbookattemptstoachieveagoodbalancebetweentheoryandpractice. I
shouldcommentthatthetheoryisespeciallyimportantinthenonsymmetriccase.
In essence what differentiates the Hermitian from the non-Hermitian eigenvalue
problem is that in the first case we can always manage to compute an approxi-
mationwhereastherearenonsymmetricproblemsthatcanbearbitrarilydifficult
tosolveandcanessentiallymakeanyalgorithmfail. Statedmorerigorously,the
eigenvalue of a Hermitian matrix is always well-conditioned whereas this is not
true for nonsymmetric matrices. On the practical side, I tried to give a general
view of algorithms and tools that have proved efficient. Many of the algorithms
described correspond to actual implementations of research software and have
beentestedonrealisticproblems. Ihavetriedtoconveyourexperiencefromthe
xv
xvi PREFACE
practiceinusingthesetechniques.
As a result of the partial emphasis on theory, there are a few chapters that
maybefoundhardtodigestforreadersinexperiencedwithlinearalgebra. These
areChapterIIIandtosomeextent,asmallpartofChapterIV.Fortunately,Chap-
terIIIisbasicallyindependentoftherestofthebook. Theminimalbackground
neededtousethealgorithmicpartofthebook,namelyChaptersIVthroughVIII,
iscalculusandlinearalgebraattheundergraduatelevel. Thebookhasbeenused
twicetoteachaspecialtopicscourse;onceinaMathematicsdepartmentandonce
in a Computer Science department. In a quarter period representing roughly 12
weeksof2.5hourslectureperweek,ChapterI,III,andIV,toVIhavebeencov-
eredwithoutmuchdifficulty. Inasemesterperiod,18weeksof2.5hourslecture
weekly,allchapterscanbecoveredwithvariousdegreesofdepth. ChaptersIIand
Xneednotbetreatedinclassandcanbegivenasremedialreading.
Finally, I would like to extend my appreciation to a number of people to
whomIamindebted. Franc¸oiseChatelin,whowasmythesisadviser,introduced
me to numerical methods for eigenvalue problems. Her influence on my way of
thinkingiscertainlyreflectedinthisbook.BeresfordParletthasbeenencouraging
throughoutmycareerandhasalwaysbeenarealinspiration. Partofthemotiva-
tioningettingthisbookcompleted,ratherthan‘neverfinished’,isowedtoL.E.
ScrivenfromtheChemicalEngineeringdepartmentandtomanyothersinapplied
scienceswhoexpressedinterestinmywork. IamindebtedtoRolandFreundwho
hasreadthismanuscriptwithgreatcareandhaspointedoutnumerousmistakes.
