Table Of ContentJournal of Computational and Applied Mathematics
Copyright © 2003 Elsevier B.V. All rights reserved
Volume 123, Issues 1-2, Pages 1-531 (1 November 2000)
Display Checked Docs | E-mail Articles | Export Citations View: Citations
Iterative solution of linear systems in the 20th century, Pages 1-33
1. gfedc
Yousef Saad and Henk A. van der Vorst
SummaryPlus | Full Text + Links | PDF (188 K)
Eigenvalue computation in the 20th century, Pages 35-65
2. gfedc
Gene H. Golub and Henk A. van der Vorst
SummaryPlus | Full Text + Links | PDF (181 K)
QR-like algorithms for eigenvalue problems, Pages 67-83
3. gfedc
David S. Watkins
SummaryPlus | Full Text + Links | PDF (130 K)
The ubiquitous Kronecker product, Pages 85-100
4. gfedc
Charles F. Van Loan
SummaryPlus | Full Text + Links | PDF (130 K)
Preconditioning eigenvalues and some comparison of solvers, Pages 101-115
5. gfedc
Ronald B. Morgan
SummaryPlus | Full Text + Links | PDF (150 K)
For tridiagonals T replace T with LDLt, Pages 117-130
6. gfedc
Beresford N. Parlett
SummaryPlus | Full Text + Links | PDF (306 K)
An overview of relative sin theorems for invariant subspaces of complex matrices, Pages 131-153
7. gfedc
Ilse C. F. Ipsen
SummaryPlus | Full Text + Links | PDF (148 K)
The trace minimization method for the symmetric generalized eigenvalue problem, Pages 155-175
8. gfedc
Ahmed Sameh and Zhanye Tong
SummaryPlus | Full Text + Links | PDF (202 K)
Successive overrelaxation (SOR) and related methods, Pages 177-199
9. gfedc
A. Hadjidimos
SummaryPlus | Full Text + Links | PDF (181 K)
On asynchronous iterations, Pages 201-216
10. gfedc
Andreas Frommer and Daniel B. Szyld
SummaryPlus | Full Text + Links | PDF (128 K)
Iterative methods for large continuation problems, Pages 217-240
11. gfedc
D. Calvetti and L. Reichel
SummaryPlus | Full Text + Links | PDF (458 K)
12.
The matrix and polynomial approaches to Lanczos-type algorithms, Pages 241-260
C. Brezinski, M. Redivo-Zaglia and H. Sadok
gfedc
SummaryPlus | Full Text + Links | PDF (157 K)
Analysis of acceleration strategies for restarted minimal residual methods, Pages 261-292
13. gfedc
Michael Eiermann, Oliver G. Ernst and Olaf Schneider
SummaryPlus | Full Text + Links | PDF (250 K)
Refining an approximate inverse, Pages 293-306
14. gfedc
Robert Bridson and Wei-Pai Tang
SummaryPlus | Full Text + Links | PDF (102 K)
Scalable preconditioned conjugate gradient inversion of vector finite element mass matrices, Pages 307-321
15. gfedc
Joe Koning, Garry Rodrigue and Dan White
SummaryPlus | Full Text + Links | PDF (429 K)
Robust multigrid methods for nonsmooth coefficient elliptic linear systems, Pages 323-352
16. gfedc
Tony F. Chan and W. L. Wan
SummaryPlus | Full Text + Links | PDF (275 K)
The Rook's pivoting strategy, Pages 353-369
17. gfedc
George Poole and Larry Neal
SummaryPlus | Full Text + Links | PDF (241 K)
Numerical methods in control, Pages 371-394
18. gfedc
Volker Mehrmann and Hongguo Xu
Abstract | PDF (169 K)
Krylov-subspace methods for reduced-order modeling in circuit simulation, Pages 395-421
19. gfedc
Roland W. Freund
Abstract | PDF (320 K)
Tikhonov regularization and the L-curve for large discrete ill-posed problems, Pages 423-446
20. gfedc
D. Calvetti, S. Morigi, L. Reichel and F. Sgallari
SummaryPlus | Full Text + Links | PDF (527 K)
Symbiosis between linear algebra and optimization, Pages 447-465
21. gfedc
Dianne P. O'Leary
SummaryPlus | Full Text + Links | PDF (124 K)
Some computational problems arising in adaptive optics imaging systems, Pages 467-487
22. gfedc
Robert J. Plemmons and Victor P. Pauca
SummaryPlus | Full Text + Links | PDF (264 K)
Numerical linear algebra algorithms and software, Pages 489-514
23. gfedc
Jack J. Dongarra and Victor Eijkhout
SummaryPlus | Full Text + Links | PDF (158 K)
The impact of high-performance computing in the solution of linear systems: trends and problems, Pages 515-530
24. gfedc
Iain S. Duff
SummaryPlus | Full Text + Links | PDF (193 K)
Index, Page 531
25. gfedc
Unknown
PDF (40 K)
Numerical Analysis 2000 Vol. III: Linear Algebra, Pages ix-xii
26. gfedc
SummaryPlus | Full Text + Links | PDF (37 K)
Journal of Computational and Applied Mathematics 123 (2000) ix{xii
www.elsevier.nl/locate/cam
Foreword
Numerical Analysis 2000
Vol. III: Linear Algebra
With the year 2000 being elected \The World Mathematical Year", the Journal of Computational
and Applied Mathematics decided to publish a series of volumes dedicated to various disciplines
of applied mathematics and numerical analysis. The series received the ambitious title \Numerical
Analysis in the 20th Century" and contains seven volumes of which the present one is devoted to
\Linear Algebra".
From the early days of scienti(cid:12)c computing, numerical linear algebra has been driven by the
necessity to be able to solve linear systems, to solve eigenproblems, and to understand the meaning
of the results. Because many of these problems have to be solved repeatedly in other computational
problems, the algorithms have to be robust and as fast as possible. This has led to much activity,
and other than only developing algorithms on demand, the involved research has been equally intel-
lectually challenging as in other sciences. The behavior of algorithms under rounding errors was a
great source of inspiration for the further development of perturbation theory.
Also, the possibility and the necessity to solve larger problems has led to algorithms for the
reduction of the information to lower dimensional subspaces. The theories of iterative methods have
been pushed forward by curiosity-driven research as well as by strong pressure from applications.
Numerical analysis and numerical linear algebra in particular, have strongly contributed to the
giant leaps that could be made, in scienti(cid:12)c computing in recent decades. The scienti(cid:12)c problems
that can be solved nowadays are bigger by many orders of magnitude than those that could be
solved, say, some 30 years ago. Roughly, half of the increased computational power must be at-
tributed to improved algorithms, and the other half has been made possible by the increase of
computational speeds in hardware. This impressive increase in scale of computation has led to more
complicated applications and this in turn has led to more complicated numerical linear algebra prob-
lems, such as Kronecker structures, highly nonnormal matrices, ill-posed problems, nonlinear eigen-
problems, etc.
At this point in time, we can conclude that numerical linear algebra has reached a certain level
of maturity. There is a solid theoretical basis for the study of various phenomena and the theory
is still in (cid:13)ux. There have been times, not so long ago, when leading researchers believed that the
theory for this area was more or less complete and that further progress in scienti(cid:12)c computing
was simply a matter of scaling. Simply stated: one had only to increase the order of the problem
and to implement the well-known algorithms e(cid:14)ciently on modern computers. It has turned out that
this was a too simple and too pessimistic point of view. Not only have we seen new challenging
0377-0427/00/$-see front matter (cid:13)c 2000 Elsevier Science B.V. All rights reserved.
PII: S0377-0427(00)00453-2
x
problems, but also the rapidly growing problem sizes led to new algorithms. Even parallel processing
led to new classes of problems, such as domain decomposition techniques.
Research in numerical linear algebra is active as ever before and witnesses many new develop-
ments. As a consequence, we collected in this volume some survey and=or tutorial papers, that illus-
trate the current high level of knowledge and progress, as well as papers with emerging or promising
ideas. In order to cover most research areas of linear algebra, we invited leading researchers to sub-
mit a paper in their respective (cid:12)eld of interest. Our initial list of invitations contained 30 names, well
spread over the areas that we considered as being most representative for Numerical Linear Algebra.
The response was quite impressive as can be seen below. In hindsight with respect to our original
list, the only major topics missing on the (cid:12)nal list are error analysis, perturbation analysis, and the
Krylov methods for eigenproblems. Impressive progress has been made in these areas and we wish
to mention Wilkinson’s book \The Algebraic Eigenvalue Problem" as a landmark in numerical linear
algebra.
All papers have been refereed in the usual way, and it was certainly a surprise for us that the
whole process could be completed almost as scheduled. The involvement of the leading researchers
in numerical linear algebra is nicely illustrated by the observation that most of those who could not
(cid:12)nd time to write a contribution helped us in the refereeing process.
This volume starts with two historical surveys, one on iterative methods for linear systems, by
Y. Saad and H. van der Vorst, and the other on eigenproblems, written by G. Golub and H. van
der Vorst.
These two surveys cover the major developments that have taken place in the twentieth century.
The reader may (cid:12)nd interesting details on how the major algorithmic developments evolved in time.
The two papers contain many references, which may serve as starting points for further research in
these areas (including the \missing" topics mentioned before).
The papers in this volume can be roughly subdivided into the following groups:
1. Eigenproblems (including SVD)
The papers in this group re(cid:13)ect established as well as current research. The QR methods represent
a success story in numerical linear algebra. In these methods, we see various ideas that click
together in an algorithm that leads to very fast and robust algorithms. D. Watkins presents new
views on QR-like methods for the eigenproblem, which leads to a better understanding of how
the various approaches are related.
C. Van Loan shows that Kronecker products do occur in many contexts of linear algebra and he
gives a nice survey on the topic. Preconditioning for eigenproblems, that is the idea to solve nearby
easier problems in order to get good starting values for iterative solution procedures, is currently
an important research topic. This aspect is taken care of in a contribution by R. Morgan. An
important problem, related to stability, is how to compute the eigenvalues of tridiagonal matrices.
B. Parlett gives a brief motivation for the new class of tridiagonal eigensolvers and shows that
the key feature here is to represent them as a product of two bidiagonals.
I. Ipsen has contributed a paper on the separation of invariant subspaces of complex ma-
trices. This is an interesting problem, in particular when classical eigenvector computations
are suspect because of small mutual angles. An alternative approach for studying sensitivity
in eigenproblems, the theory of pseudo-spectra, is not represented in our collection of papers.
We refer to the discussion in the Golub-van der Vorst paper for further background and
information.
xi
The Jacobi{Davidson method is a relatively new branch in the tree of eigenproblem solvers.
The underlying idea permits to attack non-standard eigenproblems, such as polynomial
eigenproblems. A. Sameh and Z. Tong show that a variant of the trace minimization algorithm
is related to the Jacobi{Davidson method and they present a numerical comparison.
2. Linear systems
This has traditionally been the core business of numerical linear algebra, with more emphasis
on iterative approaches during the last few decades of the past century. The current issues are
represented by the following contributions. A. Hadjidimos highlights the SOR methods, which
have played a dominant role in iterative solution approaches for a long time. D. Szyld and A.
Pfrommer consider asynchronous iteration methods, inspired by parallel processing possibilities.
Iterative techniques, based on the block-Lanczos algorithm, for the computation of solution
paths for continuation problems are discussed by D. Calvetti and L. Reichel.
Two di(cid:11)erent views on the Lanczos method are discussed in a paper by C. Brezinski, M.
Redivo{Zaglia, and H. Sadok: the matrix approach and the formal orthogonal polynomial ap-
proach. This leads to convenient treatment of breakdowns in the two-sided Lanczos method for
unsymmetric linear systems.
Minimal residual methods (including GMRES) are powerful tools for the iterative solution of
large linear systems. A common approach is to restart them when the storage requirements or
CPU-time per iteration becomes too high. M. Eiermann, O. Ernst, and O. Schneider present an
analysis for e(cid:14)cient restarting techniques.
Preconditioning has always been an essential ingredient for many iterative methods. In the
1990s, the concept of sparse approximate inverses became popular. R. Bridson and W.-P. Tang
consider re(cid:12)nement techniques for this way of preconditioning. This includes symbolic factoriza-
tion algorithms, reorderings, and blocking techniques.
Parallel aspects of the popular conjugate gradients method, for problems related to (cid:12)nite element
discretization techniques, are discussed in a paper by G. Rodrigue, J. Koning and D. White.
Many of the large-scale linear problems originate from PDEs, and the study of such systems
has signi(cid:12)cant overlap with research in numerical analysis. Multigrid methods is one particular
area where insights from linear algebra and analysis merge fruitfully. T. Chan and J. Wan survey
robust multigrid methods for elliptic PDEs with non-smooth coe(cid:14)cients. They highlight how to
recover the usual multigrid e(cid:14)ciency for this more di(cid:14)cult class of problems.
The paper by G. Poole and L. Neal on pivoting strategies for direct linear solution methods
goes back to the basics. These pivoting strategies and relevant for exploiting the possibilities of
modern computer architectures.
3. Miscellaneous problems
As we have stated before, numerical linear algebra plays an important role in many other research
(cid:12)elds and scienti(cid:12)c applications. V. Mehrmann and H. Xu give a compact survey of some key
numerical linear algebra problems of control theory and discuss the new developments in the area.
R. Freund describes how to use Krylov subspace methods for generating reduced-order models
of linear electric circuits.
D. Calvetti, S. Morigi, L. Reichel and F. Sgallari present existing and new iterative methods
for the determination of the Tikhonov regularization parameter for classes of ill-posed problems.
D. O’Leary’s paper gives a good impression on how numerical linear algebra has intruded
other research areas. She discusses, in particular, the role of linear algebra in Optimization.
xii
Imaging problems give rise to large linear systems for reconstruction from ray tracing infor-
mation. Computational problems, related to image reconstruction in Adaptive Optics Imaging, are
discussed in a paper by R. Plemmons and V. Pauca.
4. Software
Numerical linear algebra has a long tradition in high-quality software. This started with the fa-
mous Wilkinson{Reinsch collection, which formed the basis for well-known packages such as
EISPACK, LINPACK, and, more recently, LAPACK and ScaLAPACK. This has been very im-
portant for the current popularity and in(cid:13)uence of our research area. J. Dongarra and V. Eijkhout
present an overview of the linear algebra algorithms for which mature software is available.
Modern computer architectures have had a signi(cid:12)cant impact on the design of linear algebra
software, and the linear algebra algorithms have, in turn, in(cid:13)uenced the design of computer ar-
chitectures. Think, for instance, of the famous LINPACK benchmark. I. Du(cid:11) discusses the trends
and current problems related to high-performance computing.
We would like to thank all the people who have contributed to the successful completion of this
volume: Luc Wuytack for taking the initiative and for inviting us to be the editors, the authors
for their contributions and, last but not least, the referees for their careful reading and constructive
criticisms.
Apostolos Hadjidimos
Department of Mathematics
University of Crete; GR-714 09 Heraklion
Greece
Henk van der Vorst
Mathematisch Instituut; Universiteit Utrecht
Budapestlaan 6; NL-3584 CD Utrecht
The Netherlands
E-mail address: [email protected]
Paul Van Dooren
Department of Mathematical Engineering
Universite(cid:19) Catholique de Louvain
Av. Lemaitre 4; B-1348 Louvain la Neuve
Belgium
Journal of Computational and Applied Mathematics 123 (2000) 1{33
www.elsevier.nl/locate/cam
Iterative solution of linear systems in the 20th century
Yousef Saada;1, Henk A. van der Vorstb;(cid:3)
aDepartment of Computer Science and Engineering, University of Minnesota, Minneapolis, USA
bDepartment of Mathematics, Utrecht University, P.O. Box 80.010, 3508 TA Utrecht, Netherlands
Received 23 January 2000; received in revised form 2 March 2000
Abstract
This paper sketches the main research developments in the area of iterative methods for solving linear systems during
the 20th century. Although iterative methods for solving linear systems (cid:12)nd their origin in the early 19th century (work
by Gauss), the (cid:12)eld has seen an explosion of activity spurred by demand due to extraordinary technological advances in
engineering and sciences. The past (cid:12)ve decades have been particularly rich in new developments, ending with the avail-
abilityoflargetoolboxofspecializedalgorithmsforsolvingtheverylargeproblemswhichariseinscienti(cid:12)candindustrial
computational models. As in any other scienti(cid:12)c area, research in iterative methods has been a journey characterized by a
chain of contributions building on each other. It is the aim of this paper not only to sketch the most signi(cid:12)cant of these
contributions during the past century, but also to relate them to one another. (cid:13)c 2000 Elsevier Science B.V. All rights
reserved.
Keywords: ADI; Krylov subspace methods; Multigrid; Polynomial acceleration; Preconditioning; Relaxation methods;
SOR; Sparse approximate inverse
1. Introduction
Numerical linear algebra is an exciting (cid:12)eld of research and much of this research has been
triggered by a problem that can be posed simply as: given A 2 Cm(cid:2)n; b 2 Cm, (cid:12)nd solution vector(s)
x 2 Cn such that Ax=b. Many scienti(cid:12)c problems lead to the requirement to solve linear systems
of equations as part of the computations. From a pure mathematical point of view, this problem
can be considered as being solved in the sense that we explicitly know its solution in terms of
determinants. The actual computation of the solution(s) may however lead to severe complications,
when carried out in (cid:12)nite precision and when each basic arithmetic operation takes (cid:12)nite time. Even
(cid:3)Corresponding author.
E-mail addresses: [email protected] (Y. Saad), [email protected] (H.A. van der Vorst).
1Work supported by NSF=CCR and by the Minnesota Supercomputer Institute.
0377-0427/00/$-see front matter (cid:13)c 2000 Elsevier Science B.V. All rights reserved.
PII: S0377-0427(00)00412-X
2 Y. Saad, H.A. van der Vorst/Journal of Computational and Applied Mathematics 123 (2000) 1{33
the \simple" case when n=m and A is nonsingular, which is a trivial problem from a mathematical
point of view, may become very complicated, from a computational point of view, and may even
turn out to be impossible.
The traditional way to solve a nonsingular linear system is to employ Gaussian elimination, and,
with all its enhancements, to overcome numerical instabilities. This process can be carried out
in O(n3) basic (cid:13)oating point operations (additions and multiplications, assuming n = m). Many
applications lead to linear systems with a large n (where the notion of \large" depends, of course,
on the capacity of the available computer), and it became soon evident that one has to exploit speci(cid:12)c
properties of the A at hand in order to make solution of the system feasible. This has led to variants
of Gaussian elimination in which the nonzero structure of A is exploited, so that multiplications with
zero result are avoided and that savings in computer storage could be realized.
Another direction of approach was based on the solution of a nearby linear system, with a ma-
trix that admits a computationally inexpensive process (in terms of computing time and computer
storage), and to embed this in an iterative process. Both approaches aim at making the impos-
sible possible, and for the novice in this (cid:12)eld this may seem to be just a collection of clever
programming tricks: \in principle solving the problem is well understood but one has to be well
organized to make the computational process a little faster". For this novice it will certainly come
as a big surprise that a whole, still incomplete, mathematical framework had to be developed with
deep and elegant results. As a result, relevant systems could be solved many orders of magnitude
faster (and also often more accurate) than by a straightforward Gaussian elimination approach. In
this paper, we will sketch the developments and progress that has taken place in the 20th century
with respect to iterative methods alone. As will be clear, this sub(cid:12)eld could not evolve in isola-
tion, and the distinction between iterative methods and Gaussian elimination methods is sometimes
arti(cid:12)cial { and overlap between the two methodologies is signi(cid:12)cant in many instances. Neverthe-
less, each of the two has its own dynamics and it may be of interest to follow one of them more
closely.
It is likely that future researchers in numerical methods will regard the decade just passed as the
beginning of an era in which iterative methods for solving large linear systems of equations started
gaining considerable acceptance in real-life industrial applications. In looking at past literature, it is
interesting to observe that iterative and direct methods have often been in competition for solving
large systems that arise in applications. A particular discovery will promote a given method from
one camp only to see another discovery promote a competing method from the other camp. For
example, the 1950s and 1960s saw an enormous interest in relaxation-type methods { prompted by
the studies on optimal relaxation and the work by Young, Varga, Southwell, Frankel and others.
A little later, sparse direct methods appeared that were very competitive { both from the point of
view of robustness and computational cost. To this day, there are still applications dominated by
direct solvers and others dominated by iterative solvers. Because of the high memory requirement
of direct solvers, it was sometimes thought that these would eventually be replaced by iterative
solvers, in all applications. However, the superior robustness of direct solvers prevented this. As
computers have become faster, very large problems are routinely solved by methods from both
camps.
Iterative methods were, even halfway in the 20th century, not always viewed as promising. For
instance, Bodewig [23, p. 153], in 1956, mentioned the following drawbacks of iterative methods:
nearly always too slow (except when the matrix approaches a diagonal matrix), for most problems
Y. Saad, H.A. van der Vorst/Journal of Computational and Applied Mathematics 123 (2000) 1{33 3
they do not converge at all, they cannot easily be mechanised2 and so they are more appropriate
for computing by hand than with machines, and do not take advantage of the situation when the
equations are symmetric. The only potential advantage seen was the observation that Rounding
errors do not accumulate, they are restricted to the last operation. It is noteworthy that Lanczos’
method was classi(cid:12)ed as a direct method in 1956.
The penetration of iterative solvers into applications has been a slow process that is still ongoing.
At the time of this writing for example, there are applications in structural engineering as well as
in circuit simulation, which are dominated by direct solvers.
This review will attempt to highlight the main developments in iterative methods over the past
century. It is clear that a few pages cannot cover an exhaustive survey of 100 years of rich devel-
opments. Therefore, we will emphasize the ideas that were successful and had a signi(cid:12)cant impact.
Among the sources we used for our short survey, we would like to mention just a few that are
notable for their completeness or for representing the thinking of a particular era. The books by
Varga [188] and Young [205] give a complete treatise of iterative methods as they were used in the
1960s and 1970s. Varga’s book has several excellent historical references. These two masterpieces
remained the handbooks used by academics and practitioners alike for three decades. Householder’s
book [102] contains a fairly good overview of iterative methods { speci(cid:12)cally oriented towards
projection methods. Among the surveys we note the outstanding booklet published by the National
Bureau of Standards in 1959 which contains articles by Rutishauser [150], Engeli [68] and Stiefel
[170]. Later Birkho(cid:11) [21], who supervised David Young’s Ph.D. thesis in the late 1940s, wrote
an excellent historical perspective on the use of iterative methods as he experienced them himself
from 1930 to 1980. The more recent literature includes the books by Axelsson [7], Brezinski [29],
Greenbaum [88], Hackbusch [97], and Saad [157], each of which has a slightly di(cid:11)erent perspective
and emphasis.
2. The quest for fast solvers: a historical perspective
Iterative methods have traditionally been used for the solution of large linear systems with diag-
onally dominant sparse matrices. For such systems the methods of Gauss{Jacobi and Gauss{Seidel
could be used with some success, not so much because of the reduction in computational work,
but mainly because of the limited amount of memory that is required. Of course, reduction of the
computational work was also a serious concern, and this led Jacobi (1846) to apply plane rotations
to the matrix in order to force stronger diagonal dominance, giving up sparsity. Jacobi had to solve
many similar systems in the context of eigenvalue computations; his linear systems were rather
small: of order 7.
In this century, simple iterative methods were predominantly applied for solving discretized elliptic
self-adjoint partial di(cid:11)erential equations, together with a local parameter for accelerating the iteration
process. The (cid:12)rst and simplest of these methods in Richardson’s method [146]. Actually, this method
2This remark was removed from the second edition (in 1959); instead Bodewig included a small section on methods
for automatic machines [24, Chapter 9]. The earlier remark was not as puzzling as it may seem now, in view of the very
small memories of the available electronic computers at the time. This made it necessary to store intermediate data on
punched cards. It required a regular (cid:13)ow of the computational process, making it cumbersome to include techniques with
row interchanging.
