Table Of ContentSpringer Optimization and Its Applications 158
Neculai Andrei
Nonlinear Conjugate
Gradient Methods
for Unconstrained
Optimization
Springer Optimization and Its Applications
Volume 158
SeriesEditors
PanosM.Pardalos,UniversityofFlorida
MyT.Thai,UniversityofFlorida
HonoraryEditor
Ding-ZhuDu,UniversityofTexasatDallas
AdvisoryEditors
RomanV.Belavkin,MiddlesexUniversity
JohnR.Birge,UniversityofChicago
SergiyButenko,TexasA&MUniversity
FrancoGiannessi,UniversityofPisa
VipinKumar,UniversityofMinnesota
AnnaNagurney,UniversityofMassachusettsAmherst
JunPei,HefeiUniversityofTechnology
OlegProkopyev,UniversityofPittsburgh
SteffenRebennack,KarlsruheInstituteofTechnology
MauricioResende,Amazon
TamásTerlaky,LehighUniversity
VanVu,YaleUniversity
GuoliangXue,ArizonaStateUniversity
YinyuYe,StanfordUniversity
Aims and Scope
Optimization has continued to expand in all directions at an astonishing rate. New
algorithmicandtheoreticaltechniquesarecontinuallydevelopingandthediffusion
into other disciplines is proceeding at a rapid pace, with a spot light on machine
learning, artificial intelligence, and quantum computing. Our knowledge of all
aspects of the field has grown even more profound. At the same time, one of the
most striking trends in optimization is the constantly increasing emphasis on the
interdisciplinarynatureofthefield.Optimizationhasbeenabasictoolinareasnot
limited to applied mathematics, engineering, medicine, economics, computer
science, operations research, and other sciences.
TheseriesSpringerOptimizationandItsApplications(SOIA)aimstopublish
state-of-the-art expository works (monographs, contributed volumes, textbooks,
handbooks)thatfocusontheory,methods,andapplicationsofoptimization.Topics
covered include, but are not limited to, nonlinear optimization, combinatorial
optimization, continuous optimization, stochastic optimization, Bayesian
optimization, optimal control, discrete optimization, multi-objective optimization,
and more. New to the series portfolio include Works at the intersection of
optimization and machine learning, artificial intelligence, and quantum computing.
Volumes from this series are indexed by Web of Science, zbMATH, Mathematical
Reviews, and SCOPUS.
More information about this series at http://www.springer.com/series/7393
Neculai Andrei
Nonlinear Conjugate
Gradient Methods
for Unconstrained
Optimization
123
Neculai Andrei
Centerfor AdvancedModeling
andOptimization
Academy of RomanianScientists
Bucharest, Romania
ISSN 1931-6828 ISSN 1931-6836 (electronic)
SpringerOptimization andIts Applications
ISBN978-3-030-42949-2 ISBN978-3-030-42950-8 (eBook)
https://doi.org/10.1007/978-3-030-42950-8
MathematicsSubjectClassification(2010): 49M37,65K05,90C30,90C06,90C90
©SpringerNatureSwitzerlandAG2020
Thisworkissubjecttocopyright.AllrightsarereservedbythePublisher,whetherthewholeorpart
of the material is concerned, specifically the rights of translation, reprinting, reuse of illustrations,
recitation, broadcasting, reproduction on microfilms or in any other physical way, and transmission
orinformationstorageandretrieval,electronicadaptation,computersoftware,orbysimilarordissimilar
methodologynowknownorhereafterdeveloped.
The use of general descriptive names, registered names, trademarks, service marks, etc. in this
publicationdoesnotimply,evenintheabsenceofaspecificstatement,thatsuchnamesareexemptfrom
therelevantprotectivelawsandregulationsandthereforefreeforgeneraluse.
The publisher, the authors and the editors are safe to assume that the advice and information in this
book are believed to be true and accurate at the date of publication. Neither the publisher nor the
authors or the editors give a warranty, expressed or implied, with respect to the material contained
hereinorforanyerrorsoromissionsthatmayhavebeenmade.Thepublisherremainsneutralwithregard
tojurisdictionalclaimsinpublishedmapsandinstitutionalaffiliations.
ThisSpringerimprintispublishedbytheregisteredcompanySpringerNatureSwitzerlandAG
Theregisteredcompanyaddressis:Gewerbestrasse11,6330Cham,Switzerland
Preface
This book is on conjugate gradient methods for unconstrained optimization. The
concept of conjugacy was introduced by Magnus Hestenes and Garrett Birkhoff in
1936 in the context of the variational theory. The history of conjugate gradient
methods, surveyed by Golub and O’Leary (1989), began with the research studies
of Cornelius Lanczos, Magnus Hestenes, George Forsythe, Theodore Motzkin,
Barkley Rosser, and others at the Institute for Numerical Analysis as well as with
the independent research of Eduard Steifel at Eidgenössische Technische
Hochschule,Zürich. Thefirst presentation ofconjugatedirection algorithmsseems
to be that of Fox, Huskey, and Wilkinson (1948), who considered them as direct
methods, and of Forsythe, Hestenes, and Rosser (1951), Hestenes and Stiefel
(1952), and Rosser (1953). The landmark paper published by Hestenes and Stiefel
in 1952 presented both the method of the linear conjugate gradient and the con-
jugatedirectionmethods,includingconjugateGram–Schmidtprocessesforsolving
symmetric, positive definite linear algebraic systems. A closely related algorithm
was proposed by Lanczos (1952), who worked on algorithms for determining the
eigenvalues of a matrix (Lanczos, 1950). His iterative algorithm yielded the simi-
laritytransformationofamatrixintothetridiagonalformwhichtheeigenvaluescan
be well approximated. Hestenes, who worked on iterative methods for solving
linear systems (Hestenes, 1951, 1955), was also interested in the Gram–Schmidt
process for finding conjugate diameters of an ellipsoid. He was interested in
developing a general theory of quadratic forms in Hilbert space (Hestenes, 1956a,
1956b). Initially, the linear conjugate gradient algorithm was called the Hestenes–
Stiefel–Lanczos method (Golub & O’Leary, 1989).
The initial numerical experience with conjugate gradient algorithms was not
very encouraging. Although widely used in the 1960s, their application to
ill-conditioned problems gave rather poor results. At that time, preconditioning
techniques were not well understood. They were developed in the 1970s together
with methods intended for large sparse linear systems; these methods were
prompted by the paper of Reid (1971), who reinforced them by showing their
potential as iterative methods for sparse linear systems. Although Hestenes and
Stiefel stated their algorithm for sets of linear systems of equations with positive
v
vi Preface
definitematrices,fromthebeginningitwasviewedasanoptimizationtechniquefor
minimizing quadratic functions. In the 1960s, conjugate gradient and conjugate
directionmethodswereextendedtotheoptimizationofnonquadraticfunctions.The
first algorithm for nonconvex problems was proposed by Feder (1962), who sug-
gested using conjugate gradient algorithms for solving some problems in optics.
Thealgorithmsandtheconvergencestudyofseveralversionsofconjugategradient
algorithms for nonquadratic functions were discussed by Fletcher and Reeves
(1964), Polak and Ribière (1969), and Polyak (1969).
It is interesting to see that the work of Davidon (1959) on variable metric
algorithms was followed by that of Fletcher and Powell (1963). Other variants
of these methods were established by Broyden (1970), Fletcher (1970), Goldfarb
(1970), and Shanno (1970), who established one of the most effective techniques
for minimizing nonquadratic functions—the BFGS method. The main idea behind
variable metric methods is the construction of a sequence of matrices to approxi-
mate the Hessian matrix (or its inverse) by applying a sequence of rank-one (or
rank-two) update formulae. Details on the BFGS method can be found in the
landmark papers of Dennis and Moré (1974, 1977). When applied to a quadratic
function and under an exact evaluation to the Hessian, these methods give a
solution in a finite number of iterates, and they are exactly conjugate gradient
methods.VariablemetricapproximationstotheHessianmatrixaredensematrices,
and therefore, they are not suitable for large-scale problems, i.e., problems with
many variables. However, the work of Nocedal (1980) on limited-memory
quasi-Newtonmethodswhichuseavariablemetricupdatingprocedurebutwithina
prespecified memory storage enlarged the applicability of quasi-Newton methods.
At the same time, the introduction of the inexact (truncated) Newton method by
Dembo,Eisenstat,andSteihaug(1982)anditsdevelopmentbyNash(1985),andby
Schlick and Fogelson (1992a, 1992b) gave the possibility of solving large-scale
unconstrainedoptimization problems.Theidea behindtheinexactNewtonmethod
wasthatfarawayfromalocalminimum,itisnotnecessarytospendtoomuchtime
computing an accurate Newton search vector. It is better to approximate the
solution of the Newton system for the search direction computation. The
limited-memoryquasi-NewtonandthetruncatedNewtonarereliablemethods,able
to solve large-scale unconstrained optimization problems. However, as it is to be
seen, there is a close connection between the conjugate gradient and the
quasi-Newton methods. Actually, conjugate gradient methods are precisely the
BFGS quasi-Newton method, where the approximation to the inverse Hessian
oftheminimizingfunctionisrestartedastheidentitymatrixateveryiteration.The
developmentsoftheconjugategradientmethodssubjectbothtothesearchdirection
and to the stepsize computation yielded algorithms and the corresponding reliable
software with better numerical performances than the limited-memory
quasi-Newton or inexact Newton methods.
Thebookisstructuredinto12chapters.Chapter1hasanintroductorycharacter
by presenting the optimality conditions for unconstrained optimization and a thor-
ough description and the properties of the main methods for unconstrained
Preface vii
optimization (steepest descent, Newton, quasi-Newton, modifications of the BFGS
method, quasi-Newton methods with diagonal updating of the Hessian,
limited-memoryquasi-Newtonmethods,truncatedNewton,conjugategradient,and
trust-region methods). It is common knowledge that the final test of a theory is its
capacity to solve the problems which originated it. Therefore, in this chapter a
collection of 80 unconstrained optimization test problems with different structures
and complexities, as well as five large-scale applications from the MINPACK-2
collectionfortestingthenumericalperformancesofthealgorithmsdescribedinthis
book, is presented. Some problems from this collection are quadratic, and some
others are highly nonlinear. For some problems, the Hessian has a block-diagonal
structure, for others it has a banded structure with small bandwidth. There are
problemswithsparseordense Hessian. InChapter 2, thelinearconjugategradient
algorithm is detailed. The general convergence results for conjugate gradient
methods are assembled in Chapter 3. The purpose is to put together the main con-
vergence results both for conjugate gradient methods with standard Wolfe line
searchandforconjugategradientmethodswithstrongWolfelinesearch.Sincethe
search direction depends on a parameter, the conditions on this parameter which
ensuretheconvergenceofthealgorithmaredetailed.Theglobalconvergenceresults
ofconjugategradientalgorithmspresentedinthischapterfollowfromtheconditions
given by Zoutendijk and by Nocedal under classical assumptions. The remaining
chaptersarededicatedtothenonlinearconjugategradientmethodsforunconstrained
optimization, insisting both on the theoretical aspects of their convergence and on
their numerical performances forsolvinglarge-scaleproblems and applications.
Plenty of nonlinear conjugate gradient methods are known. The difference
among them is twofold: the way in which the search direction is updated and the
procedure for the stepsize computation along this direction. The main requirement
ofthesearchdirectionoftheconjugategradientmethodsistosatisfythedescentor
the sufficient descent condition. The stepsize is computed by using the Wolfe line
searchconditionsorsomevariantsofthem.Inabroadsense,theconjugategradient
algorithms may be classified as standard, hybrid, modifications of the standard
conjugate gradient algorithms, memoryless BFGS preconditioned, three-term con-
jugate gradient algorithms, and others.
ThemostimportantstandardconjugategradientmethodsdiscussedinChapter 4
are: Hestenes–Stiefel, Fletcher–Reeves, Polak–Ribière–Polyak, conjugate descent
of Fletcher, Liu–Storey, and Dai–Yuan. If the minimizing function is strongly
convex quadratic and the line search is exact, then, in theory, all choices for the
searchdirectioninstandardconjugategradientalgorithmsareequivalent.However,
for nonquadratic functions, each choice of the search direction leads to standard
conjugate gradient algorithms with very different performances. An important
ingredient in conjugate gradient algorithms is the acceleration, discussed in
Chapter 5.
viii Preface
HybridconjugategradientalgorithmspresentedinChapter 6trytocombinethe
standard conjugate gradient methods in order to exploit the attractive features of
each one. To obtain hybrid conjugate gradient algorithms, the standard schemes
may be combined in two different ways. The first combination is based on the
projection concept. The idea of these methods is to consider a pair of standard
conjugate gradient methods and use one of them when a criterion is satisfied. As
soon as the criterion has been violated, then the other standard conjugate gradient
fromthepairisused.Thesecondclassofthehybridconjugategradientmethodsis
based on the convex combination of the standard methods. This idea of these
methodsistochoose apair ofstandardmethodsand tocombine them ina convex
way, where the parameter in the convex combination is computed by using the
conjugacyconditionortheNewtonsearchdirection.Ingeneral,thehybridmethods
based on the convex combination of the standard schemes outperform the hybrid
methods based on the projection concept. The hybrid methods are more efficient
and more robust than the standard ones.
An important class of conjugate gradient algorithms discussed in Chapter 7 is
obtained by modifying the standard algorithms. Any standard conjugate gradient
algorithmmaybemodifiedinsuchawaythatthecorrespondingsearchdirectionis
descent, and the numerical performances are improved. In this area of research,
only some modifications of the Hestenes–Stifel standard conjugate gradient algo-
rithmarepresented.Today’sbest-performingconjugategradientalgorithmsarethe
modificationsoftheHestenes–Stiefelconjugategradientalgorithm:CG-DESCENT
of Hager and Zhang (2005) and DESCON of Andrei (2013c). CG-DESCENT is a
conjugate gradient algorithm with guaranteed descent. In fact, CG-DESCENT can
beviewedasanadaptiveversionoftheDaiandLiaoconjugategradientalgorithm
with a special value for its parameter. The search direction of CG-DESCENT is
related tothememoryless quasi-Newton directionofPerry–Shanno.DESCONisa
conjugategradientalgorithmwithguaranteeddescentandconjugacyconditionsand
with a modified Wolfe line search. Mainly, it is a modification of the Hestenes–
Stiefel conjugate gradient algorithm. In CG-DESCENT, the stepsize is computed
by using the standard Wolfe line search or an approximate Wolfe line search
introducedbyHagerandZhang(2005,2006a,2006b),whichisresponsibleforthe
highperformancesofthealgorithm.InDESCON,thestepsizeiscomputedbyusing
the modified Wolfe line search introduced by Andrei (2013c), in which the
parameterinthecurvatureconditionoftheWolfelinesearchisadaptivelymodified
at every iteration. Besides, DESCON is equipped with an acceleration scheme
which improves its performances.
The first connection between the conjugate gradient algorithms and the
quasi-Newton ones was presented by Perry (1976), who expressed the Hestenes–
Stiefel search direction as a matrix multiplying the negative gradient. Later on,
Shanno(1978a)showedthattheconjugategradientmethodsareexactlytheBFGS
quasi-Newtonmethods,wheretheapproximationtotheinverseHessianisrestarted
astheidentitymatrixateveryiteration.Inotherwords,conjugategradientmethods
arememorylessquasi-Newtonmethods.Thiswasthestartingpointofaveryprolific
Preface ix
research area of memoryless quasi-Newton conjugate gradient methods, which is
discussed in Chapter 8. The point was how the second-order information of the
minimizing function should be introduced in the formula for updating the search
direction.Usingthisideatoincludethecurvatureoftheminimizingfunctioninthe
search direction computation, Shanno (1983) elaborated CONMIN as the first
conjugategradientalgorithmmemorylessBFGSpreconditioned.Lateron,byusing
a combination of the scaled memoryless BFGS method and the preconditioning,
Andrei (2007a, 2007b, 2007c, 2008a) elaborated SCALCG as a double-quasi-
Newtonupdatescheme.DaiandKou(2013)elaboratedtheCGOPTalgorithmasa
family of conjugate gradient methods based on the self-scaling memoryless BFGS
method in which the search direction is computed in a one-dimensional manifold.
The search direction in CGOPTis chosen tobe closest tothe Perry–Shanno direc-
tion.ThestepsizeinCGOPTiscomputedbyusinganimprovedWolfelinesearch
introducedbyDaiandKou(2013).CGOPTwithimprovedWolfelinesearchanda
special restart condition is one of the best conjugate gradient algorithms. New
conjugategradientalgorithmsbasedontheself-scalingmemorylessBFGSupdating
usingthedeterminantorthetraceoftheiterationmatrixorthemeasurefunctionof
Byrd and Nocedalare presented inthis chapter.
Beale (1972) and Nazareth (1977) introduced the three-term conjugate gradient
methods, presented, and analyzed in Chapter 9. The convergence rate of the con-
jugate gradient method may be improved from linear to n-step quadratic if the
method is restarted with the negative gradient direction at every n iterations. One
such restart technique was proposed by Beale (1972). In his restarting procedure,
the restart direction is a combination of the negative gradient and the previous
search direction which includes the second-order derivative information achieved
by searching along the previous direction. Thus, a three-term conjugate gradient
was obtained. In order to achieve finite convergence for an arbitrary initial search
direction, Nazareth (1977) proposed a conjugate gradient method in which the
searchdirectionhasthreeterms.Plentyofthree-termconjugategradientalgorithms
are known. This chapter presents only the three-term conjugate gradient with
descent and conjugacy conditions, the three-term conjugate gradient method with
subspace minimization, and the three-term conjugate gradient method with mini-
mization of one-parameter quadratic model of the minimizing function. The
three-term conjugate gradient concept is an interesting innovation. However, the
numerical performances of these algorithms are modest.
PreconditioningoftheconjugategradientalgorithmsispresentedinChapter 10.
This is a technique for accelerating the convergence of algorithms. In fact, pre-
conditioning was used in the previous chapters as well, but it is here where the
proper preconditioning by a change of variables which improves the eigenvalues
distribution of the iteration matrix is emphasized.
Some other conjugate gradient methods, like those based on clustering the
eigenvalues of the iteration matrix or on minimizing the condition number of this
matrix, including the methods with guaranteed descent and conjugacy conditions
