Table Of ContentLinear and Nonlinear
Programming
Recent titles in the INTERNATIONAL SERIES IN OPERATIONS
RESEARCH & MANAGEMENT SCIENCE
Frederick S. Hillier, Series Editor, StanfordUniversity
Sethi,Yan&Zhang/INVENTORYANDSUPPLYCHAINMANAGEMENTWITHFORECAST
UPDATES
Cox/QUANTITATIVEHEALTHRISKANALYSISMETHODS:ModelingtheHumanHealthImpacts
ofAntibioticsUsedinFoodAnimals
Ching&Ng/MARKOVCHAINS:Models,AlgorithmsandApplications
Li&Sun/NONLINEARINTEGERPROGRAMMING
Kaliszewski/SOFTCOMPUTINGFORCOMPLEXMULTIPLECRITERIADECISIONMAKING
Bouyssouetal/EVALUATIONANDDECISIONMODELSWITHMULTIPLECRITERIA:Stepping
stonesfortheanalyst
Blecker&Friedrich/MASSCUSTOMIZATION:ChallengesandSolutions
Appa,Pitsoulis&Williams/HANDBOOKONMODELLINGFORDISCRETEOPTIMIZATION
Herrmann/HANDBOOKOFPRODUCTIONSCHEDULING
Axsäter/INVENTORYCONTROL,2nd Ed.
Hall/PATIENTFLOW:ReducingDelayinHealthcareDelivery
Józefowska&We¸glarz/PERSPECTIVESINMODERNPROJECTSCHEDULING
Tian&Zhang/VACATIONQUEUEINGMODELS:TheoryandApplications
Yan,Yin&Zhang/STOCHASTICPROCESSES,OPTIMIZATION,ANDCONTROLTHEORY
APPLICATIONSINFINANCIALENGINEERING,QUEUEINGNETWORKS,
ANDMANUFACTURINGSYSTEMS
Saaty&Vargas/DECISIONMAKINGWITHTHEANALYTICNETWORKPROCESS:Economic,
Political,Social&TechnologicalApplicationsw.Benefits,Opportunities,Costs&Risks
Yu/TECHNOLOGYPORTFOLIOPLANNINGANDMANAGEMENT:PracticalConceptsandTools
Kandiller/PRINCIPLESOFMATHEMATICSINOPERATIONSRESEARCH
Lee&Lee/BUILDINGSUPPLYCHAINEXCELLENCEINEMERGINGECONOMIES
Weintraub/MANAGEMENTOFNATURALRESOURCES:AHandbookofOperationsResearch
Models,Algorithms,andImplementations
Hooker/INTEGRATEDMETHODSFOROPTIMIZATION
Dawandeetal/THROUGHPUTOPTIMIZATIONINROBOTICCELLS
Friesz/NETWORKSCIENCE,NONLINEARSCIENCEANDINFRASTRUCTURESYSTEMS
Cai,Sha&Wong/TIME-VARYINGNETWORKOPTIMIZATION
Mamon&Elliott/HIDDENMARKOVMODELSINFINANCE
delCastillo/PROCESSOPTIMIZATION:AStatisticalApproach
Józefowska/JUST-IN-TIMESCHEDULING:Models&AlgorithmsforComputer&Manufacturing
Systems
Yu,Wang&Lai/FOREIGN-EXCHANGE-RATEFORECASTINGWITHARTIFICIALNEURAL
NETWORKS
Beyeretal/MARKOVIANDEMANDINVENTORYMODELS
Shi&Olafsson/NESTEDPARTITIONSOPTIMIZATION:MethodologyandApplications
Samaniego/SYSTEMSIGNATURESANDTHEIRAPPLICATIONSINENGINEERING
RELIABILITY
Kleijnen/DESIGNANDANALYSISOFSIMULATIONEXPERIMENTS
Førsund/HYDROPOWERECONOMICS
Kogan&Tapiero/SUPPLYCHAINGAMES:OperationsManagementandRiskValuation
Vanderbei/LINEARPROGRAMMING:Foundations&Extensions,3rd Edition
Chhajed&Lowe/BUILDINGINTUITION:InsightsfromBasicOperationsMgmt.Modelsand
Principles
∗Alistoftheearlypublicationsintheseriesisattheendofthebook∗
Linear and Nonlinear
Programming
Third Edition
David G. Luenberger
Stanford University
Yinyu Ye
Stanford University
123
DavidG.Luenberger YinyuYe
Dept.ofMgmt.Science&Engineering Dept.ofMgmt.Science&Engineering
StanfordUniversity StanfordUniversity
Stanford,CA,USA Stanford,CA,USA
SeriesEditor:
FrederickS.Hillier
StanfordUniversity
Stanford,CA,USA
ISBN:978-0-387-74502-2 e-ISBN:978-0-387-74503-9
LibraryofCongressControlNumber:2007933062
©2008bySpringerScience+BusinessMedia,LLC
Allrightsreserved.Thisworkmaynotbetranslatedorcopiedinwholeorinpartwithoutthewritten
permissionofthepublisher(SpringerScience+BusinessMedia,LLC,233SpringStreet,NewYork,
NY 10013, USA), except for brief excerpts in connection with reviews or scholarly analysis. Use in
connectionwithanyformofinformationstorageandretrieval,electronicadaptation,computersoftware,
orbysimilarordissimilarmethodologynowknownorhereafterdevelopedisforbidden.
The use in this publication of trade names, trademarks, service marks and similar terms, even if the
arenotidentifiedassuch,isnottobetakenasanexpressionofopinionastowhetherornottheyare
subjecttoproprietaryrights.
Printedonacid-freepaper
987654321
springer.com
To Susan, Robert, Jill, and Jenna;
Daisun and Fei
PREFACE
Thisbookisintendedasatextcoveringthecentralconceptsofpracticaloptimization
techniques.Itisdesignedforeitherself-studybyprofessionalsorclassroomworkat
the undergraduate or graduate level for students who have a technical background
in engineering, mathematics, or science. Like the field of optimization itself,
which involves many classical disciplines, the book should be useful to system
analysts, operations researchers, numerical analysts, management scientists, and
otherspecialistsfromthehostofdisciplinesfromwhichpracticaloptimizationappli-
cations are drawn. The prerequisites for convenient use of the book are relatively
modest; the prime requirement being some familiarity with introductory elements
of linear algebra. Certain sections and developments do assume some knowledge
ofmoreadvancedconceptsoflinearalgebra,suchaseigenvectoranalysis,orsome
backgroundinsetsofrealnumbers,butthetextisstructuredsothatthemainstream
ofthedevelopmentcanbefaithfullypursuedwithoutrelianceonthismoreadvanced
background material.
Although the book covers primarily material that is now fairly standard, it
is intended to reflect modern theoretical insights. These provide structure to what
mightotherwisebesimplyacollectionoftechniquesandresults,andthisisvaluable
bothasameansforlearningexistingmaterialandfordevelopingnewresults.One
majorinsightofthistypeistheconnectionbetweenthepurelyanalyticalcharacter
ofanoptimizationproblem,expressedperhapsbypropertiesofthenecessarycondi-
tions, and the behavior of algorithms used to solve a problem. This was a major
theme of the first edition of this book and the second edition expands and further
illustrates this relationship.
As in the second edition, the material in this book is organized into three
separate parts. Part I is a self-contained introduction to linear programming, a key
component of optimization theory. The presentation in this part is fairly conven-
tional,coveringthemainelementsoftheunderlyingtheoryoflinearprogramming,
manyofthemosteffectivenumericalalgorithms,andmanyofitsimportantspecial
applications. Part II, which is independent of Part I, covers the theory of uncon-
strainedoptimization,includingbothderivationsoftheappropriateoptimalitycondi-
tions and an introduction to basic algorithms. This part of the book explores the
generalpropertiesofalgorithmsanddefinesvariousnotionsofconvergence.PartIII
extends the concepts developed in the second part to constrained optimization
vii
viii Preface
problems.Exceptforafewisolatedsections,thispartisalsoindependentofPartI.
It is possible to go directly into Parts II and III omitting Part I, and, in fact, the
bookhasbeenusedinthiswayinmanyuniversities.Eachpartofthebookcontains
enough material to form the basis of a one-quarter course. In either classroom use
orforself-study,itisimportantnottooverlookthesuggestedexercisesattheendof
eachchapter.Theselectionsgenerallyincludeexercisesofacomputationalvariety
designed to test one’s understanding of a particular algorithm, a theoretical variety
designed to test one’s understanding of a given theoretical development, or of the
varietythatextendsthepresentationofthechaptertonewapplicationsortheoretical
areas. One should attempt at least four or five exercises from each chapter. In
progressing through the book it would be unusual to read straight through from
cover to cover. Generally, one will wish to skip around. In order to facilitate this
mode, we have indicated sections of a specialized or digressive nature with an
asterisk∗.
There are several features of the revision represented by this third edition. In
Part I a new Chapter 5 is devoted to a presentation of the theory and methods
of polynomial-time algorithms for linear programming. These methods include,
especially,interiorpointmethodsthathaverevolutionizedlinearprogramming.The
firstpartofthebookcanitselfserveasamodernbasictextforlinearprogramming.
Part II includes an expanded treatment of necessary conditions, manifested by
not only first- and second-order necessary conditions for optimality, but also by
zeroth-order conditions that use no derivative information. This part continues to
presenttheimportantdescentmethodsforunconstrainedproblems,butthereisnew
material on convergence analysis and on Newton’s methods which is frequently
used as the workhorse of interior point methods for both linear and nonlinear
programming. Finally, Part III now includes the global theory of necessary condi-
tionsforconstrainedproblems,expressedaszero-thorderconditions.Alsointerior
point methods for general nonlinear programming are explicitly discussed within
the sections on penalty and barrier methods. A significant addition to Part III is
an expanded presentation of duality from both the global and local perspective.
Finally, Chapter 15, on primal–dual methods has additional material on interior
point methods and an introduction to the relatively new field of semidefinite
programming, including several examples.
We wish to thank the many students and researchers who over the years have
givenuscommentsconcerningthesecondeditionandthosewhoencouragedusto
carry out this revision.
Stanford, California D.G.L.
July 2007 Y.Y.
CONTENTS
Chapter 1. Introduction 1
1.1. Optimization 1
1.2. TypesofProblems 2
1.3. SizeofProblems 5
1.4. IterativeAlgorithmsandConvergence 6
PART I Linear Programming
Chapter 2. Basic Properties of Linear Programs 11
2.1. Introduction 11
2.2. ExamplesofLinearProgrammingProblems 14
2.3. BasicSolutions 19
2.4. TheFundamentalTheoremofLinearProgramming 20
2.5. RelationstoConvexity 22
2.6. Exercises 28
Chapter 3. The Simplex Method 33
3.1. Pivots 33
3.2. AdjacentExtremePoints 38
3.3. DeterminingaMinimumFeasibleSolution 42
3.4. ComputationalProcedure—SimplexMethod 46
3.5. ArtificialVariables 50
3.6. MatrixFormoftheSimplexMethod 54
3.7. TheRevisedSimplexMethod 56
∗3.8. TheSimplexMethodandLUDecomposition 59
3.9. Decomposition 62
3.10. Summary 70
3.11. Exercises 70
Chapter 4. Duality 79
4.1. DualLinearPrograms 79
4.2. TheDualityTheorem 82
4.3. RelationstotheSimplexProcedure 84
4.4. SensitivityandComplementarySlackness 88
∗4.5. TheDualSimplexMethod 90
ix
x Contents
∗4.6. ThePrimal–DualAlgorithm 93
∗4.7. ReductionofLinearInequalities 98
4.8. Exercises 103
Chapter 5. Interior-Point Methods 111
5.1. ElementsofComplexityTheory 112
∗5.2. TheSimplexMethodisnotPolynomial-Time 114
∗5.3. TheEllipsoidMethod 115
5.4. TheAnalyticCenter 118
5.5. TheCentralPath 121
5.6. SolutionStrategies 126
5.7. TerminationandInitialization 134
5.8. Summary 139
5.9. Exercises 140
Chapter 6. Transportation and Network Flow Problems 145
6.1. TheTransportationProblem 145
6.2. FindingaBasicFeasibleSolution 148
6.3. BasisTriangularity 150
6.4. SimplexMethodforTransportationProblems 153
6.5. TheAssignmentProblem 159
6.6. BasicNetworkConcepts 160
6.7. MinimumCostFlow 162
6.8. MaximalFlow 166
6.9. Summary 174
6.10. Exercises 175
PART II Unconstrained Problems
Chapter 7. Basic Properties of Solutions and Algorithms 183
7.1. First-OrderNecessaryConditions 184
7.2. ExamplesofUnconstrainedProblems 186
7.3. Second-OrderConditions 190
7.4. ConvexandConcaveFunctions 192
7.5. MinimizationandMaximizationofConvexFunctions 197
7.6. Zero-OrderConditions 198
7.7. GlobalConvergenceofDescentAlgorithms 201
7.8. SpeedofConvergence 208
7.9. Summary 212
7.10. Exercises 213
Chapter 8. Basic Descent Methods 215
8.1. FibonacciandGoldenSectionSearch 216
8.2. LineSearchbyCurveFitting 219
8.3. GlobalConvergenceofCurveFitting 226
8.4. ClosednessofLineSearchAlgorithms 228
8.5. InaccurateLineSearch 230
8.6. TheMethodofSteepestDescent 233
