Table Of Content

Éric Walter Numerical Methods and Optimization A Consumer Guide Numerical Methods and Optimization Éric Walter Numerical Methods and Optimization A Consumer Guide 123 Éric Walter Laboratoire desSignaux etSystèmes CNRS-SUPÉLEC-Université Paris-Sud Gif-sur-Yvette France ISBN 978-3-319-07670-6 ISBN 978-3-319-07671-3 (eBook) DOI 10.1007/978-3-319-07671-3 Springer ChamHeidelberg New YorkDordrecht London LibraryofCongressControlNumber:2014940746 (cid:2)SpringerInternationalPublishingSwitzerland2014 Thisworkissubjecttocopyright.AllrightsarereservedbythePublisher,whetherthewholeorpartof the material is concerned, specifically the rights of translation, reprinting, reuse of illustrations, recitation,broadcasting,reproductiononmicrofilmsorinanyotherphysicalway,andtransmissionor informationstorageandretrieval,electronicadaptation,computersoftware,orbysimilarordissimilar methodology now known or hereafter developed. Exempted from this legal reservation are brief excerpts in connection with reviews or scholarly analysis or material supplied specifically for the purposeofbeingenteredandexecutedonacomputersystem,forexclusiveusebythepurchaserofthe work. Duplication of this publication or parts thereof is permitted only under the provisions of theCopyright Law of the Publisher’s location, in its current version, and permission for use must always be obtained from Springer. Permissions for use may be obtained through RightsLink at the CopyrightClearanceCenter.ViolationsareliabletoprosecutionundertherespectiveCopyrightLaw. The use of general descriptive names, registered names, trademarks, service marks, etc. in this publicationdoesnotimply,evenintheabsenceofaspecificstatement,thatsuchnamesareexempt fromtherelevantprotectivelawsandregulationsandthereforefreeforgeneraluse. While the advice and information in this book are believed to be true and accurate at the date of publication,neithertheauthorsnortheeditorsnorthepublishercanacceptanylegalresponsibilityfor anyerrorsoromissionsthatmaybemade.Thepublishermakesnowarranty,expressorimplied,with respecttothematerialcontainedherein. Printedonacid-freepaper SpringerispartofSpringerScience+BusinessMedia(www.springer.com) À mes petits-enfants Contents 1 From Calculus to Computation. . . . . . . . . . . . . . . . . . . . . . . . . . 1 1.1 Why Not Use Naive Mathematical Methods?. . . . . . . . . . . . . 3 1.1.1 Too Many Operations. . . . . . . . . . . . . . . . . . . . . . . 3 1.1.2 Too Sensitive to Numerical Errors . . . . . . . . . . . . . . 3 1.1.3 Unavailable . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4 1.2 What to Do, Then?. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4 1.3 How Is This Book Organized?. . . . . . . . . . . . . . . . . . . . . . . 4 References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6 2 Notation and Norms. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7 2.1 Introduction. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7 2.2 Scalars, Vectors, and Matrices. . . . . . . . . . . . . . . . . . . . . . . 7 2.3 Derivatives . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8 2.4 Little o and Big O . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11 2.5 Norms. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12 2.5.1 Vector Norms . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12 2.5.2 Matrix Norms . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14 2.5.3 Convergence Speeds. . . . . . . . . . . . . . . . . . . . . . . . 15 Reference . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16 3 Solving Systems of Linear Equations. . . . . . . . . . . . . . . . . . . . . . 17 3.1 Introduction. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17 3.2 Examples. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18 3.3 Condition Number(s) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19 3.4 Approaches Best Avoided . . . . . . . . . . . . . . . . . . . . . . . . . . 22 3.5 Questions About A. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22 3.6 Direct Methods . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23 3.6.1 Backward or Forward Substitution . . . . . . . . . . . . . . 23 3.6.2 Gaussian Elimination . . . . . . . . . . . . . . . . . . . . . . . 25 3.6.3 LU Factorization . . . . . . . . . . . . . . . . . . . . . . . . . . 25 3.6.4 Iterative Improvement. . . . . . . . . . . . . . . . . . . . . . . 29 3.6.5 QR Factorization . . . . . . . . . . . . . . . . . . . . . . . . . . 29 3.6.6 Singular Value Decomposition. . . . . . . . . . . . . . . . . 33 vii viii Contents 3.7 Iterative Methods . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 35 3.7.1 Classical Iterative Methods . . . . . . . . . . . . . . . . . . . 35 3.7.2 Krylov Subspace Iteration . . . . . . . . . . . . . . . . . . . . 38 3.8 Taking Advantage of the Structure of A . . . . . . . . . . . . . . . . 42 3.8.1 A Is Symmetric Positive Definite. . . . . . . . . . . . . . . 42 3.8.2 A Is Toeplitz . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 43 3.8.3 A Is Vandermonde . . . . . . . . . . . . . . . . . . . . . . . . . 43 3.8.4 A Is Sparse . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 43 3.9 Complexity Issues . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 44 3.9.1 Counting Flops. . . . . . . . . . . . . . . . . . . . . . . . . . . . 44 3.9.2 Getting the Job Done Quickly . . . . . . . . . . . . . . . . . 45 3.10 MATLAB Examples. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 47 3.10.1 A Is Dense . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 47 3.10.2 A Is Dense and Symmetric Positive Definite . . . . . . . 52 3.10.3 A Is Sparse . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 53 3.10.4 A Is Sparse and Symmetric Positive Definite. . . . . . . 54 3.11 In Summary. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 55 References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 56 4 Solving Other Problems in Linear Algebra . . . . . . . . . . . . . . . . . 59 4.1 Inverting Matrices . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 59 4.2 Computing Determinants. . . . . . . . . . . . . . . . . . . . . . . . . . . 60 4.3 Computing Eigenvalues and Eigenvectors . . . . . . . . . . . . . . . 61 4.3.1 Approach Best Avoided . . . . . . . . . . . . . . . . . . . . . 61 4.3.2 Examples of Applications . . . . . . . . . . . . . . . . . . . . 62 4.3.3 Power Iteration. . . . . . . . . . . . . . . . . . . . . . . . . . . . 64 4.3.4 Inverse Power Iteration. . . . . . . . . . . . . . . . . . . . . . 65 4.3.5 Shifted Inverse Power Iteration . . . . . . . . . . . . . . . . 66 4.3.6 QR Iteration. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 67 4.3.7 Shifted QR Iteration . . . . . . . . . . . . . . . . . . . . . . . . 69 4.4 MATLAB Examples. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 70 4.4.1 Inverting a Matrix . . . . . . . . . . . . . . . . . . . . . . . . . 70 4.4.2 Evaluating a Determinant . . . . . . . . . . . . . . . . . . . . 71 4.4.3 Computing Eigenvalues. . . . . . . . . . . . . . . . . . . . . . 72 4.4.4 Computing Eigenvalues and Eigenvectors . . . . . . . . . 74 4.5 In Summary. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 75 References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 76 5 Interpolating and Extrapolating . . . . . . . . . . . . . . . . . . . . . . . . . 77 5.1 Introduction. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 77 5.2 Examples. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 78 Contents ix 5.3 Univariate Case . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 79 5.3.1 Polynomial Interpolation. . . . . . . . . . . . . . . . . . . . . 80 5.3.2 Interpolation by Cubic Splines. . . . . . . . . . . . . . . . . 84 5.3.3 Rational Interpolation . . . . . . . . . . . . . . . . . . . . . . . 86 5.3.4 Richardson’s Extrapolation . . . . . . . . . . . . . . . . . . . 88 5.4 Multivariate Case. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 89 5.4.1 Polynomial Interpolation. . . . . . . . . . . . . . . . . . . . . 89 5.4.2 Spline Interpolation . . . . . . . . . . . . . . . . . . . . . . . . 90 5.4.3 Kriging . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 90 5.5 MATLAB Examples. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 93 5.6 In Summary. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 95 References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 97 6 Integrating and Differentiating Functions . . . . . . . . . . . . . . . . . . 99 6.1 Examples. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 100 6.2 Integrating Univariate Functions. . . . . . . . . . . . . . . . . . . . . . 101 6.2.1 Newton–Cotes Methods. . . . . . . . . . . . . . . . . . . . . . 102 6.2.2 Romberg’s Method. . . . . . . . . . . . . . . . . . . . . . . . . 106 6.2.3 Gaussian Quadrature. . . . . . . . . . . . . . . . . . . . . . . . 107 6.2.4 Integration via the Solution of an ODE. . . . . . . . . . . 109 6.3 Integrating Multivariate Functions . . . . . . . . . . . . . . . . . . . . 109 6.3.1 Nested One-Dimensional Integrations. . . . . . . . . . . . 110 6.3.2 Monte Carlo Integration . . . . . . . . . . . . . . . . . . . . . 111 6.4 Differentiating Univariate Functions. . . . . . . . . . . . . . . . . . . 112 6.4.1 First-Order Derivatives . . . . . . . . . . . . . . . . . . . . . . 113 6.4.2 Second-Order Derivatives . . . . . . . . . . . . . . . . . . . . 116 6.4.3 Richardson’s Extrapolation . . . . . . . . . . . . . . . . . . . 117 6.5 Differentiating Multivariate Functions. . . . . . . . . . . . . . . . . . 119 6.6 Automatic Differentiation . . . . . . . . . . . . . . . . . . . . . . . . . . 120 6.6.1 Drawbacks of Finite-Difference Evaluation . . . . . . . . 120 6.6.2 Basic Idea of Automatic Differentiation . . . . . . . . . . 121 6.6.3 Backward Evaluation . . . . . . . . . . . . . . . . . . . . . . . 123 6.6.4 Forward Evaluation. . . . . . . . . . . . . . . . . . . . . . . . . 127 6.6.5 Extension to the Computation of Hessians. . . . . . . . . 129 6.7 MATLAB Examples. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 131 6.7.1 Integration. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 131 6.7.2 Differentiation . . . . . . . . . . . . . . . . . . . . . . . . . . . . 134 6.8 In Summary. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 137 References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 138 7 Solving Systems of Nonlinear Equations . . . . . . . . . . . . . . . . . . . 139 7.1 What Are the Differences with the Linear Case? . . . . . . . . . . 139 7.2 Examples. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 140 x Contents 7.3 One Equation in One Unknown . . . . . . . . . . . . . . . . . . . . . . 141 7.3.1 Bisection Method. . . . . . . . . . . . . . . . . . . . . . . . . . 142 7.3.2 Fixed-Point Iteration. . . . . . . . . . . . . . . . . . . . . . . . 143 7.3.3 Secant Method. . . . . . . . . . . . . . . . . . . . . . . . . . . . 144 7.3.4 Newton’s Method. . . . . . . . . . . . . . . . . . . . . . . . . . 144 7.4 Multivariate Systems. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 148 7.4.1 Fixed-Point Iteration. . . . . . . . . . . . . . . . . . . . . . . . 148 7.4.2 Newton’s Method. . . . . . . . . . . . . . . . . . . . . . . . . . 149 7.4.3 Quasi–Newton Methods . . . . . . . . . . . . . . . . . . . . . 150 7.5 Where to Start From?. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 153 7.6 When to Stop?. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 154 7.7 MATLAB Examples. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 155 7.7.1 One Equation in One Unknown . . . . . . . . . . . . . . . . 155 7.7.2 Multivariate Systems. . . . . . . . . . . . . . . . . . . . . . . . 160 7.8 In Summary. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 164 References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 165 8 Introduction to Optimization . . . . . . . . . . . . . . . . . . . . . . . . . . . 167 8.1 A Word of Caution. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 167 8.2 Examples. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 167 8.3 Taxonomy. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 168 8.4 How About a Free Lunch? . . . . . . . . . . . . . . . . . . . . . . . . . 172 8.4.1 There Is No Such Thing . . . . . . . . . . . . . . . . . . . . . 173 8.4.2 You May Still Get a Pretty Inexpensive Meal . . . . . . 174 8.5 In Summary. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 174 References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 175 9 Optimizing Without Constraint. . . . . . . . . . . . . . . . . . . . . . . . . . 177 9.1 Theoretical Optimality Conditions . . . . . . . . . . . . . . . . . . . . 177 9.2 Linear Least Squares. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 183 9.2.1 Quadratic Cost in the Error . . . . . . . . . . . . . . . . . . . 183 9.2.2 Quadratic Cost in the Decision Variables . . . . . . . . . 184 9.2.3 Linear Least Squares via QR Factorization . . . . . . . . 188 9.2.4 Linear Least Squares via Singular Value Decomposition . . . . . . . . . . . . . . . . . . . . . . . . . . . 191 9.2.5 What to Do if FTF Is Not Invertible?. . . . . . . . . . . . 194 9.2.6 Regularizing Ill-Conditioned Problems . . . . . . . . . . . 194 9.3 Iterative Methods . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 195 9.3.1 Separable Least Squares . . . . . . . . . . . . . . . . . . . . . 195 9.3.2 Line Search. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 196 9.3.3 Combining Line Searches . . . . . . . . . . . . . . . . . . . . 200 Contents xi 9.3.4 Methods Based on a Taylor Expansion of the Cost. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 201 9.3.5 A Method That Can Deal with Nondifferentiable Costs. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 216 9.4 Additional Topics. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 220 9.4.1 Robust Optimization. . . . . . . . . . . . . . . . . . . . . . . . 220 9.4.2 Global Optimization. . . . . . . . . . . . . . . . . . . . . . . . 222 9.4.3 Optimization on a Budget . . . . . . . . . . . . . . . . . . . . 225 9.4.4 Multi-Objective Optimization. . . . . . . . . . . . . . . . . . 226 9.5 MATLAB Examples. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 227 9.5.1 Least Squares on a Multivariate Polynomial Model. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 227 9.5.2 Nonlinear Estimation . . . . . . . . . . . . . . . . . . . . . . . 236 9.6 In Summary. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 241 References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 243 10 Optimizing Under Constraints . . . . . . . . . . . . . . . . . . . . . . . . . . 245 10.1 Introduction. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 245 10.1.1 Topographical Analogy. . . . . . . . . . . . . . . . . . . . . . 245 10.1.2 Motivations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 245 10.1.3 Desirable Properties of the Feasible Set . . . . . . . . . . 246 10.1.4 Getting Rid of Constraints. . . . . . . . . . . . . . . . . . . . 247 10.2 Theoretical Optimality Conditions . . . . . . . . . . . . . . . . . . . . 248 10.2.1 Equality Constraints . . . . . . . . . . . . . . . . . . . . . . . . 248 10.2.2 Inequality Constraints. . . . . . . . . . . . . . . . . . . . . . . 252 10.2.3 General Case: The KKT Conditions . . . . . . . . . . . . . 256 10.3 Solving the KKT Equations with Newton’s Method . . . . . . . . 256 10.4 Using Penalty or Barrier Functions. . . . . . . . . . . . . . . . . . . . 257 10.4.1 Penalty Functions. . . . . . . . . . . . . . . . . . . . . . . . . . 257 10.4.2 Barrier Functions . . . . . . . . . . . . . . . . . . . . . . . . . . 259 10.4.3 Augmented Lagrangians . . . . . . . . . . . . . . . . . . . . . 260 10.5 Sequential Quadratic Programming. . . . . . . . . . . . . . . . . . . . 261 10.6 Linear Programming. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 261 10.6.1 Standard Form. . . . . . . . . . . . . . . . . . . . . . . . . . . . 265 10.6.2 Principle of Dantzig’s Simplex Method. . . . . . . . . . . 266 10.6.3 The Interior-Point Revolution. . . . . . . . . . . . . . . . . . 271 10.7 Convex Optimization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 272 10.7.1 Convex Feasible Sets . . . . . . . . . . . . . . . . . . . . . . . 273 10.7.2 Convex Cost Functions. . . . . . . . . . . . . . . . . . . . . . 273 10.7.3 Theoretical Optimality Conditions . . . . . . . . . . . . . . 275 10.7.4 Lagrangian Formulation . . . . . . . . . . . . . . . . . . . . . 275 10.7.5 Interior-Point Methods . . . . . . . . . . . . . . . . . . . . . . 277 10.7.6 Back to Linear Programming. . . . . . . . . . . . . . . . . . 278 10.8 Constrained Optimization on a Budget . . . . . . . . . . . . . . . . . 280

ebook img

Numerical Methods and Optimization: A Consumer Guide PDF

485 Pages·2014·6.896 MB·English
by  Éric Walter (auth.)
#Mathematics #Optimization. Operations Research
Save to my drive
Quick download
Download

Download Numerical Methods and Optimization: A Consumer Guide PDF Free - Full Version

by Éric Walter (auth.)| 2014| 485 pages| 6.896| English

Download Numerical Methods and Optimization: A Consumer Guide by Éric Walter (auth.) in PDF format completely FREE. No registration required, no payment needed. Get instant access to this valuable resource on PDFdrive.to!

Free Download PDF

About Numerical Methods and Optimization: A Consumer Guide

No description available for this book.

Detailed Information

Author:Éric Walter (auth.)
Publication Year:2014
Pages:485
Language:English
File Size:6.896
Format:PDF
Price:FREE
Download Free PDF

Safe & Secure Download - No registration required

Why Choose PDFdrive for Your Free Numerical Methods and Optimization: A Consumer Guide Download?

  • 100% Free: No hidden fees or subscriptions required for one book every day.
  • No Registration: Immediate access is available without creating accounts for one book every day.
  • Safe and Secure: Clean downloads without malware or viruses
  • Multiple Formats: PDF, MOBI, Mpub,... optimized for all devices
  • Educational Resource: Supporting knowledge sharing and learning

Free Books Similar to Numerical Methods and Optimization: A Consumer Guide

Frequently Asked Questions

Is it really free to download Numerical Methods and Optimization: A Consumer Guide PDF?

Yes, on https://PDFdrive.to you can download Numerical Methods and Optimization: A Consumer Guide by Éric Walter (auth.) completely free. We don't require any payment, subscription, or registration to access this PDF file. For 3 books every day.

How can I read Numerical Methods and Optimization: A Consumer Guide on my mobile device?

After downloading Numerical Methods and Optimization: A Consumer Guide PDF, you can open it with any PDF reader app on your phone or tablet. We recommend using Adobe Acrobat Reader, Apple Books, or Google Play Books for the best reading experience.

Is this the full version of Numerical Methods and Optimization: A Consumer Guide?

Yes, this is the complete PDF version of Numerical Methods and Optimization: A Consumer Guide by Éric Walter (auth.). You will be able to read the entire content as in the printed version without missing any pages.

Is it legal to download Numerical Methods and Optimization: A Consumer Guide PDF for free?

https://PDFdrive.to provides links to free educational resources available online. We do not store any files on our servers. Please be aware of copyright laws in your country before downloading.

The materials shared are intended for research, educational, and personal use in accordance with fair use principles.

We use cookies

We use cookies to ensure you get the best browsing experience on our website. By clicking "Accept Cookies", you agree that we can store cookies on your device in accordance with our Terms and Privacy.