Introduction to optimization and semidifferential calculus

362 Pages·2012·3.133 MB·English

by  Michael C. Delfour

I
ntroductIon to
o
ptImIzatIon and
S
emIdIfferentIal
c
alculuS
MO12_Delfour_FM-A.indd 1 1/11/2012 11:28:44 AM
MOS-SIAM Series on Optimization
This series is published jointly by the Mathematical Optimization Society and the Society for Industrial
and Applied Mathematics. It includes research monographs, books on applications, textbooks at all
levels, and tutorials. Besides being of high scientific quality, books in the series must advance the
understanding and practice of optimization. They must also be written clearly and at an appropriate
level for the intended audience.
Editor-in-Chief
Thomas Liebling
École Polytechnique Fédérale de Lausanne

Editorial Board
William Cook, Georgia Tech
Gérard Cornuejols, Carnegie Mellon University
Oktay Gunluk, IBM T.J. Watson Research Center
Michael Jünger, Universität zu Köln
Adrian S. Lewis, Cornell University
Pablo Parrilo, Massachusetts Institute of Technology
Wiliam Pulleyblank, United States Military Academy at West Point
Daniel Ralph, University of Cambridge
Éva Tardos, Cornell University
Ariela Sofer, George Mason University
Laurence Wolsey, Université Catholique de Louvain

Series Volumes
Delfour, M. C., Introduction to Optimization and Semidifferential Calculus
Ulbrich, Michael, Semismooth Newton Methods for Variational Inequalities and Constrained
Optimization Problems in Function Spaces
Biegler, Lorenz T., Nonlinear Programming: Concepts, Algorithms, and Applications to
Chemical Processes
Shapiro, Alexander, Dentcheva, Darinka, and Ruszczyn´ski, Andrzej, Lectures on Stochastic
Programming: Modeling and Theory
Conn, Andrew R., Scheinberg, Katya, and Vicente, Luis N., Introduction to Derivative-Free
Optimization
Ferris, Michael C., Mangasarian, Olvi L., and Wright, Stephen J., Linear Programming with MATLAB
Attouch, Hedy, Buttazzo, Giuseppe, and Michaille, Gérard, Variational Analysis in Sobolev
and BV Spaces: Applications to PDEs and Optimization
Wallace, Stein W. and Ziemba, William T., editors, Applications of Stochastic Programming
Grötschel, Martin, editor, The Sharpest Cut: The Impact of Manfred Padberg and His Work
Renegar, James, A Mathematical View of Interior-Point Methods in Convex Optimization
Ben-Tal, Aharon and Nemirovski, Arkadi, Lectures on Modern Convex Optimization: Analysis,
Algorithms, and Engineering Applications
Conn, Andrew R., Gould, Nicholas I. M., and Toint, Phillippe L., Trust-Region Methods
MO12_Delfour_FM-A.indd 2 1/11/2012 11:28:44 AM
I
ntroductIon to
o
ptImIzatIon and
S
emIdIfferentIal
c
alculuS

M. C. Delfour
Centre de Recherches Mathématiques
and
Département de Mathématiques et de Statistique
Université de Montréal
Montréal, Canada
Society for Industrial and Applied Mathematics Mathematical Optimization Society
Philadelphia Philadelphia
MO12_Delfour_FM-A.indd 3 1/11/2012 11:28:44 AM
Copyright © 2012 by the Society for Industrial and Applied Mathematics and the Mathematical
Optimization Society
10 9 8 7 6 5 4 3 2 1
All rights reserved. Printed in the United States of America. No part of this book may be
reproduced, stored, or transmitted in any manner without the written permission of the
publisher. For information, write to the Society for Industrial and Applied Mathematics,
3600 Market Street, 6th Floor, Philadelphia, PA 19104-2688.
Trademarked names may be used in this book without the inclusion of a trademark symbol.
These names are used in an editorial context only; no infringement of trademark is intended.
Library of Congress Cataloging-in-Publication Data
Delfour, Michel C., 1943-
Introduction to optimization and semidifferential calculus / M. C. Delfour.
p. cm. -- (MOS-SIAM series on optimization)
Includes bibliographical references and index.
ISBN 978-1-611972-14-6
1. Mathematical optimization. 2. Differential calculus. I. Title.
QA402.5.D348 2012
515’.642--dc23
2011040535
is a registered trademark.
MO12_Delfour_FM-A.indd 4 1/11/2012 11:28:44 AM
To Francis and Guillaume
MO12_Delfour_FM-A.indd 5 1/11/2012 11:28:45 AM
Contents
ListofFigures xi
Preface xiii
AGreatandBeautifulSubject. . . . . . . . . . . . . . . . . . . . . . . . . . . . xiii
IntendedAudienceandObjectivesoftheBook. . . . . . . . . . . . . . . . . . . xiv
NumberingandReferencingSystems. . . . . . . . . . . . . . . . . . . . . . . . xv
Acknowledgments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xv
1 Introduction 1
1 MinimaandMaxima . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1
2 CalculusofVariationsandItsOffsprings . . . . . . . . . . . . . . . . . . . 2
3 ContentsoftheBook . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3
4 SomeBackgroundMaterialinClassicalAnalysis . . . . . . . . . . . . . . 4
4.1 GreatestLowerBoundandLeastUpperBound . . . . . . . . . . . 5
4.2 EuclideanSpace . . . . . . . . . . . . . . . . . . . . . . . . . . . 6
4.2.1 CartesianProduct,Balls,andContinuity . . . . . . . . . 6
4.2.2 Open,Closed,andCompactSets . . . . . . . . . . . . . 7
4.3 Functions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9
4.3.1 DefinitionsandConvention . . . . . . . . . . . . . . . . 9
4.3.2 ContinuityofaFunction. . . . . . . . . . . . . . . . . . 10
2 Existence,Convexities,andConvexification 11
1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11
2 WeierstrassExistenceTheorem . . . . . . . . . . . . . . . . . . . . . . . . 11
3 ExtremaofFunctionswithExtendedValues . . . . . . . . . . . . . . . . . 12
4 LowerandUpperSemicontinuities . . . . . . . . . . . . . . . . . . . . . . 16
5 ExistenceofMinimizersinU . . . . . . . . . . . . . . . . . . . . . . . . . 21
5.1 U Compact . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21
5.2 U ClosedbutnotNecessarilyBounded . . . . . . . . . . . . . . . 24
5.3 GrowthPropertyatInfinity . . . . . . . . . . . . . . . . . . . . . . 26
5.4 SomePropertiesoftheSetofMinimizers . . . . . . . . . . . . . . 28
6 (cid:1)Ekeland’sVariationalPrinciple . . . . . . . . . . . . . . . . . . . . . . 29
7 Convexity,Quasiconvexity,StrictConvexity,andUniqueness. . . . . . . . 32
7.1 ConvexityandConcavity. . . . . . . . . . . . . . . . . . . . . . . 32
7.2 Quasiconvexity . . . . . . . . . . . . . . . . . . . . . . . . . . . . 39
vii
viii Contents
7.3 StrictConvexityandUniqueness . . . . . . . . . . . . . . . . . . . 40
8 LinearandAffineSubspaceandRelativeInterior . . . . . . . . . . . . . . 43
8.1 Definitions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 43
8.2 DomainofConvexFunctions . . . . . . . . . . . . . . . . . . . . 45
9 ConvexificationandFenchel–LegendreTransform. . . . . . . . . . . . . . 46
9.1 ConvexlscFunctionsasUpperEnvelopesofAffineFunctions . . . 46
9.2 Fenchel–LegendreTransform . . . . . . . . . . . . . . . . . . . . 51
9.3 LscConvexificationandFenchel–LegendreBitransform . . . . . . 55
9.4 InfimaoftheObjectiveFunctionandofItslscConvexified
Function . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 58
9.5 PrimalandDualProblemsandFenchelDualityTheorem . . . . . . 59
10 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 63
3 Semidifferentiability,Differentiability,Continuity,andConvexities 67
1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 67
2 Real-ValuedFunctionsofaRealVariable . . . . . . . . . . . . . . . . . . 69
2.1 ContinuityandDifferentiability . . . . . . . . . . . . . . . . . . . 72
2.2 MeanValueTheorem . . . . . . . . . . . . . . . . . . . . . . . . . 73
2.3 PropertyoftheDerivativeofaFunctionDifferentiable
Everywhere . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 74
2.4 Taylor’sTheorem . . . . . . . . . . . . . . . . . . . . . . . . . . . 75
3 Real-ValuedFunctionsofSeveralRealVariables . . . . . . . . . . . . . . 76
3.1 GeometricalApproachviatheDifferential . . . . . . . . . . . . . . 76
3.2 Semidifferentials,Differentials,Gradient,andPartial
Derivatives . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 79
3.2.1 Definitions . . . . . . . . . . . . . . . . . . . . . . . . . 79
3.2.2 ExamplesandCounterexamples. . . . . . . . . . . . . . 82
3.2.3 Gradient . . . . . . . . . . . . . . . . . . . . . . . . . . 86
3.2.4 FréchetDifferential . . . . . . . . . . . . . . . . . . . . 88
3.3 HadamardDifferentialandSemidifferential . . . . . . . . . . . . . 91
3.4 OperationsonSemidifferentiableFunctions . . . . . . . . . . . . . 96
3.4.1 AlgebraicOperations,LowerandUpperEnvelopes . . . 96
3.4.2 ChainRulefortheCompositionofFunctions. . . . . . . 98
3.5 LipschitzianFunctions . . . . . . . . . . . . . . . . . . . . . . . . 103
3.5.1 DefinitionsandTheirHadamardSemidifferential . . . . 103
3.5.2 (cid:1)DiniandHadamardUpperandLower
Semidifferentials. . . . . . . . . . . . . . . . . . . . . . 104
3.5.3 (cid:1)ClarkeUpperandLowerSemidifferentials. . . . . . . 105
3.5.4 (cid:1)PropertiesofUpperandLowerSubdifferentials . . . . 107
3.6 Continuity,HadamardSemidifferential,andFréchet
Differential . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 110
3.7 MeanValueTheoremforFunctionsofSeveralVariables . . . . . . 111
3.8 FunctionsofClassesC(0)andC(1) . . . . . . . . . . . . . . . . . . 113
3.9 FunctionsofClassC(p)andHessianMatrix . . . . . . . . . . . . . 116
4 ConvexandSemiconvexFunctions. . . . . . . . . . . . . . . . . . . . . . 119
4.1 DirectionallyDifferentiableConvexFunctions . . . . . . . . . . . 119
4.2 (cid:1)SemidifferentiabilityandContinuityofConvexFunctions . . . . 122
Contents ix
4.2.1 ConvexityandSemidifferentiability . . . . . . . . . . . 123
4.2.2 ConvexityandContinuity . . . . . . . . . . . . . . . . . 126
4.3 (cid:1)LowerHadamardSemidifferentialataBoundaryPointof
theDomain . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 131
4.4 (cid:1)SemiconvexFunctionsandHadamardSemidifferentiability . . . 132
5 (cid:1)SemidifferentialofaParametrizedExtremum . . . . . . . . . . . . . . . 139
5.1 SemidifferentialofanInfimumwithrespecttoaParameter . . . . . 139
5.2 InfimumofaParametrizedQuadraticFunction . . . . . . . . . . . 143
6 SummaryofSemidifferentiabilityandDifferentiability . . . . . . . . . . . 148
7 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 149
4 OptimalityConditions 153
1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 153
2 UnconstrainedDifferentiableOptimization . . . . . . . . . . . . . . . . . 154
2.1 SomeBasicResultsandExamples . . . . . . . . . . . . . . . . . . 154
2.2 LeastandGreatestEigenvaluesofaSymmetricMatrix . . . . . . . 164
2.3 (cid:1)HadamardSemidifferentialoftheLeastEigenvalue . . . . . . . 166
3 OptimalityConditionsforU Convex . . . . . . . . . . . . . . . . . . . . . 168
3.1 Cones . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 168
3.2 ConvexGateauxDifferentiableObjectiveFunction . . . . . . . . . 170
3.3 SemidifferentiableObjectiveFunction . . . . . . . . . . . . . . . . 177
3.4 (cid:1)ArbitraryConvexObjectiveFonction . . . . . . . . . . . . . . . 178
4 AdmissibleDirectionsandTangentConestoU . . . . . . . . . . . . . . . 180
4.1 SetofAdmissibleDirectionsorHalf-Tangents . . . . . . . . . . . 180
4.2 PropertiesoftheTangentConesT (x)andS (x) . . . . . . . . . . 184
U U
4.3 (cid:1)Clarke’sandOtherTangentCones . . . . . . . . . . . . . . . . 187
5 Orthogonality,Transposition,andDualCones . . . . . . . . . . . . . . . . 190
5.1 OrthogonalityandTransposition . . . . . . . . . . . . . . . . . . . 190
5.2 DualCones . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 192
6 NecessaryOptimalityConditionsforU Arbitrary . . . . . . . . . . . . . . 197
6.1 NecessaryOptimalityCondition . . . . . . . . . . . . . . . . . . . 197
6.1.1 HadamardSemidifferentiableObjectiveFunction . . . . 197
6.1.2 (cid:1)ArbitraryObjectiveFunction . . . . . . . . . . . . . . 199
6.2 DualNecessaryOptimalityCondition . . . . . . . . . . . . . . . . 200
7 AffineEqualityandInequalityConstraints . . . . . . . . . . . . . . . . . . 202
7.1 CharacterizationofT (x) . . . . . . . . . . . . . . . . . . . . . . 202
U
7.2 DualConesforLinearConstraints . . . . . . . . . . . . . . . . . . 203
7.3 LinearProgrammingProblem . . . . . . . . . . . . . . . . . . . . 208
7.4 SomeElementsofTwo-PersonZero-SumGames . . . . . . . . . . 217
7.5 FenchelPrimalandDualProblemsandtheLagrangian . . . . . . . 220
7.6 QuadraticProgrammingProblem . . . . . . . . . . . . . . . . . . 223
7.6.1 TheoremofFrank–Wolfe . . . . . . . . . . . . . . . . . 223
7.6.2 NonconvexObjectiveFunction . . . . . . . . . . . . . . 226
7.6.3 ConvexObjectiveFunction . . . . . . . . . . . . . . . . 228
7.7 FréchetDifferentiableObjectiveFunction . . . . . . . . . . . . . . 234
7.8 Farkas’LemmaandItsExtension . . . . . . . . . . . . . . . . . . 234
x Contents
8 (cid:1)GlimpseatOptimalityviaSubdifferentials . . . . . . . . . . . . . . . . 235
9 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 237
5 ConstrainedDifferentiableOptimization 241
1 ConstrainedProblems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 241
2 EqualityContraints: LagrangeMultipliersTheorem . . . . . . . . . . . . . 242
2.1 TangentConeofAdmissibleDirections . . . . . . . . . . . . . . . 242
2.2 JacobianMatrixandImplicitFunctionTheorem. . . . . . . . . . . 243
2.3 LagrangeMultipliersTheorem . . . . . . . . . . . . . . . . . . . . 245
3 InequalityContraints: Karush–Kuhn–TuckerTheorem . . . . . . . . . . . 256
4 SimultaneousEqualityandInequalityConstraints . . . . . . . . . . . . . . 270
5 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 286
A InverseandImplicitFunctionTheorems 291
1 InverseFunctionTheorem . . . . . . . . . . . . . . . . . . . . . . . . . . 291
2 ImplicitFunctionTheorem . . . . . . . . . . . . . . . . . . . . . . . . . . 292
B AnswerstoExercises 295
1 ExercisesofChapter2 . . . . . . . . . . . . . . . . . . . . . . . . . . . . 295
2 ExercisesofChapter3 . . . . . . . . . . . . . . . . . . . . . . . . . . . . 309
3 ExercisesofChapter4 . . . . . . . . . . . . . . . . . . . . . . . . . . . . 315
4 ExercisesofChapter5 . . . . . . . . . . . . . . . . . . . . . . . . . . . . 327
ElementsofBibliography 339
IndexofNotation 349
Index 351
List of Figures
2.1 Discontinuousfunctionshavingaminimizingpointin[0,1] . . . . . . . . . 12
2.2 Exampleofanlscfunction . . . . . . . . . . . . . . . . . . . . . . . . . . 20
2.3 Lscfunctionthatisnotuscat0 . . . . . . . . . . . . . . . . . . . . . . . . 23
2.4 Convexfunctionandconcavefunction . . . . . . . . . . . . . . . . . . . . 33
2.5 Exampleofaquasiconvexfunctionthatisnotconvex . . . . . . . . . . . . 40
2.6 Examplesofconvexfunctions: f (notlsc),cl f,andg(lsc) . . . . . . . . 46
2.7 Thefunctionf(x,y)=x2/y fory≥ε>0andsomesmallε . . . . . . . . 47
2.8 Casesy∈domf (left)andy(cid:3)∈domf (right) . . . . . . . . . . . . . . . . 48
3.1 Exampleofrightandleftdifferentiability . . . . . . . . . . . . . . . . . . 71
3.2 Regiondeterminedbythefunctionsαandβ . . . . . . . . . . . . . . . . . 74
3.3 Thefunctionf(x)=|x|inaneighborhoodofx=0forn=1 . . . . . . . 81
3.4 Example3.5 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 83
3.5 Examples3.6and3.8inlogarithmicscale . . . . . . . . . . . . . . . . . . 85
3.6 Example3.7 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 87
3.7 Upperenvelopeoftwofunctions . . . . . . . . . . . . . . . . . . . . . . . 97
3.8 Upperenvelopeofthreefunctions . . . . . . . . . . . . . . . . . . . . . . 97
3.9 Thetwoconvexfunctionsg andg on[0,1]ofExample4.1 . . . . . . . . 122
1 2
3.10 FunctionofExercise7.1 . . . . . . . . . . . . . . . . . . . . . . . . . . . 150
4.1 Banana-shapedlevelsetsoftheRosenbrockfunctionat0.25,1,4,9,16 . . 159
4.2 Basisfunctions(φ ,...,φ ,...,φ )ofP 1(0,1) . . . . . . . . . . . . . . . . 161
0 i n n
4.3 Anonconvexclosedconein0 . . . . . . . . . . . . . . . . . . . . . . . . 169
4.4 Closedconvexconein0generatedbytheclosedconvexsetV . . . . . . . 169
4.5 Closedconvexconein0generatedinR3byanonconvexsetV . . . . . . . 169
4.6 ConvexsetU tangenttothelevelsetoff throughx∈U . . . . . . . . . . 171
4.7 TangencyoftheaffinesubspaceAorthelinearsubspaceS
toalevelsetoff . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 172
4.8 TangencyofU toalevelsetofthefunctionf atx∈U . . . . . . . . . . . 177
4.9 Half-tangentdh(0;+1)tothepathh(t)inU atthepointh(0)=x. . . . . . 180
4.10 Cuspatx∈∂U: U andT (x) . . . . . . . . . . . . . . . . . . . . . . . . 182
U
4.11 Firstexample: U andT (x) . . . . . . . . . . . . . . . . . . . . . . . . . 183
U
4.12 Secondexample: U andT (x) . . . . . . . . . . . . . . . . . . . . . . . . 183
U
4.13 Thirdexample: U andT (x) . . . . . . . . . . . . . . . . . . . . . . . . . 183
U
4.14 Fourthexample: U andT (x) . . . . . . . . . . . . . . . . . . . . . . . . 184
U
xi