Selected Applications of Convex Optimization

150 Pages·2015·2.392 MB·English

by  Li Li

Springer Optimization and Its Applications 103
Li Li

Selected
Applications
of Convex
Optimization
Springer Optimization and Its Applications
Volume 103
ManagingEditor
PanosM.Pardalos(UniversityofFlorida)
Editor–CombinatorialOptimization
Ding-ZhuDu(UniversityofTexasatDallas)
AdvisoryBoard
J.Birge(UniversityofChicago)
C.A.Floudas(PrincetonUniversity)
F.Giannessi(UniversityofPisa)
H.D.Sherali(VirginiaPolytechnicandStateUniversity)
T.Terlaky(McMasterUniversity)
Y.Ye(StanfordUniversity)
AimsandScope
Optimizationhasbeenexpandinginalldirectionsatanastonishingrateduringthe
lastfewdecades.Newalgorithmicandtheoreticaltechniqueshavebeendeveloped,
thediffusionintootherdisciplineshasproceededatarapidpace,andourknowledge
ofallaspectsofthefieldhasgrownevenmoreprofound.Atthesametime,oneof
themoststrikingtrendsinoptimizationistheconstantlyincreasingemphasisonthe
interdisciplinarynatureofthefield.Optimizationhasbeenabasictoolinallareas
ofappliedmathematics,engineering,medicine,economics,andothersciences.
The series Springer Optimization and Its Applications publishes undergraduate
and graduate textbooks, monographs and state-of-the-art expository work that
focusonalgorithmsforsolvingoptimizationproblemsandalsostudyapplications
involvingsuchproblems.Someofthetopicscoveredincludenonlinearoptimization
(convex and nonconvex), network flow problems, stochastic optimization, optimal
control, discrete optimization, multi-objective programming, description of soft-
warepackages,approximationtechniquesandheuristicapproaches.
Moreinformationaboutthisseriesathttp://www.springer.com/series/7393

Li Li
Selected Applications
of Convex Optimization
123
LiLi
DepartmentofAutomation
TsinghuaUniversity
Beijing,China
Additionalmaterialtothisbookcanbedownloadedfromhttp://extras.springer.com.
ISSN1931-6828 ISSN1931-6836 (electronic)
SpringerOptimizationandItsApplications
ISBN978-3-662-46355-0 ISBN978-3-662-46356-7 (eBook)
DOI10.1007/978-3-662-46356-7
JointlypublishedwithTsinghuaUniversityPress,Beijing
ISBN:978-7-302-39029-9TsinghuaUniversityPress,Beijing
LibraryofCongressControlNumber:2015932423
MathematicsSubjectClassification:90C25,90C90,65K10,65K15,49M20
SpringerHeidelbergNewYorkDordrechtLondon
©TsinghuaUniversityPress,BeijingandSpringer-VerlagBerlinHeidelberg2015
Thisworkissubjecttocopyright.AllrightsarereservedbythePublishers,whetherthewholeorpartof
thematerialisconcerned,specificallytherightsoftranslation,reprinting,reuseofillustrations,recitation,
broadcasting,reproductiononmicrofilmsorinanyotherphysicalway,andtransmissionorinformation
storageandretrieval,electronicadaptation,computersoftware,orbysimilarordissimilarmethodology
nowknownorhereafterdeveloped.
Theuseofgeneraldescriptivenames,registerednames,trademarks,servicemarks,etc.inthispublication
doesnotimply,evenintheabsenceofaspecificstatement,thatsuchnamesareexemptfromtherelevant
protectivelawsandregulationsandthereforefreeforgeneraluse.
Thepublishers,theauthorsandtheeditorsaresafetoassumethattheadviceandinformationinthisbook
arebelievedtobetrueandaccurateatthedateofpublication.Neitherthepublishersnortheauthorsor
theeditorsgiveawarranty,expressorimplied,withrespecttothematerialcontainedhereinorforany
errorsoromissionsthatmayhavebeenmade.
Printedonacid-freepaper
Springer-VerlagGmbHBerlinHeidelbergispartofSpringerScience+BusinessMedia(www.springer.
com)
ToMyBelovedParents!

Preface
This book focuses on the applications of convex optimization, a special class
of mathematical optimization techniques. Related problems arise in a variety of
applicationsandcanbenumericallysolvedinanefficientway.Likewise,diversified
applicationshavecontributedtoconvexoptimizationandurgedthedevelopmentof
new optimization techniques. This intergrowth continues to produce new achieve-
ments.JohnTukeysaid,“Thebestthingaboutbeingastatisticianisthatyougetto
playineveryone’sbackyard.”While,inourhumbleopinion,“Thebestthingabout
being a convex optimization researcher is that you get to help build many ones’
backyards.”
As with any science worth its salt, convex optimization has a coherent formal
theory and a rambunctious experimental wing. There are already a number of
textbooks on the theory of convex optimization. So, we address the applications
inthisbook.
Tothoroughlysurveyapplicationsofconvexoptimizationinthisbookiscertainly
impossible due to the width and depth of covered topics. Here, we emphasize our
discussions on formulating empirical problems into formal convex optimization
problemsinsixrepresentativecategories.
In Chap.2, we introduce the supporting vector machines that are supervised
learning models for data classification. In Chap.3, we study the parameter esti-
mation problems particularly on maximum likelihood estimation and expectation
maximizationalgorithms.InChap.4,wediscussthenormapproximationandsome
widely adopted regularization tricks with a special emphasis on recently hottest
sparsity features. In Chap.5, we present the positive semidefinite programming
problems, linear matrix inequalities, and their applications in control theory. In
Chap.6, we give some interesting examples of convex relaxation methods. In
Chap.7,wevisitsomefrequentlyusedgeometricproblemsthatcanbehandledas
convexoptimizationproblems.
Webelievethattheserepresentativeapplicationswelldemonstratetheinterplay
ofconvexoptimizationtheoryandapplications.
The selection of topics is significantly influenced by the valuable textbook
ConvexOptimizationwrittenbyProf.StephenBoydandProf.LievenVandenberghe
vii
viii Preface
(a book that we require all my graduate students to carefully read for at least one
time). Convex Optimization contains much more topics than this book. However,
wemakenotablydeeperdiscussionsineachcategorythatarementionedaboveand
arrangesomeissuesinamoresyntheticway.
Similar to the writing purpose of Convex Optimization, our major goal is to
help the reader develop the skills needed to recognize and formulate convex
optimization problems that might be encountered in practice, since experiences of
manyresearchershadproventhegreatadvantagestorecognizingorformulatinga
problem as a convex optimization problem. All derivation processes are presented
indetailssothatreaderscanteachthemselveswithoutanydifficulties.
Thedraftofthisbookhadbeenusedforfiveyearsinagraduatecourse“Convex
Optimization and Applications” in Tsinghua University. The required background
of readers includes a solid knowledge of advanced calculus, linear algebra, basic
probability theory, and basic knowledge of convex optimization. This book is not
a text primarilyabout convex analysis or the mathematics of convex optimization.
Indeed,wehopethisbookwillbeusedasasupplementtextbookforseveraltypes
ofcourses,includingoperationsresearch,computerscience,statistics,datamining,
andmanyfieldsofscienceandengineering.Ourteachingexperiencesshowedthat
the covered materials also serve as useful references for beginners who are major
inmachinelearningandcontroltheoryfield.Moreover,upontherequestsofmany
readers,weplantoenrichthetopicsofthisbookinthecomingyearssothatitcould
servestudentsinotherfields.
Finally, we would like to thank Prof. Stephen Boyd at the Department of
ElectricalEngineering,StanfordUniversity,forthehelpfuldiscussions.Wewould
liketothankDr.XiaolingHuang,Dr.ZhengpengWu,Mr.KaidiYang,Mr.Yingde
Chen, as well as Mr. Wei Guo for typing some parts of this book and Mr. Jiajie
ZhangfordevelopingmostMatlabcodesnippetsaswellasallfiguresforthisbook.
We would also like to thank many of our students for debugging the typos of this
book.Withouttheirkindhelp,wecannotfinishthetediouswritingtasksinsucha
shorttime.
Due to our limited knowledge, this book might contain mistakes and typos.
Please be kind to e-mail us when you find any problems within this book. We are
morethanhappytorevisethisbookuponyournoticeandextenditscontentsinthe
followingteachingprocess.
Beijing,China LiLi
Contents
1 PreliminaryKnowledge ..................................................... 1
1.1 Nomenclatures .......................................................... 1
1.2 ConvexSetsandConvexFunctions.................................... 2
1.3 ConvexOptimization ................................................... 5
1.3.1 GradientDescentandCoordinateDescent .................... 5
1.3.2 Karush-Kuhn-Tucker(KKT)Conditions...................... 7
1.4 SomeLemmasinLinearAlgebra ...................................... 10
1.5 ABriefIntroductionofCVXToolbox................................. 11
Problems....................................................................... 13
References..................................................................... 15
2 SupportVectorMachines................................................... 17
2.1 BasicSVM .............................................................. 17
2.2 SoftMarginSVM....................................................... 22
2.3 KernelSVM............................................................. 28
2.4 Multi-kernelSVM ...................................................... 35
2.5 Multi-classSVM........................................................ 38
2.6 DecompositionandSMO............................................... 45
2.7 FurtherDiscussions..................................................... 49
Problems....................................................................... 49
References..................................................................... 51
3 ParameterEstimations...................................................... 53
3.1 MaximumLikelihoodEstimation...................................... 53
3.2 MeasurementswithiidNoise........................................... 59
3.3 ExpectationMaximizationforMixtureModels....................... 61
3.4 TheGeneralExpectationMaximization ............................... 66
3.5 ExpectationMaximizationforPPCAModelwithMissingData..... 68
3.6 K-MeansClustering..................................................... 73
Problems....................................................................... 76
References..................................................................... 77
ix