Table Of ContentComputational
Optimization
This textbook offers a guided tutorial reviewing the theoretical fundamentals while going
through the practical examples used for constructing the computational frame, applied to
various real-life models.
Computational Optimization: Success in Practice will lead the readers through the entire
process. They will start from the simple calculus examples of fitting data and the basics of
optimal control methods and end with the construction of a multi-component framework
for running PDE-constrained optimization. This framework will be assembled piece by
piece; the readers may apply this process at the levels of complexity matching their current
projects or research needs.
By connecting examples with the theory and discussing the proper “communication” be-
tween them, the readers will learn the process of creating a “big house.” Moreover, they can
use the framework exemplified in the book as the template for their research or course prob-
lems – they will know how to change the single “bricks” or add extra “floors” on top of that.
This book is authored for students, faculty, and researchers.
Features:
• The main optimization framework builds through the course exercises and centers on
MATLAB®.
• All other scripts to implement computations for solving optimization problems with
various models use only open-source software, e.g., FreeFEM.
• All computational steps are platform-independent; readers may freely use Windows,
macOS, or Linux systems.
• All scripts illustrating every step in building the optimization framework will be avail-
able to the readers online.
• Each chapter contains problems based on the examples provided in the text and asso-
ciated scripts. The readers will not need to create the scripts from scratch, but rather
modify the codes provided as a supplement to the book.
This book will prove valuable for graduate students of math, computer science, engineer-
ing, and all who explore optimization techniques at different levels for educational or
research purposes. It will benefit many professionals in academic and industry-related
research: professors, researchers, postdoctoral fellows, and the personnel of R&D depart-
ments.
Textbooks in Mathematics
Series editors:
Al Boggess, Kenneth H. Rosen
Geometry and Its Applications, Third Edition
Walter J. Meyer
Transition to Advanced Mathematics
Danilo R. Diedrichs and Stephen Lovett
Modeling Change and Uncertainty
Machine Learning and Other Techniques
William P. Fox and Robert E. Burks
Abstract Algebra
A First Course, Second Edition
Stephen Lovett
Multiplicative Differential Calculus
Svetlin Georgiev, Khaled Zennir
Applied Differential Equations
The Primary Course
Vladimir A. Dobrushkin
Introduction to Computational Mathematics: An Outline
William C. Bauldry
Mathematical Modeling the Life Sciences
Numerical Recipes in Python and MATLABTM
N. G. Cogan
Classical Analysis
An Approach through Problems
Hongwei Chen
Classic Vector Algebra
Vladimir Leptic
Introduction to Number Theory
Mark Hunacek
Probability and Statistics for Engineering and the Sciences with Modeling using R
William P. Fox and Rodney X. Sturdivant
Computational Optimization: Success in Practice
Vladislav Bukshtynov
https://www.routledge.com/Textbooks-in-Mathematics/book-series/CANDHTEXBOOMTH
Computational
Optimization
Success in Practice
Vladislav Bukshtynov
MATLAB® is a trademark of The MathWorks, Inc. and is used with permission. The MathWorks
does not warrant the accuracy of the text or exercises in this book. This book’s use or discussion of
MATLAB® software or related products does not constitute endorsement or sponsorship by The
MathWorks of a particular pedagogical approach or particular use of the MATLAB® software.
First edition published 2023
by CRC Press
6000 Broken Sound Parkway NW, Suite 300, Boca Raton, FL 33487-2742
and by CRC Press
4 Park Square, Milton Park, Abingdon, Oxon, OX14 4RN
CRC Press is an imprint of Taylor & Francis Group, LLC
© 2023 Vladislav Bukshtynov
Reasonable efforts have been made to publish reliable data and information, but the author and pub-
lisher cannot assume responsibility for the validity of all materials or the consequences of their use.
The authors and publishers have attempted to trace the copyright holders of all material reproduced
in this publication and apologize to copyright holders if permission to publish in this form has not
been obtained. If any copyright material has not been acknowledged, please write and let us know so
we may rectify in any future reprint.
Except as permitted under U.S. Copyright Law, no part of this book may be reprinted, reproduced,
transmitted, or utilized in any form by any electronic, mechanical, or other means, now known or
hereafter invented, including photocopying, microfilming, and recording, or in any information stor-
age or retrieval system, without written permission from the publishers.
For permission to photocopy or use material electronically from this work, access www.copyright.com
or contact the Copyright Clearance Center, Inc. (CCC), 222 Rosewood Drive, Danvers, MA 01923, 978-
750-8400. For works that are not available on CCC, please contact [email protected]
Trademark notice: Product or corporate names may be trademarks or registered trademarks and are
used only for identification and explanation without intent to infringe.
ISBN: 978-1-032-22947-8 (hbk)
ISBN: 978-1-032-23005-4 (pbk)
ISBN: 978-1-003-27516-9 (ebk)
DOI: 10.1201/9781003275169
Typeset in CMR10
by KnowledgeWorks Global Ltd.
Publisher’s note: This book has been prepared from camera-ready copy provided by the authors.
To my wife Katya and daughter Ksenia.
Contents
Preface xiii
Author xix
Acronyms and Abbreviations xxi
List of Algorithms xxiii
1 Introduction to Optimization 1
1.1 Optimization Models . . . . . . . . . . . . . . . . . . . . . . 1
1.2 General Notations for Optimization Problem . . . . . . . . . 2
1.3 Data Fitting Examples . . . . . . . . . . . . . . . . . . . . . 5
1.4 Optimization Fundamentals . . . . . . . . . . . . . . . . . . 11
1.4.1 Feasibility . . . . . . . . . . . . . . . . . . . . . . . . . 11
1.4.2 Optimality . . . . . . . . . . . . . . . . . . . . . . . . 13
1.4.3 Convexity . . . . . . . . . . . . . . . . . . . . . . . . . 14
1.5 General Optimization Algorithm . . . . . . . . . . . . . . . . 15
1.5.1 Solving Optimization Problems in Iterations. . . . . . 16
1.5.2 Termination Criteria . . . . . . . . . . . . . . . . . . . 17
1.6 Convergence . . . . . . . . . . . . . . . . . . . . . . . . . . . 18
1.7 Homework Problems . . . . . . . . . . . . . . . . . . . . . . . 20
2 Minimization Approaches for Functions of One Variable 23
2.1 Minimizing a Function in 1D . . . . . . . . . . . . . . . . . . 23
2.2 Bisection Method . . . . . . . . . . . . . . . . . . . . . . . . 25
2.3 Golden Section Search . . . . . . . . . . . . . . . . . . . . . . 27
2.4 Newton’s Method . . . . . . . . . . . . . . . . . . . . . . . . 31
2.5 Brute-Force Search . . . . . . . . . . . . . . . . . . . . . . . 32
2.6 Monte Carlo Method . . . . . . . . . . . . . . . . . . . . . . 34
2.7 Practical Examples . . . . . . . . . . . . . . . . . . . . . . . 36
2.8 Computational Analysis for Convergence . . . . . . . . . . . 44
2.9 Homework Problems . . . . . . . . . . . . . . . . . . . . . . . 48
2.10 Lab Assignment #1: Review Chapters 1–2 . . . . . . . . . . 50
vii
viii Contents
3 Generalized Optimization Framework 51
3.1 Parameter Identification for Least-Square Data Fitting . . . 51
3.2 Generalized Optimization Framework . . . . . . . . . . . . . 54
3.2.1 Computational Components . . . . . . . . . . . . . . . 54
3.2.2 Choice of Proper Software . . . . . . . . . . . . . . . . 56
3.3 Choosing and Adjusting Optimization Algorithms . . . . . . 58
3.4 Visualization and Analysis of Obtained Solutions . . . . . . . 63
3.5 Testing and Dealing with Problems (Debugging) . . . . . . . 68
3.6 TEST Mode for the Gradient-based Framework . . . . . . . 70
3.7 Accuracy and Performance . . . . . . . . . . . . . . . . . . . 73
3.8 Communication within the Framework . . . . . . . . . . . . 81
3.9 Homework Problems . . . . . . . . . . . . . . . . . . . . . . . 83
4 Exploring Optimization Algorithms 87
4.1 Iterative Optimization Algorithms Revisited . . . . . . . . . 87
4.2 Gradient-based Strategies: Line Search vs. Trust Region . . . 89
4.2.1 Line Search . . . . . . . . . . . . . . . . . . . . . . . . 89
4.2.2 Trust Region . . . . . . . . . . . . . . . . . . . . . . . 90
4.2.3 Comparing Strategies . . . . . . . . . . . . . . . . . . 91
4.3 Heuristic Algorithms . . . . . . . . . . . . . . . . . . . . . . 92
4.4 Particle Swarm Optimization . . . . . . . . . . . . . . . . . . 93
4.5 Overview of Optimization Algorithms . . . . . . . . . . . . . 96
4.6 Homework Problems . . . . . . . . . . . . . . . . . . . . . . . 97
5 Line Search Algorithms 99
5.1 Local and Global Minimums: Theory Review . . . . . . . . . 99
5.1.1 Necessary Conditions . . . . . . . . . . . . . . . . . . 99
5.1.2 Necessary vs. Sufficient Conditions . . . . . . . . . . . 101
5.1.3 Existence of Global Minimums . . . . . . . . . . . . . 102
5.2 Selecting Search Direction . . . . . . . . . . . . . . . . . . . 103
5.2.1 Principal Generalization . . . . . . . . . . . . . . . . . 103
5.2.2 Some Theory for Convergence . . . . . . . . . . . . . . 104
5.2.3 Steepest Descent . . . . . . . . . . . . . . . . . . . . . 105
5.3 Newton’s Method and Newton-based Modifications . . . . . 107
5.3.1 Pure Newton’s Method . . . . . . . . . . . . . . . . . 107
5.3.2 Discretized Newton’s Method . . . . . . . . . . . . . . 108
5.3.3 Modified Newton’s Method . . . . . . . . . . . . . . . 109
5.3.4 Diagonally Scaled SD . . . . . . . . . . . . . . . . . . 109
5.3.5 Quasi-Newton Methods . . . . . . . . . . . . . . . . . 110
5.3.6 Gauss-Newton Approach. . . . . . . . . . . . . . . . . 111
5.4 Conjugate Gradient . . . . . . . . . . . . . . . . . . . . . . . 113
5.5 Line Search Performance Comparison . . . . . . . . . . . . . 114
5.6 Homework Problems . . . . . . . . . . . . . . . . . . . . . . . 124
5.7 Lab Assignment #2: Review Chapters 3–5 . . . . . . . . . . 127
Contents ix
6 Choosing Optimal Step Size 129
6.1 Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 129
6.2 Simple Approaches . . . . . . . . . . . . . . . . . . . . . . . 131
6.3 Inexact Line Search . . . . . . . . . . . . . . . . . . . . . . . 132
6.3.1 Armijo Rule. . . . . . . . . . . . . . . . . . . . . . . . 132
6.3.2 Wolfe Conditions . . . . . . . . . . . . . . . . . . . . . 133
6.3.3 Goldstein Conditions . . . . . . . . . . . . . . . . . . . 135
6.3.4 Backtracking Line Search . . . . . . . . . . . . . . . . 136
6.4 Step Size Performance Comparison . . . . . . . . . . . . . . 137
6.5 Advanced Methods for 1D Search . . . . . . . . . . . . . . . 148
6.6 Brent’s Method . . . . . . . . . . . . . . . . . . . . . . . . . 149
6.7 Bracketing-Brent Toolbox in MATLAB . . . . . . . . . . . . . . 150
6.7.1 General Description . . . . . . . . . . . . . . . . . . . 151
6.7.2 Initial Bracketing with fn min brack.m . . . . . . . . 153
6.7.3 Brent Minimization with fn brent.m. . . . . . . . . . 154
6.7.4 Technicalities for MATLAB Implementation . . . . . . . 155
6.8 Bracketing-Brent Toolbox Performance . . . . . . . . . . . . 157
6.9 Homework Problems . . . . . . . . . . . . . . . . . . . . . . . 164
7 Trust Region and Derivative-Free Methods 167
7.1 Trust Region Outline . . . . . . . . . . . . . . . . . . . . . . 167
7.2 General Algorithm . . . . . . . . . . . . . . . . . . . . . . . . 169
7.3 Cauchy Point Calculation . . . . . . . . . . . . . . . . . . . . 170
7.4 Improving Cauchy Point by Dogleg Method . . . . . . . . . . 171
7.5 Checking and Tuning Performance . . . . . . . . . . . . . . . 174
7.6 Exploring Derivative-Free Options . . . . . . . . . . . . . . . 181
7.7 Homework Problems . . . . . . . . . . . . . . . . . . . . . . . 186
7.8 Lab Assignment #3: Review Chapters 6–7 . . . . . . . . . . 189
7.9 Midterm Assignment: Review Chapters 1–7 . . . . . . . . . . 191
8 Large-Scale and Constrained Optimization 193
8.1 Generalization of Large-Scale Optimization . . . . . . . . . . 193
8.2 LSO Examples . . . . . . . . . . . . . . . . . . . . . . . . . . 194
8.2.1 Solving Systems of Equations . . . . . . . . . . . . . . 194
8.2.2 Space-Dependent Parameter Reconstruction . . . . . . 195
8.2.3 Parameter Identification – Another Example . . . . . 200
8.2.4 State-Dependent Parameter Reconstruction . . . . . . 204
8.3 Improving Performance of LSO . . . . . . . . . . . . . . . . . 207
8.4 General Theory for Constrained Optimization . . . . . . . . 208
8.5 Lagrange Multiplier Approach . . . . . . . . . . . . . . . . . 210
8.6 Lagrange Multiplier vs. Penalization . . . . . . . . . . . . . . 211
8.7 Extending Complexity – DE-constrained
Optimization . . . . . . . . . . . . . . . . . . . . . . . . . . . 225
8.8 KKT Optimality Conditions . . . . . . . . . . . . . . . . . . 227
8.9 Homework Problems . . . . . . . . . . . . . . . . . . . . . . . 229
