Table Of ContentKenneth V. Price ·Rainer M. Storn
Jouni A. Lampinen
Differential Evolution
APractical Approach to Global Optimization
With 292 Figures, 48 Tables and CD-ROM
123
Authors Series Editors
Kenneth V. Price G. Rozenberg (Managing Editor)
Owl Circle 836 [email protected]
Vacaville, CA 95687
USA Th. Bäck, J.N. Kok, H.P. Spaink
Leiden Center for Natural Computing
Rainer M. Storn Leiden University
Rohde & Schwarz GmbH & Co.KG Niels Bohrweg 1
Mühldorfstraße 15 2333 CA Leiden,
81671 München The Netherlands
Germany
A.E. Eiben
Jouni A. Lampinen Vrije Universiteit Amsterdam
Lappeenranta University of Technology
Department of Information Technology
P.O.Box 20
53851 Lappeenranta
Finland
Library of Congress Control Number: 2005926508
ACM Computing Classification (1998): F.1–2, G.1.6, I.2.6, I.2.8, J.6
ISBN-10 3-540-20950-6 Springer Berlin Heidelberg New York
ISBN-13 978-3-540-20950-8 Springer Berlin Heidelberg New York
This work is subject to copyright. All rights are reserved, whether the whole or part of the material is
concerned, specifically the rights of translation, reprinting, reuse of illustrations, recitation, broadcasting,
reproduction on microfilm or in any other way, and storage in data banks. Duplication of this publication
or parts thereof is permitted only under the provisions of the German Copyright Law of September 9,
1965, in its current version, and permission for use must always be obtained from Springer. Violations
are liable for prosecution under the German Copyright Law.
The publisher and the authors accept no legal responsibility for any damage caused by improper
use of the instructions and programs contained in this book and the CD-ROM. Although the
software has been tested with extreme care, errors in the software cannot be excluded.
Springer is a part of Springer Science+Business Media
springer.com
© Springer-Verlag Berlin Heidelberg 2005
Printed in Germany
The use of general descriptive names, registered names, trademarks, etc. in this publication does not
imply, even in the absence of a specific statement, that such names are exempt from the relevant
protective laws and regulations and therefore free for general use.
Cover Design: KünkelLopka, Werbeagentur, Heidelberg
Typesetting: by the Authors
Production: LE-TEX Jelonek, Schmidt & Vöckler GbR, Leipzig
Printed on acid-free paper 45/3142/YL – 5 4 3 2 1 0
KP: To my father
RS: To my ever-supportive parents, to my beloved wife, Marion, and to
my wonderful children, Maja and Robin
JL: To the memory of my little dog and best friend Tonique, for all the
happy countryside and city memories we shared
Preface
Optimization problems are ubiquitous in science and engineering. What
shape gives an airfoil maximum lift? Which polynomial best fits the given
data? Which configuration of lenses yields the sharpest image? Without
question, very many researchers need a robust optimization algorithm for
solving the problems that are fundamental to their daily work.
Ideally, solving a difficult optimization problem should not itself be dif-
ficult, e.g., a structural engineer with an expert knowledge of mechanical
principles should not also have to be an expert in optimization theory just
to improve his designs. In addition to being easy to use, a global optimiza-
tion algorithm should also be powerful enough to reliably converge to the
true optimum. Furthermore, the computer time spent searching for a solu-
tion should not be excessive. Thus, a genuinely useful global optimization
method should be simple to implement, easy to use, reliable and fast.
Differential Evolution (DE) is such a method. Since its inception in
1995, DE has earned a reputation as a very effective global optimizer.
While DE is not a panacea, its record of reliable and robust performance
demands that it belongs in every scientist and engineer’s “bag of tricks”.
DE originated with the Genetic Annealing algorithm developed by
Kenneth Price and published in the October 1994 issue of Dr. Dobb’s
Journal (DDJ), a popular programmer’s magazine. Genetic Annealing is a
population-based, combinatorial optimization algorithm that implements
an annealing criterion via thresholds. After the Genetic Annealing algo-
rithm appeared in DDJ, Ken was contacted by Dr. Rainer Storn, (then with
Siemens while at the International Computer Science Institute at the Uni-
versity of California at Berkeley; now at Rohde & Schwarz GmbH, Mu-
nich, Germany) about the possibility of using Genetic Annealing to solve
the Chebyshev polynomial fitting problem. Determining the coefficients of
the Chebyshev polynomials is considered by many to be a difficult task for
a general-purpose optimizer.
Ken eventually found the solution to the five-dimensional Chebyshev
problem with the Genetic Annealing algorithm, but convergence was very
slow and effective control parameters were hard to determine. After this
initial find, Ken began modifying the Genetic Annealing algorithm to use
floating-point instead of bit-string encoding and arithmetic operations in-
VIII Preface
stead of logical ones. He then discovered the differential mutation operator
upon which DE is based. Taken together, these alterations effectively
transformed what had been a combinatorial algorithm into the numerical
optimizer that became the first iteration of DE. To better accommodate
parallel machine architectures, Rainer suggested creating separate parent
and child populations. Unlike Genetic Annealing, DE has no difficulty de-
termining the coefficients of even the 33-dimensional Chebyshev polyno-
mial.
DE proved effective not only on the Chebyshev polynomials, but also
on many other test functions. In 1995, Rainer and Ken presented some
early results in the ICSI technical report TR-95-012, “Differential Evolu-
tion – A Simple and Efficient Adaptive Scheme for Global Optimization
over Continuous Spaces”. These successes led Rainer and Ken to enter DE
in the First International Contest on Evolutionary Optimization in Nagoya,
Japan, that was held during May of 1996 in conjunction with the IEEE In-
ternational Conference on Evolutionary Computation. DE finished third
behind two methods that scored well on the contest functions, but which
were not versatile enough to be considered general-purpose optimizers.
The first-place method explicitly relied on the fact that the contest func-
tions were separable, while the second-place algorithm was not able to
handle a large number of parameters due to its dependence on Latin
squares. Buoyed by this respectable showing, Ken and Rainer wrote an ar-
ticle on DE for DDJ that was published in April 1997 (Differential Evolu-
tion - A Simple Evolution Strategy for Fast Optimization). This article was
very well received and introduced DE to a large international audience.
Many other researchers in optimization became aware of DE’s potential
after reading, “Differential Evolution – A Simple and Efficient Heuristic
for Global Optimization over Continuous Spaces”, by Rainer and Ken.
Published in the December 1997 issue of The Journal of Global Optimiza-
tion, this paper gave extensive empirical evidence of DE’s robust perform-
ance on a wide variety of test functions. Also about this time, Rainer estab-
lished a DE web site (http://www.icsi.berkeley.edu/~storn/code/html) to
post code, links to DE applications and updates for the algorithm.
Ken entered DE in the Second International Contest on Evolutionary
Optimization that was to be held in Indianapolis, Indiana, USA in April
1997. A lack of valid entries forced the cancellation of the actual contest,
although those that qualified were presented. Of these, DE was the best
performer. At this conference, Ken met Dr. David Corne who subsequently
invited him to write an introduction to DE for the compendium, New Ideas
in Optimization (1999). Since then, Ken has focused on refining the DE
algorithm and on developing a theory to explain its performance. Rainer
has concentrated on implementing DE on limited-resource devices and on
Preface IX
creating software applications in a variety of programming languages. In
addition, Rainer has explored DE’s efficacy as a tool for digital filter de-
sign, design centering and combinatorial optimization.
Prof. Jouni Lampinen (Lappeenranta University of Technology, Lap-
peenranta, Finland) began investigating DE in 1998. In addition to con-
tributing to the theory on DE and demonstrating DE’s effectiveness as a
tool for mechanical engineering, Jouni has also developed an exceptionally
simple yet effective method for adapting DE to the particular demands of
both constrained and multi-objective optimization. Jouni also maintains a
DE bibliography (http://www.lut.fi/~jlampine/debiblio.html).
Like DE, this book is designed to be easy to understand and simple to
use. It details how DE works, how to use it and when it is appropriate.
Chapter 1, “The Motivation for DE”, opens with a statement of the general
optimization problem that is followed by a discussion of the strengths and
weaknesses of the traditional methods upon which DE builds. Classical
methods for optimizing differentiable functions along with conventional
direct search methods like those of Hooke–Jeeves and Nelder–Mead are
discussed. Chapter 1 concludes with a look at some of the more advanced
optimization techniques, like simulated annealing and evolutionary algo-
rithms.
Chapter 2, “The Differential Evolution Algorithm”, introduces the DE
algorithm itself, first in an overview and then in detail. Chapter 3,
“Benchmarking DE”, compares DE’s performance to that reported for
other EAs. Several versions of DE are included in the comparison. Chapter
4, “Problem Domains”, extends the basic algorithm to cover a variety of
optimization scenarios, including constrained, mixed-variable and multi-
objective optimization as well as design centering. All these adaptations
are of great practical importance, since many real-world problems belong
to these domains.
Chapter 5, “Architectural Aspects”, gives explicit advice on how to im-
plement DE on both parallel and sequential machine architectures. In addi-
tion, Chapter 5 presents algorithms for auxiliary operations. Chapter 6,
“Computer Code”, provides instructions for using the software that ac-
companies this book on CD-ROM. Chapter 7, “Applications”, presents a
collection of 12 DE applications that have been contributed by experts
from many disciplines. Applications include structure determination by X-
ray analysis, earthquake relocation, multi-sensor fusion, digital filter de-
sign and many other very difficult optimization problems. An appendix
contains descriptions of the test functions used throughout this book.
Dr. Storn would like to thank Siemens corporate research, especially
Prof. Dr. H. Schwärtzel, Dr. Yeung-Cho Yp and Dr. Jean Schweitzer for
supporting DE research. In addition, Prof. Lampinen would like to express
X Preface
his gratitude to members of his DE research group, Jani Rönkkönen, Jun-
hong Liu and Saku Kukkonen, for their help preparing this book. We espe-
cially wish to thank the researchers who have contributed their DE applica-
tions to Chapter 7.
J.-P. Armspach, Institut de Physique Biologique, Université Louis Pasteur,
Strasbourg, UMR CNRS-ULP 7004, Faculté de Médecine, F-67085,
Strasbourg Cedex, France ; (Sect. 7.6)
Keith D. Bowen, Bede Scientific Incorporated, 14 Inverness Drive East,
Suite H-100, Englewood, CO, USA; (Sect. 7.10)
Nirupam Chakraborti, Department of Metallurgical and Materials Engi-
neering, Indian Institute of Technology, Kharagpur (W.B) 721 302,
India; (Sect. 7.1)
David Corcoran, Department of Physics, University of Limerick, Ireland;
(Sect. 7.2)
Robert W. Derksen, Department of Mechanical and Industrial Engineering
University of Manitoba, Canada; (Sect. 7.3)
Drago Dolinar, University of Maribor, Faculty of Electrical Engineering
and Computer Science, Smetanova 17, 2000 Maribor, Slovenia; (Sect.
7.9)
Steven Doyle, Department of Physics, University of Limerick, Ireland;
(Sect. 7.2)
Kay Hameyer, Katholieke Universiteit Leuven, Department E.E. (ESAT),
Division ELEN, Kaardinal Mercierlaan 94, B-3001 Leuven, Belgium;
(Sect. 7.9)
Evan P. Hancox, Department of Mechanical and Industrial Engineering,
University of Manitoba, Canada; (Sect. 7.3)
Fabrice Heitz, LSIIT-MIV, Université Louis Pasteur, Strasbourg, UMR
CNRS-ULP 7005, Pôle API, Boulevard Sébastien Brant, F-67400 Ill-
kirch, France ; (Sect. 7. 6)
Rajive Joshi, Real-Time Innovations Inc., 155A Moffett Park Dr, Sunny-
vale, CA 94089, USA; (Sect. 7.4)
Michal Kvasni(cid:254)ka, ERA a.s, Pod(cid:268)bradská 186/56, 180 66 Prague 9, Czech
Republic; (Sect. 7.5)
Kevin M. Matney, Bede Scientific Incorporated, 14 Inverness Drive East,
Suite H-100, Englewood, CO, USA; (Sect. 7.10)
Lars Nolle, School of Computing and Mathematics, The Nottingham Trent
University, Burton Street, Nottingham, NG1 4BU, UK; (Sect. 7.12)
Guy-René Perrin, LSIIT-ICPS, Université Louis Pasteur, Strasbourg,
UMR CNRS-ULP 7005, Pôle API, Boulevard Sébastien Brant, F-
67400 Illkirch, France ; (Sect. 7. 6)
Preface XI
Bohuslav R(cid:292)žek, Geophysical Institute, Academy of Sciences of the Czech
Republic, Bo(cid:254)ní II/1401, 141 31 Prague 4, Czech Republic; (Sect.
7.5)
Michel Salomon, LSIIT-ICPS, Université Louis Pasteur, Strasbourg, UMR
CNRS-ULP 7005, Pôle API, Boulevard Sébastien Brant, F-67400 Ill-
kirch, France ; (Sect. 7. 6)
Arthur C. Sanderson, Rensselaer Polytechnic Institute, 110 8th St, Troy,
NY 12180, USA; (Sect. 7.4)
Amin Shokrollahi, Laboratoire d’algorithmique Laboratoire de mathé-
matiques algorithmiques, EPFL, I&C-SB, Building PSE-A, 1015
Lausanne, Switzerland; (Sect. 7.7)
Rainer M. Storn, Rohde & Schwarz GmbH & Co. KG, Mühldorfstr. 15,
81671 München, Germany; (Sects. 7.7 and 7.8)
Gorazd Štumberger, University of Maribor, Faculty of Electrical Engineer-
ing and Computer Science, Smetanova 17, 2000 Maribor, Slovenia;
(Sect. 7.9)
Matthew Wormington, Bede Scientific Incorporated, 14 Inverness Drive
East, Suite H-100, Englewood, CO, USA; (Sect. 7.10)
Ivan Zelinka, Institute of Information Technologies, Faculty of Technol-
ogy, Tomas Bata University, Mostni 5139, Zlin, Czech Republic;
(Sects. 7.11 and 7.12)
We are also indebted to everyone who has contributed the public do-
main code that has made DE so accessible. In particular, we wish to thank
Eric Brasseur for making plot.h available to the public, Makoto Matsu-
moto and Takuji Nishimura for allowing the Mersenne Twister random
number generator to be freely used, Lester E. Godwin for writing the C++
version of DE, Feng-Sheng Wang for providing the Fortran90 version of
DE, Walter Di Carlo for porting DE to Scilab(cid:163), Jim Van Zandt and Arnold
Neumaier for helping with the MATLAB(cid:163) version of DE and Ivan Zelinka
and Daniel Lichtblau for providing the MATHEMATICA(cid:163) version of DE.
A special debt of gratitude is owed to David Corne for his unflagging
support and to A. E. Eiben and the editors of Springer-Verlag’s Natural
Computing Series for their interest in DE. In addition, we want to thank
Ingeborg Meyer for her patience and professionalism in bringing our book
to print. We are also indebted to Neville Hankins for his exquisitely de-
tailed copyediting and to both Ronan Nugent and Ulrike Stricker at
Springer-Verlag for helping to resolve the technical issues that arose dur-
ing the preparation of this manuscript.
XII Preface
Additionally, this book would not be possible were it not for the many
engineers and scientists who have helped DE become so widespread. Al-
though they are too numerous to mention, we wish to thank them all.
Finally, it would have been impossible to write this book without our
families’ understanding and support, so we especially want to thank them
for their forbearance and sacrifice.
Kenneth V. Price
Rainer M. Storn
Jouni A. Lampinen
Table of Contents
Preface....................................................................................................VII
Table of Contents.................................................................................XIII
1 The Motivation for Differential Evolution...........................................1
1.1 Introduction to Parameter Optimization...........................................1
1.1.1 Overview...................................................................................1
1.1.2 Single-Point, Derivative-Based Optimization...........................6
1.1.3 One-Point, Derivative-Free Optimization and the Step Size
Problem...................................................................................11
1.2 Local Versus Global Optimization.................................................16
1.2.1 Simulated Annealing...............................................................18
1.2.2 Multi-Point, Derivative-Based Methods..................................19
1.2.3 Multi-Point, Derivative-Free Methods....................................20
1.2.4 Differential Evolution – A First Impression............................30
References............................................................................................34
2 The Differential Evolution Algorithm................................................37
2.1 Overview........................................................................................37
2.1.1 Population Structure................................................................37
2.1.2 Initialization.............................................................................38
2.1.3 Mutation..................................................................................38
2.1.4 Crossover.................................................................................39
2.1.5 Selection..................................................................................40
2.1.6 DE at a Glance.........................................................................41
2.1.7 Visualizing DE........................................................................43
2.1.8 Notation...................................................................................47
2.2 Parameter Representation...............................................................48
2.2.1 Bit Strings................................................................................48
2.2.2 Floating-Point..........................................................................50
2.2.3 Floating-Point Constraints.......................................................52
2.3 Initialization....................................................................................53
2.3.1 Initial Bounds..........................................................................53
