Table Of ContentExperimental Methods for the Analysis
of Optimization Algorithms
· ·
Thomas Bartz-Beielstein Marco Chiarandini
·
Luís Paquete Mike Preuss
Editors
Experimental Methods
for the Analysis of
Optimization Algorithms
123
Editors
Prof.Dr.ThomasBartz-Beielstein Dr.MarcoChiarandini
CologneUniversityofAppliedSciences UniversityofSouthernDenmark
InstituteofComputerScience DepartmentofMathematicsand
FacultyofComputerand ComputerScience
EngineeringScience Campusvej55
CampusGummersbach 5230Odense
Steinmüllerallee1 Denmark
51643Gummersbach [email protected]
Germany
[email protected]
Dr.rer.nat.LuísPaquete MikePreuss
UniversityofCoimbra TUDortmund
CISUC DepartmentofComputerScience
DepartmentofInformaticsEngineering AlgorithmEngineering
PóloII Otto-Hahn-Str.14
3030-290Coimbra 44227Dortmund
Portugal Germany
[email protected] [email protected]
ISBN978-3-642-02537-2 e-ISBN978-3-642-02538-9
DOI10.1007/978-3-642-02538-9
SpringerHeidelbergDordrechtLondonNewYork
ACMCodes:G.1,G.3,F.2,I.2
(cid:2)c Springer-VerlagBerlinHeidelberg2010
Thisworkissubjecttocopyright.Allrightsarereserved,whetherthewholeorpartofthematerialis
concerned,specificallytherightsoftranslation,reprinting,reuseofillustrations,recitation,broadcasting,
reproductiononmicrofilmorinanyotherway,andstorageindatabanks.Duplicationofthispublication
orpartsthereofispermittedonlyundertheprovisionsoftheGermanCopyrightLawofSeptember9,
1965,initscurrentversion,andpermissionforusemustalwaysbeobtainedfromSpringer.Violations
areliabletoprosecutionundertheGermanCopyrightLaw.
Theuseofgeneraldescriptivenames,registerednames,trademarks,etc.inthispublicationdoesnot
imply,evenintheabsenceofaspecificstatement,thatsuchnamesareexemptfromtherelevantprotective
lawsandregulationsandthereforefreeforgeneraluse.
Coverdesign:KuenkelLopkaGmbH
Printedonacid-freepaper
SpringerispartofSpringerScience+BusinessMedia(www.springer.com)
Foreword
This book belongs on the shelf of anyone interested in carrying out experimental
researchonalgorithmsandheuristicsforoptimizationproblems.
The editors have brought together expertise from diverse sources to address
methodological issues arising in this field. The presentation is wide-ranging, con-
taining“bigpicture”discussionsaswellasmorefocusedtreatmentofspecificsta-
tisticaltechniquesandtheirapplication.Theemphasisthroughoutisoncarefulpro-
cessandscientificrigor;thediscussionisilluminatedwithmanycasestudies,small
tutorials,andreferencestotheliteratureonoptimization.
Don’t keep this book on the shelf: read it, and apply the techniques and tools
contained herein to your own algorithmic research project. Your experiments will
becomemoreefficientandmoretrustworthy,andyourexperimentaldatawilllead
tocleareranddeeperinsightsaboutperformance.
Amherst,Massachusetts,February2010 CatherineC.McGeoch
v
Foreword
Onceuponatime,moreexactlynearlyhalfacenturyago,whenthefirstcybernetic
machines,henceforthcalledcomputers,becameavailabletoacademicinstitutions,
afewpeopleseemedtohavewaitedfortheiriterativepowertoperformotherwise
boring procedures like solving sets of linear equations, etc. Among other ideas to
makeuseoftheirtirelessworkingthroughloopsofinstructionswasthesimulation
of organic evolution, the main subroutines of which are mutation, recombination,
andnaturalselection.Itwasonlyasmallsteptoimaginethatbymeansofthesame
principlesthedesignoftechnicaldevices,managerialtasksandothersystemscould
be stepwise improved, if not even optimized. Competing methods from numerical
mathematicswereknown,ofcourse,butalsotheirlimitationstolinearandquadratic
dependenciesbetweendecisionvariablesandobjectives.Inso-calledblackboxsit-
uationswheremuchmorecomplexdependenciesprevailandnonlinearconstraints,
stochastic disturbances, and the like hamper the search for optima, using evolu-
tionaryvariationandevaluationprocessesshoweduptheircapacities.Evolutionary
algorithms thus were born during the 1960s, and they have matured ever since to
a powerful and broadly accepted tool within many disciplines. Together with two
othermodernstreams,artificialneuralnetworks(NN)andfuzzysystems(FS),they
havebeensubsumedintotheso-calledcomputationalintelligence(CI)field,atleast
since1994,whenthefirstworldcongressonCItookplacewithitsthreesubbranches
NN,FS,andEC(evolutionarycomputation).
InthebeginningofEConehadtobehappyifonecouldrerunanumericalexper-
imentafewtimes,forexamplewithdifferentseedsofthepseudorandomgenerator
or different start positions in the search space. Gathering a whole set of statistical
datawasunimaginablethen,sothatmanyopenquestionsremainedabouttheper-
formanceofthealgorithms.Whatarethosequestions?It’snotonlyanaveragevalue
anditsvarianceandskewness,orthebestresultoutofafewrunsthatareinterest-
ing.Onewantstoknowwhethertheultimateresult,i.e.usingthealgorithmspecific
stoppingcriterion,isalwaysthesame,whetherandhowitdependsontherandom
numbersandstartingpointsused.Furtheritmayhappenornotthatfatalexecution
errorsoccur,likedivisionbyzeroorextractingthesquarerootofanegativenum-
ber,orthatthestoppingcriteriondoesnotworkproperly–eveniftheoptimumwas
vii
viii Foreword
found exactly. And what exactly does it mean to talk of four or eight precise dig-
its or even more (which may depend on the hardware and on the system software
handlingrealnumbers,theirmantissaeandexponents)?Introducingcommonstop-
ping criteria to compare different methods can easily deliver controversial results
dependingontheslopesofbest(oraverage)intermediateresultsoverthenumberof
objectivefunctionvalues.Suchslopescannotonlyhaveonecrossing,buteventwo
or more. Then the result depends heavily upon the number of admitted iterations,
one method being quicker at the beginning while finally delivering mediocre final
results,ortheotherwayround,orevenmorecomplicated.
Ifyouareinterestedinsuchquestions,THISisthebooktolookinto.Hereyou
willfindevenmoreaspectsthataretreatedscientificallybytheexpertsinthatexcit-
ingdomainofferingtheirup-to-dateknow-howandevenleadingintophilosophical
domains.
Dortmund,February2010 Hans-PaulSchwefel
Preface
Optimization algorithms are used to solve problems that arise in relevant research
andapplicationareassuchasoperationsresearch,computerscience,andengineer-
ing. During recent decades the experimental approach has been recognized and
accepted in the analysis of these algorithms and a considerable body of research
has been devoted to the development and establishment of an adequate scientific
methodologyforpursuingthiskindofanalysis.Statisticaltoolshavebecomemore
and more popular. This book is written for researchers and practitioners of opera-
tionsresearchandcomputersciencewhowishtoimprovetheexperimentalassess-
mentoftheiroptimizationalgorithmswiththefinalgoalofimprovingtheirdesign.
Itcollectsprominentmethodologicalworksondifferentscenariosofexperimental
analysis.
The book consists of an introduction and 4 chapters written by the editors plus
11chapters(includinganappendix)writtenbyinvitedcontributors.Alltogetherthe
projectinvolved30authorsof16world-wideacademicinstitutions.
The first part of the book lays the basis giving an all-round view of the issues
involvedintheexperimentalanalysisofalgorithms.Thesecondparttreatsthechar-
acterization by means of statistical distributions of the algorithm performance in
termsofsolutionquality,run-time,andothermeasures.Thethirdpartcollectsad-
vanced methods from experimental design for configuring and tuning algorithms
on a specific class of instances with the goal of using the least amount of experi-
mentationandattainingsoundconclusions.Severalchaptersareenrichedwithcase
studies.
Acknowledgments
Weareindebtedtomanypeople,especiallytothecontributorsofthisvolume.
Thebookistheresultofaninspiringandfruitfulcooperationamongtheeditors.
First contacts were established during the Metaheuristic International Conference
inViennain2005—followedbyresearchvisitsandmeetingsduringconferences.
ix
x Preface
We realized that problems that we had been trying to solve are of interest for a
broaderaudienceandnotrestrictedtoourownresearchareas(computationalintel-
ligence,operationsresearch,mathematicaloptimization).Certainly,amilestonein
thepathwaytothisbookwastheWorkshoponEmpiricalMethodsfortheAnalysis
ofAlgorithmsheldinReykjavikin2006.Wearethankfultoalltheparticipantsof
thatworkshop,manyofwhombecameauthorsinthisbookandcontributedtode-
fineitsscope.Amongthem,aspecialmentiongoestoCatherineMcGeochwhose
perspectiveintheexperimentalanalysisofalgorithmshighlyinfluencedourviews.
Onanindividualbasis,Bartz-Beielsteingratefullyacknowledgesthesupportof
colleaguesatCologneUniversityofAppliedSciences:WolfgangKonenforthein-
spiringintellectualexchangeandMichaelBongardsforprovidingvaluableinputfor
real-world applications. Some of Thomas Bartz-Beielstein’s work was conducted
underaBundesministeriumfürForschungundBildung(BMBF)grant(FIWA,AIF
FKZ17N2309)andunderaCologneUniversityofAppliedSciencesgrant(COSA).
ThomasBartz-BeielsteinwouldliketothankDeborahMayoforexchangingideas
about error and inference. Many thanks go to Jack P.C. Kleijnen, Ekaterina Vla-
dislavleva, and Dick den Hertog for invitations to the wonderful mini-symposia at
TilburgUniversity.ThomasBartz-BeielsteinandMikePreussaregratefultoHans-
Paul Schwefel and Günter Rudolph for detailed comments, valuable suggestions
and inspiring advice over many years. We thank Gwenn Volkert, Jürgen Branke,
MarkWineberg,andSteffenChristensenforpromotingtheideaofexperimentalre-
search during the leading conferences in evolutionary computation. Ruxandra and
Catalin Stoean also deserve thanks for the open-mindedness to jointly investigate
every possible algorithmic variation. This also applies to the colleagues at RWTH
Aachen,ChristophKausch,FrankHenrichandClaudeBouvy,whoaddedvaluable
insightintoindustrialoptimizationproblemsinworkconductedtogetherunderDFG
projectgrant252441.MarcoChiarandiniisthankfultoYuriGoegebeurforhisad-
viceonstatisticalissuesduringthereviewingprocessofthechapters.LuísPaquete
is grateful to Carlos Fonseca and Thomas Stützle for the encouraging advice and
stimulatingexchangeofideasaboutthemaintopicofthisbook.Healsoacknowl-
edges his colleagues at University of Coimbra, mainly Ernesto Costa and Alberto
Moraglio,fortheenlighteningdiscussionsondesignandanalysisofstochastical-
gorithmsforoptimization.
Finally,weallwouldliketothankRonanNugentofSpringer,whoprovidedgreat
supportduringallproductionstagesofthisbook.
Gummersbach,Odense,Coimbra,Dortmund, ThomasBartz-Beielstein
February2010 MarcoChiarandini
LuísPaquete
MikePreuss
