Hierarchical Bayesian Optimization Algorithm: Toward a new Generation of Evolutionary Algorithms

186 Pages·2005·1.8 MB·English

by  Martin Pelikan (auth.)

MartinPelikan
HierarchicalBayesianOptimizationAlgorithm
170
StudiesinFuzzinessandSoftComputing,Volume
Editor-in-chief
Prof.JanuszKacprzyk
SystemsResearchInstitute
PolishAcademyofSciences
ul.Newelska6
01-447Warsaw
Poland
E-mail:[email protected]
Furthervolumeofthisseries Vol.163.A.Ghosh,L.C.Jain(Eds.)
canbefoundonourhomepage: EvolutionaryComputationinDataMining,
2005
springeronline.com
ISBN3-540-22370-3
Vol.156.R.Tadeusiewicz,M.R.Ogiela Vol.164.M.Nikravesh,L.A.Zadeh,
MedicalImageUnderstandingTechnology, J.Kacprzyk(Eds.)
2004 SoftComputingforInformationProdessing
ISBN3-540-21985-4 andAnalysis,2005
ISBN3-540-22930-2
Vol.157.R.A.Aliev,F.Fazlollahi,R.R.Aliev
SoftComputinganditsApplicationsin Vol.165.A.F.Rocha,E.Massad,A.PereiraJr.
BusinessandEconomics,2004 TheBrain:FromFuzzyArithmeticto
ISBN3-540-22138-7 QuantumComputing,2005
ISBN3-540-21858-0
Vol.158.K.K.Dompere
Cost-BenefitAnalysisandtheTheoryof Vol.166.W.E.Hart,N.Krasnogor,
FuzzyDecisions–Identificationand J.E.Smith(Eds.)
MeasurementTheory,2004 RecentAdvancesinMemeticAlgorithms,
ISBN3-540-22154-9 2005
ISBN3-540-22904-3
Vol.159.E.Damiani,L.C.Jain,
M.Madravia(Eds.) Vol.167.Y.Jin(Ed.)
SoftComputinginSoftwareEngineering, KnowledgeIncorporationinEvolutionary
2004 Computation,2005
ISBN3-540-22030-5 ISBN3-540-22902-7
Vol.160.K.K.Dompere Vol.168.YapP.Tan,KimH.Yap,
Cost-BenefitAnalysisandtheTheoryof LipoWang(Eds.)
FuzzyDecisions–FuzzyValueTheory,2004 IntelligentMultimediaProcessingwithSoft
ISBN3-540-22161-1 Computing,2005
ISBN3-540-22902-7
Vol.161.N.Nedjah,L.deMacedo
Mourelle(Eds.) Vol.169.C.R.Bector,SureshChandra
EvolvableMachines,2005 FuzzyMathematicalProgrammingand
ISBN3-540-22905-1 FuzzyMatrixGames,2005
ISBN3-540-23729-1
Vol.162.R.Khosla,N.Ichalkaranje,L.C.Jain
DesignofIntelligentMulti-AgentSystems, Vol.170.MartinPelikan
2005 HierarchicalBayesianOptimization
ISBN3-540-22913-2 Algorithm,2005
ISBN3-540-23774-7
Martin Pelikan
Hierarchical Bayesian
Optimization Algorithm
Toward a New Generation
of Evolutionary Algorithms
123
MartinPelikan
UniversityofMissouri
Dept.ofMathematics
andComputerScience
St.Louis,MO63121
USA
E-mail:[email protected]
ISSNprintedition:1434-9922
ISSNelectronicedition:1860-0808
ISBN3-540-23774-7SpringerBerlinHeidelbergNewYork
LibraryofCongressControlNumber:2004116659
Thisworkissubjecttocopyright.Allrightsarereserved,whetherthewholeorpartofthematerial
isconcerned,specificallytherightsoftranslation,reprinting,reuseofillustrations,recitation,
broadcasting,reproductiononmicrofilmorinanyotherway,andstorageindatabanks.Dupli-
cationofthispublicationorpartsthereofispermittedonlyundertheprovisionsoftheGerman
CopyrightLawofSeptember9,1965,initscurrentversion,andpermissionforusemustalwaysbe
obtainedfromSpringer.ViolationsareliableforprosecutionundertheGermanCopyrightLaw.
SpringerisapartofSpringerScience+BusinessMedia
springeronline.com
©Springer-VerlagBerlinHeidelberg2005
PrintedinGermany
Theuseofgeneraldescriptivenames,registerednames,trademarks,etc.inthispublicationdoes
notimply,evenintheabsenceofaspecificstatement,thatsuchnamesareexemptfromtherelevant
protectivelawsandregulationsandthereforefreeforgeneraluse.
Typesetting:bytheauthorandTechBooksusingaSpringerLATEXmacropackage
Coverdesign:E.Kirchner,SpringerHeidelberg
Printedonacid-freepaper 62/3141/jl-543210
To my parents, Jaroslava and Peter

Foreword
At the 1997 Genetic Programming Conference at Stanford University, a self-
described“long red-haired guy” walked up to me and introduced himself. His
name was Martin Pelikan, he came from Slovakia, and he wanted to visit my
lab at Illinois. He was doing interesting work, and he seemed like a bright
enough fellow, so we exchanged e-mail addresses and worked on overcoming
the logistical problems of a visit. Doing so took longer than I expected, but
on September 2,1998 he arrived in Champaign and immediately got towork.
The initial visit turned into a stint as a PhD student, the stint turned into
a dissertation, and that dissertation has now been transformed into the book
before you. It is a remarkable piece of work.
Simply stated Martin hasshown how totake (a)population-based search,
(b) Bayesian networks, (c) niching or clustering techniques and (d) compact
representations and create a fairly general purpose optimizer of remarkable
breadth and capability. His system is called the hierarchical Bayesian opti-
mization algorithm (hBOA). I have used the term competent to characterize
genetic and other search procedures that solve a large class of hard problems
quickly, reliably, and accurately, and the book before you is a comprehensive
description of the thinking that went into the design of a competent hBOA,
its precursors, the analytical theory of hBOA operation, and empirical tests
ofhBOAonbothboundingtestfunctionsandproblemsofinterestinthereal
world; it is a thorough discussion of what in my view is the most competent
solver to have emerged from these and related lines of thinking.
The book is original in many respects, so it is difficult to single out any
one of its many contributions. Four items do deserve particular comment:
• Decomposition of the problem of hierarchical competence.
• Analysis of hBOA scalability.
• Characterization of a bounding class of hierarchically difficult problems.
• Demonstration of hBOA in well-chosen test and real-world functions.
The key that opened the door to a fuller understanding of the everyday
solution of hierarchically difficult problems was a tripartite decomposition
VIII Foreword
of their solution. Martin’s work clearly articulates the need for (a) capable
learning of important substructures, (b) preservation of alternate competing
substructuresthatappeartosolveaproblem,and(c)theircompactrepresen-
tation. This decomposition is the subject of a recent patent filing, but more
importantly, doing all three things simultaneously was absolutely essential to
the progress made.
Thisqualitativedecompositionoffunctionwascritical,butthisstudywas
nomereexerciseincleverintuition.Martinfollowedhisdesigncreativitywith
an especially careful analysis of algorithmic scalability by better understand-
ing the population size and run duration necessary to achieve high quality
solutions.
TheseboundingresultswouldnothavebeenasmeaningfulhadMartinnot
carefully considered boundedly difficult test functions for his initial testing.
Parsimonious testing at the edge of the design envelope ensures that hBOA
performsaswellifnotbetteronproblemseasierthanthosetested.Real-world
problemssometimesofferuptheirownsurprises,butextensivetestingonspin
glassesandsolvableinstancesofmaximumsatisfiabilityproblemsofferedmore
than a little assurance that the design methodology adopted is sound.
These extraordinary results would be sufficient for me to recommend this
book to genetic algorithmists and artificial evolutionaries around the world,
but the melding of population thinking, traditional topics of artificial intel-
ligence and machine learning, and analytics drawn from population biology
(both real and artificial) suggest that this text may be of interest to a much
broader audience than the usual treatise on genetic algorithms. The commu-
nities mentioned above have a long history of talking past one another, but
Martin’s work hurries past the intellectual chauvinism to integrate important
contributions of many fields into a helluva solver. I urge anyone interested
in the design, theory, implementation, and application of a broadly compe-
tent solver to buy this book, read it, and engage its lessons in thought or in
practice.
Urbana, Illinois, USA David E. Goldberg
January 2005
Preface
A black-box optimization problem may be defined by specifying (1) a set of
all potential solutions to the problem and (2) a measure for evaluating the
quality of each candidate solution. The task is to find a solution or a set of
solutions that perform best with respect to the specified measure. An impor-
tant feature of black-box optimization is that a black-box optimizer does not
know anything about the semantics of potential solutions or the relationship
between potential solutions and the evaluation procedure. Potential solutions
can represent anything from aircraft wings, to chess strategies, to musical
compositions, to job schedules, or to configurations of a molecule. The eval-
uation procedure can be based on an experiment in a wind tunnel, a traffic
simulation,oracomputerprocedure.Theonlywayhowablack-boxoptimizer
can learn something about the problem is to sample new solutions, evaluate
them, and process the results of the evaluation. Since many challenging real-
world problems can be formulated as black-box optimization problems, the
design of automated, robust, and scalable black-box optimization methods is
one of the most important challenges in computational optimization.
The primary goal of this book is to design an advanced black-box opti-
mization algorithm for automated, robust, and scalable solution of a broad
class of real-world problems without the need for interaction with the user or
problem-specific knowledge in addition to the set of potential solutions and
theperformancemeasure.Despitethatthebooktriestoalleviatetheneedfor
expensive manual problem analysis, the methods presented here enable the
use of prior problem-specific knowledge of various forms. Consequently, if the
practitioner has additional knowledge about the problem, this can likely be
used for further efficiency enhancement.
To meet this difficult challenge, the book derives inspiration from the way
the best problem solvers – humans – solve their problems. To simplify prob-
lemsandtodeveloptractablemodelsofcomplexsystems,humansoftenbreak
up the problem or system into several subproblems or subsystems. This can
be done on a single or multiple levels, but the basic idea remains the same –
decompose the big problem into smaller subproblems, solve the subproblems