Table Of ContentSpringer Series in Operations Research
and Financial Engineering
Marcia Fampa
Jon Lee
Maximum-Entropy
Sampling
Algorithms and Application
Springer Series in Operations Research
and Financial Engineering
Series Editors
Thomas V. Mikosch, Køebenhavns Universitet, Copenhagen, Denmark
Sidney I. Resnick, Cornell University, Ithaca, USA
Stephen M. Robinson, University of Wisconsin-Madison, Madison, USA
Editorial Board
Torben G. Andersen, Northwestern University, Evanston, USA
Dmitriy Drusvyatskiy, University of Washington, Seattle, USA
Avishai Mandelbaum, Technion - Israel Institute of Technology, Haifa, Israel
Jack Muckstadt, Cornell University, Ithaca, USA
Per Mykland, University of Chicago, Chicago, USA
Philip E. Protter, Columbia University, New York, USA
Claudia Sagastizabal, IMPA – Instituto Nacional de Matemáti, Rio de Janeiro,
Brazil
David B. Shmoys, Cornell University, Ithaca, USA
David Glavind Skovmand, Køebenhavns Universitet, Copenhagen, Denmark
Josef Teichmann, ETH Zürich, Zürich, Switzerland
Bert Zwart, Centrum Wiskunde & Informatica, Amsterdam, The Netherlands
The Springer Series in Operations Research and Financial Engineering publishes
monographsandtextbooksonimportanttopicsintheoryandpracticeofOperations
Research, Management Science, and Financial Engineering. The Series is distin-
guishedbyhighstandardsincontentandexposition,andspecialattentiontotimely
or emerging practice inindustry, business, and government. Subject areas include:
Linear, integer and non-linear programming including applications; dynamic
programming and stochastic control; interior point methods; multi-objective
optimization; Supply chain management, including inventory control, logistics,
planningandscheduling;GametheoryRiskmanagementandriskanalysis,including
actuarial science and insurance mathematics; Queuing models, point processes,
extremevalue theory, andheavy-tailed phenomena;Networked systems, including
telecommunication,transportation,andmanyothers;Quantitativefinance:portfolio
modeling,options,andderivativesecurities;Revenuemanagementandquantitative
marketingInnovativestatisticalapplicationssuchasdetectionandinferenceinvery
largeand/orhighdimensionaldatastreams;Computationaleconomics
Marcia Fampa • Jon Lee
Maximum-Entropy Sampling
Algorithms and Application
123
Marcia Fampa Jon Lee
Department of Systems Engineering Department of Industrial
and Computer Science, COPPE & Operations Engineering
Federal University of Rio de Janeiro University of Michigan
Rio de Janeiro, RJ, Brazil Ann Arbor, MI, USA
ISSN 1431-8598 ISSN 2197-1773 (electronic)
Springer Series in Operations Research and Financial Engineering
ISBN 978-3-031-13077-9 ISBN 978-3-031-13078-6 (eBook)
https://doi.org/10.1007/978-3-031-13078-6
Mathematics Subject Classification (2020): 90-08, 90C26, 90C25, 90C27, 90C10, 90C22
© The Editor(s) (if applicable) and The Author(s), under exclusive license to Springer Nature Switzerland
AG 2022
This work is subject to copyright. All rights are solely and exclusively licensed by the Publisher, whether
the whole or part of the material is concerned, specifically the rights of translation, reprinting, reuse of
illustrations, recitation, broadcasting, reproduction on microfilms or in any other physical way, and
transmission or information storage and retrieval, electronic adaptation, computer software, or by similar
or dissimilar methodology now known or hereafter developed.
The use of general descriptive names, registered names, trademarks, service marks, etc. in this publication
does not imply, even in the absence of a specific statement, that such names are exempt from the relevant
protective laws and regulations and therefore free for general use.
The publisher, the authors, and the editors are safe to assume that the advice and information in this book
are believed to be true and accurate at the date of publication. Neither the publisher nor the authors or
the editors give a warranty, expressed or implied, with respect to the material contained herein or for any
errors or omissions that may have been made. The publisher remains neutral with regard to jurisdictional
claims in published maps and institutional affiliations.
This Springer imprint is published by the registered company Springer Nature Switzerland AG
The registered company address is: Gewerbestrasse 11, 6330 Cham, Switzerland
DedicatedtoKurt,ontheoccasionofhis
65+X birthday.Acontinuousordiscrete
celebration?Allone!
Preface
“Don’tworryabouttheoverallimportanceoftheproblem;workonitifitlooksinteresting.
Ithinkthere’sasufficientcorrelationbetweeninterestandimportance.”
-DavidBlackwell,1984[DeG86]
Thisbookisaimedatoptimizationstudentsandresearcherslookingforsomething
a little different. The TSP (traveling-salesperson problem) is a great problem for
learning about 0/1 linear optimization (see [LLKS85], for example). The MESP
(maximum-entropy sampling problem) is a great problem for learning about 0/1
nonlinearoptimization.Evenifyouarenotdirectlyinterestedinourproblem,study
thisbook—weguaranteethatitwillopenyourmind!
Briefly, this book is mainly about algorithms for MESP: find a maximum
(log-)determinant principal submatrix (cid:24)»((cid:150)(… of a given order B, from an input
positive-semidefinite matrix (cid:24) of order = ¡ B. The problem is nice to study for a
varietyofreasons:
• MESPhasasimplestatement(liketheTSP).
• If (cid:24) is the covariance matrix for a multivariate Gaussian vector Y 2 R=, then
MESP is equivalent to finding a maximum differential-entropy subvector Y( of
sizeBfromY.
• MESP has application to the problem of designing environmental-monitoring
networksandtospatialstatisticsingeneral.
• MESPisa“submodularmaximization”problem,soithaslinkstoawell-studied
(NP-Hard)problemamenabletoapproximationalgorithms.
• MESPisNP-Hard(liketheTSP)andalreadyverychallengingwith==100(not
liketheTSP!).
• MESP is a very challenging and unusual problem from the perspective of 0/1
optimization;theconstraintsaresimple,andallofthecomplexityisinthe(non-
separable)objectivefunction(thereverseoftheusualformulationoftheTSP).
• Like the TSP, useful mathematical-optimization techniques for MESP include:
convexoptimization,semidefiniteoptimization,Lagrangianrelaxation,Lagrangian
vii
viii Preface
duality, integer-linear optimization, matching, dynamic programming, branch-
and-bound,andlocalsearch.
• MethodsforMESPuselotsofmatrixalgebraandmatrixcalculus .
Wehavetriedtomakethebookasself-containedasispractical.Someprevious
exposure to linear, integer-linear, and nonlinear optimization will make the going
easier.Wehaveanappendixontoolsthatweusefrommatrixalgebraandcalculus;
if you are unfamiliar with or rusty on those subjects, then at least breeze through
theappendixtostart.Wedonotincluderesultsofcomputationalexperiments;but
formanyoftheresultsthatwediscuss,resultsofcomputationalexperimentscanbe
foundinthesourcepapers.
“No,thisissomethingthatIcan’tunderstand,positivelycan’tunderstand.Butthestrangest,
themostincomprehensiblethingofall,ishowauthorscanchoosesuchsubjects.Iconfess
thatthisisquiteinconceivable;itisindeed...no,no,Ijustcan’tunderstanditatall!”
-NikolaiGogol(TheNose)
JonfirstgotinfectedwithMESP,byemail(butnotavirus),viaMauriceQueyranne
(University of British Columbia), who had heard about it from his infected UBC
colleague Jim Zidek, an environmental statistician. This was during Jon’s 1991-2
stayattheCenterforOperationsResearchandEconometrics(CORE)atUniversité
CatholiquedeLouvain(Belgium).Jon’sworkonMESPthatyear,withChunKo(then
apost-doctoralfellowatCentrumWiskunde&Informatica(CWI),Amsterdam;now
recovered,aTraderatCitigroup)andMaurice,wasseminalinJon’snowlonginterest
inmixed-integernonlinearoptimization.Soonafter,Jonhadadoctoralstudentatthe
UniversityofKentucky,JoyWilliams.KurtAnstreicher(UniversityofIowa),along-
time friend and collaborator of Jon’s, had a visiting Doutorado Sanduíche student
from Universidade Federal do Rio de Janeiro (UFRJ), Marcia Fampa. The four
collaboratedonMESP,leadingtoPh.D.sforJoy([Wil98])andMarcia([Fam96]),
andsonowthreemorewereinfectedwithMESP(Joyrecovered,nowVicePresident
forAcademicAffairsattheUniversityofSiouxFalls).
Another key moment was when Jon gave a talk on MESP at the Iowa Center
forGlobal Environmental Research,University ofIowa, April1994. Therehe met
Dale Zimmerman (Iowa) and Werner Müller (Johannes Kepler University, Linz),
which lead to his participation at MODA 1995 (Spetses), where he met Henry
Wynn, and at MODA 2001 (Puchberg/Schneeberg). Via all of this, some of the
mathematical-optimizationworkofJonandhisMESPcollaboratorswasexposedto
theoptimal-designcommunity.
DeGroot:Onecomponentoftheinterestisprobablythatothersareinterestedinit,anyway.
Blackwell:That’sabigcomponent.Youwanttotellsomebodyaboutitafteryou’vedoneit.
-see[DeG86]
Weconceivedtheideaofthismonographinthepre-pandemicautumnof2019,and
started it in earnest during the pandemic summer of 2020. We continued working
hard on it during the pandemic summer of 2021, completed a first draft in the
pandemicautumnof2021,andrefinedandfinalizeditduringthepandemicfirsthalf
of2022.Hopefullythereaderisatapost-pandemictime.
Preface ix
Entropy(andcannabis)“togo”atAnnArborinpandemictimes
Marcia’sworkonMESPwassupportedinpartbyCNPqgrants303898/2016-0
and434683/2018-3.JonwassupportedovertheyearstostudyMESPbyNSFgrants
DMI-9401424andCMMI-1160915,ONRgrantN00014-17-1-2296,AFOSRgrants
FA9550-19-1-0175andFA9550-22-1-0172,andadditionallybyConservatoireNa-
tionaldesArtsetMétiers.WewerefurthersupportedbyfundingfromtheSimons
Foundation and the Centre de Recherches Mathématiques (at Université de Mon-
tréal),throughtheSimons-CRMscholar-in-residenceprogram(inconnectionwith
the thematic activity Mixed-Integer Nonlinear Programming: Theory and Compu-
tation,organizedbyAndreaLodiandBruceShepherd,October2019).
We also gratefully acknowledge Leo Liberti, Sebastian Sager, and Angelika
Wiegele, for organizing the 2019 Oberwolfach workshop on Mixed-Integer Non-
linear Optimization, which catalyzed our renewed interest in maximum-entropy
sampling,viathetalkofKurtAnstreicheronthetopic.
We thank our collaborators on MESP: Hessa Al-Thani, Kurt Anstreicher, Sam
Burer, Zhongzhu Chen, Alan Hoffman, Chun Ko, Amélie Lambert, Yongchun Li,
MauriceQueyranne,JoyWilliams,andWeijunXie.Additionally,ontherelatedtopic
of submodular maximization, Jon wishes to acknowledge his collaborators: Vahab
Mirrokni,ViswanathNagarajan,AndrewOrso,SiqianShen,MaximSviridenko,and
JanVondrák.Andontheclosely-relatedtopicofconvexMINLO,Marciawishesto
acknowledgeherclosecollaboratorsWendelMeloandFernandaPereira.
AnnArbor,RiodeJaneiro,andParis MarciaFampa
June2022 JonLee
Overview
• Notation.Tablesofmuchofthenotationthatweuse.
• Chapter1:Theproblemandbasicproperties.Wedefineandgivebasicprop-
ertiesofdifferentialentropy,andthenwedefinethemaximum-entropysampling
problem(MESP)anditsconstrainedversion(CMESP).Thereminderofthechap-
terisdevotedtobasicpropertiesofMESP/CMESP.WedemonstratethatMESPis
NP-Hard,viaitsrelationshipwiththestable-set(graph)problem,andweestablish
theefficientlysolvablecaseinwhichtheinputmatrix(cid:24)oritsinverseistridiago-
nal.WeintroducetheequivalentcomplementaryproblemofMESP/CMESP.We
introducetheequivalentscaledversionofMESP/CMESP.Weintroducetheim-
portantideaof“masking”acovariancematrix(i.e.,takingitsHadamardproduct
withacorrelationmatrix),anoperationthatneverdecreasesentropy.Finally,we
demonstratethatentropyisasubmodularsetfunction.
• Chapter 2: Branch-and-bound. Our main algorithmic framework for attack-
ing MESP/CMESP is of branch-and-bound type. We describe the framework
and some methods for MESP for getting good lower bounds, i.e., good feasible
solutions.Themethodsareinthecategoriesofgreedysolutions,local-searchso-
lutions,andasubtlecombinationofthoseideastogetanefficientapproximation
algorithmwithaperformanceguarantee.
• Chapter 3: Upper bounds. A large part of the book is on upper bounds for
MESP/CMESP,akeyingredientformakingbranch-and-boundpractical.Upper
boundsarebasedonmanydifferentideasincludingconvexrelaxations,semidef-
inite optimization, eigenvalues, Lagrangian relaxation, Lagrangian duality, 0/1
linearoptimization,matching,anddynamicprogramming.
• Chapter4:Environmentalmonitoring.Animportantapplicationforallofthe
algorithmicworkonMESP/CMESPisintheareaofenvironmentalmonitoring.
Wegointosomedepthforthisapplication,includingthedetailsofhowtopass
fromrawdatatoacovariancematrixthatservesasinputforMESP/CMESP.
• Chapter5:Opportunities.TherearestillmanychallengesforMESP/CMESP.
Inthischapter,wedescribesomepromisingdirectionsforfurtherresearch.
• Appendix A: Basic formulae and inequalities. A collection of formulae and
inequalities that we make use of. This is more than a list — we put a lot of
xi
