Laplace's Demon and the Adventures of His Apprentices

29 Pages·2014·0.41 MB·English

’
Laplace s Demon and the
Adventures of His Apprentices
Roman Frigg, Seamus Bradley,
Hailiang Du, and Leonard A. Smith*y
ThesensitivedependenceoninitialconditionsðSDICÞassociatedwithnonlinearmodels
imposeslimitationsonthemodels’predictivepower.Wedrawattentiontoanadditional
limitationthanhasbeenunderappreciated,namely,structuralmodelerrorðSMEÞ.Amodel
hasSMEifthemodeldynamicsdifferfromthedynamicsinthetargetsystem.Ifanon-
linearmodelhasonlytheslightestSME,thenitsabilitytogeneratedecision-relevantpre-
dictionsiscompromised.Givenaperfectmodel,wecantaketheeffectsofSDICinto
accountbysubstitutingprobabilisticpredictionsforpointpredictions.Thisrouteisfore-
closedinthecaseofSME,whichputsusinaworseepistemicsituationthanSDIC.
1. Introduction. The sensitive dependence on initial conditions ðSDICÞ
associated with nonlinear models imposes limitations on the models’ pre-
dictive power. These limitations have been widely recognized and exten-
ReceivedDecember2012;revisedJune2013.
*Tocontacttheauthors,pleasewriteto:RomanFrigg,DepartmentofPhilosophy,Logic,and
ScientificMethod,LondonSchoolofEconomicsandPoliticalScience;e-mail:r.p.frigg@lse
.ac.uk.SeamusBradley,MunichCentreforMathematicalPhilosophy,Ludwig-Maximilians-
Universität München; e-mail: [email protected]. Hailiang Du, Centre
fortheAnalysisofTimeSeries,LondonSchoolofEconomicsandPoliticalScience;e-mail:
[email protected],CentrefortheAnalysisofTimeSeries,LondonSchool
ofEconomicsandPoliticalScience;e-mail:[email protected].
yWorkforthisarticlehasbeensupportedbytheLondonSchoolofEconomics’sGrantham
ResearchInstituteonClimateChangeandtheEnvironmentandtheCentreforClimateChange
EconomicsandPolicyfundedbytheEconomicsandSocialScienceResearchCounciland
Munich Re. Frigg acknowledges financial support from the Arts andHumanitiesResearch
Council–fundedManagingSevereUncertaintyprojectandgrantFFI2012-37354oftheSpanish
MinistryofScienceandInnovationðMICINNÞ.Bradley’sresearchwassupportedbytheAlex-
andervonHumboldtFoundation.Smithwouldalsoliketoacknowledgecontinuingsupport
fromPembrokeCollege,Oxford.WewouldliketothankWendyParker,DavidStainforth,
EricaThompson,andCharlotteWerndlforcommentsonearlierdraftsandhelpfuldiscussions.
PhilosophyofScience,81(January2014)pp.31–59.0031-8248/2014/8101-0009$10.00
Copyright2014bythePhilosophyofScienceAssociation.Allrightsreserved.
31
32 ROMANFRIGGETAL.
sivelydiscussed.1Inthisarticlewedrawattentiontoanadditionalproblem
than has been underappreciated, namely, structural model error ðSMEÞ. A
model has SME if the model dynamics differ from the dynamics in the
target system. The central claim of this article is that if a nonlinear model
has only the slightest SME, then its ability to generate decision-relevant
probabilistic predictions is compromised. We will also show that SME in
fact puts us in a worse epistemic situation than SDIC. Given a perfect
model, we can take the effects of SDIC into account by substituting
probabilistic predictions for point predictions. This route is foreclosed in
thecaseofSME,whichrelegatesbothpointpredictionsandaccurateprob-
abilistic predictions to the sweet land of idle dreams.
Toreachourconclusion,weretellthetaleofLaplace’sdemon,butwitha
twist.Inourrenderingofthetale,theDemonhastwoapprentices,aSenior
ApprenticeandaFreshmanApprentice.Theabilitiesoftheapprenticesfall
short of the Demon’s in ways that turn them into explorers of SDIC and
SME.By assumption, theDemoncancomputetheunabridgedtruth about
everything; comparing his predictions with those of the apprentices will
reveal the ways in which SDIC and SME curtail our predictive abilities.2
Insection2weintroduceourthreeprotagonistsaswellasbasicelements
of dynamical systems theory, which provides the theoretical backdrop
against which our story is told. In section 3 we follow the apprentices on
various adventures that show how predictions break down in the presence
of SME.In section 4 we provide a general mathematical argument for our
conclusion, thereby defusing worries that the results in section 3 are idio-
syncrasiesofourexampleandthattheythereforefailtocarryovertoother
nonlinear models. In section 5 we briefly discuss a number of scientific
modeling endeavors whose success is threatened by problems with SME,
which counters the charge that our analysis of SME is philosophical hair-
splitting without scientific relevance. Insection6wesuggestawayofem-
bracingtheproblem,andinsection7wedrawsomegeneralconclusions.
2. TheDemonandHisApprentices. Laplaceð1814Þinvitesustoconsider
asupremeintelligencewhoisablebothtoidentifyallbasiccomponentsof
1.Foradiscussionoftheunpredictabilityassociatedwithnonlinearsystems,seeWerndl
ð2009Þandreferencestherein.Fordiscussionsofchaosmoregenerally,see,e.g.,Smith
ð1992,1998,2007Þ,Battermanð1993Þ,andKellertð1993Þ.
2.Inothertellingsofthetale,wehavereferredtothistriadastheDemon,hisAppren-
tice,andtheNovice;theimpactofchaosontheDemonisdiscussedinSmithð1992Þ,
andhisApprenticewasintroducedinSmithð2007Þ.Ofcourse,iftheuniverseisinfact
stochastic, then the Demon will make perfect probability forecasts and appears rather
similartoI.J.Good’sInfiniteRationalOrg.Inadeterministicuniverse,itistheðseniorÞ
Apprenticewhosharesthesimilarityofperfectprobabilisticforecasts.
LAPLACE’SDEMONANDHISAPPRENTICES 33
nature and the forces acting between them and to observe these compo-
nents’initialconditions.Onthebasisofthisinformation,theDemonknows
the deterministic equations of motion of the world and uses his unlimited
computational powertosolve them.Thesolutionsofthe equationsofmo-
tion together with the initial conditions tell him everything he wants to
knowsothat“nothingwouldbeuncertainandthefuture,asthepast,would
bepresentto½his(cid:2)eyes”ð4Þ.Thisoperationallyomniscientcreatureisnow
known as Laplace’s Demon.
Let us introduce some formal apparatus in order to givea precise state-
mentoftheDemon’scapabilities.Inordertopredictthefuture,theDemon
possessesamathematicalmodeloftheworld.ItispartofLaplace’soriginal
scenariothatthemodelisamodeloftheentireworld.However,nothingin
what follows depends on the model being global in this sense, and so we
consider a scenario in which the Demon predicts the behavior of a partic-
ular part or aspect of the world. In line with much of the literature on
modeling, we refer to this part or aspect of the world as the target system.
Mathematically modeling a target system amounts to introducing a dy-
namicalsystem,ðX;f;mÞ,whichrepresentsthattargetsystem.Asindicated
t
by the notation, a dynamical system consists of three elements. The first
element,thesetX,isthesystem’sstatespace,whichrepresentstatesofthe
target system. The second element, f, is a family offunctions mapping X
t
onto itself, which is known as the time evolution: if the system is in state
x ∈X attimet50,thenitisiny5fðx Þatsomelatertimet.Thestatex
0 t 0 0
iscalledthesystem’sinitialcondition.Inwhatfollowsweassumethatf is
t
deterministic.3Forthisreason,calculatingy5fðx Þforsomefuturetimet
t 0
andagiveninitialconditionismakingapointprediction.Inthedynamical
systemsweareconcernedwithinthisarticle,thetimeevolutionofasystem
is generated by the repeated application of a map U at discrete time steps:
f 5Ut, for t50;1;2;:::,4 where Ut is the result of applying U t times.
t
Thethirdelement,m,isthesystem’smeasure,allowingustosaythatpartsof
X have certain sizes.
With this in place,we candescribe Laplace’s Demonasa creaturewith
the following capabilities:
1. ComputationalOmniscience:heisabletocalculatey5fðxÞforallt
t
and for any x arbitrarily fast.
2. Dynamical Omniscience: he is able to formulate the true time evo-
lution f of the target system.
t
3.Infact,itsufficesforf tobeforwarddeterministic;seeEarmanð1986,chap.2Þ.
t
4.Thisisacommonassumption.Foranintroductiontodynamicalsystems,seeArnold
andAvezð1968Þ.
34 ROMANFRIGGETAL.
3. Observational Omniscience: he is able to determine the true initial
condition x of the target system.
0
If these conditions were met, the Demon could compute the future with
certainty. Laplace is quick to point out that the human mind “will always
remain infinitely removed” from the Demon’s intelligence, of which it
offersonlya“feebleidea”ð1814,4Þ.Thequestiontheniswhattheseshort-
comingsareandhowtheyaffectourpredictiveabilities.Itisacuriousfact
that while the failure of computational and observational omniscience has
been discussed extensively, relatively little has been said about how not
being dynamically omniscient affects our predictive abilities.5 The aim of
this article is to fill this gap.
To aidourexplorations,we providetheDemonwithtwoapprentices—
theSeniorApprenticeandtheFreshmanApprentice.Likethemaster,both
apprenticesarecomputationallyomniscient.TheDemonhassharedthegift
of dynamical omniscience with the Senior Apprentice: they both have the
perfectmodel.ButtheDemonhasnotgrantedtheSeniorobservationalom-
niscience:shehasonlynoisyobservationsandcanspecifythesystem’sini-
tial condition only within a certain margin of error. The Freshman has not
yetbeengrantedeitherobservationalordynamicalomniscience:hehasnei-
theraperfectmodelnorpreciseobservations.
Both apprentices are aware of their limitations and come up with cop-
ing strategies. They have read Poincaré and Lorenz, and they know that a
chaoticsystem’stimeevolutionexhibitsSDIC:evenarbitrarilycloseinitial
conditionswillfollowverydifferenttrajectories.Thiseffect,alsoknownas
the butterfly effect, makes it misinformative to calculate y5fðz Þ for an
t 0
approximateinitialconditionz becauseevenif z isarbitrarilyclosetothe
0 0
trueinitialconditionx ,fðz Þandfðx Þwilleventuallydiffersignificantly.
0 t 0 t 0
Toaccountfortheirlimitedknowledgeaboutinitialconditions,eachap-
prenticecomesupwithaprobabilitydistributionoverrelevantinitialstates,
which accounts for their observational uncertainty about the system’s ini-
tial condition. Call such a distribution p ðxÞ; the subscript indicates that
0
the distribution describes uncertainty in x at t50.6 The relevant question
thenishowinitialprobabilitieschangeover thecourseoftime. Toanswer
thisquestion,theyusef toevolvep ðxÞforwardintimeði.e.,tocalculate
t 0
pðxÞÞ.Weusesquarebracketstoindicatethatf½p ðxÞ(cid:2)istheforwardtime
t t 0
image of p ðxÞ. The time evolution of the distribution is given by the
0
Frobenius-Perronoperator ðBerger2001, 126–27Þ. If thetime evolution is
one-to-one, this operator reduces to pðxÞ5p ðf ðxÞÞ.
t 0 2t
5.See,however,Smithð2002ÞandMcWilliamsð2007Þ.
6.Ourargumentdoesnottradeonthespecificformof p0ðxÞ;weassumep0ðxÞisideal
giventheinformationavailable.
LAPLACE’SDEMONANDHISAPPRENTICES 35
The idea is simple and striking: if p ðxÞ provides them with the proba-
0
bility of finding the system’s state at a particular place in X at t50, then
pðxÞ is the probability of finding the system’s state at a particular place at
t
any later time t. And the apprentices do not only make the ðtrivialÞ state-
mentthat pðxÞ isa probabilitydistribution in a purely formalsense ofbe-
t
inganobjectthatsatisfiesthemathematicalaxiomsofprobability;theyare
committed to the ðnontrivialÞ claim that the probabilities are decision rel-
evant. In other words, the apprentices take pðxÞ to provide us with pre-
t
dictions about the future of sufficient quality that we ought to place bets,
set insurance premiums, or make public policy decisions according to the
probabilities given to us by pðxÞ.
t
This solves the Senior Apprentice’s problem, but the Freshman has a
furtherobstacletoovercome:thefactthathismodelhasastructuralmodel
error ðSMEÞ. We face a SME when the model’s functional form is rele-
vantly different from that of the true system. In technical terms, by SME
we mean the condition when the dynamical equations of the model differ
from the true equations describing the system under study: in some cases
we can write fM 5fT 1d, where fM is the dynamics of the model, fT is
t t t t t
thetrue dynamicsofthe system,and d is thedifferencebetween thetwo.7
t
The Freshman’s solution to this problem is to adopt what he calls the
closeness-to-goodnesslink.Theleadingideabehindthislinkisthemaxim
that a model that is close enough to the truth will produce predictions that
arecloseenoughtowhatactuallyhappenstobegoodenoughforacertain
predictive task. Given that we consider time evolutions that are generated
by the iterative application of a map, this idea can be made precise as
follows. Let U be the Demon’s map ðwhere the subscript T stands for
T
‘True’, as the Demon has the true modelÞ, and let U be the Freshman’s
F
approximatetimeevolution.ThenD :5U 2U isthedifferencebetween
U T F
the two maps, assuming they share the same state space. Furthermore, let
pTðxÞ be probabilities obtained under the true time evolution ðwhere
t
fT 5UtÞ,andpFðxÞtheprobabilitiesthatresultfromtheapproximatetime
t T t
evolutionðwherefF 5UtÞ;D ðx;tÞisthedifferencebetweenthetwo.The
t F p
closeness-to-goodnesslinksaysthatifD issmall,thenD ðx;tÞissmalltoo
U p
for all times t, presupposing an appropriate notion of being small. The
notionofbeingsmallcanbeexplainedindifferentwayswithoutalteringthe
7.Notethatthisequationassumesthatthemodelandthesystemsharethesamestate
space,thatis,thattheyaresubtractableðseeSmith2006Þ.Theyneednotbe.Alsonote
thatSMEcontrastswithparameteruncertainty,wherethemodelsharesthetruesystem’s
mathematical structure, yet the true values of certain parameters are uncertain in the
model.Parametersmaybeuncertainwhenthemathematicalstructureisperfect,butthey
are indeterminate given SME: no set of parameter values will suffice to perfect the
model.
36 ROMANFRIGGETAL.
conclusion. Below we quantify D in terms of the maximal one-step error
U
and D ðx;tÞ in terms of the relative entropy of the two distributions.
p
3. TheApprentices’Adventures. TheDemonschedulesatutorial.TheSe-
nior Apprentice claims that while her inability to identify the true initial
conditionpreventsherfrommakingvalidpointpredictions,herprobability
forecastsaregoodinthesensethat,conditionedontheinformationtheDe-
mon allows her ðspecifically her initial probability distribution p ðxÞÞ, she
0
isable toproducea decision-relevantdistributionpðxÞ for all later times t.
t
The Freshman does not want to play second fiddle and ventures the bold
claim that dynamical omniscience is as unnecessary as observational om-
niscienceandthathecanachievethedecisionrelevanceusinganimperfect
model and the closeness-to-goodness link.
The all-knowing Demon requires them to put their skills to test in a
concrete situation in ecology: the evolution over time of a population of
rapidlyreproducingfishinapond.Tothisend,theyagreetointroducethe
population density ratio r: the number of fish per cubic meter at time t
t
dividedbythemaximumnumberoffishthepondcouldaccommodateper
cubicmeter.Hencer liesintheunitinterval½0;1(cid:2).Thentheygoawayand
t
study the situation.
Afterawhiletheyreconveneandcomparenotes.TheFreshmansuggests
thatthedynamicsofthesystemcanbemodeledsuccessfullywiththewell-
known logistic map:
r 54rð12rÞ; ð1Þ
t11 t t
wherethe differencebetween timest andt11 isa generationðwhich,for
easeofpresentation,weassumetobe1weekÞ.Recallfromsection2thata
dynamical system is a three-partite entity consisting of a state space X, a
time evolution operator f ðwhere f 5Ut if the time evolution is gener-
t t
ated by the repeated application of a map U at discrete time stepsÞ, and a
measure m. The Freshman’s model is a dynamical system that consists of
thestatespaceX 5½0;1(cid:2);histimeevolutionfF isgeneratedbyiteratively
t
applying4rð12rÞ, whichis U ; mis the standard Lebesguemeasure on
t t F
½0;1(cid:2).
TheDemonandtheSeniorApprenticeknowthetruedynamicallawfor
r:
t
16(cid:4) (cid:2) (cid:3)(cid:5)
~r 5ð12εÞ4~rð12~rÞ1ε ~r 122~r21~r3 ; ð2Þ
t11 t t 5 t t t
where ε is a small parameter. The tilde notation is introduced and justified
in Smith ð2002Þ. The right-hand side of equation ð2Þ, which we call the
quartic map, is U ; applying U iteratively yields fT.
T T t
LAPLACE’SDEMONANDHISAPPRENTICES 37
ItisimmediatelyclearthattheFreshman’smodellacksasmallstructural
perturbation:asε→0theDemon’smapconvergestowardtheFreshman’s.
Figure1showsbothU andU forε50:1,illustratinghowsmallthedif-
T F
ference between the two is.
We now associate the D with fF’s one-step error: the maximum dif-
U t
ference between fF and fTðxÞ for x ranging over the entire X. The maxi-
t t
mum one-step error of the model is 5(cid:3)1023 at x50:85344, where r
t11
50:50031and~r 50:49531,andhenceitisreasonabletosaythatD is
t11 U
small.Applyingthecloseness-to-goodnesslink,theFreshmannowexpects
D ðx;tÞ to be small too. That is, starting with the same initial probability
p
distribution p ðxÞ, he would expect pTðxÞ and pFðxÞ to be least broadly
0 t t
similar. We will now see that the Freshman is mistaken.
SinceitisimpossibletocalculatepTðxÞandpFðxÞwithpencilandpaper,
t t
weresorttocomputersimulation.Tothisend,wepartitionX into32cells,
which,inthiscontext,arereferredtoasbins.Thesebinsarenowtheatoms
of our space for evaluating predictions: in what follows we calculate the
Figure1. Equationð1Þindottedlineandequationð2Þinshadedline,withr and~r
t t
ontheX-axisandr and~r ontheY-axis.Colorversionavailableasanonline
t11 t11
enhancement.
38 ROMANFRIGGETAL.
probabilitiesofthesystem’sstatexbeinginacertainbin.Thisisofcourse
notthesameascalculatingacontinuousprobabilitydistribution,butsince
nothing in what follows hangs on the difference between a continuous
distributionandoneoverbins,andforthesakeofnotationalease,werefrain
from introducing a new variable and take ‘pTðxÞ’ and ‘pFðxÞ’ to refer to
t t
the probabilities of bins. Similarly, a computer cannot handle analytical
functions ðor real numbersÞ, and so we represent p ðxÞ by an ensemble of
0
1,024 points. We first draw a random initial condition ðaccording to the
invariant measure of the logistic mapÞ. By assumption this is the true ini-
tial condition of the system at t50, and it is designated by the cross in
figure 2a. Wethenchooseanensembleof1,024pointsconsistentwiththe
true initial condition. These 1,024 points form our ensemble, shown as a
distributioninfigure2a.DividingthenumbersontheY-axisby1,024yields
anestimateoftheprobabilityforthesystem’sstatetobeinaparticularbin.
Figure 2. Evolution of the initial probability distribution under the Freshman’s
approximate dynamics ðblackÞ and the Senior’s true dynamics ðgrayÞ. The gray
crossmarkstheDemon’sevolutionofthetrueinitialcondition;theblackcrossis
theFreshman’sevolutionofthetrueinitialcondition.Y-axisindisrescaledtomake
thedetailsmorevisible.Colorversionavailableasanonlineenhancement.
LAPLACE’SDEMONANDHISAPPRENTICES 39
We now evolve all these points forward both under the Senior’s dy-
namicsðgraylinesÞandtheFreshman’sdynamicsðblacklinesÞ.Figures2b–
2d show how many points there are in each bin at t52, t54, and t58.
While the two distributions overlap relatively well after 2 and 4 weeks,
they are almost completely disjoint after 8 weeks. Hence, for this x these
0
calculations show the failure of the closeness-to-goodness link: D being
U
small does not imply that D ðx;tÞ is also small for all t. In fact, for t58,
p
D ðx;tÞ isas large ascan bebecausethereis nooverlapat all between the
p
two distributions.8
Two important points emerge from this example. The first point is that
eventhoughchaosundercutspointpredictions,onecanstillmakeinforma-
tive probabilistic predictions. The position of the gray cross is appropri-
atelyreflectedbythegraydistributionatalltimes:thegrayprobabilitydis-
tributionremainsmaximallyinformativeaboutthesystem’sstategiventhe
informationavailable.
Thesecondandmoreunsettlingpointisthattheabilitytoreliablymake
decision-relevant probabilistic forecasts is lost if nonlinearity is combined
withSME.EventhoughtheFreshman’sdynamicsareveryclosetotheDe-
mon’s, hisprobabilitiesareoff track:he regardseventsthat donothappen
asverylikely,whileheregardswhatactuallyhappensasveryunlikely.So
his predictions here are worse than useless: they are fundamentally mis-
leading.Hence,simplymovinganinitialdistributionforwardintimeunder
thedynamicsofamodelðevenagoodoneÞneednotyielddecision-relevant
evidence.Even models that yielddeep physical insightcanproduce disas-
trousprobabilityforecasts.ThefactthatasmallSMEcandestroytheutility
of a model’s predictions is called the hawkmoth effect.9 The effect illus-
tratesthatthecloseness-to-goodnesslinkfails.
This example shows that what truly limits our predictive ability is not
SDIC but SME. In other words, it is the hawkmoth effect rather than the
butterfly effect that decimates our capability to make decision-relevant
forecasts. We can mitigate against the butterfly effect by replacing point
forecasts with probabilistic forecasts, but we have no comparable move
withforceagainstthehawkmotheffect.Andthesituationdoesnotchange
in the long run. It is true that distributions will spread with time and as
t→`. As the distribution approaches the system’s natural measure it be-
comes uninformative. But becoming uninformative and being misleading
are very different vices.
8.Thisnotionismadepreciseintermsofrelativeentropybelow.
9.Thompson ð2013Þintroduced this term in analogy to the butterfly effect.The term
alsoemphasizesthatSMEyieldsaworseepistemicpositionthanSDIC:hawkmothsare
bettercamouflagedandlessphotogenicthanbutterflies.
40 ROMANFRIGGETAL.
One could object that the presentation of our case is biased in various
ways. The first alleged bias is the choice of the particular initial distribu-
tion shown in figure 2a. This distribution, so the argument goes, has been
carefullychosentodriveourpointhome,butmostotherdistributionswould
notbemisleadinginsuchaway,andourresultonlyshowsthatunexpected
results can occur every now and then but does not amount to a wholesale
rejection of the closeness-to-goodness link.
There is of course no denying that the above calculations rely on a
particular initial distribution, but that realization does not rehabilitate the
closeness-to-goodness link. We have repeated the same calculations with
2,048differentinitialdistributionsðchosenrandomlyaccordingtothenat-
ural measure of the logistic mapÞ, and so we obtain 2,048 pairs of pTðxÞ
t
and pFðxÞ for t52, t54, and t58.
t
So far we operated with an intuitive notion of the difference between
twodistributions.Butinordertoanalyzethe2,048pairsofdistributions,we
need a formal measure of the difference between two distributions. We
choose the so-called relative entropy:
E
(cid:6) (cid:7)
1
pF
SðpFjpTÞ :5 pFln t dx;
t t t pT
0 t
where ‘ln’ is the natural logarithm.10 The relative entropy provides a mea-
sure for the overlap of two distributions. If the distributions overlap per-
fectly—pF equals pT—their ratio is then one in the logarithm, and the
t t
entropyiszero;themoredissimilarthedistributions,thehigherthevalueof
SðpFjpTÞ.Hence,itisreasonabletoconsiderD ðx;tÞ:5SðpFjpTÞ.Figure3
t t p t t
showsahistogramoftherelativeentropyofour2,048distributionsatt58.
The histogram shows that the Freshman’s probabilities are in line with
the Senior’s only in about a quarter of the cases. Almost half of the dis-
tributionpairshaverelativeentropy7ormore.Thetwodistributionsshown
infigure2dhave arelative entropyof8.23.11 So ourhistogramshows that
at t58 almost half of all distribution pairs are as disconnected as those
in figure 2d and, hence, are seriously misleading.
Thereisatemptationtorespondthatthisdoesnotshowthatprobabilities
are useless; it only shows that we should not use these probabilities when
they are misleading. The problem with this suggestion is that outside our
10.Inourcasetheintegralbecomesasumoverthebinsofthepartition.Foradiscussion
ofrelativeentropyandinformationtheory,seeCurdandThomasð1991Þ.
11.Giventhatourensembleisonlyfinite,weassigntheprobability1=ð1;024(cid:3)32Þto
anybinwithnoensemblememberatall.Ifthatbinoccurs,thentheentropywouldbe
~10.4nats.Hence,~10.4reflectsthemaximumvalueoftheentropythatcanbeobserved
intheseexperiments.