On Probability Estimation by Exponential Smoothing

0.35 MB·English

by  Christopher Mattern

On Probability Estimation by Exponential Smoothing
Christopher Mattern
TechnischeUniversitätIlmenau
Ilmenau,Germany
[email protected]
Abstract. Probabilityestimationisessentialforeverystatisticaldatacompressionalgorithm.
In practice probability estimation should be adaptive, i.e. recent observations should receive
5 ahigherweightthanolderobservations. Wepresentaprobabilityestimationmethodbasedon
1 exponentialsmoothingthatsatisfiesthisrequirementandrunsinconstanttimeperletter. Our
0
main contribution is a theoretical analysis in case of a binary alphabet for various smoothing
2
n rate sequences: W√e show that the redundancy w.r.t. a piecewise stationary model with s
segments is O(s n) for any bit sequence of length n, an improvement over redundancy
a √
(cid:0) (cid:1)
J O s nlogn ofpreviousapproacheswithsimilartimecomplexity.

9

1 Introduction
]
T
I Background. Sequentialprobabilityassignmentisanelementarycomponentofeverysta-
.
s tistical data compression algorithm, such as Prediction by Partial Matching, Context Tree
c
[ Weighting and PAQ (“pack”). Statistical compression algorithms split compression into

2 modeling and coding and process an input sequence letter-by-letter. During modeling a
v modelcomputesadistributionpandduringcodinganencodermapsthenextletterx,given
2
p,toacodewordofalengthcloseto−logp(x)bits(thisistheidealcodelength). Decoding
0
2 is the reverse: Given p and the codeword the decoder restores x. Arithmetic Coding is the
1
de facto standard en-/decoder, it closely approximates the ideal code length [1]. All of the
0
. mentionedcompressionalgorithmsrequiresimple,elementarymodelstopredictaprobabil-
1
0 itydistribution. Elementarymodelsaretypicallybasedonsimpleclosed-formexpressions,
5 suchasrelativeletterfrequencies.Nevertheless,elementarymodelshaveabigimpactonboth
1
theoretical guarantees [9, 11] and empirical performance [4, 10] of statistical compression
:
v
algorithms.(Commonly,weexpresstheoreticalguaranteesonamodelbytheamountofbits
i
X
themodelrequiresaboveanidealcompetingschemeassumingidealencoding,theso-called
r
redundancy.) Itiswisetochooseelementarymodelscarefullyanddesirabletoanalyzethem
a
theoreticallyandtostudythemexperimentally. Inthisworkwefocusonelementarymodels
with the ability to adapt to changing statistics (see next paragraph) whose implementation
meets practical requirements, that is O(Nn) (arithmetic) operations and O(N) data words
(holdinge.g.acounterorarationalnumber)whileprocessingasequenceoflengthnoveran
alphabetofsizeN.
Previous Work. Relative frequency-based elementary models, such as the Laplace- and
KT-Estimator,arewell-knownandwell-understood[1].Amajordrawbackoftheseclassical
techniquesisthattheydon’texploitrecencyeffects(adaptivity): Foranaccurateprediction
novelobservationsareofhigherimportancethanpastobservations[2,7].Fromatheoretical
pointofviewadaptivityisevidentinlowredundancyw.r.t.anadaptivecompetingscheme
suchasaPiecewiseStationaryModel(PWS).APWSpartitionsasequenceoflengthnarbi-
trarilyintossegmentsandpredictsanarbitraryfixedprobabilitydistributionforeveryletter
within a segment. (Since both segmentation and prediction within a segment are arbitrary,
wemayassumebothtobeoptimal.)
Toliftthelimitationofclassicalrelativefrequency-basedelementarymodelswetypically
age observation counts, aging takes place immediately before incrementing the count of a
novelletter. Foragingfrequency-basedelementarymodelsthereexisttwomajorstrategies,
whichareheavilyusedinpractice. InStrategy 1(countrescaling)wedivideallcountsbya
factorinwell-definedintervals(e.g.whenthesumofallcountsexceedsathreshold)[2],for
Strategy 2(countsmoothing)wemultiplyallcountsbyafactorin(0,1)ineveryupdate[7].
√
Strategy1wasanalyzedin[5]andhasredundancyO(s nlogn). Similarly,aKT-estimator
which completely discards all counts periodically was analyzed in [8] and has redundancy
√
O(s nlogn).Strategy2wasstudiedmainlyexperimentally[7,9,10].
Anotherapproachforadaptiveprobabilityestimation,commonlyusedinPAQ,issmooth-
ing of probabilities, Strategy 3. Given a probability distribution (i.e. the prediction of the
previousstep)andanovelletterwecarryoutanupdateasfollows:Firstwemultiplyallprob-
abilitieswithsmoothingrateα ∈ (0,1)andafterwardsweincrementtheprobabilityofthe
novelletterby1−α.Smoothingrateαdoesnotvaryfromsteptostep.Toourknowledgethis
common-senseapproachwasfirstmentionedin[3].Afinitestatemachinethatapproximates
smoothing was analyzed in [6] and has redundancy O(nK−2/3) w.r.t. PWS with s = 1,
whereK isthenumberofstates.
Allaforementionedapproachesmeetpracticaldemands,theyrequireO(Nn)(arithmetic)
operationsandO(N)datawords.Morecomplex(butunpractical)methodsarebasedonmix-
turesoverelementarymodelsassociatedtoso-calledtransitiondiagrams[8,12]orassociated
to(PWS-)partitions[10].
OurContribution. Inthisworkweanalyzeageneralizationofstrategies2and3forabinary
alphabet. Basedonmildassumptionsonsequenceα ,α ,... ofsmoothingrates(α isused
1 2 k
for an update after observing the k-th letter) we explicitly identify an input sequence with
maximumredundancyw.r.t.PWSwiths = 1andsubsequentlyderiveredundancybounds
fors ≥ 1(Section3).ForPWSwitharbitraryswegiveredundancyboundsforthreechoices
ofsmoothingratesinSection4.First,weconsiderafixedsmoothingrateα = α = α = ...
√ 1 2
(asinPAQ)andprovideα∗(n)thatguaranteesredundancyO(s n)forasequenceoflength
n; second, weproposeavaryingsmoothingrate, whereα ≈ α∗(k); andfinallyavarying
k
smoothingratethatisequivalenttoStrategy2fromtheprevioussection. Bytuningparam-
√
eters we obtain redundancy O(s n) for all smoothing rate choices, an improvement over
redundancyguaranteesknownsofarformodelsrequiringO(Nn)(arithmetic)operationsper
inputsequence. Section5supportsourboundswithasmallexperimentalstudyandfinally
Section6summarizesandevaluatesourresultsandgivesperspectivesforfutureresearch.
2 Preliminaries
Sequences. Weusex todenoteasequencex x ...x ofobjects(numbers,letters,...).
i:j i i+1 j
Unless stated differently, sequences are bit sequences (have letters {0, 1}). If i > j, then
x := φ, where φ is the empty sequence; if j = ∞, then x = x x ... has infinite
i:j i:j i i+1
length. Forsequencex definex := x andx := x ;wecallx deterministic,if
1:n <i 1:i−1 ≤i 1:i 1:n
x = ··· = x ,andnon-deterministic,otherwise.
1 n
Code Length and Entropy. Code length is measured in bits, thus log := log . For prob-
2
ability distribution p over {0,1} and letter x we define (cid:96)(x;p) := −logp(x). The binary
entropy function is denoted as H(q) := −qlog(q) − (1 − q)log(1 − q), for a probability
q. Forsequencex andrelativefrequencyq ofa1-bitinx leth(x ) := nH(q)bethe
1:n 1:n 1:n
empiricalentropyofx .
1:n
Partitions and Sets. Calligraphiclettersdenotesets. Apartitionofanon-emptysegment
(interval) (a,b] of integers is a set of non-overlapping segments (i ,i ],...,(i ,i ] s.t.
0 1 n−1 n
a = i < i < ··· < i = b.Thephrasek-thsegmentuniquelyrefersto(i ,i ].
0 1 n k−1 k
ModelsandExponentialSmoothingofProbabilities. Wefirstcharacterizethetermmodel
fromstatisticaldatacompression,inorderdefineourmodelingmethod. AmodelMDLmaps
asequencex oflengthn ≥ 0toaprobabilitydistributionpon{0,1}.Wedefinetheshort-
1:n
handsMDL(x≤n) := p(thisisnottheprobabilityofsequencex≤n!)andMDL(x;x≤n) := p(x).
(cid:80)
Model MDL assigns (cid:96)(x≤n;MDL) := − 1≤k≤nlogMDL(xk;x<k) bits to sequence x≤n. We
arenowreadytoformallydefineourmodelofinterest.
Definition2.1. Forsequenceα ,where0 < α ,α ,··· < 1,andprobabilitydistribution
1:∞ 1 2
p, where p(0),p(1) > 0, we define model ESP = (α1:∞,p) by the sequential probability
assignmentrule

αkESP(x;x<k)+1−αk, ifk > 0andx = xk
ESP(x;x≤k) = αkESP(x;x<k), ifk > 0andx (cid:54)= xk . (1)

p(x), ifk = 0
SmoothingratescontroltheadaptionofESP,largeαi’sgivehighweighttooldobservations
andlowweighttonewobservations,theconverseholdsforsmallα ’s. Forouranalysiswe
i
mustassumethatthesmoothingratesaresufficientlylarge:
Assumption2.2. ESP = (α1:∞,p)satisfies 12 < α1,α2,··· < 1andw.l.o.g.p(0) ≤ p(1).
For the upcoming analysis the product of smoothing rates plays an important role. Hence,
givensmoothingratesequenceα ,wedefineβ = 1andβ := α ·...·α ,fori > 0.
1:∞ 0 i 1 i
3 Redundancy Analysis
First Main Result. NowwecanstateourfirstmainresultwhichcomparesESPtothecode
lengthofanoptimalfixedcodeforx ,thatistheempiricalentropyh(x ).Beforeweprove
1:n 1:n
thetheorem,wediscussitsimplications.
Theorem3.1. IfAssumption2.2holds,thenwehave
(cid:96)(x1:n;ESP)−h(x1:n)
(cid:40) (cid:80)n−1log 1 , ifx isdeterministic
≤ i=0 1−p(1)βi 1:n . (2)
log 1 +(cid:80)n−2log 1 −nH(cid:0)1(cid:1), otherwise
p(1)βn−1 i=0 1−p(1)βi n
RecallthatbyAssumption2.2wehavep(0) ≤ p(1).First,consideradeterministicsequence
of0-bits.By(1)wehaveESP(1;x≤i) = βip(1),thusESP(0;x≤i) = 1−βip(1),bothfor0 ≤
i < n.Thetotalcodelengthis(cid:96)(x1:n;ESP) = −(cid:80)in=−01log(1−βip(1))andclearlyh(x1:n) =
0,sotheredundancy(cid:96)(x1:n;ESP)−h(x1:n)matches(2). Nowconsideranon-deterministic
sequencex = 00...01withsingle1-bitatpositionn. Similartothedeterministiccase
1:n
thesameequationsforESP(·;x≤i)hold,fori < n. Thetotalcodelengthis(cid:96)(x1:n;ESP) =
−(cid:80)n−2log(1 − β p(1)) − log(β p(1))andtheempiricalentropyish(x ) = nH(1).
i=0 i n−1 1:n n
Again,theredundancymatches(2). Insummary,ifp(0) ≤ p(1),then00...0isadetermin-
isticsequencewithmaximumredundancyand00...01isanon-deterministicsequencewith
maximumredundancy. Similarstatementshold,ifp(0) ≥ p(1): bysymmetrywemusttog-
gle0-bitsand1-bits. Whenp(0) = p(1)wehaveequalredundancy,e.g.(cid:96)(00...0;ESP) =
(cid:96)(11...1;ESP)inthedeterministiccase. Insummary,foragiveninstanceofESP(thatsatis-
fiesAssumption2.2)theworst-caseinputiseither00...0(only0-bits)or00...01(single1-
bit),amongall2nbitsequencesoflengthn.Forfixednwecannoweasilycomparetheredun-
danciesofthosetwoinputsandimmediatelydepicttheworst-caseinputanditsredundancy.
FortheproofofTheorem3.1werequirethefollowinglemma.
Lemma3.2. Anynon-deterministicsequencex oflengthn ≥ 2satisfies
1:n
(cid:40) (cid:0) (cid:1)
nH 1 , ifx isdeterministic
h(x )−h(x ) ≥ n 2:n .
1:n 2:n nH(cid:0)1(cid:1)−(n−1)H(cid:0) 1 (cid:1), otherwise
n n−1
Proof. Let1−pbetherelativefrequencyofx inx ,thush(x )−h(x ) = nH(p)−
1 1:n 1:n 2:n
(cid:0) (cid:1)
(n−1)H n ·p =: f(p).Wedistinguishtwocases:
n−1
(cid:0) (cid:1) (cid:0) (cid:1)
Case1: x isdeterministic.Wehavep = n−1 andf(p) = nH n−1 = nH 1 .
2:n n n n
Case2: x isnon-deterministic. Since H(p) is concave, H(cid:48)(p) is decreasing and f(cid:48)(p) =
2:n
(cid:2) (cid:0) (cid:1)(cid:3)
n H(cid:48)(p)−H(cid:48) n ·p ≥ 0, i.e. f(p) is increasing and minimal for minimum p. Since
n−1
(cid:0) (cid:1) (cid:0) (cid:1)
x isnon-deterministictheminimumvalueofpis 1 andwegetf(p) ≥ f 1 = nH 1 −
1:n n n n
(cid:0) (cid:1)
(n−1)H 1 .
n−1
Nowletusproceedwiththemajorpieceofworkinthissection.
ProofofTheorem3.1. Wedefiner(x1:n,ESP) := (cid:96)(x1:n;ESP)−h(x1:n)anddistinguish:
Case1: x1:n isdeterministic. By p(0) ≤ p(1) (Assumption 2.2) we have ESP(x;x<i) ≥
ESP(0;x<i) = 1−p(1)βi−1andh(x1:n) = 0,weget
(cid:88) 1 (cid:88) 1
r(x1:n,ESP) = log ≤ log .
ESP(xi;x<i) 1−p(1)βi
1≤i≤n 0≤i<n
Case2: x isnon-deterministic.Wehaven ≥ 2andbyinductiononnweprove
1:n
1 (cid:88) 1 (cid:0) (cid:1)
r(x1:n,ESP) ≤ log p(1)β + log 1−p(1)β −nH n1 .
n−1 i
0≤i<n−1
(cid:0) (cid:1)
Base: n = 2. — We have x ∈ {01,10}, in either case h(x ) = nH 1 = 2 and
1:n 1:n n
(cid:96)(x1:n;ESP) = log p(x1)β11p(x2) = log p(11)β1 +log 1−p1(1),theclaimfollows.
Step: n > 2. — By defining ESP(cid:48) = (α1(cid:48):∞,p(cid:48)), where αi(cid:48) = αi+1, βi(cid:48) = α1(cid:48) · ... · αi(cid:48),
p(cid:48) = ESP(x≤1)wemaywrite
1
r(x1:n,ESP) = log +r(x2:n,ESP(cid:48))−(h(x1:n)−h(x2:n)). (3)
p(x )
1
Now w.l.o.g. fix p(cid:48) s.t. p(cid:48)(0) ≤ p(cid:48)(1). Since we want to bound (3) from above, we must
choosex s.t.p(x )isminimal(andther.h.s.of(3)ismaximal).Todoso,distinguish:
1 1
Case1: x = 0. For some distribution q with q(0) > 0 we have p(x ) = q(0) and 1 ≥
1 1 2
(cid:2) (cid:3)
p(cid:48)(0) = α q(0) + 1 − α , thus q(0) ≤ α − 1 /α . (Notice the subtle detail: α ≤ 1
1 1 1 2 1 1 2
impliesq(0) ≤ 0,whichcontradictsq(0) > 0andwouldmakeCase1impossible;however
weassumedα > 1.)Furthermore,wehaveq(0) ≤ 1.
1 2 2
Case2: x = 1. For some distribution r with r(1) > 0 we have p(x ) = r(1) and 1 ≤
1 1 2
(cid:2) (cid:3)
p(cid:48)(1) = α r(1)+1−α ,thusr(1) ≥ α − 1 /α .
1 1 1 2 1
Since q(0) ≤ r(1) (i.e. Case 1 minimizes p(x )) and q(0) ≤ 1 we may now w.l.o.g.
1 2
assumethatx = 0,p(cid:48)(1) = α p(1),p(x ) = 1−p(1)andp(0) ≤ p(1).Wedistinguish:
1 1 1
Case1: x isdeterministic. We must have x = 11...1, since x = 0 and x is non-
2:n 2:n 1 1:n
deterministic,thus
(cid:88) 1 1 (cid:88) 1
r(x2:n,ESP(cid:48)) = log 1−p(cid:48)(0)β(cid:48) ≤ log p(1)β + log 1−p(1)β , (4)
0≤i<n−1 i n−1 1≤i<n−1 i
where we obtain the inequality by p(cid:48)(0)β(cid:48) ≤ p(cid:48)(1)β(cid:48) = p(1)β , for i < n − 2 and
i i i+1
1 − p(cid:48)(0)β(cid:48) = 1 − [1 − p(1)α ]β /α ≥ p(1)β , for i = n − 2. To obtain the
n−2 1 n−1 1 n−1
claimweplugtheinequalities(4)andh(x )−h(x ) ≥ nH(1)(byLemma3.2)into(3)
1:n 2:n n
andnotethatp(x ) = 1−p(1)β (sinceβ = 1).
1 0 0
Case2: x isnon-deterministic.Thehypothesisandp(cid:48)(1)β(cid:48) = p(1)β yield
2:n i i+1
1 (cid:88) 1 (cid:0) (cid:1)
r(x2:n,ESP(cid:48)) ≤ log p(cid:48)(1)β(cid:48) + log 1−p(cid:48)(1)β(cid:48) −(n−1)H n−11
n−2 0≤i<n−2 i
1 (cid:88) 1 (cid:0) (cid:1)
= log + log −(n−1)H 1 . (5)
p(1)β 1−p(1)β n−1
n−1 i
1≤i<n−1
Weplugtheinequalities(5)andh(x )−h(x ) ≥ nH(1)−(n−1)H( 1 )(byLemma3.2),
1:n 2:n n n−1
into(3)andnotethatp(x ) = 1−p(1)β (sinceβ = 1)toendtheproof.
1 0 0
Second Main Result. LetusnowextendthecompetingschemeofTheorem3.1,towhich
wecompareESPto. Supposethecompetingschemesplitstheinputsequencex1:naccording
toanarbitrarypartitionS of[1,n]andmayuseanoptimalfixedcodewithineverysegment
[a,b] ∈ S. The competing scheme has total coding cost h(x ) for x , thus coding cost
a:b a:b
(cid:80)
h(x )forx Notice,thatthisalowerboundonthecodingcostofanyPWSwith
[a,b]∈S a:b 1:n
partitionS. SincethesituationwithinasegmentresemblesthesituationofTheorem3.1,we
maynownaturallyextendtheredundancyanalysistotheaforementionedcompetitor.
Theorem3.3. LetS beanarbitrarypartitionof[1,n]. IfAssumption2.2holds,then
(cid:88) 1 (cid:88) (cid:88) 1
(cid:96)(x1:n;ESP)− h(xa:b) ≤ |S|log + log . (6)
p(0)β 1−β /β
n−1 i a
[a,b]∈S (a,b]∈Sa<i<b
Proof. Letr(x1:n,ESP) := (cid:96)(x1:n;ESP)−h(x1:n). Ourplanfortheproofistosimplify(2)
(seecalculationsbelow)toyield
1 (cid:88) 1
(cid:96)(x1:n;ESP)−h(x1:n) ≤ log + log (7)
p(0)β 1−β
n−1 i
1≤i<n
anduseto(7)toboundtheredundancyforanarbitrarysegment(a,b]fromS(seecalculations
below)via
(cid:88) 1 1 (cid:88) 1
log −h(x ) ≤ log + log . (8)
a+1:b
ESP(xi;x<i) p(0)βb−1 1−βi/βa
a<i≤b a<i<b
Wenowobtain(6)easilybysumming(8)overallsegments(a,b]fromSandbyβ ≥ β .
b−1 n−1
Simplifying (2).Observethat(cid:80) log 1 = log 1 +(cid:80) log 1 andfur-
0≤i<n 1−p(1)βi p(0) 1≤i<n 1−p(1)βi
thermorep(0) ≤ p(1).Sobound(2)becomes
1 (cid:88) 1 1 (cid:88) 1
r(x1:n,ESP) ≤ log + log ≤ log + log ,
p(0) 1−p(1)β p(0) 1−β
i i
1≤i<n 1≤i<n
(cid:0) (cid:1)
ifx isdeterministicandbylog 1 −nH 1 ≤ 0(sincep(1) ≥ 1 andn ≥ 2)
1:n p(1) n 2
1 (cid:88) 1 (cid:0) (cid:1)
r(x1:n,ESP) ≤ log p(0)p(1)β + log 1−p(1)β −nH n1
n−1 i
1≤i<n−1
1 (cid:88) 1
≤ log + log ,
p(0)β 1−β
n−1 i
1≤i<n
ifx isnon-deterministic.Ineithercasebound(7)holds.
1:n
Redundancyof(a,b]. For segment (a,b] we define sequence x(cid:48)1:b−a = xa+1:b and ESP(cid:48) =
(α1(cid:48):∞,p(cid:48)),s.t.ESP(x;x<i) = ESP(x(cid:48);x(cid:48)<i−a)fori ∈ (a,b]. Therefore,letα1(cid:48):∞ = αa+1:∞,
β(cid:48) = α(cid:48) ·...·α(cid:48) andw.l.o.g.p(cid:48)(0) ≤ p(cid:48)(1).Weobtain
i 1 i
(cid:88) 1
log −h(x )
a+1:b
ESP(xi;x<i)
a<i≤b
(cid:88) 1
= log ESP(cid:48)(x(cid:48);x(cid:48) ) −h(x(cid:48)1:b−a) = (cid:96)(x1:n,ESP)−h(x1:n)
1≤i≤b−a i <i
(7) 1 (cid:88) 1 1 (cid:88) 1
≤ log + log ≤ log + log ,
p(cid:48)(0)β(cid:48) 1−β(cid:48) p(0)β 1−β /β
b−a−1 1≤i<b−a i b−1 a<i<b i a
wherethelaststepisduetop(cid:48)(0) ≥ p(0)β (alsop(cid:48)(1) ≥ p(0)β )andβ(cid:48) = β /β .
a a i a+i a
4 Choice of Smoothing Rate Sequence
FixedSmoothingRate. Astraight-forwardchoiceforthesmoothingratesistousethesame
rateαineverystep. Thisleadstoasimpleandfastimplementation,sincenosmoothingrate
sequenceneedstobecomputedorstored.Werequirethefollowinglemmafortheanalysis:
Lemma4.1. For0 < α < 1wehave(cid:80) log 1 ≤ (πloge)2.
1≤i≤m 1−αi 6log 1
α
Proof. Form = 0theboundtriviallyholds,letm ≥ 1.Sincelog 1 isdecreasinginzand
1−αz
integrableforzin[0,∞)wemayboundtheseriesbyanintegral,
(cid:88) 1 (cid:90) m 1 (cid:90) m(cid:88) αjz
log ≤ log dz = log(e) dz. (9)
1−αi 1−αz j
0 0
1≤i≤m j≥1
Theequalityin(9)followsfromtheseriesexpansionln 1 = (cid:80) yj/j,for|y| < 1. To
1−y j≥1
endtheproof,itremainstoboundtheintegralin(9)asfollows(notice(cid:80) j−2 = π2/6):
j≥1
(cid:90) m(cid:88) αjz (cid:88) 1 (cid:90) m loge (cid:88) 1−αjm π2loge
dz = αjzdz = ≤ .
j j log 1 j2 6log 1
0 j≥1 j≥1 0 α j≥1 α
Corollary 4.2. Let S be an arbitrary partition of [1,n]. If α = α = α = ... and
1 2
Assumption2.2holds,then
(cid:88) (cid:20) 1 (πloge)2 1(cid:21)
(cid:96)(x1:n;ESP)− h(xa:b) ≤ |S|· log p(0) + 6log 1 +(n−1)log α . (10)
[a,b]∈S α
Proof. Wehaveβ = αi,thusfori ∈ (a,b]weplugtheestimate
i
(cid:88) 1 (cid:88) 1 Lem.4.1(πloge)2
log = log ≤
1−β /β 1−αi−a 6log 1
a<i<b i a 0<i−a<b−a α
andlogβ = (n−1)logαinto(6)toconcludetheproof.
n−1
−√ π
Choosingα = e 6(n−1) minimizesther.h.s.ofbound(10)andsatisfiesα > 1 (Assump-
2
tion2.2),whenn ≥ 5.Theoptimalchoicegivesredundancyatmost
(cid:20)2πloge √ 1 (cid:21) (cid:20) √ 1 (cid:21)
|S|· √ · n+log < |S|· 3.701· n+log . (11)
6 p(0) p(0)
VaryingSmoothingRate. Itisimpossibletochooseanoptimalfixedsmoothingrate,when
nisunknown. Astandardtechniquetohandlethissituationisthedoublingtrick,whichwill
√ √ √
increasethe n-termin(11)byafactorof 2/( 2−1) ≈ 3.41.However,wecandobetter
√
byslowlyincreasingthesmoothingratestep-by-step,whichonlyleadstoafactor 2 ≈ 1.41.
√
Corollary4.3. LetS beanarbitrarypartitionof[1,n]. Ifα = e−π/ 12(k+1) (i.e.α > 1)
k k 2
andAssumption2.2holds,then
(cid:88) (cid:20) 1 2πloge √ (cid:21)
(cid:96)(x1:n;ESP)− h(xa:b) ≤ |S|· log + √ · n . (12)
p(0) 3
[a,b]∈S
(cid:16) (cid:17)
Proof. Wehaveβ = exp −√π (cid:80) k−1/2 andboundthetermsdependingonthe
i 12 1<k≤i+1
β ’sin(6)fromabove.First,observethat
i
(cid:88) k−1/2 ≤ (cid:90) n √dz ≤ 2√n D=e⇒f.βi log 1 ≤ π√loge√n, (13)
z β 3
1 n−1
1<k≤n
second, for a < i < b we have β /β = α · ... · α ≤ (α )i−a, since i < n and
i a a+1 i n−1
α ,α ,... isincreasing,consequentlyweobtain
1 2
(cid:88) 1 (cid:88) 1 Lem.4.1(πloge)2 πloge√
log ≤ log ≤ = √ n. (14)
1−β /β 1−(α )i−a 6log 1 3
a<i<b i a a<i<b n−1 αn−1
Weplug(13)and(14)into(6),theresultis(12).
Count Smoothing. Consider aging Strategy 2 from Section 1 with smoothing rate λ ∈
(0,1). WewillnowshowthatStrategy2isaninstanceofESP.Fors0,s1 > 0wedefinethe
smoothedcounts(x;x )ofbitxandthesmoothedtotalcountt asfollows
≤k k

λs(x;x )+1, ifk > 0andx = x (cid:40)
 <k k λt +1, ifk > 0
k−1
s(x;x ) := λs(x;x ), ifk > 0andx (cid:54)= x andt := .
≤k <k k k
s +s , ifk = 0
s , ifk = 0 0 1
x
Strategy2predictsp(x;x ) = s(x;x )/t .Incasex = xweget
≤k ≤k k k
λs(x;x )+1 λt s(x;x ) 1 t −1 1
<k k−1 <k k
p(x;x ) = = + = p(x;x )+ ,
≤k <k
t t t t t t
k k k−1 k k k
similarlyp(x;x ) = tk−1p(x;x ), ifx (cid:54)= x. Ifwenowchooseα = tk−1 andp(x) =
≤k t <k k k t
sx theabovesequentiaklprobabilityassignmentruleresembles(1). Thisinskightallowsus
s0+s1
toadoptouranalysismethod.Todoso,werequirethefollowingtechnicalstatementfirst.
Lemma4.4. For1 ≤ a ≤ band0 < λ < 1wehave 1−λa ≥ a.
1−λb b
Proof. Letf(z) := ln((1−λz)/z),itsufficestoprovethatf(a) ≥ f(b).Bylnλz ≥ 1−1/λz
wegetf(cid:48)(z) = [(1−lnλz)·λz −1]/[a(1−λa)] ≤ 0,sof isdecreasing.
Corollary 4.5. Let S be an arbitrary partition of [1,n]. Fix 0 < λ < 1 and m ≥ 1, define
t := λt + 1 for k ≥ 1 and t = 1 + λ + ··· + λm−1 for k = 0. If α = tk−1 and
k k−1 0 k t
k
Assumption2.2holds,then
(cid:88) (cid:20) n (πloge)2 1(cid:21)
(cid:96)(x1:n;ESP)− h(xa:b) ≤ |S|· log p(0) + 6log 1 +(n−1)log λ . (15)
[a,b]∈S λ
Proof. Let k ≥ 1 and note that by t = λt + 1 we may write α = λt /t and
k k−1 k k−1 k
t = 1+λ+···+λk+m−1 = (1−λk+m)/(1−λ)andget
k
λt λt λt t 1−λm
β = α ·...·α = 0 1 ... i−1 = 0 ·λi = ·λi.
i 1 i t t t t 1−λm+i
1 2 i i
Wenowproceedbyboundingthetermsdependentonβ in(6):
i
(1−λm)λiLem.4.4 mλi m≥1 λi β 1−λm+a a≤i
β = ≥ ≥ and i = ·λi−a ≤ λi−a
i 1−λm+i m+i i+1 β 1−λm+i
a
Fromtheaboveinequalitiesweobtain
1 n (cid:88) 1 (cid:88) 1 Lem.4.1(πloge)2
log ≤ log and log ≤ log ≤ .
β λn−1 1−β /β 1−λi−a 6log 1
n−1 a<i<b i a a<i<b λ
Finallyweplugtheaboveinequalitiesinto(6)andrearrengingyields(15).
Fork → ∞wehavet → 1 ,thusα → λ,i.e.weexpectthesmoothedcountsmethodto
k 1−λ k
performsimilartoESPwithfixedsmoothingrateλ,whentheinputislargeenough. Bound
(15) reflects this behavior, it differs from (10) only by the additive term |S|logn. Further-
more,theoptimalvalueofλin(15)matchestheoptimalvalueofαin(10).
5 Experiments
Forinputsoflengthnweexperimentallycheckedthetightnessofourboundsfromtheprevi-
oussectionforawiderangeofESP-instanceswithsmoothingratechoices(i)fixed“optimal”
(cid:112)
smoothing rate α = exp(−π/ 6(n−1)) (here “optimal” means that the corresponding
bound,c.f.Corollary4.2,isminimized),(ii)varyingsmoothingfromCorollary4.3and(iii)
(cid:112)
varyingsmoothingfromCorollary4.5with“optimal”λ = exp(−π/ 6(n−1))andm = 1.
Sinceourboundsfromcorollaries4.2,4.3and4.5areworst-caseboundswecomparethemto
theempiricallymeasured(approximate)worst-caseredundancy. Furthermore,wecompare
the(approximate)worst-caseredundancyof(i),(ii)and(iii)toeachother. Wenowexplain
thedetailsbelow.
Experimental Setup. Inthefollowingletsmoothingratesequenceα ,inputlengthn =
1:∞
1000,partitionS = {(0,200],(200,700],(700,1000]}andε = 0.05befixed.(Weinspected
theoutcomeofourexperimentsfordifferentparametersandgotsimilarresults,hencethese
values.) WewanttojudgeonourboundsonawiderangeofESP-instances,inparticularwe
chooseclassC = {(α1:∞,p) | 0 < ε ≤ p(0),p(1)}ofESP-instances. Todoso,wehaveto
modifyourboundslightly,wemustreplacethetermp(0)byε: Forinstance,inSituation(i),
wemayboundtheredundancyofanyESP ∈ Cofprefixx1:kofgivenx1:nasfollows
(cid:88) (cid:20)2πloge √ 1(cid:21)
(cid:96)(x1:k;ESP)− h(xa:min{k,b}) ≤ |S|· √ · k −1+log , (16)
6 ε
[a,b]∈S
for 1 ≤ k ≤ n. Since the resulting bounds remain worst-case bounds, we compare the
resultingboundsforsituations(i)-(iii)totheworst-caseredundancy
(cid:16) (cid:88) (cid:17)
r(k) := max (cid:96)(x1:k;ESP)− h(xa:min{k,b}) . (17)
ESP∈C,x1:n
[a,b]∈S
Unfortunately, computing the maximum is intractable, since C is uncountably infinite and
there are exponentially many sequences x . To lift this limitation we take the maximum
1:n
overafinitesubsetofESP-instancesfromC andinputsx1:n,specifiedasfollows: Fornum-
bers q0,...,q|S| ∈ {0.05,...,0.95} we consider pairs (ESP,x1:n) s.t. ESP(0;φ) = q0 (q0
determinesanESP-instance)andx1:n isdrawnuniformatrandomfromallsequenceswhere
for the i-th segment [a,b] ∈ S subsequence x has exactly (cid:98)q · (b − a + 1)(cid:99) 1-bits (q
a:b i i
determinesthe(approximate)fractionof1-bitsinthei-thsegment). Wenowtakethemaxi-
mumin(17)overallcombinations(q ,...,q )andrepeattherandomexperiment100times
0 |S|
foreverycombination(q ,...,q )(intotal19|S|+1 ·100simulations). Figure1depictsthe
0 |S|
(cid:80)
approximationofr(k)(solidlines)andourboundson(cid:96)(x≤k;ESP)− [a,b]∈S h(xa:min{b,k})
(dashedlines).(Forinstance,bound(16)isdepictedasdashedlineintheleftplotofFigure1.)
ApproximateWorst-CaseRedundancy. Wenowcompare(approximate)r(k)forsmooth-
ingratechoices(i)-(iii)andobserve: Ononehand,aslongaskissmall,varyingsmoothing
rates,(ii)and(iii),yieldlowerredundancythan(i),and(iii)performsbetterthan(ii). Onthe
otherhand,whenkislarge(i),(ii)and(iii)don’tdiffertoomuch.Theincreaseinredundancy
atk = 201andk = 701isnearlyidenticalinallcases,thedifferenceinredundancyisalmost
entirelycausedbysegment(0,200].
Bounds Behavior. Now we compare the bounds to (approximate) r(k). In general, the
tightness of our bounds decreases as the number of segments increases. This is plausible,
500 500 500
400 400 400
300 300 300
200 200 200
100 100 100
0 0 0
0 200 400 600 800 1,000 0 200 400 600 800 1,000 0 200 400 600 800 1,000
Figure 1: Redundancy bound (dashed line) and approximate worst-case redundancy r(k)
(solid line) of class {(α ,p) | 0 < ε ≤ p(0),p(1)} for ε = 0.05 w.r.t. competitor with
1:∞
partition S = {[1,200],(200,700],(700,1000]} on the length-k prefix, 1 ≤ k ≤ n of a
sequencewithlengthn = 1000. Thex-axisisprefixlengthk andthey-axisisredundancyin
bit. Everyplotcorrespondstoadifferentsmoothingratechoice: (i)fixed“optimal”smoothing
(cid:112)
rate α = exp(−π/ 6(n−1)), (ii) varying smoothing from Corollary 4.3 and (iii) varying
(cid:112)
smoothingfromCorollary4.5with“optimal”λ = exp(−π/ 6(n−1))andm = 1.
since we essentially concatenated the worst-case bound for |S| = 1. However, we don’t
know,whetherornottheworst-caseredundancyfor|S| = 1canappearinmultipleadjacent
segmentsatthesametime. Experimentsindicatethatthismaynotbethecase. Furthermore,
in(i)theboundistightest,especiallywithinsegment(0,200].Incases(ii)and(iii)thebounds
aremoreloose. Anexplanationis, thatinthecorrespondingproofsweworkedwithrather
(cid:80)
generoussimplifications,e.g.whenbounding− log(1−β /β )fromabove. Ifwe
a<i<b i a
compare(ii)to(i)andto(iii)wecansee,thatbound(ii)istighterforverysmallk.Thereason
issimple:Bound(ii)doesnotdependonasmoothingrateparameter,whereas(i)containsthe
term1/log 1 and(iii)containstheterm1/log 1. Thesetermsdominatethebounds,whenk
α λ
issmallandαandλarecloseto1. (Wehaveα = λ ≈ 0.96,sinceαandλwerechosento
minimizethecorrespondingboundforn = 1000.)
6 Conclusion
Inthisworkweanalyzedaclassofpracticalandadaptiveelementarymodelswhichassign
probabilities by exponential smoothing, ESP. Our analysis is valid for a binary alphabet.
Bychoosingsmoothingratesappropriatelyourstrategygeneralizescountsmoothing(Strat-
egy2)andprobabilitysmoothingfromPAQ(Strategy3).Duetoitslowmemoryfootprintand
linearper-sequencetimecomplexityESPisattractivefromapracticalpointofview. Froma
theoreticpointofview√ESPisattractiveaswell:ForvarioussmoothingratesequencesESPhas
redundancyonlyO(s n)aboveanyPWSwithssegments,animprovementoverprevious
approaches.Ashortexperimentalstudysupportsourbounds.
Nevertheless, experiments indicate that there is room for an improved analysis. Despite
minortechnicalissuesamajorapproachwouldbetoobtainredundancyboundsw.r.t.PWS
thattakethesimilarityofadjacentsegmentsintoaccount. Thatis,ifadjacentsegmentshave
verysimilardistributions,theincreaseinredundancyshouldbesmall,comparedtoadjacent
segments with drastically different distributions. Furthermore, it is desirable to generalize
theanalysistoanon-binaryalphabet.Wedeferathoroughexperimentalstudythatcompares
ESPtootheradaptiveelementarymodelstofutureresearch.