Joint universal lossy coding and identification of stationary mixing sources with general alphabets

0.3 MB·English

by  Maxim Raginsky

Joint Universal Lossy Coding and
Identification of Stationary Mixing
Sources with General Alphabets
Maxim Raginsky, Member, IEEE
Abstract—We consider the problem of joint universal nonparametricsettings(e.g.,theclass ofallstationaryergodic
9 variable-rate lossy coding and identification for parametric discrete-alphabet sources), universal schemes such as Ziv–
0 classes of stationary β-mixing sources with general (Polish) Lempel [4] amount to constructing a probabilistic model for
0 alphabets. Compression performance is measured in terms
2 of Lagrangians, while identification performance is measured the source. In the reverse direction, Kieffer [5] and Merhav
by the variational distance between the true source and the [6], among others, have addressed the problem of statistical
n
estimated source. Provided that the sources are mixing at modeling (parameter estimation or model identification) via
a
J a sufficiently fast rate and satisfy certain smoothness and universal lossless coding.
3 Vapnik–Chervonenkis learnability conditions, it is shown that, Once we consider lossy coding, though, the relationship
for bounded metric distortions, there exist universal schemes
1 between coding and modeling is no longer so simple. On
for joint lossy compression and identification whose Lagrangian
the one hand, having full knowledge of the source statis-
] redundancies converge to zero as pVnlogn/n as the block
T length n tendsto infinity,where Vn istheVapnik–Chervonenkis tics is certainly helpful for designing optimal rate-distortion
dimensionofacertainclassofdecisionregionsdefinedbythen- codebooks. On the other hand, apart from some special cases
I
s. dimensional marginal distributions of the sources; furthermore, (e.g., for i.i.d. Bernoulli sources and the Hamming distortion
c for each n, the decoder can identify n-dimensional marginal measure or for i.i.d. Gaussian sources and the squared-error
[ of the active source up to a ball of radius O(pVnlogn/n) in
variational distance,eventuallywith probabilityone. Theresults distortion measure), it is not at all clear how to extract a

1 are supplemented by several examples of parametric sources reliable statistical model of the source from its reproduction
v satisfying the regularity conditions. via a rate-distortion code (although, as shown recently by
4
Weissman and Ordentlich [7], the joint empirical distribution
0 Index Terms—Learning, minimum-distance density estimation,
of the source realization and the corresponding codeword of
9 two-stage codes, universal vector quantization, Vapnik–
1 Chervonenkis dimension. a “good” rate-distortion code converges to the distribution
. solving the rate-distortion problem for the source). This is
1
not a problem when the emphasis is on compression, but
0
9 I. INTRODUCTION there are situations in which one would like to compress
0 It is well known that lossless source coding and statis- the source and identify its statistics at the same time. For
: tical modeling are complementary objectives. This fact is instance, in indirect adaptive control (see, e.g., Chapter 7 of
v
i captured by the Kraft inequality (see Section 5.2 in Cover Tao [8]) the parameters of the plant (the controlled system)
X
and Thomas [1]), which provides a correspondence between are estimated on the basis of observation, and the controller
r uniquely decodable codes and probability distributions on a is modified accordingly. Consider the discrete-time stochastic
a
discrete alphabet. If one has full knowledge of the source setting, in which the plant state sequence is a randomprocess
statistics, thenonecandesignan optimallosslesscodeforthe whose statistics are governed by a finite set of parameters.
source,andviceversa.However,inpracticeitisunreasonable Suppose that the controller is geographically separated from
to expectthatthe sourcestatistics are knownprecisely,soone the plant and connected to it via a noiseless digital channel
has to design universal schemes that perform asymptotically whosecapacityisRbitsperuse.Then,giventhetimehorizon
optimallywithin a givenclass ofsources.Inuniversalcoding, T, the objective is to design an encoder and a decoder for
too,asRissanenhasshownin[2],[3],thecodingandmodeling the controller to obtain reliable estimates of both the plant
objectives can be accomplished jointly: given a sufficiently parametersandtheplantstatesequencefromthe2TR possible
regularparametricfamilyofdiscrete-alphabetsources,theen- outputs of the decoder.
codercanacquirethesourcestatisticsviamaximum-likelihood To state the problem in general terms, consider an infor-
estimation on a sufficiently long data sequence and use this mation source emitting a sequence X = Xi i Z of random
{ }∈
knowledge to select an appropriate coding scheme. Even in variables taking values in an alphabet . Suppose that the
processdistributionof X is notspecifiedXcompletely,butit is
Thematerial inthis paperwaspresented inpartattheIEEEInternational
knowntobeamemberofsomeparametricclass P :θ Λ .
SymposiumonInformationTheory,Nice,France,June2007.Thisworkwas { θ ∈ }
supportedbytheBeckmanInstitute Fellowship. We wish to answer the following two questions:
M. Raginsky was with the Beckman Institute for Advanced Science and 1) Is the class P : θ Λ universally encodable with
θ
Technology, University of Illinois, Urbana, IL 61801 USA. He is now with { ∈ }
respect to a given single-letter distortion measure ρ, by
the Department of Electrical and Computer Engineering, Duke University,
Durham,NC27708USA(e-mail: [email protected]). codes with a given structure (e.g., all fixed-rate block
2
codeswith a givenper-letterrate, allvariable-rateblock dimensional marginals of the source process distributions.
codes, etc.)? In other words, does there exist a scheme This result shows very clearly that the price to be paid for
that is asymptotically optimal for each P , θ Λ? universality, in terms of both compression and identification,
θ
∈
2) If the answer to Question 1) is positive, can the codes grows with the richness of the underlying model class, as
be constructed in such a way that the decoder can not captured by the VC dimension sequence V . The richer
n
{ }
only reconstructthe source, but also identifyits process the model class, the harder it is to learn, which affects the
distribution P , in an asymptotically optimal fashion? compression performance of our scheme because we use the
θ
source parameters learned from past data to decide how to
In previous work [9], [10], we have addressed these two
encode the current block. Furthermore, comparing the rate at
questions in the context of fixed-rate lossy block coding
which the Lagrangian redundancy decays to zero under our
of stationary memoryless (i.i.d.) continuous-alphabet sources
scheme with the O(logn/n) result of Chou, Effros and Gray
with parameter space Λ a bounded subset of Rk for some
[14],whoseuniversalschemeisnotaimedatidentification,we
finite k. We have shown that, under appropriate regularity
immediatelyseethat,inensuringtosatisfythetwinobjectives
conditionsonthedistortionmeasureandonthesourcemodels,
of compression and modeling, we inevitably sacrifice some
thereexistjointuniversalschemesforlossycodingandsource
compression performance.
identification whose redundancies (that is, the gap between
The paper is organized as follows. Section II introduces
the actual performance and the theoretical optimum given by
notation and basic concepts related to sources, codes and
the Shannon distortion-rate function) and source estimation
Vapnik–Chervonenkis classes. Section III lists and discusses
fidelity both converge to zero as O logn/n , as the block
theregularityconditionsthathavetobesatisfiedbythesource
length n tends to infinity. The code operates by coding
(cid:0)p (cid:1) model class, and contains the statement of our result. The
each block with the code matched to the source with the
result is proved in Section IV. Next, in Section V we give
parameters estimated from the preceding block. Comparing
three examples of parametric source families (namely, i.i.d.
thisconvergenceratetothelogn/nconvergencerate,whichis
Gaussiansources,Gaussianautoregressivesourcesandhidden
optimalfor redundanciesof fixed-rate lossy block codes [11],
Markovprocesses)whichfittheframeworkofthispaperunder
we see that there is, in general, a price to be paid for doing
suitable regularity conditions.We conclude in Section VI and
compression and identification simultaneously. Furthermore,
outline directions for future research. Finally, the Appendix
the constant hidden in the O() notation increases with the
containssome technicalresultson Lagrange-optimalvariable-
·
“richness” of the model class P : θ Λ , as measured by
θ rate quantizers.
{ ∈ }
the Vapnik–Chervonenkis (VC) dimension [12] of a certain
class of measurable subsets of the source alphabet associated
with the sources. II. PRELIMINARIES
The main limitation of the results of [9], [10] is the i.i.d. A. Sources
assumption, which is rather restrictive as it excludes many
In this paper, a source is a discrete-time stationary ergodic
practicallyrelevantmodelclasses(e.g.,autoregressivesources, random process X = Xi i Z with alphabet . We assume
or Markov and hidden Markov processes). Furthermore, the that is a Polish spa{ce }(i.∈e., a complete seXparable metric
assumption that the parameter space Λ is bounded may not spaceX1) and equip with its Borel σ-field. For any pair of
always hold, at least in the sense that we may not know the indices i,j Z wXith i < j, let Xj denote the segment
diameter of Λ a priori. In this paper we relax both of these (X ,X ,..∈.,X ) of X. If P is thie process distribution
i i+1 j
assumptions and study the existence and the performance of of X, then we let E denote expectation with respect
P
universal schemes for joint lossy coding and identification to P, and let Pn deno{te·}the marginal distribution of Xn.
1
of stationary sources satisfying a mixing condition, when
Whenever P carries a subscript, e.g., P = P , we write
θ
the sources are assumed to belong to a parametric model E instead. We assume that there exists a fixed σ-finite
class Pθ : θ Λ , Λ being an open subset of Rk for mθe{a·s}ure µ on , such that the n-dimensional marginal of
{ ∈ }
some finite k. Because the parameter space is not bounded, X
any process distribution of interest is absolutely continuous
we have to use variable-rate codes with countably infinite with respect to the product measure µn, for all n 1. We
codebooks, and the performance of the code is assessed by denote the correspondingdensities dPn/dµn by pn.≥To avoid
a composite Lagrangian functional [13] which captures the notational clutter, we omit the superscript n from µn, Pn
trade-off between the expected distortion and the expected and pn wheneverit is clear from the argument,as in dµ(xn),
rate of the code. Our result is that, under certain regularity dP(xn) or p(xn).
conditions on the distortion measure and on the model class,
GiventwoprobabilitymeasuresP,Qonameasurablespace
thereexistuniversalschemesforjointlossysourcecodingand
( , ), the variational distance between them is defined by
identificationsuchthat,asthe blocklengthntendsto infinity, Z A
△
the gap between the actual Lagrangian performance and the d(P,Q)= sup P(A ) Q(A ),
i i
| − |
optimal Lagrangian performance achievable by variable-rate {Ai}⊆AXi
codes at that block length, as well as the source estimation wherethesupremumisoverallfinite -measurablepartitions
fidelity at the decoder, converge to zero as O Vnlogn/n , of (see, e.g., Section 5.2 of GrayA[15]). If p and q are
where V is the VC dimension of a certain class of decision Z
n (cid:0)p (cid:1)
regions induced by the collection Pn : θ Λ of the n- 1Thecanonical exampleistheEuclidean spaceRd forsomed<∞.
{ θ ∈ }
3
the densities of P and Q, respectively, with respect to a where ℓ(s) denotes the length of s in bits. The mapping of
dominating measure ν, then we can write the source X into the reproduction process X is defined by
d(P,Q)= |p(z)−q(z)|dν(z). Xnnk(k++11) =ϕ f Xnnk(k++11),Xnnkk−m+1 c, k∈Z.
ZZ That is, the encod(cid:16)ing(cid:0)is done in blocks o(cid:1)f(cid:17)length n, but the
A useful property of the variational distance is that, for any b
measurablefunctionf : [0,1], E f E f d(P,Q). encoderisalsoallowedtoobservethemsymbolsimmediately
Z → | P − Q |≤ precedingeachblock.TheeffectivememoryofCn,misdefined
When P and Q are n-dimensional marginals of Pθ and Pθ′,
respectively,i.e.,P =Pθn andQ=Pθn′,wewritedn(θ,θ′)for as the set M⊆{1,...,m}, such that
ddi(sPtaθnn,cPeθnd′)(.PI,fQA;′is)abσe-tswubefieneldPoafnAd,Qwewdiethfinreestpheecvtatroiationbayl f(xm)=f(xm), ∀xm,xm ∈Xm :xi =xi,∀i∈M.
′ ′
A A The size of is called the effective memory length of
d(P,Q; ′)=△ sup P(Ai) Q(Ai), Cn,m. We|Msh|aell oMften use Cn,em to also denote tehe composite
wherethesupermAumis{oAvie}r⊆aAll′fiXniit|e -me−asurable|partitions hmaaspzpeirnogmϕe◦mfor:yX(m1n ==0C),nw,me(Xsh1anl,lXde−0nmo+te1)it.mWohreencothmepaccotdlye
′
of . Given a δ > 0 and a probabiliAty measure P, the vari- by Cn. b
Z
ational ball of radius δ around P is the set of all probability The performance of the code on the source with process
measures Q with d(P,Q) δ. distribution P is measured by its expected distortion
≤
GivenasourceX withprocessdistributionP,letP0 and
D (Cn,m)=△ E ρ (Xn,Xn) ,
P denote the marginal distributions of P on X −∞ and P P n 1 1
{reX1g∞ui}lai≥ri1ty, rceosepfeficctiievnelty(.oFroβr-emaicxhinkg≥co1e,ffitchieenktt)ho-ofr{Pderii}said≤bes0ofilnuetde nwh1ere nforρ(xxn,x∈) iXsntheanpdernx-lnette∈r diXsbtonr,tiooρnn(xinnc,uxrrne)d =i△n
as [16], [17]: − i=1 i i
reproducing xn by xn, and by ibts expectebd rate b
P
βP(k)=△ sup |P(Ai∩Bj)−P−0∞(Ai)P1∞(Bj)|, RP(Cnb,m)b=△ EP ℓn f X1n,X−0m+1 ,
Xi Xj  where ℓn(s) denotes the le(cid:8)ngth(cid:0) o(cid:0)f a binary str(cid:1)in(cid:1)(cid:9)g s in bits,
where the supremum is over all finite σ(X0 )-measurable normalized by n. (We follow Neuhoff and Gilbert [18] and
 
partitions A and all finite σ(X )-measu−r∞able partitions normalize the distortion and the rate by the length n of the
B . Obs{ervie}that k∞ reproduction block, not by the combined length n + m of
j
{ } the source blockplusthe memoryinput.)When workingwith
βP(k)=d P,P−0∞×P1∞;σ(X−0∞,Xk∞) , (1) variable-rate quantizers, it is convenient [13], [19] to absorb
the distortion and the rate into a single performancemeasure,
thevariationaldistan(cid:0)cebetweenP andtheproductd(cid:1)istribution
the Lagrangian distortion
P0 P withrespecttotheσ-algebraσ(X0 ,X ).Since
X−∞is×stat1∞ionary, we can “split” its process di−st∞ributik∞on at any L (Cn,m,λ)=△ D (Cn,m)+λR (Cn,m),
P P P
point l Z and define β (k) equivalently by
P
∈ where λ > 0 is the Lagrange multiplier which controls the
βP(k)=△ d P,P−l∞×Pl∞+1;σ(X−l∞,Xl∞+k) . (2) distortion-rate trade-off. Geometrically, LP(Cn,m) is the y-
interceptof the line with slope λ, passing through the point
Again, if P is subs(cid:0)cripted by some θ, P =Pθ, then(cid:1)we write (R (Cn,m),D (Cn,m)) in the−rate-distortion plane [20]. If
β (k). P P
θ P carries a subscript, P = P , then we write D (), R ()
θ θ θ
· ·
and L ().
θ
·
B. Codes
The class of codes we consider here is the collection of C. Vapnik–Chervonenkis classes
all finite-memory variable-rate vector quantizers. Let be a In this paper, we make heavy use of Vapnik–Chervonenkis
X
reproductionalphabet,also assumed to be Polish. We assume theory (see Devroye, Gyo¨rfi and Lugosi [21], Vapnik [22],
that is a subset of a Polish metric space bwith a DevroyeandLugosi[23]orVidyasagar[24]fordetailedtreat-
X ∪ X Y
bounded metric ρ0(, ): there exists some ρmax < + , such ments). This section contains a brief summary of the needed
· · ∞
that ρ0(y,yb′) ρmax for all y,y′ . We take ρ: concepts and results. Let ( , ) be a measurable space. For
[0,ρmax], ρ(x≤,x) =△ ρ0(x,x), as o∈urY(single-letter)Xdis×toXrti→on any collection of mZeasAurable subsets of and any n-
C ⊆A Z
function. A variable-rate vector quantizer with block lebngth tuple zn n, define the set (zn) 0,1 n consisting of
fn a:ndnmemomryblengthismthbies eancpoadirerC, nϕ,m: = (f,ϕ)n, wishtehree aAlldist.inTc∈htebnZinarystringsof theCform(⊆1{{z1∈A}},...,1{zn∈A}),
sdtercinogXdsers,a×taisnfXdyiSng⊆→th{e0Sp,r1e}fi∗xicsonadcitoiuonntoarb,leeqcuoivlSlaelce→tniotlnyX,bothfebKinraarfyt ∈C Sn(C)=△ zmn∈aZxn|C(zn)|
inequality is called the nth shatter coefficient of . The Vapnik–
C
2 ℓ(s) 1, Chervonenkis dimension (or VC-dimension) of , denoted by
− ≤ V( ), is defined as the largest n for which SC( ) = 2n (if
Xs∈S C n C
4
S ( ) = 2n for all n = 1,2,..., then we set V( ) = ). If interior. We wish to design a sequence of variable-rate vector
n
V( C)< , then is called a Vapnik–ChervonenCkis cla∞ss (or quantizers, such that the decoder can reliably reconstruct the
VCCclass∞). If isCa VC class with V( ) 2, then it follows original source sequence X and reliably identify the active
C C ≥
from the results of Vapnik and Chervonenkis [12] and Sauer sourcein an asymptoticallyoptimalmannerfor allθ Λ. We
[25] that S ( ) nV( ). begin by listing the regularity conditions. ∈
n C
C ≤
For a VC class , the so-called Vapnik–Chervonenkis in-
C
equalities (see Lemma 2.1 below) relate its VC dimension
Condition 1. The sources in P :θ Λ are algebraically
V( ) to maximal deviations of the probabilities of the events { θ ∈ }
β-mixing: there exists a constant r>0, such that
C
in from their relative frequencies with respect to an i.i.d.
samCple of size n. For any zn n, let βθ(k)=O(k−r), θ Λ
∈Z ∀ ∈
1 n where the constant implicit in the O() notation may depend
Pzn = n δzi on θ. ·
i=1
X
denote the induced empirical distribution, where δ is the
zi This condition ensuresthat certain finite-block functionsof
Dirac measure (point mass) concentrated at zi. We then have the source X can be approximated in distribution by i.i.d.
the following:
processes, so that we can invoke the Vapnik–Chervonenkis
Lemma 2.1 (Vapnik–Chervonenkisinequalities): Let P be
machinery of Section II-C. This “blocking” technique, which
a probability measure on ( , ), and Zn = (Z ,...,Z )
Z A 1 1 n we exploit in the proof of our Theorem 3.1, dates back to
an n-tuple of independent random variables with Z P,
i ∼ Bernstein [28], and was used by Yu [29] to derive rates
1 i n. Let be a Vapnik–Chervonenkis class with
V(≤) 2≤. Then forCevery δ >0, of convergence in the uniform laws of large numbers for
C ≥ stationarymixingprocesses,andbyMeir[30]inthecontextof
nonparametricadaptivepredictionofstationarytimeseries.As
P sup|PZ1n(A)−P(A)|>δ ≤8nV(C)e−nδ2/32 (3) an example of when an even stronger decay condition holds,
and (cid:26)A∈C (cid:27) let X = {Xi}i∈Z be a finite-order autoregressive moving-
average(ARMA) processdrivenby a zero-meani.i.d. process
V( )logn Y = Y ,i.e., thereexistpoisitiveintegersp,q andp+q+1
E(cid:26)Asu∈pC|PZ1n(A)−P(A)|(cid:27)≤cr Cn , (4) real c{onsi}tants a0,a1...,ap,b1,...,bq such that
where c > 0 is a universal constant. The probabilities and p q
expectations are with respect to the product measure Pn on aiXn i = bjYn j, n Z.
− − ∈
( n, n). Xi=0 Xj=1
Z A
Remark 2.1: A more refined technique involving metric Mokkadem [31] has shown that, providedthe common distri-
entropiesandempiricalcoveringnumbers,duetoDudley[26], bution of the Y is absolutely continuous and the roots of the
i
canyieldamuchbetterO(1/√n)boundontheexpectedmax- polynomial A(z) = p a zi lie outside the unit circle in
i=0 i
imaldeviationbetweenthetrueandtheempiricalprobabilities. the complex plane, the β-mixing coefficients of X decay to
P
This improvement,however, comes at the expense of a much zero exponentially.
larger constant hidden in the O() notation.
·
Finally, we shall need the following lemma, which is a
Condition 2. For each θ Λ, there exist constants δ > 0
θ
simple corollary of the results of Karpinski and Macintyre ∈
and c >0, such that
θ
[27] (see also Section 10.3.5 of Vidyasagar [24]):
meLaesmurmabale2.s2u:bsLeetst oCf=Rd{,Asuξch: tξha∈t Rk} be a collection of sunpdn√(θn,θ′) ≤cθkθ−θ′k
A = z Rd :Π(z,ξ)>0 , ξ Rk for all θ′ in the open ball of radius δθ centered at θ, where
ξ
{ ∈ } ∈ is the Euclidean norm on Λ.
where for each z Rd, Π(z, ) is a polynomial of degree s k·k
in the components∈of ξ. Then· is a VC class with V( )
C C ≤ This condition guarantees that, for any sequence δn n N
2klog(4es). { } ∈
of positive reals such that
δ 0,√nδ 0, as n
III. STATEMENTOF RESULTS n → n → →∞
In this section we state our result concerning universal and any sequence θn n N in Λ satisfying θn θ <δn for
{ } ∈ k − k
schemes for joint lossy compression and identification of a given θ Λ, we have
∈
stationarysourcesundercertainregularityconditions.Wework
d (θ,θ ) 0, as n .
in the usual setting of universal source coding: we are given n n → →∞
a source X = Xi i Z whose process distribution is known Itisweaker(i.e.,moregeneral)thantheconditionsofRissanen
{ }∈
to be a member of some parametric class P : θ Λ . The [2],[3] which controlthe behaviorof the relative entropy(in-
θ
{ ∈ }
parameter space Λ is an open subset of the Euclidean space formationdivergence)asafunctionofthesourceparametersin
Rk for some finite k, and we assume that Λ has nonempty termsof theFisher informationandrelatedquantities.Indeed,
5
for each n let is passed by the null hypothesis H under the likelihood-
0
be thDen(nPoθrmkPalθi′z)ed≡=nth-nn11orEZdXθern(cid:26)plrneθl(adxdtPPinvθ)θe′l(nXenpp1ntθθr′)(o((cid:27)xpxnyn))(diµnf(oxrnm)ation rCZcicnalla1aantdilsseols=sptiiwdfiefi(neeeoZcdrrsise1sκen,iytoA.tsilne.yAl.:dr,f,XurZAioldenmm′e.)n∈→N,PtioAaθcnwn{a.ndl0,T,ecswo1qlua}urpesiiptsavdeiocafieshAlficeeanAn∼ttetihdow∈Zant1nbitAZyphAan1rκt′,et,Awe.sirp.(fen.exs,tccnh:Zta)etnmio=r△ataahs1srsees{oosxcdacnimrai∈aatAeptwel}dnae.
divergence) between Pθ and Pθ′. Suppose that, for each θ,
Dn(Pθ Pθ′)is twice continuouslydifferentiableasa function κA(Z1),...,κA(Zm) = κA′(Z1),...,κA′(Zm) .
k
of θ . Let θ lie in an open ball of radius δ around θ. Since
′ ′ It is(cid:0)easy to see that Zn(cid:1)is a(cid:0)n equivalence relation.(cid:1)From
D(Pθ Pθ′) attains its minimum at θ′ = θ, the gradient the definitionsof the sh∼att1er coefficients S ( ) and the VC
k m n
θ′Dn(Pθ Pθ′) evaluatedatθ′ =θ iszero,andwe can write dimensionV( )(cf.SectionII-C),weseethAatthecardinality
∇ k n
the second-order Taylor expansion of Dn about θ as of the quotienAt set n/ Zn is equal to 2m for all sample
1 sizes m V( ),Awhere∼as1for m > V( ), it is bounded
Dn(PθkPθ′)= 2(θ−θ′)TJn(θ)(θ−θ′)+o(kθ−θ′k2), (5) from abo≤ve byAmnV(An), which is strictly lAesns than 2m. Thus,
where the Hessian matrix the fact that the Yatracos class n has finite VC dimension
A
implies that the problem of estimating the density pn from a
∂2 large i.i.d. sample reduces, in a sense, to a finite nuθmber (in
Jn(θ)="∂θi′∂θj′Dn(PθkPθ′)(cid:12)θ′=θ#i,j=1,...,k, fact, polynomial in the sample size m, for m > V(An)) of
(cid:12) simple hypothesis tests of the type (6). Our Condition 1 will
under additional regularity conditions(cid:12), is equal to the Fisher
then allow us to transfer this intuition to (weakly) dependent
information matrix
samples.
1 ∂2
I (θ)= E lnp (Xn) Now that we have listed the regularityconditionsthat must
n (cid:20)−n θ(cid:26)∂θi∂θj θ 1 (cid:27)(cid:21)i,j=1,...,k hold for the sources in {Pθ : θ ∈ Λ}, we can state our main
result.
(seeClarkeandBarron[32]).Assumenow,followingRissanen
[b2o]u,n[d3e]d, (tbhyatatchoenssteaqnutednecpeenodfinmgaotrnixθ)n.oTrhmesn w{keInca(nθ)wk}riteis souTrhceeosresmati3sf.1y:ingLeCt o{nPdθiti:oθns∈1–Λ3}. bTeheanpfaorrameevterriyc cλlas>s o0f
Dn(PθkPθ′) ≤ 21(kIn(θ)k+o(1))·kθ−θ′k2 avnna(rdnia+ebvleen-r(ry2a+tηeη)v/>rec+t0o,r1t)hqeuaranendteiezxfeifsretssctwiavietshemqmeumeenmocroeyry{leClne∗ngn,tmghtnhn}2nm,∈nNsuco≤hf
≤ c′θkθ−θ′k2, that, for all θ Λ,
∈
i.e., the normalized relative entropies Dn(PθkPθ′) are lo- Lθ(Cn,mn,λ) inf inf Lθ(Cn,m,λ)
cally quadratic in θ′. Then Pinsker’s inequality (see, e.g., ∗ −m≥0Cn,m
Lemma 5.2.8 of Gray [15]) implies that 2Dn(Pθ Pθ′) V( )logn
d (θ,θ )/√n, and we recover our Condition 2. Rikssanen≥’s =O An , (7)
connditio′n, while stronger than our Conditpion 2, is easier to r n !
check,thefactwhichweexploitinourdiscussionofexamples where the constants implicit in the O() notation depend on
·
of Section V. θ. Furthermore, for each n, the binary description produced
Condition3.Foreachn, let be thecollectionofallsets by the encoder is such that the decoder can identify the n-
n
A dimensional marginal of the active source up to a variational
of the form
ball of radius O V( )logn/n with probability one.
n
Aθ,θ′ = xn n :pθ(xn)>pθ′(xn) , θ =θ′. A
∈X 6 (cid:0)p (cid:1)
Then we req(cid:8)uire that, for each n, is(cid:9)a VC class, and What (7) says is that, for each block length n and each
V( )=o(n/logn). An θ Λ, the code Cn,mn, which is independent of θ, performs
An alm∈ost as well as t∗he best finite-memoryquantizerwith block
This condition is satisfied, for example, when V( ) =
V < independently of n, or when V( ) = logAnn. The length n that can be designed with full a priori knowledgeof
∞ An the n-dimensional marginal Pn. Thus, as far as compression
use of the class dates back to the work of Yatracos θ
An goes,ourschemecancompetewithallfinite-memoryvariable-
[33] on minimum-distance density estimation. The ideas of
rate lossy block codes (vector quantizers), with the additional
Yatracoswere furtherdevelopedby Devroyeand Lugosi[34],
bonusof allowing the decoder to identify the active source in
[35], who dubbed the Yatracos class (associated with
An an asymptotically optimal manner.
the densities pn). We shall adhere to this terminology. To
θ It is not hard to see that the double infimum in (7) is
give an intuitive interpretation to , let us consider a pair
n
A achieved already in the zero-memory case, m=0. Indeed, it
θ,θ Λ of distinct parameter vectors and note that the set
′
∈ is immediate that having nonzero memory can only improve
xn : pθ(xn) > pθ′(xn) consists of all xn for which the
{ } the Lagrangian performance, i.e.,
simple hypothesis test
H0 :X1n ∼Pθn versus H1 :X1n ∼Pθn′ (6) min≥f0Cinn,fmLθ(Cn,m,λ)≤iCnnfLθ(Cn,λ), ∀θ ∈Λ.
6
On the other hand, given any code Cn,m = (f,ϕ), we can A sequence δ of positive reals is given, such that
n
• { }
construct a zero-memory code Cn = (f ,ϕ ), such that
0 0 0 δ 0,√nδ 0, as n
Lθ(C0n,λ) ≤ Lθ(Cn,m,λ) for all θ ∈ Λ. To see this, define n → n → →∞
for each xn n the set (we shall specify the sequence δ later in the proof).
S(xn)=△ {s∈∈X{0,1}∗ :s=f(xn,zm) for some zm ∈Xm}, • mFoermeoarcyhnn-b∈locNk acnoddeeaCchnθ=∈{(Λnf},nt,hϕenre) etxhiasttsacahzieevroes-
θ θ θ
and let (or comes arbitrarily close to) the nth-order Lagrangian
optimum for P : L (Cn,λ)=Ln(λ).
f (xn)=argmin ρ (xn,ϕ(s))+λℓ(s) , xn n θ θ θ θ
0 n
∀ ∈X
s (xn) Fix the block length n. Because the source is stationary, it
and ϕ0 ϕ∈.SThen(cid:0), given any (xn,zm)(cid:1) n m, let sufficestodescribethemappingofXb1nintoX1n.Theencoding
s=f(xn≡,zm). We then have ∈ X × X is done as follows:
b
ρn(xn,C0n(xn))+λℓ(f0(xn)) 1) aTshePenn,cwodheerreesθti=maθte(XsP0θn0 from). themn-blockX−0mn+1
=ρn(xn,ϕ(f0(xn))+λℓ(f0(xn)) 2) Theθeencoder then comp−umtens+t1he waiting time
ρ (xn,ϕ(s))+ℓ(s)
n e e
≤=ρn(xn,f(xn,zm))+ℓ(f(xn,zm)). Tn =△ inf i≥1:dn θ(i),θ(X−0mn+1) ≤√nδn ,
TfoarkianllgθexpΛec,tawtihoincsh,pwroevseeseththatat Lθ(C0n,λ) ≤ Lθ(Cn,m,λ) wemitphtythseetstinasnedqaurdalctoonv(cid:0)+enti.onTehtahtatist,htehein(cid:1)efinmcoudmerolfooothkes
∈ throughthe database c an∞d findsthe first θ(i), such that
infLθ(Cn,λ) inf inf Lθ(Cn,m,λ), θ Λ. the n-dimensionaldistributionPn is in the variational
Cn ≤m 0Cn,m ∀ ∈ θ(i)
≥ ball of radius √nδ around Pn.
TheinfimumofLθ(Cn,λ)overallzero-memoryvariable-rate n θe
quantizersCnwithblocklengthnistheoperationalnth-order 3) If Tn < + , the encoder sets θ =θ(i); otherwise, the
∞
distortion-rate Lagrangian Ln(λ) [20]. Because each P is encoder sets θ = θd, where θd Λ is some default
θ θ ∈
ergodic, Ln(λ) converges to the distortion-rate Lagrangian parameter vector, say, θ(1). b
θ 4) The binary debscription of Xn is a concatenation of the
b 1
△
L (λ)=min D (R)+λR , following three binary strings: (i) a 1-bit flag b to tell
b θ θ
R
whether T is finite (b = 0) or infinite (b = 1); (ii) a
n
where Dθ(R) is the Shannon(cid:0) distortion-rat(cid:1)e function of Pθ binary string s1 which is equal to s(Tn) if Tn < +
([s1e4e]).LTemhums,aou2risnchtehmeeAipspuennidveixrsatolnCohtooun,lyEifnfrtohseanntdh-oGrrdaeyr Torhetostarninegmsp=ty bstsrinigsicfaTllend=th+e∞fir;st(-isitia)gse2d=esfcrθbi(pXti1on∞n).,
1
sense of (7), but also in the distortion-rate sense, i.e., while s is called the second-stage description.
2
foreveryLθθ(CΛ∗n,.mTnh,uλs),−inLthθe(λte)rm→in0o,logyoafs[n14→],o∞urscheme tThheerdeepcrooddeurcrteiocenievXes1nbs1=s2,ϕdθbe(tse2rm).inNesotθefrtohmatsw,ahnednprbod=uce0s
is weakly m∈inimax universal for P :θ Λ . (which, as we shall show, will happben eveentually almost
{ θ ∈ } surely), Pn lies inbthe variationalball of radius√nδ around
b n
θ
the estimated source Pn. If the latter is a good estimate
e
IV. PROOF OF THEOREM3.1 of Pn, i.e., d (θ ,θ) θ 0 as n a.s., then the
θ0 n 0 → → ∞
A. The main idea estimate of the true source computed by the decoder is only
In this section, we describe the main idea behind the proof slightlyworse.Furtheermore,asweshallshow,thealmost-sure
convergence of d (θ ,θ) to zero as n implies that the
andfix somenotation.We havealreadyseenthatitsufficesto n 0
→ ∞
LagrangianperformanceofCn onP isclosetotheoptimum
construct a universal scheme that can compete with all zero- θb θ0
tmheamt tohreyrevaerxiiasbtslea-rasetequqeunacnetizeCrsn.,mTnhatoifs,coitdessu,ffisucecshttohasthow LθF0(oCrmθn0a,lλly),≡theLnθc0o(dλe).Cbn,mn is comprised by the following
{ ∗ } maps: b ∗
V( )logn
Lθ(C∗n,mn,λ)−Lnθ(λ)=O r Ann !,∀θ ∈Λ. • the parameter estimator θ :Xmn →Λ;
the parameter encoder g : Λ , where
(8)
This is what we shabll prove. • 0s(i) i N 1 ; e → S S ≡
{ }∈ ∪{ }
Weassumethroughoutthatthe“true”sourceisP forsome the parameter decoder ψe: Λ. e e
θ0 • S →
θ0 Λ. Our code operates as follows. Suppose that: Let f denote the compositiong θ of the parameterestimator
∈ e ◦e
Both the encoder and the decoder have access to a and the parameter encoder, which we refer to as the first-
• θco(iu)ntablΛy.inUfisninitge “Edlaiatasb’ausen”ivcer=sal{θre(pi)r}esi∈enNt,awtiohnereofeatchhe pstaargaemeeentecroddeecr,odanerdalnedt θthdeefinrostete-stthaeegecoemncpoodseirt.ioTnhψe d◦efcoodferthψe
intege∈rs [36], we can associate to each θ(i) a unique isthefirst-stagedecodebr.Thecollection Cn :θ eΛ edefines
{ θ ∈ }
binarystrings(i)with ℓ(s(i))=logi+O(loglogi)bits. thesecond-stagecodes.Theencoderf : n mn e
∗ X ×X →S×S
e
7
and the decoder ϕ : n of Cn,mn are defined as independent. Thus, the set
∗ S×S →X ∗
f∗(X1n,X−0mn+1e)=△ f(X−0bmn+1)fθb(X−0mn+1)(X1n) = n (j 1)(n+ln)+1 i j(n+ln) ln
M { − ≤ ≤ − }
and j=1
e [
△
respectively.ϕT∗o(ssas)s=essϕψteh(ees)(pse)r,formasnc∈eSo,fsC∈nS,mn, consider ilsentghteheifsfenc2ti.ve memory of C∗n,mn, and the effective memory
e e ∗ e Let Q(n) denote the marginal distribution of Zn, and let
the function
Q(n) denotetheproductofncopiesofPn.Wenowshowthat
g(X1n,X−0mn+1)=△ ρn(cid:16)X1n,Cθbn(X−0mn+1)(X1n)(cid:17) cwereeacsainngalyppfirnoexliymwatiethQn(.nN) boyteQth(ant)bionthvaQθr0i(ant)ioannadlQdi(snta)nacree,dine--
+λ ℓn fθb(X−0mn+1)(X1n) +ℓn f(X−0mn+1) . fined onthe σ-algebraF(n), geeneratedby allXi exceptthose
tTohePexpisecptarteicohinselE(cid:16)yθ0theg(LXa1gnr,aXng−0imann(cid:17)+p1e)rforo(cid:16)mfeagncewiothf Crens(cid:17),pmienct, TinheYr1e,fo..re.,,Yusni,ngsointdhuatctdio(nQa(nn)d,Qth(en)d)efi=nitdio(Qn(onf),thQee(nβ)-;mFi(xni)n)g.
θ0 (cid:8) (cid:9) ∗ coefficient (cf. Section II-A), weehave e
at Lagrange multiplier λ, on the source P . We consider
θ0
separately the contributions of the first-stage and the second- d(Q(n),Q(n)) (n 1)β (l )=O(1/n1+η),
stage codes. Define another function h : n mn R+ ≤ − θ0 n
X ×X → where the last equality follows from Condition 1 and from
by e
ourchoiceofl .Thisinturnimpliesthefollowingusefulfact
n
h(Xn,X0 )=△ ρ Xn,Cn (Xn) (see also Lemma 4.1 of Yu [29]), which we shall heavily use
1 −mn+1 n 1 θb(X−0mn+1) 1 in the proof: for any measurable function σ : n2 [0,M]
(cid:16) (cid:17)
+λℓn fθb(X−0mn+1)(X1n) , with M <∞, X →
(cid:16) (cid:17)
sgoratnhgaitaEnθp0enrfho(rmXa1nn,cXe−o0fmtnh+e1c)o(cid:12)(cid:12)(cid:12)Xde−0Cmθbnn(+X1−0omnis+1t)heon(rPanθd0.oHme)nLcae-, (cid:12)(cid:12)(cid:12)EQ(n=)(cid:8)Oσ((1Z/nn)1(cid:9)+η−),EQe(n)(cid:8)σ(Zn)(cid:9)(cid:12)(cid:12)(cid:12)≤M(n−1)βθ0((1ln0))
wheretheconstanthiddenin theO() notationdependsonM
g(X1n,X−0mn+1)=h(X1n,X−0mn+1)+λℓn f(X−0m+1) , and on θ0. ·
(cid:16) (cid:17)
so, taking expectations, we get
e
L (Cn,mn,λ)=E h(Xn,X0 )
θ0 ∗ θ0 1 −mn+1 C. Construction of the database
+λEθ0 ℓn f(X(cid:8)−0mn+1) . (cid:9) (9) The database, or the first-stage codebook, c is constructed
Our goal is to show thant the(cid:16)first term in(cid:17)Eoq. (9) converges byrandomselection.LetW beaprobabilitydistributiononΛ
e
to the nth-orderoptimum Ln (λ), and that the second term is which is absolutely continuous with respect to the Lebesgue
θ0 measure and has an everywhere positive and continuous den-
o(1).
The proof itself is organbized as follows. First we motivate sity w(θ). Let C = Θ(i) i N be a collection of independent
{ }∈
random vectors taking values in Λ, each generated according
the choice of the memory lengths m in Section IV-B. Then
we indicate how to select the databasne C (Section IV-C) and to W independently of X. We use W to denote the process
distribution of C.
how to implement the parameter estimator θ (Section IV-D)
andthe parameterencoder/decoderpair (g,ψ) (SectionIV-E). Note that the first-stage codebook is countably infinite,
The proof is concluded by estimating the Laegrangian perfor- which means that, in principle, both the encoder and the
decoder must have unbounded memory in order to store it.
mance of the resulting code (Section IVe-Fe) and the fidelity
This difficulty can be circumvented by using synchronized
of the source identification at the decoder (Section IV-G).
In the following, (in)equalities involving the relevant random random number generators at the encoder and at the decoder,
so that the entries of C can be generated as needed. Thus,
variables are assumed to hold for all realizations and not just
a.s., unless specified otherwise. by construction,the encoderwill generateTn samples(where
T is the waiting time) and then communicate (a binary
n
encoding of) T to the decoder. Since the decoder’s random
n
B. The memory length number generator is synchronized with that of the encoder’s,
the decoder will be able to recover the required entry of C.
Let l = n(2+η)/r , where r is the common decay
n
⌈ ⌉
exponent of the β-mixing coefficients β (k) in Condition 1,
θ
and let m = n(n+l ). We divide the m -block X0
into n blocnks Z ,...,Zn of length n interlenaved by n−bmlonc+k1s D. Parameter estimation
1 n
Y1,...,Yn of length ln (see Figure 1). The parameter esti- The parameter estimator θ : mn Λ is constructed
mator θ, although defined as acting on the entire X0 , as follows. Because the source XX is s→tationary, it suffices
effectively will make use only of Zn = (Z ,...,Z−)m.nT+h1e to describe the action of θeon X0 . In the notation
Zj’s aree each distributed according to Pθn0, b1ut theynare not of Section IV-A, let PZn be the e−mmpni+ri1cal distribution of
e
8
. . .
n l n l n l
n n n
. . .
Z Y Z Y Z Y Xn {Cn} X^ n
1 1 2 2 n n 1 q^ 1
~ ~
... q ~g y q^(X-0m+1)
n
parameter parameter parameter
estimator encoder decoder
Fig.1. ThestructureofthecodeC∗n,mn.Theshadedblocks arethoseusedforestimating thesourceparameters.
Zn =(Z ,...,Z ). For every θ Λ, define holds for every θ Λ, regardless of whether T is finite or
1 n n
∈ ∈
infinite.
Uθ(Zn) =△ sup |Pθn(A)−PZn(A)|
A∈An F. Performance of the code
sup pθ(xn)dµ(xn) PZn(A) , Given the random codebook C, the expected Lagrangian
≡ −
A∈An(cid:12)(cid:12)ZA (cid:12)(cid:12) performance of our code on the source Pθ0, is
where is the Yatrac(cid:12)os class defined by the nth(cid:12)-order
θd˜(eXns−0itmieA+s1n{)panθs a:nyθθ∈∗ ∈ΛΛ}(cid:12) s(asteisefySinegction III). Finally,(cid:12)define Lθ0(C=∗nE,mθ0n,λh)=X1nE,θX0n−0gm(cid:16)nX+11n,X−0mn+1)(cid:17)o
Uθ∗(Zn)<θinΛf Uθ(Zn)+ n1, +(cid:8)λE(cid:0)θ0 ℓn f X−0(cid:1)m(cid:9)n+1 . (13)
∈ We now upper-boundthe twonter(cid:16)ms(cid:0)in (13). W(cid:1)e(cid:17)sotart with the
where the extra 1/n term is there to ensure that at least one e
second term.
such θ exists. This is the so-called minimum-distance (MD)
∗ We need to bound the expectation of the waiting time
density estimator of Devroye and Lugosi [34], [35] (see also
T = T (θ(X0 )). Our strategy borrows some elements
Devroye and Gyo¨rfi [37]), adapted to the dependent-process frnom thenpaper−omfnK+o1ntoyiannisand Zhang[38]. Consider the
setting of the present paper. The key property of the MD
probabilitye
estimator is that
dn θ(Xm0n+1),θ0 ≤4Uθ0(Zn)+ n3 (11) qn =△ W dn Θ,θ(X−0mn+1) ≤√nδn ,
(see, e.g., The(cid:16)oerem 5.1 of D(cid:17)evroye and Gyo¨rfi [37]). This withfioclhlowissaforranndosmu(cid:16)ffifcuine(cid:0)cnttiloyenlaorfgeXt−0hmatn+(cid:1)1. From C(cid:17)ondition 2,
holds regardless of whether the samples Z ,...,Z are inde-
1 n
pendent or not.
qn ≥W kΘ−θ(X−0mn+1)k≤δn/cθe ,
(cid:16) (cid:17)
E. Encoding and decoding of parameter estimates where θ θ(X0 ).eBecause the density w(θ) is ev-
erywhere ≡positive−,mthn+e1latter probability is strictly positive
Next we construct the parameter encoder-decoder pair
for almeost alel X0 , and so q > 0 eventually almost
(g,ψ). Given a θ ∈Λ, define the waiting time surely. Thus, the−wmanit+in1g times T nwill be finite eventually
n
T (θ)=△ inf i 1:d (θ,Θ(i)) √nδ , almost surely (with respect to both the source X and the
e e n { ≥ n ≤ n} first-stage codebook C). Now, if q > 0, then, conditioned
withthestandardconventionthattheinfimumoftheemptyset n
on X0 = x0 , the waiting time T is a geometric
liosoekqsuathlrtoou+gh∞t.hTehcaotdies,bgoiovkenCaaθn∈dΛfin,dthsetphaerapmoseitteiornenocfodtheer rando−mmvna+ri1ablew−imthnp+a1rameterqn, anditis nnothardto show
(see, e.g., Lemma 3 of Kontoyiannisand Zhang [38]) that for
first Θ(i) such that the variational distance between the nth-
any ǫ>0
order distributions Pn and Pn is at most √nδ . If no such
θ Θ(i) n
Θma(ip)sigs faonudndψ,tbhyeencodersets Tn =+∞.We thendefinethe P log[(Tn−1)qn]≥ǫ X−0mn+1 =x0−mn+1 ≤e−2ǫ.
(cid:16) (cid:12) (cid:17)
Setting ǫ = log(2logn), w(cid:12)e have, for almost all Xn ,
e eg(θ)= 10,s(Tn), iiff TTn =<∞ that (cid:12) −mn+1
n
(cid:26) ∞ P log[(T 1)q ] log(2logn) X0 =x0
and e n− n ≥ −mn+1 −mn+1
ψ(0s(i))=Θ(i), ψ(1)=θ(1) (cid:16) e 2logn n 2. (cid:12) (cid:17)
− − (cid:12)
≤ ≤ (cid:12)
respectively. Thus, = 0s(i) 1 , and the bound Then, by the Borel–Cantelli lemma,
e S { }∪{e}
ℓ(g(θ)) logT +O(loglogT ) (12) log(T q ) loglogn+2
e≤ n n n n ≤
e
9
eventually almost surely, so that by 1 O(1/n). Combining this with (15), taking logarithms,
−
and then taking expectations, we obtain
E logT loglogn+2 E logq (14)
θ0{ n}≤ − θ0{ n} E logq
for almost every realization of the random codebook C and θ0{ n}
for sufficiently large n. We now obtain an asymptotic lower log(1 O(1/n))+klogδn+2c(k,θ0)
≥ −
bound on E logq . Define the events = log(1 O(1/n))
θ0{ n} −
GFnn ==△△ nd(dnn(Θ(cid:16)θe,(θX0)−0≤mn√+n1)δ,nθ/02(cid:17)≤, √nδn/2o, ++k32klolgohgpn12+04c8((kV,(θA0)n)+1)nlnn+6i
Hn =△ (cid:8){kΘ−θ0k≤δn/2cθ0}(cid:9). log(1 O(1/n))+ 3k log 1 +c(k,θ0),
≥ − 2 n
Then by the triangle inequality we have
where c(k,θ ) is a constant that depends only on k and θ .
0 0
F and G = d Θ,θ(X0 ) √nδ , Using this and (14), we get that
n n ⇒ n −mn+1 ≤ n
and, for n sufficiently large,(cid:16)we cean write (cid:17) Eθ0{logTn}≤loglogn+O(logn)
(a) for W-almost every realization of the random codebook C,
q W(G )P (F )
n ≥ n θ0 n for n sufficiently large. Together with (12), this implies that
(=b) W(G )Q(n)(F )
n n E ℓ f X0
(≥c) W(Hn)Q(n)(Fn), θ=0n nO(cid:16)el(cid:0)ogn−mn++1O(cid:1)(cid:17)ologlogn + 3 +o(1)
where (a) follows from the independence of X and C, (b) n n n
(cid:18) (cid:19) (cid:18) (cid:19)
fdoelploewndssforonmlytohneZfanc,tathnadt(tch)efpoallroawmsetferromestCimonatdoirtiθo(nX2−0amndn+th1e) for W-almost all realizations of the first-stage codebook.
fact that δ 0. Since the density w is everywheere positive
n
→
and continuous at θ , w(θ) w(θ )/2 for all θ H for n
0 0 n
≥ ∈
sufficiently large, so
k
1 δ
W(H )= w(θ)dθ w(θ )v n , (15) We now turn to the first term in (13). Recall that, for each
n 0 k
ZHn ≥ 2 (cid:18)2cθ0(cid:19) θ ∈ Λ, the code Cθn is nth-order optimal for Pθ. Using this
where v is the volume of the unit sphere in Rk. Next, the fact together with the boundedness of the distortion measure
k
fact thatthe minimum-distanceestimate θ(X0 ) depends ρ, we can invoke Lemma A.3 in the Appendix and assume
onlyonZn impliesthattheeventFn belongs−tmonth+e1σ-algebra without loss of generality that each Cθn has a finite codebook
(n), and from (10) we get e (of size not exceeding 2nρmax/λ), and each codevector can
F be described by a binary string of no more than 2nρ /λ
max
Q(n)(Fn)≥Q(n)(Fn)−O(1/n1+η). (16) bthites.mHaergnicnea,lhd(iXst1rni,bXut−i0omnsn+o1f)P≤ 3ρomnaσx.(XLe0t P)−anadndσP(X+ b)e,
Under Q(n), the n-blockseZ1,...,Zn are i.i.d. according to respectively. Note that h(Xn,Xθ00 ) do−e∞s not depend1∞on
Pθn0, and we can invoke the Vapnik–Chervonenkismachinery X0 .This,togetherwithCon−dimtino+n11andthechoiceofl ,
tolowere-boundQ(n)(Fn).InthenotationofSec.IV-D,define im−pllnie+s1that n
the event
I e=△ 4U (Zn)+ 3 √nδn . Eθ0 h X1n,X−0mn+1
n (cid:26) θ0 n ≤ 2 (cid:27) (cid:8)≤(cid:0)EP−×P+ h X(cid:1)(cid:9)1n,X−0mn+1 +βθ0(ln)
Then In implies Fn by (11), and =EP−×P+(cid:8)h(cid:0)X1n,X−0mn+1(cid:1)(cid:9)+O(1/n2+η).
Q(n)(Fnc)≤Q(n)(Inc)≤8nV(An)e−n(√nδn−6/n)2/2048, Furthermore, (cid:8) (cid:0) (cid:1)(cid:9)
wheere the seconed bound is by the Vapnik–Chervonenkis(1in7-) EP−×P+ h X1n,X−0mn+1
equality (3) of Lemma 2.1. Combining the bounds (16) and = (cid:8) (cid:0) h(xn,zmn(cid:1))(cid:9)dP (xn)dP (zmn)
θ0 θ0
(17) and using Condition 1, we obtain ZXn×Xmn
Pθm(Fn)≥1−8nV(An)e−n(√nδn−6/n)2/2048−O(1/n1+(1η8)) (=a)ZXmn Eθ0{h(X1n,zmn)}dPθ0(zmn)
Now, if we choose2048(V( n)+1)lnn 6 where(a)foll(=obw)Esθb0y(cid:26)FLubθ0in(cid:16)iC’sθbnt(hXe−0omrenm+1a)n,λd(cid:17)th(cid:27)e,boundednessof
δ = A + ,
n n n3/2 h, while (b) follows from the definition of h. The Lagrangian
p
then the right-handside of (18) can be further lower-bounded performance of the code Cn, where θ =θ(X0 ), can be
θb −mn+1
b b
10
further bounded as eventually almost surely, so the corresponding expectation is
O V( )logn/n aswell.Summingtheestimates(21)and
Lθ0 Cθbn,λ (22), weAonbtain
(cid:0)p (cid:1)
(a(cid:16)) (cid:17)
(≤=b)LLθbnθb((cid:16)λC)θbn+,λ3(cid:17)ρm+ax3dρnmaθx(dXn(cid:16)−0θbm(nX+−01m),nθ+01),θ0(cid:17) FEinθa0l(cid:8)lyh,(cid:0)pXut1nti,nXg−0emvenr+y1th(cid:1)i(cid:9)ng=toLbgnθe0t(hλe)r,+wOe s eerthVa(tA, envn)elnotguanll!y,.
(c) (cid:16) (cid:17)
≤ Lbnθ0(λ)+4ρmaxdn bθ(X−0mn+1),θ0 L Cn,mn =Ln (λ)+O V(An)logn
(≤d)Lbnθ0(λ)+4ρmax d(cid:16)nbθ(X−0mn+1),θ(cid:17)(X−0mn+1) θ0(cid:16) ∗ log(cid:17)n bθ0 loglog nr 3 n !
h (cid:16) (cid:17) +λ O +O + +o(1) (23)
b +dn(cid:16)θ(bX−0mn+1),θ0e(cid:17)i, for W-almosthev(cid:16)erynrea(cid:17)lization(cid:16)of tnhe firs(cid:17)t-stange codebiookC.
where (a) follows from Lemma A.3 in the Appendix, (b)
e This proves (8), and hence (7).
follows from the nth-order optimality of Cn for Pn, (c)
b b
θ θ
follows,overboundingslightly,fromtheLagrangianmismatch
bound of Lemma A.2 in the Appendix, and (d) follows from G. Identification of the active source
the triangle inequality. Taking expectations, we obtain
We have seen that the expected variational distance
E d θ ,θ(X0 ) between the n-dimensional
Eθ0(cid:26)Lθ0 Cθbn(X−0mn+1),λ (cid:27)≤Lnθ0(λ) mθa0rgninnal(cid:0)s 0of the−tmrune+1so(cid:1)uorce Pθ0 and the estimated source
+4ρm(cid:0)ax·Eθ0 dn θ(cid:1)(X−0mbn+1),θ(X−0mn+1) tPoθb(Xsh−0omwn+t1h)acbtotnhviesrgceosntvoerzgeernoceasalsoV(hAonld)slogevne/nntu.aWllye wwiisthh
n (cid:16) (cid:17) p
+d θ(X0 ),θ . (19) probability one, i.e.,
n −mn+1 b0 e
The second d ((cid:16), ) term in (19)(cid:17)ocan be interpreted as the V( )logn
estimation erronr ·du·ee to estimating Pθn0 by Pθen, while the first dn(θ0,θ(X−0mn+1))=O r Ann ! (24)
d (, ) is the approximation error due to quantization of the
n · · P -almost surebly.
parameter estimate θ. We examine the estimation error first. θ0
Given an ǫ > 0, we have by the triangle inequality that
Using (11), we can write
d (θ ,θ(X0 ))>ǫ implies
E d Pn e ,Pn 4E U (Zn) + 3. (20) n 0 −mn+1
mbaNpyoepwartθsho,0uexrneieamecxQ(cid:16)ahpt(eenZθc)t∗tjh:a(eXtiis−o0enxmdp+ioes1fct)rtiaUbtuiθθo0t0en((cid:17)dZoonfa≤)cUcwθo0ri(dtZhθin0ng{r)estwpθo0eitcPhtθnr0teo,s}patenhcdetntpworeoQdcu(ancn)t ofwdrfohnmte(cid:16)hreθeX0fiθbe,−0r(θesXm(tX-n−0s+t−0ma1mgne+(nc+1ef)n1.)cSi(cid:17)osedc+tehtireod,nnwm(cid:16)eIiθVens(-ieXmEe)u−0.tmmhRa-netd+citas1htl)ial,sinnθbcfg(ueXrote−h0usermtirmcnio+mant1eps)tl(cid:17)riouef>sctPiǫoθn,n0
EQ(n){Ueθ0(Zn)} ≤ EQe(n){Uθ0(Zn)}+(n−1)βθ(ln) dn θ0,θ(X−0mn+1) >ǫ−√nδn.
V( )logn 1 Finally, using t(cid:16)he property (11(cid:17)) of the minimum-distance
c An +O e
≤ n n1+η estimator, we obtain that
r (cid:18) (cid:19)
= O V(Ann)logn , dn θ0,θ(X−0mn+1) >ǫ
r ! (cid:16) (cid:17)
implies
b
where the second estimate follows from the Vapnik– 1 3
U (Zn)> ǫ √nδ .
Chervonenkis inequality (4) and from the choice of ln. This, θ0 4 − n− n
(cid:18) (cid:19)
together with (20), yields
Therefore,
V( )logn
Eθ0 d Pθn∗(X−0m+1),Pθn0 =O r Ann !. (21) Q(n) dn θ0,θ(X−0mn+1) >ǫ
n (cid:16) (cid:17)o n (cid:16) (cid:17)1 o 3
Q(n) bU (Zn)> ǫ √nδ
As for the first dn(, ) term in (19), we have, by construction ≤ θ0 4 − n− n
of the first-stage en·co·der, that (cid:26) (cid:18) (cid:19)(cid:27)
(a) 1 3
Q(n) U (Zn)> ǫ √nδ
d θ(X0 ),θ(X0 ) ≤ θ0 4 − n− n
n −mn+1 −mn+1 (cid:26) +(n 1)β(cid:18)(l ) (cid:19)(cid:27)
(cid:16) V(cid:17)( )logn e − θ0 n
b ≤√nδne=O r Ann ! (22) (≤b) 8nV(An)exp −n(ǫ−√n5δ1n2−3/n)2
(cid:18) (cid:19)
+(n 1)β (l ), (25)
− θ0 n