Table Of ContentAsymptotic Accuracy of Bayes Estimation
for Latent Variables with Redundancy
3
1
0
2
Keisuke Yamazaki
n
a [email protected]
J
Department of Computational Intelligence and Systems Science,
0
3
Tokyo Institute of Technology
] G5-19 4259 Nagatsuta Midori-ku Yokohama, Japan
L
M
.
t
a Abstract
t
s
[ Hierarchical parametric models consisting of observable and latent vari-
ables are widely used for unsupervised learning tasks, such as clustering.
3
v These models can be regular or singular. In the singular case, there is no
4
one-to-one relation between a probability function and the parameter. Oth-
3
erwise, in the regular case, the Fisher information matrix is positive definite,
2
3 and the estimation accuracy of both observable and latent variables has been
.
5 studied. In the singular case, however, conventional statistical analysis based
0 on theinverse Fisher matrix is notapplicable, and soan algebraic geometrical
2
analysis has been developed and is used to elucidate the Bayes estimation of
1
: observablevariables. Thepresentpaperappliesthisanalysistolatent-variable
v
i estimation and determines its theoretical performance. Our results show that
X
the posterior distribution of the parameter in the observable-variable estima-
r
a tion can be different from the one in the latent-variable estimation, which
implies that the Markov chain Monte Carlo method based on the parameter
and the latent variable cannot construct the proper posterior distribution.
Keywords: unsupervised learning, hierarchical parametric models, Bayes
statistics, algebraic geometry, singularities
1 Introduction
Hierarchical parametric models are employed for unsupervised learning in many
data-mining and machine-learning applications. Statistical analysis of the models
plays an important role for not only revealing the theoretical properties but also the
practical applications. For example, theasymptotic formsofthe generalizationerror
1
Table 1: Estimation classification according to the target variable and the model
case
Estimation Target \Model Case Regular Case Singular Case
Observable Variable Reg-OV estimation Sing-OV estimation
Latent Variable Reg-LV estimation Sing-LV estimation
and the marginal likelihood are used for model selection in the maximum-likelihood
and Bayes methods, respectively [1, 11, 8].
Parametric models generally fall into two cases: regular and singular. The
present paper focuses on the models, the function of which are continuous and
sufficiently smooth with respect to the parameter. In regular cases, the Fisher in-
formation matrix is positive definite, and there is a one-to-one relation between the
parameter and the expression of the model as a probability function. Otherwise,
the model is singular, and the parameter space includes singularities. Due to these
singularities, the Fisher information matrix is not positive definite, and so the con-
ventional analysis methods that rely on its inverse matrix are not applicable. In this
case, an algebraic geometrical approach can be used to analyze the Bayes method
[12, 13].
Hierarchical models have both observable and latent variables. The latent vari-
ables represent the underlying structure of the model, while the observable ones
correspond to the given data. For example, unobservable labels in clustering are
expressed as the latent variables in mixture models, and the system dynamics of
time-series data is a sequence of the variables in hidden Markov models. Hierar-
chical models thus have two estimation targets: observable and latent variables.
The well-known generalization error measures the performance of the prediction of
a future observable variable. Combining the two model cases and the two estima-
tion targets, there are four estimation cases, which are summarized in Table 1. We
will use the abbreviations shown in the table to specify the target variable and the
model case; for example, Reg-OV estimation stands for estimation of the observable
variable in the regular case.
In the present paper, we will focus on the Sing-LV estimation. One of the main
concerns in unsupervised learning is the estimation of unobservable parts and in
practical situations, the ranges of the latent variables are unknown, which corre-
sponds to the singular case. The other estimation cases have already been studied;
the accuracy of the Reg-OV estimation has been clarified on the basis of the conven-
tional analysis method, and the results have been used for model selection criteria,
such as AIC [1]. The primary purpose for using the algebraic geometrical method
is to analyze the Sing-OV estimation, and the asymptotic generalization error of
the Bayes method has been derived for many models [3, 2, 10, 16, 17, 18, 19, 20].
Recently, an error function for the latent-variable estimation was formalized in a
2
distribution-based manner, and its asymptotic form was determined for the Reg-LV
estimation of both the maximum likelihood and Bayes methods [14]. Hereinafter,
the estimation method will be assumed to be the Bayes method unless it is explicitly
stated otherwise.
The main contributions of the present paper are summarized as follows:
1. The algebraic geometrical method for the Sing-OV estimation is applicable to
the analysis of the Sing-LV estimation.
2. The asymptotic form of the error function is obtained, and its dominant order
is larger than that of the Reg-LV estimation.
3. We found that the posterior parameter distribution of the Sing-LV estimation
is different from that of the Sing-OV estimation.
The third result is important for practical applications: in some cases, methods
based on latent variables, such as the Markov chain Monte Carlo (MCMC) method
with Gibbs sampling, cannot construct the proper posterior distribution of the pa-
rameter because the joint distribution of the parameter and the latent variable can
have a different peak from that of the marginal distribution. Asymptotic analysis
shows the condition when these methods fail.
The rest of this paper is organized as follows. The next section summarizes the
observable-variable estimation, which includes the Reg-OV and the Sing-OV esti-
mations, and introduces the asymptotic prediction performance based on algebraic
geometrical analysis. In Section 3, the latent-variable estimation and its evaluation
function are formulated in a distribution-based manner. Sections 4 and 5 show the
main results: the former section presents the derivation of the asymptotic form of
the error function in the Sing-LV estimation, and the latter presents an investigation
of the asymptotic accuracy of mixture models and a demonstration of how to calcu-
late the form in a concrete model. Finally, Sections 6 and 7 present the discussion
and conclusions, respectively.
2 Accuracy of the Observable-Variable Estima-
tion and the Algebraic Geometrical Analysis
Inthissection, weformalizetheBayes methodfortheobservable-variableestimation
and introduce the algebraic geometrical method, which is applicable to both the
Reg-OV and the Sing-OV estimations.
Let a learning model be defined by
K K
p(x|w) = p(x,y|w) = p(y|w)p(x|y,w),
y=1 y=1
X X
3
where x ∈ RM is an observable variable, y ∈ {1,...,K} is a latent one, and w is
a parameter in a continuous space. For the discrete x such that x ∈ {1,2,...,M},
M
all results hold by replacing dx with . Assume that the observable data
x=1
Xn = {x ,··· ,x } are independent and identically distributed from the true model,
1 n
R P
which is expressed as
K∗
q(x) = q(y)q(x|y).
y=1
X
Notethatthevaluerangeofthelatent variabley described as[1,...,K∗]isgenerally
unknown and can be different from the one in the learning model. We also assume
thatthetruemodel satisfies theminimality condition: K∗ istheminimum rangeofy
for function representation of q(x). For example, consider a three-component model
such that q(x|y = 1) 6= q(x|y = 2) = q(x|y = 3) for all x ∈ RM. The minimality
condition requires the two-component expression
q(x) =q(y = 1)q(x|y = 1)+{q(y = 2)+q(y = 3)}q(x|y = 2)
=q(y = 1)q(x|y = 1)+q(y = ¯2)q(x|y = ¯2),
where ¯2 = {2,3}.
The present paper focuses on the case in which the true model is in the class of
the learning model. More formally, there is a set of parameters expressing the true
model such that
Wt ={w∗;p(x|w∗) = q(x)} =6 ∅,
X
which is referred to as the true parameter set for x. This means that the latent
variable range satisfies K = K∗ or K > K∗. The former relation corresponds to
the regular case and the latter one to the singular case. The true parameter set Wt
X
includes K! isolated points in the regular case due to the symmetry of the parameter
space. On the other hand, it consists of an analytic set in the singular case. We
explain this structure using the following model settings.
Example 1 Let a learning model be a two-component mixture model, and let the
true model be a single component. The mathematical expressions are given by
p(x|w) =af(x|b )+(1−a)f(x|b ),
1 2
∗
q(x) =f(x|b ),
where x ∈ R1, f(x|b) is a component distribution, and w = {a,b ,b } such that
1 2
a ∈ [0,1] and b ,b ∈ R1.
1 2
4
The latent variable y describes the labels of the components. The model expression
with the labels is defined by
2 2
p(x|w) = p(y = k)p(x|y = k,w) = p(x,y = k|w).
k=1 k=1
X X
We can confirm that the true parameter set consists of the following analytic set:
Wt ={a = 1,b = b∗}∪{a = 0,b = b∗}∪{b = b = b∗}.
X 1 2 1 2
The Fisher information matrix is not positive definite in this set. Moreover, the
intersection of these subsets are singularities. If the true model has two components,
∗ ∗ ∗ ∗
q(x) =a f(x|b )+(1−a )f(x|b )
1 2
for a∗ 6= 0,1 and b∗ 6= b∗, the estimation will be in the regular case. Due to
1 2
K! = 2! = 2, the set consists of two isolated points;
Wt ={(a = a∗,b = b∗,b = b∗),(a = 1−a∗,b = b∗,b = b∗)}.
X 1 1 2 2 1 2 2 1
In Bayesian statistics, estimation of the observable variables is defined by
p(x|Xn) = p(x|w)p(w|Xn)dw,
Z
n
p(x |w)ϕ(w;η)
p(w|Xn) = i=1 i ,
Z(Xn)
Q
where ϕ(w;η) is a prior distribution with the hyperparameter η, p(w|Xn) is the
posterior distribution of the parameter, and its normalizing factor is given by
n
Z(Xn) = p(x |w)ϕ(w;η)dw.
i
Z i=1
Y
This formulation is available for both the Reg-OV and Sing-OV estimations. The
estimation accuracy is measured by the average Kullback-Leibler divergence:
q(x)
G(n) =E q(x)ln dx ,
X
p(x|Xn)
(cid:20)Z (cid:21)
where the expectation is
E [f(Xn)] = f(Xn)q(Xn)dXn.
X
Z
5
Let us define the free energy as
F(Xn) =−lnZ(Xn),
which plays an important role in Bayes statistics as a criterion for selecting the
optimal model. In the Reg-OV estimation, the Bayesian information criterion (BIC)
[11] and the minimum-description-length principle (MDL) [8] are both based on the
asymptotic formofF(Xn). Theoretical studies oftenanalyze theaverage freeenergy
given by
F (n) =−nS +E [F(Xn)]
X X X
, where the entropy function is defined by
S =− q(x)lnq(x)dx.
X
Z
The model that minimizes F(Xn) is then selected as optimal from among the can-
didate models. The energy function F (n) allows us to investigate the average
X
behavior of the selection. Note that the entropy term does not affect the selec-
tion result because it is independent of the candidate models. According to the
definitions, the average free energy and the generalization error have the relation
G(n) =F (n+1)−F (n),
X X
which implies that the asymptotic form of F(n) also relates to that of G(n).
The algebraic geometrical analysis [12, 13] is applicable to both the regular and
singular cases for deriving the asymptotic form of F (n). The rest of the section
X
discusses the case Wt 6= ∅, although it is also important to consider the case Wt =
X X
∅, where the learning model cannot attain the true model.
Let us define another Kullback-Leibler divergence,
q(x)
H (w) = q(x)ln dx,
X
p(x|w)
Z
which is assumed to be analytic. We consider the prior distribution ϕ(w;η) =
ψ (w;η)ψ (w;η), where ψ (w;η) is a positive function of class C∞ and ψ (w;η) is
1 2 1 2
a nonnegative analytic function. Then, the zeta function of a statistical model is
given by
ζ (z) = H (w)zϕ(w;η)dw,
X X
Z
6
where z is a complex variable. From algebraic analysis, we know that its poles are
real, negative, and rational [4]. Let the largest pole and its order be z = −λ and
X
m , respectively. The zeta function includes the term
X
f (z)
c
ζ (z) = +...,
X (z +λX)mX
where f (z) is a holomorphic function. We define the state density function of t > 0
c
as
v(t) = δ(t−H (w))ϕ(w;η)dw.
X
Z
The zeta function is its Mellin transform:
∞
ζ (z) = v(t)tzdt.
X
Z0
Moreover, it is known that the inverse Laplace transform of v(t) has the same
asymptotic form as F (n). Following the transforms from ζ (z) to F (n) through
X X X
v(t), we obtain the asymptotic form
F (n) =λ lnn−(m −1)lnlnn+O(1).
X X X
Using the same coefficients, the asymptotic form of the generalization error is given
by
λ m −1 1
X X
G(n) = − +o . (1)
n nlnn nlnn
(cid:18) (cid:19)
Since the learning model can attain the true model, we can confirm that the gen-
eralization error converges to zero for n → ∞. The coefficients are λ = d/2 and
X
m = 1 in the regular case, which shows that the asymptotic form holds in both
X
the Reg-OV and Sing-OV estimations.
3 Formal Definitions of the Latent-Variable Esti-
mation and its Accuracy
This section formulates the Bayes latent-variable estimation and an error function
that measures its accuracy.
We first consider a detailed definition of a latent variable. Let Yn = {y ,...,y }
1 n
be unobservable data, which correspond to the latent parts of the observable Xn.
Then, the complete form of the data is (x ,y ), and (Xn,Yn) and Xn are referred
i i
7
to as complete and incomplete data, respectively. The true model generates the
complete data (Xn,Yn), where the range of the latent variables is y ∈ {1,...,K∗}.
i
The learning model, on the other hand, has the range y ∈ {1,...,K}. For a unified
i
description, we define that the true model has probabilities q(y) = 0 and q(x,y) = 0
for y > K∗.
We define the true parameter set for (x,y) as
Wt = {w∗;p(x,y|w∗) = q(x,y)},
XY
which is part of Wt . In Example 1,
X
Wt ={a = 1,b = b∗}.
XY 1
The subsets {a = 0,b = b∗} and {b = b = b∗} in Wt are excluded since Wt
2 1 2 X XY
takes account of the representation with respect to not only x but also y. Due to the
assumption Wt 6= ∅, Wt is not empty. The set Wt again consists of an analytic
X XY XY
set in the singular case, and it is a unique point in the regular case.
While latent-variable estimation falls into various types according to the target
of the estimation, the present paper focuses on the Type-I estimation of [14]: the
joint probability of (y ,...,y ) is the target and is written as p(Yn|Xn). The Bayes
1 n
estimation has two equivalent definitions:
n
p(x ,y |w)
p(Yn|Xn) = i i p(w|Xn)dw (2)
p(x |w)
i
Z i=1
Y
Z(Xn,Yn)
= , (3)
Z(Xn)
where the marginal likelihood for the complete data is given by
n
Z(Xn,Yn) = p(x ,y |w)ϕ(w;η)dw.
i i
Z i=1
Y
The true probability of Yn is uniquely given by
q(Xn,Yn) n q(x ,y )
q(Yn|Xn) = = i i . (4)
q(Xn) q(x )
i
i=1
Y
The accuracy of the estimation is measured by the difference between q(Yn|Xn)
and p(Yn|Xn). Thus, we define the error function as the average Kullback-Leibler
divergence,
1 q(Yn|Xn)
D(n) = E ln , (5)
n XY p(Yn|Xn)
(cid:20) (cid:21)
where the expectation is defined as
K K
E [f(Xn,Yn)] = ··· f(Xn,Yn)q(Xn,Yn)dXn.
XY
Z yX1=1 yXn=1
8
4 Asymptotic Analysis of the Error Function
In this section, we show that the algebraic geometrical analysis is applicable to the
Sing-LV estimation, and present the asymptotic form of the error function D(n).
Let us define the zeta function on the complete data (x,y) as
ζ (z) = H (w)zϕ(w;η)dw,
XY XY
Z
where the Kullback-Leibler divergence H (w) is given by
XY
K
q(x,y)
H (w) = q(x,y)ln dx.
XY
p(x,y|w)
y=1 Z
X
Let the largest pole of ζ (z) be z = −λ , and let its order be m .
XY XY XY
We consider the following conditions:
(A1) The divergence functions H (w) and H (w) are analytic.
XY X
(A2) The prior distribution has the compact support, which includes Wt , and has
X
the expression ϕ(w;η) = ψ (w;η)ψ (w;η), where ψ (w;η) > 0 is a function of
1 2 1
class C∞ and ψ (w;η) ≥ 0 is analytic on the support of ϕ(w;η).
2
The following theorem is the main result of the present paper:
Theorem 2 Let the true distribution of the latent variables and the estimated dis-
tribution be defined by Eqs. 3 and 4, respectively. By assuming the conditions (A1)
and (A2), the asymptotic form of D(n) is expressed as
lnn lnlnn lnlnn
D(n) =(λ −λ ) −(m −m ) +o .
XY X XY X
n n n
(cid:18) (cid:19)
Proof: Let us define another average free energy as
F (n) = −nS +E −lnZ(Xn,Yn) ,
XY XY XY
(cid:20) (cid:21)
where the entropy function is given by
K
S = − q(x,y)lnq(x,y)dx.
XY
y=1 Z
X
9
According to the definitions of the error function D(n) and the Bayes estimation
method Eq. 3, it holds that
q(Xn,Yn) q(Xn)
nD(n) =E ln −E ln
XY X
Z(Xn,Yn) Z(Xn)
(cid:20) (cid:21) (cid:20) (cid:21)
=−nS −E lnZ(Xn,Yn) +nS +E lnZ(Xn)
XY XY X X
(cid:20) (cid:21) (cid:20) (cid:21)
=F (n)−F (n).
XY X
Based on (A1), (A2), and algebraic geometrical analysis, we obtain the asymptotic
forms of F (n) and F (n):
XY X
F (n) =λ lnn−(m −1)lnlnn+O(1),
XY XY XY
F (n) =λ lnn−(m −1)lnlnn+O(1),
X X X
which proves the theorem. (End of Proof)
5 Accuracy of the Sing-LV Estimation in Mixture
Models
Inthissection, wederivetheasymptoticerroroftheBayeslatent-variableestimation
for mixture models. In Theorem 2, the possible dominant order was calculated as
lnn/n. However, there is no guarantee that this is the actual maximum order; the
order can decrease to a constant if the zeta functions ζ (z) and ζ (z) have their
XY X
largest poles in the same position and their multiple orders are also the same. The
result in this section clearly shows that the dominant order is lnn/n in the mixture
models.
5.1 Model Settings
Assume that a learning model is a mixture of distributions described by
K
p(x|w) = a f(x|b ), (6)
k k
k=1
X
where the components are defined by the distribution function f of x, which is a
regular model, and the parameter w that consists of the mixing ratios {a ,...,a },
2 K
and d is the dimension of the component parameters such that b ∈ Rdc. The
c k
K
mixing ratios have constraints a ≥ 0 and a = 1. We regard a as a function
k k=1 k 1
K
of the parameters a = 1− a .
1 k=2 k P
P
10
