Nonparametric semi-supervised learning of class proportions

3.6 MB·

by  Shantanu Jain

JournalofMachineLearningResearch (2015) Submitted12/15;Published00/00
Nonparametric semi-supervised learning of class proportions
Shantanu Jain [email protected]
Department of Computer Science and Informatics
Indiana University
Bloomington, IN 47405, USA
Martha White [email protected]
Department of Computer Science and Informatics
6 Indiana University
1 Bloomington, IN 47405, USA
0
2 Michael W. Trosset [email protected]
Department of Statistics
n
Indiana University
a
J Bloomington, IN 47408, USA

8
Predrag Radivojac [email protected]

] Department of Computer Science and Informatics
L
Indiana University
M
Bloomington, IN 47405, USA
.
t
a
t
s
[

1
v Abstract
4
4 The problem of developing binary classifiers from positive and unlabeled data is often
9 encountered in machine learning. A common requirement in this setting is to approximate
1
posterior probabilities of positive and negative classes for a previously unseen data point.
0
Thisproblemcanbedecomposedintotwosteps: (i)thedevelopmentofaccuratepredictors
.
1 that discriminate between positive and unlabeled data, and (ii) the accurate estimation of
0
thepriorprobabilitiesofpositiveandnegativeexamples. Inthisworkweprimarilyfocuson
6
the latter subproblem. We study nonparametric class prior estimation and formulate this
1
: problemasanestimationofmixingproportionsintwo-componentmixturemodels,givena
v
sample from one of the components and another sample from the mixture itself. We show
i
X that estimation of mixing proportions is generally ill-defined and propose a canonical form
r to obtain identifiability while maintaining the flexibility to model any distribution. We
a
use insights from this theory to elucidate the optimization surface of the class priors and
propose an algorithm for estimating them. To address the problems of high-dimensional
density estimation, we provide practical transformations to low-dimensional spaces that
preserveclasspriors. Finally, wedemonstratetheefficacyofourmethodonunivariateand
multivariate data.
Keywords: Positive-unlabeled learning, mixtures of distributions, identifiability.
1. Introduction
Binary classification is often attempted in situations where the examples from one class
greatly outnumber the examples from the other class (Chawla et al., 2004). An extreme
(cid:13)c2015ShantanuJain,MarthaWhite,MichaelW.TrossetandPredragRadivojac.
Jain, White, Trosset and Radivojac
case of this scenario occurs when the examples of one class (say, positives) are relatively
easy to obtain, while the examples of the other class (say, negatives) are either too expen-
sive or practically impossible to obtain. In such problems we are often presented with data
sets containing a relatively small number of positive examples and a relatively large num-
ber of unlabeled examples that contain both positive and negative examples at unknown
proportions.
Positive and unlabeled data sets are often observed in the sciences, where the absence
of a class designation, even after a failure to label the data point as positive, cannot be
interpreted as a negative class label. For example, a protein can be experimentally tested
for a particular functionality; e.g., catalytic activity. If confirmed, the data associating a
proteinwithcatalyticactivityisreliable; however,afailuretoconfirmcatalyticactivitymay
only be due to experimental limitations (Dessimoz et al., 2013). Further compounding the
problem, unsuccessful experiments are rarely reported in the literature. Another situation
conforming to the open-world assumption occurs in social networks, where a click on the
“like” button on Facebook is a reliable indicator of preference, yet the absence of a “like”
cannot be considered as an indicator of dislike. A simple and important question in all such
situations it that of estimating class priors: What is the fraction of all proteins in a given
species that are enzymes? or How many Facebook users like a particular product?
The positive-unlabeled data sets do not conform to typical assumptions in machine
learning. Traditional supervised algorithms assume the existence of both positive and neg-
ative examples, whereas unsupervised algorithms operate without any information on class
labels. Even most semi-supervised algorithms assume both positive and negative class la-
bels, in addition to the unlabeled set, and thus require modification. This framework has
been studied in the past decade and a half, usually under the names of partially super-
vised learning and learning from positive and unlabeled data (Liu et al., 2002; Denis et al.,
2005). Regardless of the problem type, the main goal in all these approaches is to learn
classifiers that discriminate between positives and negatives using the available data, ide-
ally by estimating posterior probabilities of class labels given an input example. Here, we
are (scientifically) motivated by the problem of estimating class priors: the proportions of
positive and negative examples in the unlabeled data, given a set of positive examples and
an unlabeled set.
More formally, we consider the binary classification problem of mapping an input space
to an output space = 0,1 given i.i.d. samples of positive and unlabeled examples
X Y { }
from fixed but unknown underlying probability distributions. We formulate estimation
of the fraction of positive examples in the unlabeled data as parameter learning of two-
component mixture models
f(x) = αf (x)+(1 α)f (x), (1)
1 0
−
wherex , f andf aredistributionsofthepositiveandnegativedata, respectively, and
1 0
∈ X
α (0,1) is the mixing proportion or class prior for the positive examples. In the simplest
∈
setting, f (x) and f (x) correspond to the class-conditional distributions p(x y = 1) and
1 0
|
p(x y = 0),respectively. Moregenerally,f andf mightbeanydistributionsobtainedafter
1 0
|
applying deterministic transformations that preserve the mixing proportions. For example,
a function g : [0,1], such as a classifier trained on the labeled vs. unlabeled data,
X →
could map the original feature vector x to a scalar, resulting in a univariate f. We will later
2
Nonparametric semi-supervised learning of class proportions
discuss class-prior preserving transformations; in the meantime, we refer to f , f , and f as
1 0
data distributions.
Despite the simplicity of our formulation, we are not aware of any previous attempts to
formulate class prior estimation using two-component mixture models. We therefore begin
by discussing identifiability conditions for mixing proportions. Using insights derived from
thistheory,wethenproposealgorithmsforlearningαusingmaximum-likelihoodestimation.
Finally, we conduct experiments on both synthetic and real-world data, obtaining evidence
that our methodology is sound and effective.
2. Identifiability issues and the max-canonical form
If the mixture distribution can be written as (1) for more than one choice of α, then
estimation of α is ill-defined. In order to proceed, we require identifiability: the existence
of a unique mixing proportion. In general, mixing proportions are not identifiable. In this
section we propose a max-canonical form for α that ensures identifiability and leads to a
viable estimation algorithm.
To formalize identifiability of the mixing proportion, we more generally use probability
measures to enable discrete, continuous and mixed random variables to be considered. Let
µ, µ and µ be probability measures defined on a σ-algebra and let
1 0
A
µ(A) = αµ (A)+(1 α)µ (A),
1 0
−
forallA . ThisexpressiongeneralizesthemixturemodelfromEquation1toprobability
∈ A
measures. For example, for a continuous random variable with probability density function
(cid:82)
f(x), the associated measure for any A is µ(A) = f(x)dx.1 Let now be an
∈ A A P0
arbitrary family of probability measures defined on such that µ = . Finally, let
0 1
A P ∩{ } ∅
us define a family of non-trivial two-component mixtures as
( ,µ ) = αµ +(1 α)µ : µ , α (0,1) . (2)
0 1 1 0 0 0
F P { − ∈ P ∈ }
For ( ,µ ) to be identifiable, there needs to be a one-to-one mapping between the set of
0 1
F P
parameters (α,µ ) (0,1) and µ ( ,µ ); that is, each µ must correspond to a
0 0 0 1
∈ ×P ∈ F P
unique set of parameters. We formalize this in the following definition and Lemma 2.
Definition 1 (Identifiability) For any non-empty set of probability measures and any
0
P
probability measure µ on such that µ = , ( ,µ ) is said to be identifiable if
1 0 1 0 1
A P ∩{ } ∅ F P
a,b (0,1) and λ ,ν
0 0 0
∀ ∈ ∀ ∈ P
aµ +(1 a)λ = bµ +(1 b)ν = (a,λ ) = (b,ν ).
1 0 1 0 0 0
− − ⇒
Similarly, ( ,µ ) is said to be identifiable in α if a,b (0,1) and λ ,ν ,
0 1 0 0 0
F P ∀ ∈ ∀ ∈ P
aµ +(1 a)λ = bµ +(1 b)ν = a = b.
1 0 1 0
− − ⇒
Fortunately,usingthefollowinglemma,wecanfocusonidentifiabilityintermsofthemixing
proportion, which is the parameter we are interested in estimating. Therefore, after Lemma
2 we will drop the qualification “in α” when discussing identifiability of ( ,µ )
0 1
F P
1. Moretechnically,µ(A)=(cid:82) f(x)L(dx)forallA∈B(Rd),whereB(Rd)istheBorelσ-algebraonRd and
A
L is the Lebesgue measure on Rd.
3
Jain, White, Trosset and Radivojac
Table 1: A summary of notation.
Symbol Definition
µ, µ , µ Probability measures: mixture (µ), component one (µ ), component zero (µ )
1 0 1 0
f, f , f Probability density functions: mixture (f), component one (f ), component zero (f )
1 0 1 0
α Mixing proportion, class prior
A Sigma algebra
P An arbitrary family of distributions from which µ is selected. P ∩{µ }=∅
0 0 0 1
Pall The family of all distributions, except µ , on A from which µ is selected. Pall∩{µ }=∅
0 1 0 0 1
F(P ,µ ) F(P ,µ )={αµ +(1−α)µ :µ ∈P , α∈(0,1)}
0 1 0 1 1 0 0 0
Q Q =Pall\F(P ,µ )
0 0 0 0 1
A(µ,µ ,P ) A(µ,µ ,P )={α∈(0,1):µ=αµ +(1−α)µ , whereµ ∈P }
1 0 1 0 1 0 0 0
R(µ,µ1) R(µ,µ1)={µ(A)/µ1(A):A∈A, µ1(A)>0}
α∗ α∗ =infR(µ,µ ); A(µ,µ ,Pall)=(0,α∗]
1 1 0
µ∗ The particular µ when α=α∗
0 0
Lemma 2 ( ,µ ) is identifiable if and only if ( ,µ ) is identifiable in α.
0 1 0 1
F P F P
Proof If ( ,µ ) is identifiable, then a,b (0,1) and λ ,ν
0 1 0 0 0
F P ∀ ∈ ∀ ∈ P
aµ +(1 a)λ = bµ +(1 b)ν (a,λ ) = (b,ν ) a = b
1 0 1 0 0 0
− − ⇒ ⇒
and so ( ,µ ) is identifiable in α.
0 1
F P
If ( ,µ ) is identifiable in α, then for any a,b (0,1) and λ ,ν where
0 1 0 0 0
F P ∈ ∈ P
aµ +(1 a)λ = bµ +(1 b)ν , we know a = b. Replacing b with a, we obtain
1 0 1 0
− −
aµ +(1 a)λ = bµ +(1 b)ν = aµ +(1 a)λ = aµ +(1 a)ν
1 0 1 0 1 0 1 0
− − ⇒ − −
= λ = ν
0 0
⇒
Therefore, (a,λ ) = (b,ν ) and, by definition, ( ,µ ) is identifiable.
0 0 0 1
F P
Unfortunately, in general, ( ,µ ) is not identifiable. For µ ( ,µ ), we define the
0 1 0 1
F P ∈ F P
following set
A(µ,µ , ) = α (0,1) : µ = αµ +(1 α)µ , where µ . (3)
1 0 1 0 0 0
P { ∈ − ∈ P }
For ( ,µ ) to be identifiable, A(µ,µ , ) must be a singleton for every µ ( ,µ ).
0 1 1 0 0 1
F P P ∈ F P
In Lemma 4, we demonstrate that this is not the case for all, the set of all measures on
P0
except µ ; in fact, we will see that every mixture in ( all,µ ) corresponds to an entire
A 1 F P0 1
interval of choices for α. We next present Theorem 3, which enables us to remedy this
non-identifiability issue by restricting µ to a smaller family, without losing the modeling
0
flexibility of ( all,µ ). We will first discuss the ramifications of this theorem and our
F P0 1
proposed max-canonical form; we prove the theorem and required lemmas at the end of the
section. Table 1 provides the summary of notation for quick reference.
Theorem 3 For = all ( all,µ ) the following hold:
Q0 P0 \F P0 1
1. ( ,µ ) is identifiable and contains the same mixtures as ( all,µ ).
F Q0 1 F P0 1
2. For any µ ( all,µ ), there exists α∗ (0,1) such that A(µ,µ , all) = (0,α∗] and
∈ F P0 1 ∈ 1 P0
there exists µ∗ such that µ = α∗µ +(1 α∗)µ∗.
0 ∈ Q0 1 − 0
4
Nonparametric semi-supervised learning of class proportions
likelihood
A(¹,¹ ,Pall) = (0,®*]
1 0
0 ®* 1 class prior
Figure 1: Stylized depiction of the likelihood as a function of the mixing proportion. The
truemixingproportionisnotidentifiableandliesintheregionA(µ,µ , all). The
1 P0
end of the interval α∗ is identifiable and preserves representation of ( all,µ ).
F P0 1
A procedure estimating the likelihood is expected to show flat likelihood for any
α A(µ,µ , all) and then decrease after α∗.
∈ 1 P0
3. For any α < α∗ with µ such that µ = αµ + (1 α)µ , µ is a non-trivial mix-
0 1 0 0
−
ture containing µ and the distance between α and the upper bound α∗ is smoothly
1
related to the proportion of µ in µ ; that is, µ ( all,µ ) and α∗ α = (1
1 0 0 ∈ F P0 1 − −
α)maxA(µ ,µ , all).
0 1 P0
Interestingly, the two families ( ,µ ) and ( all,µ ) contain the same mixtures, yet
F Q0 1 F P0 1
( ,µ )isidentifiableand ( all,µ )isnot. Importantly,Theorem3suggestsacanonical
F Q0 1 F P0 1
form for the estimation of the mixing proportion that ensures identifiability by selecting
α∗ = maxA(µ,µ , all). With this max-canonical form, estimation is implicitly restricted
1 P0
to the identifiable set ( ,µ ), while maintaining the ability to model any mixture in
0 1
F Q
( all,µ ). This canonical form is intuitive in that it prefers µ that is not composed of
F P0 1 0
µ . The reason for the lack of identifiability is that we can always shift some portion of µ
1 1
into µ until all weight is on µ (i.e., α = 0). Therefore, choosing the maximum α selects
0 0
the most separated µ and µ . Moreover, statement 3 indicates that even if in practice µ
0 1 0
really does have a small proportion of µ , the over-estimate of α smoothly relates to this
1
small proportion.
Blanchard et al. (2010) give an identifiability result that captures several aspects of
Theorem 3; precisely, in identifying ’s role towards identifiability and establishing α∗ as
0
Q
the maximum mixing proportion. Our theorem additionally shows that there is no loss in
the modeling flexibility by restricting to and that the set of all valid mixing proportions
0
Q
is actually the interval (0,α∗]. This interval is significant because it directly informs our
algorithm development in Section 3. Our theorem also quantifies the error of α∗ when
µ / .
0 0
∈ Q
Non-identifiability of the mixing proportion suggests an interval of equally likely solu-
tions using inference techniques such as maximum likelihood, where µ and µ are approx-
1
imated using (integrals of) density or mass functions. The approximation is expected to
5
Jain, White, Trosset and Radivojac
progressively deteriorate as the mixing proportion increases beyond α∗. Figure 1 illustrates
the expected relationship between a likelihood function and the mixing proportion.
2.1 Proof of Theorem 3
We first prove the following two lemmas.
Lemma 4 For any µ ( all,µ ), let
∈ F P0 1
R(µ,µ1) = µ(A)/µ1(A) : A , µ1(A) > 0 .
{ ∈ A }
Then, there exists
α∗ = infR(µ,µ ) (0,1) and A(µ,µ , all) = (0,α∗].
1 ∈ 1 P0
Hence, ( all,µ ) is nonidentifiable. Moreover, α∗ can be defined in terms of the densities
F P0 1
f,f1 corresponding to µ,µ1, if they exist; precisely, α∗ = inf f(x)/f1(x) : x , f1(x) > 0 .
{ ∈ X }
Proof
Part 1: First, we show that α∗ is well defined and it is in (0,1). R(µ,µ ) is non-empty,
1
because there exists A with µ (A) > 0; thus, α∗ is well defined. Now, because
1
∈ A
µ ( all,µ ), there exists b (0,1) and λ all such that µ = bµ +(1 b)λ . For any
∈ F P0 1 ∈ 0 ∈ P0 1 − 0
A such that µ (A) > 0,
1
∈ A
µ(A) bµ (A)+(1 b)λ (A) λ (A)
1 0 0
= − = b+(1 b) b
µ (A) µ (A) − µ (A) ≥
1 1 1
giving α∗ b > 0. To show that α∗ < 1, suppose there exists µ and µ such that α∗ 1.
1
≥ ≥
Then because α∗ is the infimum of R(µ,µ )
1
µ(A)
1 when µ (A) > 0.
1
µ (A) ≥
1
Moreover, when µ (A) = 0, clearly µ(A) µ (A). Therefore, µ(A) µ (A) for all A .
1 1 1
≥ ≥ ∈ A
Now because µ / ( all,µ ), µ = µ and so there exists B such that µ(B) > µ (B).
1 ∈ F P0 1 (cid:54) 1 ∈ A 1
This leads to a contradiction as follows:
1 = µ( ) = µ( B)+µ(B)
X X \
> µ ( B)+µ (B)
1 1
X \
= 1
Thus α∗ < 1 and α∗ (0,1).
∈
Part 2: Second, we show that A(µ,µ , all) (0,α∗]. Recall that A(µ,µ , all) =
(cid:8)α (0,1) : µ = αµ +(1 α)µ , where µ1 P0 all(cid:9)⊇. We need to show that if a1 P(00,α∗],
∈ 1 − 0 0 ∈ P0 ∈
then there exists λ all such that µ = aµ +(1 a)λ . Consider
0 ∈ P0 1 − 0
µ aµ
1
λ = − .
0
1 a
−
6
Nonparametric semi-supervised learning of class proportions
Because a α∗ < 1, λ is well defined. Moreover, λ = µ because equality would imply
0 0 1
≤ (cid:54)
µ = µ (trivial mixture). If we can show λ is a probability measure, then λ all and
1 0 0 ∈ P0
Part 2 is complete.
λ (A) 0 A : By definition of α∗, for all A with µ (A) > 0, we have
0 1
≥ ∀ ∈ A ∈ A
µ(A) α∗µ (A) aµ (A) because a (0,α∗]. The inequality is trivial when µ (A) = 0.
1 1 1
≥ ≥ ∈
Thus, µ(A) aµ (A) 0 for all A ; consequently, λ (A) 0.
1 0
− ≥ ∈ A ≥
λ ( ) = 1: Because µ( ) = 1 and µ ( ) = 1,
0 1
X X X
µ( ) aµ ( ) 1 a
1
λ ( ) = X − X = − = 1.
0
X 1 a 1 a
− −
(cid:80)
Similarly, λ ( ) = 0 and λ ( A ) = λ (A ). Therefore, λ is a probability measure.
0 ∅ 0 ∪ i i 0 i 0
Part 3: Third, we show that A(µ,µ , all) (0,α∗]. Take any a A(µ,µ , all)
1 P0 ⊆ ∈ 1 P0
and corresponding λ all such that µ = aµ + (1 a)λ . For any A , we know
0 ∈ P0 1 − 0 ∈ A
(1 a)λ0(A) 0 and so µ(A) aµ1(A). Thus, for all A with µ1(A) > 0, µ(A)/µ1(A) a
− ≥ ≥ ∈ A ≥
and consequently a α∗.
≤
In summary, A(µ,µ , all) = (0,α∗] and it is not a singleton set for any µ. Therefore, we
1 P0
conclude that ( all,µ ) is not identifiable.
F P0 1
Part 4: Fourth, we show that
f αf
1
f = −
0
1 α
−
is a valid probability density, if and only if α (0,α∗]
∈
“ =” Any α A(µ,µ , all) = (0,α∗] is a valid mixture; i.e., there exists µ all such
⇐ ∈ 1 P0 0 ∈ P0
that µ = αµ +(1 α)µ . Because µ is a well defined measure and it can be expressed
1 0 0
−
as µ0 = (µ−αµ1)/(1−α), its probability density, h0, can be defined in terms of the densities
corresponding to µ and µ1 as follows: h0 = (f−αf1)/(1−α). Thus f0 = h0 and it is a well
defined probability density.
“= ” We give a proof by contradiction. Suppose f is a well defined probability
0
⇒
density for some α > α∗. Then µ , the probability measure corresponding f , can be
0 0
expressed in terms of probability measures corresponding to densities f and f as follows:
1
µ0 = (µ−αµ1)/(1−α); consequently, µ = αµ1+(1 α)µ0. Thus α is a valid mixing proportion,
−
i.e., α A(µ,µ , all) = (0,α∗], which gives the contradiction.
∈ 1 P0
Part 5: Next, we show that α∗ = inf f(x)/f1(x) : x , f1(x) > 0 . We give a proof by
{ ∈ X }
contradiction. Suppose α∗∗ = inf f(x)/f1(x) : x , f1(x) > 0 = α∗.
{ ∈ X } (cid:54)
If α∗∗ < α∗: Using the definition of α∗∗, there exists x with f (x) > 0 and
1
∈ X
f(x)/f1(x) < α∗. Thus f(x) α∗f1(x) < 0 and (f−α∗f1)/(1−α∗) is not a probability density
−
function, which contradicts part 4.
If α∗∗ > α∗: Using the definition of α∗∗, α∗∗ f(x)/f1(x) when f1(x) > 0. Thus
≤
f(x) α∗∗f (x) 0 when f (x) > 0. The inequality is trivially true when f (x) = 0.
1 1 1
− ≥ (cid:82)
Consequently f0(x) = (f(x)−α∗∗f1(x))/(1−α∗∗) ≥ 0 for all x ∈ X. Moreover, X f0(x)dx = 1.
Thus f is a well defined probability density function, which contradicts part 4.
0
Thus α∗∗ = α∗.
Lemma 5 ( ,µ ) is identifiable if and only if ( ,µ ) = .
0 1 0 1 0
F P F P ∩P ∅
7
Jain, White, Trosset and Radivojac
Proof “ =” We give a proof by contradiction. Suppose ( ,µ ) = , but ( ,µ )
0 1 0 0 1
⇐ F P ∩P ∅ F P
is not identifiable. Therefore, there exists a,b (0,1) and λ ,ν such that aµ +(1
0 0 0 1
∈ ∈ P −
a)λ = bµ +(1 b)ν , but a = b. Without loss of generality we can assume a > b. We now
0 1 0
− (cid:54)
have
aµ +(1 a)λ = bµ +(1 b)ν
1 0 1 0
− −
ν0 = (a−b)/(1−b)µ1+(1−a)/(1−b)λ0 = (a−b)/(1−b)µ1+(1 (a−b)/(1−b))λ0
⇒ −
Because (a−b)/(1−b) (0,1) and λ0 0, it follows that ν0 ( 0,µ1). Moreover, ν0 was
∈ ∈ P ∈ F P
picked from . Therefore, ( ,µ ) contains ν and so is not empty, which is a
0 0 1 0 0
P F P ∩ P
contradiction.
“= ” Again, we give a proof by contradiction. Suppose ( ,µ ) is identifiable, but
0 1
⇒ F P
( ,µ ) = . Let λ ( ,µ ) . Because λ ( ,µ ), there exists ν
0 1 0 0 0 1 0 0 0 1 0 0
F P ∩P (cid:54) ∅ ∈ F P ∩P ∈ F P ∈ P
and c (0,1) such that λ0 = cµ1 + (1 c)ν0. Let a (c,1) and b = (a−c)/(1−c). As
∈ − ∈
a,b (0,1) and ν ,λ , it follows that aµ +(1 a)ν ,bµ +(1 b)λ ( ,µ ).
0 0 0 1 0 1 0 0 1
∈ ∈ P − − ∈ F P
First, we can see that aµ +(1 a)ν = bµ +(1 b)λ because
1 0 1 0
− −
bµ +(1 b)λ = bµ +(1 b)(cµ +(1 c)ν )
1 0 1 1 0
− − −
= (b+(1 b)c)µ +(1 b)(1 c)ν
1 0
− − −
= (b(1 c)+c)µ +(1 b(1 c) c)ν
1 0
− − − −
= aµ +(1 a)ν (cid:46)a = b(1 c)+c.
1 0
− −
Because a = b(1 c)+c is a convex combination of b and 1, we know a (b,1), giving
− ∈
a > b and so a = b. Therefore aµ +(1 a)ν = bµ +(1 b)λ , but (a,ν ) = (b,λ ). It
1 0 1 0 0 0
(cid:54) − − (cid:54)
follows ( ,µ ) is not identifiable, which is a contradiction.
0 1
F P
[ProofofTheorem1:] ToproveStatement2weuseLemma4. Foranyµ ( all,µ ),
∈ F P0 1
we know there exists an α∗ (0,1) such that A(µ,µ , all) = (0,α∗]. This means that
∈ 1 P0
α∗ is a valid mixing proportion for mixture µ. Thus, there exists µ∗ all such that
0 ∈ P0
µ = α∗µ +(1 α∗)µ∗. To show that µ∗ , we need to show that µ∗ / ( all,µ ).
1 − 0 0 ∈ Q0 0 ∈ F P0 1
Using a proof by contradiction, assume µ∗ ( all,µ ). For some a (0,1) and λ all,
0 ∈ F P0 1 ∈ 0 ∈ P0
µ∗ = aµ +(1 a)λ , giving
0 1 − 0
µ = α∗µ +(1 α∗)(aµ +(1 a)λ )
1 1 0
− −
= (α∗+(1 α∗)a)µ +(1 α∗)(1 a)ν
1 0
− − −
= (α∗+(1 α∗)a)µ +(1 (α∗+(1 α∗)a))λ (cid:46)α = α∗+(1 α∗)a
1 0
− − − −
Therefore, α A(µ,µ , all) but α = α∗+(1 α∗)a > α∗, which is a contradiction.
∈ 1 P0 −
To prove Statement 1 we use Lemma 5.
all ( ,µ ) ( all,µ )
Q0 ⊆ P0 ⇒ F Q0 1 ⊆ F P0 1
( ,µ ) ( all,µ ) = .
⇒ Q0∩F Q0 1 ⊆ Q0∩F P0 1 ∅
Therefore, ( ,µ ) = and so by Lemma 5, ( ,µ ) is identifiable.
0 0 1 0 1
Q ∩F Q ∅ F Q
To prove that ( ,µ ) and ( all,µ ) contain the same set of mixtures, we need to
F Q0 1 F P0 1
show that ( ,µ ) ( all,µ ) and ( all,µ ) ( ,µ ), where here we use to
F Q0 1 ⊆ F P0 1 F P0 1 ⊆ F Q0 1 ⊆
8
Nonparametric semi-supervised learning of class proportions
mean contains a subset of the same unique probability measures. It is already clear that
( ,µ ) ( all,µ ) and ( all,µ ) ( ,µ ) follows from Statement 2, because
F Q0 1 ⊆ F P0 1 F P0 1 ⊆ F Q0 1
any µ ( all,µ ) can be represented as a mixture of µ and some µ∗ with α∗ giving
∈ F P0 1 1 0 ∈ Q0
the mixing proportion.
To prove Statement 3 From statement 2, for some µ∗ ,
0 ∈ Q0
αµ +(1 α)µ = α∗µ +(1 α∗)µ∗
1 − 0 1 − 0
⇒ µ0 = (α∗−α)/(1−α)µ1+(1−(α∗−α)/(1−α))µ∗0.
Because (α∗−α)/(1−α) ∈ (0,1), µ0 ∈ F(P0all,µ1). Let a = maxA(µ0,µ1,P0all) the maximum
proportion of µ in µ , with corresponding λ all such that µ = aµ +(1 a)λ . Then,
1 0 0 ∈ P0 0 1 − 0
µ = αµ +(1 α)µ
1 0
−
= αµ +(1 α)(aµ +(1 a)λ )
1 1 0
− −
= (α+a αa)µ +(1 α a+αa)λ .
1 0
− − −
Because the choice of a ensures the maximum proportion on µ for representing µ , λ
1 0 0
cannot be expressed as a mixture containing µ . Therefore,
1
α∗ = α+a αa = α+(1 α)a
− −
giving
α∗ α = (1 α)maxA(µ ,µ , all) < maxA(µ ,µ , all).
− − 0 1 P0 0 1 P0
3. Algorithm development
In this section, we formulate the estimation of the mixing proportion in terms of a new set
of parameters, β. We then develop an efficient algorithm by taking advantage of the special
optimization surface over α, elucidated by Theorem 3 and depicted in Figure 1.
Let X be an i.i.d. sample from the first component and X an i.i.d. sample from the
1
mixture. To approximate the mixture, a common approach is to use
k k
(cid:88) (cid:88)
fˆ(x) = w κ (x) w (0,1), w = 1, (4)
i i i i
∈
i=1 i=1
where κ ’s are probability density functions (pdfs) or probability mass functions (pmfs).
i
For continuous random variables, for example, a typical setting is a multivariate Gaussian,
κ (x) exp( x x 2/σ2) with learned or predefined centers x as in mixture models or
i ∝ −(cid:107) − i(cid:107)2 i
examples as centers as in kernel density estimation. For discrete random variables, the κ ’s
i
could be multinomials.
To relate f to f, we similarly approximate f by re-weighting the kernels. To do so, we
1 1
introduce a vector-valued variable β = (β ,...,β ), where β (0,1], and define
1 k i
∈
(cid:80)k
β w κ (x)
h (x β) = i=1 i i i
1 | (cid:80)k β w
i=1 i i
9
Jain, White, Trosset and Radivojac
and
fˆ(x) ((cid:80)k β w )h (x β) (cid:80)k (1 β )w κ (x)
h (x β) = − i=1 i i 1 | = i=1 − i i i
0 | 1 (cid:80)k β w (cid:80)k (1 β )w
− i=1 i i i=1 − i i
giving
(cid:32) k (cid:33) (cid:32) k (cid:33)
(cid:88) (cid:88)
fˆ(x) = β w h (x β)+ 1 β w h (x β). (5)
i i 1 i i 0
| − |
i=1 i=1
Notice that (cid:80)k β w 1 and, thus, h (x β) and h (x β) are pmfs for discrete x and pdfs
i=1 i i ≤ 1 | 0 |
forcontinuousx. Itfollowsthatfˆisamixturewithcomponentsh (x β)andh (x β),where
1 0
(cid:80)k | |
α = β w . Intuitively, the β ’s are larger if f is more similar to f; correspondingly,
i=1 i i i 1
(cid:80)k
the proportion β w should be larger, because f accounts for more of f.
i=1 i i 1
Our goal is to obtain an estimate for β with maximal (cid:80)k β w that minimizes the
i=1 i i
KL-divergence to the true distributions, KL(f h ( β)) and KL(f fˆ); i.e., maximizes the
1 1
likelihoodofh (x β)undersampleX andtheli|k|elih·o|odoffˆ(x)und|e|rsampleX. Notethat
1 1
fˆ(x) cannot be u|sed in the likelihood function directly because it is not really a function
of β, as they algebraically cancel. Thus, we define h(x β), similar to fˆ(x), obtained by
replacing h (x β) in Equation 5 by fˆ(x), an estimate of|f obtained from X :
1 1 1 1
|
(cid:32) k (cid:33) (cid:32) k (cid:33)
(cid:88) (cid:88)
h(x β) = β w fˆ(x)+ 1 β w h (x β). (6)
i i 1 i i 0
| − |
i=1 i=1
The combined log-likelihood of β under these models is
(β X,X ) = γ (β X)+γ (β X ) (7)
1 1 1
L | L | L |
where
(cid:88)
(β X ) = logh (x β)
1 1
L | |
x∈X1
(cid:88)
(β X) = logh(x β)
L | |
x∈X
andγ andγ arenonnegativecoefficients. Wewilllaterexploretwoscenarios: (i)γ = γ = 1
1 1
that equally weights each example, and (ii) γ = 1/|X|, γ1 = 1/|X1| that equally weights each
sample. Since the weights do not influence the remainder of this section, we will simply
assume that γ = γ = 1.
1
There are two remaining issues for the problem specification: concavity and enforcing
themax-canonicalform. First, although (β X)isconcave, (β X )mightnotbeconcave,
1
L | L |
because
k k
(cid:88) (cid:88)
logh (x β) = log β w κ (x) log β w
1 i i i i i
| −
i=1 i=1
10