Dirichlet draws are sparse with high probability

0.31 MB·English

by  Matus Telgarsky

Dirichlet draws are sparse with high probability
Matus Telgarsky
Abstract
This note provides an elementary proof of the folklore fact that draws from a Dirichlet
3
distribution (with parameters less than 1) are typically sparse (most coordinates are small).
1
0
2
1 Bounds

n
a
Let Dir(α) denote a Dirichlet distribution with all parameters equal to α.
J

1 Theorem 1.1. Suppose n ≥ 2 and (X ,...,X ) ∼ Dir(1/n). Then, for any c ≥ 1 satisfying
1 n 0
2
6c ln(n)+1<3n,
G] 0 Pr(cid:20)(cid:12)(cid:12)(cid:12)(cid:12)(cid:26)i:Xi ≥ n1c0(cid:27)(cid:12)(cid:12)(cid:12)(cid:12)≤6c0ln(n)(cid:21)≥1− n1c0.
L
The parameter is taken to be 1/n, which is standard in machine learning. The above theo-
.
s rem states that (with high probability) as the exponent on the sparsity threshold grows linearly
c
[ (n−1,n−2,n−3,...), the number of coordinates above the threshold cannot grow faster than linearly
(6ln(n),12ln(n),18ln(n),...).
1
The above statement can be parameterized slightly more finely, exposing more tradeoffs than
v
just the threshold and number of coordinates.
7
1
Theorem 1.2. Suppose n ≥ 1 and c ,c ,c > 0 with c ln(n) + 1 < 3n, and (X ,...,X ) ∼
9 1 2 3 2 1 n
4 Dir(c1/n); then
.
1
0 Pr(cid:2)|{i:X ≥n−c3}|≤c ln(n)(cid:3)≥1− 1 (cid:18)1(cid:19)c32−c1c3 − 1 (cid:18)1(cid:19)4c92 .
3 i 2 e1/3 n e4/9 n
1
:
v The natural question is whether the factor ln(n) is an artifact of the analysis; simulation exper-
i iments with Dirichlet parameter α = 1/n, summarized in Figure 1a, exhibit both the ln(n) term,
X
and the linear relationship between sparsity threshold and number of coordinates exceeding it.
r
a The techniques here are loose when applied to the case α = o(1/n). In particular, Figure 1b
suggests α = 1/n2 leads to a single nonsmall coordinate with high probability, which is stronger
than what is captured by the following theorem.
Theorem 1.3. Suppose n≥3 and (X ,...,X )∼Dir(1/n2); then
1 n
Pr(cid:2)|{i:X ≥n−2}|≤5(cid:3)≥1−e2/e−2−e−8/3 ≥0.64.
i
Moreover, for any function g :Z →R and any n satisfying 1≤ln(g(n))<3n−1,
++ ++
Pr(cid:2)|{i:X ≥n−2}|≤ln(g(n))(cid:3)≥1−e2/e−1/3(cid:18) 1 (cid:19)1/3−e−4/9(cid:18) 1 (cid:19)4/9.
i g(n) g(n)
(Take for instance g to be the inverse Ackermann function.)
1
Dir(n−1) Dir(n−2)
ln(n)ormalized by 4453....0505 ††††====nnnn−−−−1234 †eeding 221...028 ††††====nnnn−−−−1234
n c
†ng , 3.0 es ex1.6
ceedi2.5 dinat
nates ex21..05 #cooor1.4
di 1.2
oor1.0
o
c
#
0.5 1.0
0 200 400 600 800 1000 0 20 40 60 80 100
#dimensions (n) #dimensions (n)
Figure 1: For each Dirichlet parameter choice α ∈ {n−1,n−2} and each number of dimensions n
(horizontalaxis),1000Dirichletdistributionsweresampled. Foreachtrial,thenumberofcoordinates
exceeding each of 4 choices of threshold were computed. In the case of α=n−1, these counts were
thenscaledbyln(n)tobettercoordinatewiththesuggestedtrendsinTheorems1.1and1.2. Finally,
these counts values (for each (n,(cid:15))) were converted into quantile curves (25%–75%).
2 Proofs
Theorems 1.1 to 1.3 are established via the following lemma.
Lemma 2.1. Let reals (cid:15)∈(0,1] and α>0 and positive integers k,n be given with k+1<3n. Let
(X ,...,X )∼Dir(α). Then
i n
Pr(cid:2)|{i:X ≥(cid:15)}|≤k(cid:3)≥1−(cid:15)−nαe−(k+1)/3−e−4(k+1)/9.
i
The proof avoids dependencies between the coordinates of a Dirichlet draw via the following
alternate representation. Throughout the rest of this section, let Gamma(α) denote a Gamma
distribution with parameter α.
Lemma 2.2. (See for instance Balakrishnan and Nevzorov (2003, Equation 27.17).) Let α>0 and
n≥1 be given. If (X ,...,X )∼Dir(α) and {Y }n are n i.i.d. copies of Gamma(α), then
1 n i i=1
(cid:26) (cid:27)
Y
(X ,...,X )=d i .
1 n (cid:80)n Y
i=1 i
Before turning to the proof of Lemma 2.1, one more lemma is useful, which will allow a control
of the Gamma distribution’s cdf.
Lemma 2.3. For any α>0, c≥0, and z ≥1,
Pr[Gamma(α)≤zc]≤zαPr[Gamma(α)≤c].
2
Proof. Since e−zx ≤e−x for every x≥0 and z ≥1,
1 (cid:90) zc
Pr[Gamma(α)≤zc]= e−xxα−1dx
Γ(α)
0
1 (cid:90) c
= e−zx(zx)α−1zdx
Γ(α)
0
zα (cid:90) c
≤ e−xxα−1dx
Γ(α)
0
=zαPr[Gamma(α)≤c].
Proof of Lemma 2.1. Since z (cid:55)→Pr[Gamma(α)≥z] is continuous and has range [0,1], choose c≥0
so that
k+1
Pr[Gamma(α)>c]=Pr[Gamma(α)≥c]= , (2.4)
3n
where (k+1)/(3n)<1. By this choice and Lemma 2.3,
(cid:18) (cid:19)
k+1
Pr[Gamma(α)≤c/(cid:15)]≤(cid:15)−αPr[Gamma(α)≤c]=(cid:15)−α 1− ≤(cid:15)−αe−(k+1)/(3n). (2.5)
3n
Now let {Y }n be n i.i.d. copies of Gamma(α). Define the events
i i=1
A:=[∃i∈[n](cid:5)Y ≥c/(cid:15)] and B :=[|{i∈[n]:Y ≤c}|≥n−k].
i i
The remainder of the proof will establish a lower bound on Pr(A∧B). To see that this finishes the
(cid:80)
proof, define S := Y ; sinceeventA impliesthat S ≥c/(cid:15), itfollowsthat Y ≤c implies Y /S ≤(cid:15).
i i i i
Consequently, events A and B together imply that Y /S ≤ (cid:15) for at least n−k choices of i. By
i
Lemma 2.2, it follows that Pr(A∧B) is a lower bound on the event that a draw from Dir(α) has at
least n−k coordinates which are at most (cid:15).
Returning to task, note that
Pr(A∧B)=1−Pr(¬A∨¬B)≥1−Pr(¬A)−Pr(¬B). (2.6)
To bound the first term, by eq. (2.5),
Pr(¬A)=Pr[∀i∈[n](cid:5)Y <c/(cid:15)]=Pr[Y ≤c/(cid:15)]n ≤(cid:15)−nαe−(k+1)/3. (2.7)
i 1
For the second term, define indicator random variables Z :=[Y >c], whereby
i i
k+1
E(Z )=Pr[Z =1]=Pr[Y >c]=Pr[Y ≥c]= .
i i i i 3n
Then, by a multiplicative Chernoff bound (Kearns and Vazirani, 1994, Theorem 9.2),
(cid:34) (cid:35)
(cid:88)
Pr(¬B)=Pr[|{i∈[n]:Y >c}|≥k+1]=Pr Z ≥3nE(Z ) ≤exp(−4nE(Z )/3). (2.8)
i i i i
i
Inserting (2.7) and (2.8) into the lower bound on Pr(A∧B) in (2.6),
Pr(A∧B)≥1−(cid:15)−nαe−(k+1)/3−e−4(k+1)/9.
Proof of Theorem 1.2. Instantiate Lemma 2.1 with k =c2ln(n), α=c1/n, and (cid:15)=n−c3.
3
Proof of Theorem 1.1. Instantiate Theorem 1.2 with c =1, c =6c , c =c , and note
1 2 0 3 0
1 (cid:18)1(cid:19)c0 1 (cid:18)1(cid:19)8c30 1 (cid:32) 1 1 (cid:18)1(cid:19)5c30(cid:33) 1
+ ≤ + ≤ .
e1/3 n e4/9 n nc0 e1/3 e4/9 2 nc0
Proof of Theorem 1.3. Define the function f(z) := z−z over (0,∞). Note that f(cid:48)(z) = −(ln(z)+
1)z−z, which is positive for z < 1/e, zero at z = 1/e, and negative thereafter; consequently,
sup f(z) = f(1/e) = e1/e. As such, instantiating Lemma 2.1 with (cid:15) = n−2, α = n−3,
z∈(0,∞)
and any k <3n−1 gives
Pr(cid:2)|{i:X ≥n−2}|≤k(cid:3)≥1−n2/ne−(k+1)/3−e−4(k+1)/9
i
≥1−e2/ee−(k+1)/3−e−4(k+1)/9.
Plugging in k ∈{5,ln(g(n))} gives the two bounds.
Acknowledgement
The author thanks Anima Anandkumar and Daniel Hsu for relevant discussions.
References
N.BalakrishnanandV.B.Nevzorov. Aprimeronstatisticaldistributions. Wiley-Interscience,2003.
M. J. Kearns and U. V. Vazirani. An introduction to computational learning theory. MIT Press,
Cambridge, MA, USA, 1994.
4