Table Of ContentComprehend DeepWalk as Matrix Factorization
Cheng Yang Zhiyuan Liu
Tsinghua University Tsinghua University
[email protected] [email protected]
January 5, 2015
5
1
0
2 Abstract
Word2vec [2], as an efficient tool for learning vector representation of
n
a words has shown its effectiveness in many natural language processing
J tasks. Mikolov et al. issued Skip-Gramand NegativeSampling model [3]
2 for developing this toolbox. Perozzi et al. introduced the Skip-Gram
modelintothestudyofsocial networkforthefirsttime,anddesigned an
] algorithm named DeepWalk [6] for learning node embedding on a graph.
G
Weprove that the DeepWalk algorithm is actually factoring a matrix M
L where each entry Mij is logarithm of theaverage probability that node i
. randomly walks to nodej in fix steps. Wewill explain it in section 3.
s
c
[
1 Notation
1
v
Network G = (V,E). Node-context set D is generated from random walk,
8
where each piece of D is a node-context pair (v,c). V is the set of nodes and
5
3 VC is the set of context nodes. In most cases, V =VC.
0 Consider a node-context pair (v,c):
0 #(v,c)denotesthenumberoftimes(v,c)appearsinD. #(v)= #(v,c′)
1. and #(c)=Pv′∈V #(v′,c) denotes the number of times v and c apPpce′a∈rVsCin D.
0 Note that |D|= #(v′,c′).
5 DeepWalk algPorvi′t∈hVmPemc′∈bVeCds a node v into a d-dimension vector −→v ∈ Rd.
1 Also, a context node c ∈ V is represented by a d-dimension vector −→c ∈ Rd.
: C
v Let W be a |V|×d matrix where rowi is vector−→v and H be a |V |×d matrix
i C
Xi where row j is vector −→cj. Our goal is to figure out a matrix M =WHT.
r
a
2 Proof
Perozzi et al. implemented DeepWalk algorithm with Skip-Gram and Hier-
archical Softmax model. Note that Hierarchical Softmax [5] [4] is a variant of
softmax for speeding the training time. In this section, we give proofs for both
Negative Sampling and softmax with Skip-Gram model.
2.1 Negative Sampling
NegativeSamplingapproximatelymaximizestheprobabilityofsoftmaxfunc-
tion by randomly choosing k negative samples from context set. Levy and
1
Goldberg showedthat Skip-Gramwith Negative Sampling model(SGNS) is im-
plicitly factorizingaword-contextmatrix[1]byassumingthatdimensionalityd
−→ −→
is sufficiently large. In other words, we can assign each product v · c a value
independently of the others.
In SGNS model, we have
−→ −→ 1
P((v,c)∈D)=σ(v · c)=
1+e−→−v·→−c
Suppose we choose k negative samples for each node-context pair (v,c) ac-
cording to the distribution P (c ) = #(cN). Then the objective function for
D N |D|
SGNS can be written as
l = X X #(v,c)(logσ(−→v ·−→c)+k·EcN∼PD[logσ(−−→v ·−→c)])
v∈V c∈VC
−→ −→ #(cN) −→ −→
= X X #(v,c)(logσ(v · c)+k· X#(v) X logσ(−v · c)
|D|
v∈V c∈VC v∈V cN∈VC
−→ −→ #(c) −→ −→
= X X(#(v,c)(logσ(v · c)+k·#(v)· logσ(−v · c)
|D|
v∈V c∈VC
Denote x=−→v ·−→c. By solving ∂l =0, we have
∂x
−→ −→ #(v,c)·|D|
v · c =x=log −logk
#(v)·#(c)
#(vi,cj)
ThuswehaveM =log |D| −logk. M canbeinterpretedasPoint-
ij #(vi)·#(cj) ij
|D| |D|
wise Mutual Information(PMI) of node-context pair (v ,c ) shifted by logk.
i j
2.2 Softmax
SincebothNegativeSamplingandHierarchicalSoftmaxarevariantsofsoft-
max, we pay more attention to softmax model and give a further discussion in
−→ −→
next section. We also assume that the values of v · c are independent.
In softmax model,
e→−v·→−c
P((v,c)∈D)=
e→−v·→−c′
Pc′∈VC
And the objective function is
e→−v·→−c
l = X X #(v,c)·log e→−v·→−c′
v∈V c∈VC Pc′∈VC
−→ −→
After extracting all terms associated to v · c as l(v,c), we have
e→−v·→−c e→−v·→−c˜
l(v,c)=#(v,c)log e→−v·→−c′ +e→−v·→−c + X #(v,c˜)log e→−v·→−c′ +e→−v·→−c
Pc′∈VC,c′6=c c˜∈VC,c˜6=c Pc′∈VC,c′6=c
2
Note that l = 1 l(v,c). Denote x =−→v ·−→c. By solving ∂l = 0
|VC|Pv∈V Pc∈VC ∂x
for all such x, we have
−→ −→ #(v,c)
v · c =x=log +b
v
#(v)
where b can be any real constant since it will be canceled when we compute
v
P((v,c)∈D). Thus we haveM =log#(vi,cj)+b . We will discuss whatM
ij #(vi) vi ij
represents in next section.
3 Discussion
Itis clearthat the method ofsampling node-contextpairs willaffect matrix
M. In this section, we will discuss #(v), #(c) and #(v,c) based on an ideal
|D| |D| #(v)
sampling method for DeepWalk algorithm.
Assume the graph is connected and undirected and window size is t. We
can easily generalize this sampling method to directed graph by only adding
(RW ,RW ) into D.
i j
Algorithm 1 Ideal node-context pair sampling algorithm
Generate an infinite long random walk RW.
Denote RW as the node on position i of RW, where i=0,1,2,...
i
for i=0,1,2,... do
for j ∈[i+1,i+t] do
add (RW ,RW ) into D
i j
add (RW ,RW ) into D
j i
end for
end for
Each appearance of node i will be recorded 2t times in D for undirected
graph and t times for directed graph. Thus we can figure out that #(vi) is the
|D|
frequency of v appears in the random walk, which is exactly the PageRank
i
value of v . Also note that #(vi,vj) is the expectation times that v is observed
i #(vi)/2t j
in left/right t neighbors of v .
i
Denote the transition matrix in PageRank algorithm be A. More formally,
let d be the degree of node i. A = 1 if (i,j)∈E and A =0 otherwise. We
i ij di ij
use e to denote a |V|-dimension row vector where all entries are zero except
i
the i-th entry is 1.
Suppose that we start a random walk from node i and use e to denote the
i
initial state. Then e A is the distribution over all the nodes where j-th entry
i
is the probability that node i walks to node j. Hence j-th entry of e At is the
i
probability that node i walks to node j at exactly t steps. Thus [e (A+A2+
i
···+At)] is the expectation times that v appears in right t neighbors of v .
j j i
Hence
#(v ,v )
i j =2[e (A+A2+···+At)]
i j
#(v )/2t
i
3
#(v ,v ) [e (A+A2+···+At)]
i j i j
=
#(v ) t
i
This equality also holds for directed graph.
By setting b =log2t for all i, M =log #(vi,vj) is logarithmof the expec-
vi ij #(vi)/2t
tation times that v appears in left/right t neighbors of v .
j i
By setting b = 0 for all i, M = log#(vi,vj) = log[ei(A+A2+···+At)]j is
vi ij #(vi) t
logarithm of the averageprobability that node i randomly walks to node j in t
steps.
References
[1] O. Levy and Y. Goldberg.
Neural word embedding as implicit matrix factorization.
In Advances in Neural Information Processing Systems, pages 2177–2185,
2014.
[2] T. Mikolov, K. Chen, G. Corrado,and J. Dean.
Efficient estimation of word representations in vector space.
arXiv preprint arXiv:1301.3781, 2013.
[3] T. Mikolov, I. Sutskever, K. Chen, G. S. Corrado,and J. Dean.
Distributedrepresentationsofwordsandphrasesandtheircompositionality.
In Advances in Neural Information Processing Systems, pages 3111–3119,
2013.
[4] A. Mnih and G. E. Hinton.
A scalable hierarchicaldistributed language model.
In Advances in neural information processing systems, pages 1081–1088,
2009.
[5] F. Morin and Y. Bengio.
Hierarchical probabilistic neural network language model.
In AISTATS, volume 5, pages 246–252.Citeseer, 2005.
[6] B. Perozzi,R. Al-Rfou, and S. Skiena.
Deepwalk: Online learning of social representations.
arXiv preprint arXiv:1403.6652, 2014.
4
