Document Classification with Latent Dirichlet Allocation

109 Pages·2009·2.32 MB·English

Document Classification with Latent
Dirichlet Allocation
István Bíró
Supervisor: András Lukács Ph.D.
Eötvös Loránd University
Faculty of Informatics
Department of Information Sciences
Informatics Ph.D. School
János Demetrovics D.Sc.
Foundations and Methods of Informatics Ph.D. Program
János Demetrovics D.Sc.
Budapest, 2009

AlulírottBíróIstvánkijelentem,hogyeztadoktoriértekezéstmagamkészítettemésabbancsak
amegadottforrásokathasználtamfel. Mindenolyanrészt,amelyetszószerint,vagyazonostar-
talomban, de átfogalmazva más forrásból átvettem, egyértelmu˝en, a forrás megadásával meg-
jelöltem.
Budapest,2009. június30.
BíróIstván
Az értekezés bírálatai és a védésro˝l készült jegyzo˝könyv megtekintheto˝ az Eötvös Loránd Tu-
dományegyetemInformatikaiKaránakDékániHivatalában.

Acknowledgements
Firstly, I wish to thank my supervisors András Lukács and András Benczúr for introducing me
to the topic of data mining. Without their constant support and guidance this thesis would have
neverbeencompleted.
Among my colleagues, I am especially grateful to Jácint Szabó for the successful research
thatwedidtogether. Theseresultshavefinallybecomethemainconstituentsofmythesis. Ialso
expressmyacknowledgementstoCsabaSzepesváriandZoltánSzamonekforthejointworkin
my language modeling research. I would like to thank all of my co-authors for the successful
team work and the numerous fruitful discussions. I am grateful to the Data Mining and Web
Search Group in MTA SZTAKI, which proved to be a pleasant and inspiring environment.
I would like to thank Ana Maguitman for the invitation to the University of South at Bahia
Blanca as well as for the Argentine helpfulness and support during the three months research
period. IammuchindebttoAndrásBenczúr,MátyásBrendelandJácintSzabóforreadingmy
thesisandprovidingusefulcommentsandsuggestions.
My research was partially supported by the projects EU FP7 JUMAS, NKFP-07-A2 and
NKFP ASTOR 2/004/05 projects and the grant OTKA NK72845 of the Hungarian National
ScienceFund.
Last but not least, I would like to say thank you for the trust and love of my wife Ari and
mytwodaughtersVirágandDorkaprovidingmehalfofthesuccessinachievingmygoals.
v

Notation
D documentcollection(corpus),D = {d }M
m m=1
M numberofdocumentsinthecorpus
V vocabulary,V = {t }V
j j=1
V numberoftermsinthevocabulary
K setoflatenttopics
K numberoflatenttopics
N numberofwords(word-positions)inthecorpus
N documentlength,N = (cid:80)M N
m m=1 m
(cid:126)
α(cid:126),β,(cid:126)γ Dirichlethyperparameters
(cid:126)
ϑ topicmixturevectorfordocumentd
m m
Θ topicmixturematrixforthewholecorpus,Θ = {ϑ(cid:126) }M
m m=1
ϕ(cid:126) wordmixturecomponentoftopick
k
Φ wordmixturematrix,Φ = {ϕ(cid:126) }K
k k=1
z mixtureindicatorthatchoosesthetopicforthenth wordindocumentd
m,n m
w termindicatorforthenth wordindocumentd
m,n m
Abbreviations
AUC AreaUnderCurve(relatedtotheReceiverOperatorCharacteristic)
IR InformationRetrieval
F F-measure
LDA LatentDirichletAllocation
LSA/LSI LatentSemanticAnalysis/Indexing
ODP OpenDirectoryProject
PLSA/PLSI ProbabilisticLatentSemanticAnalysis/Indexing
SVD SingularValueDecomposition
SVM SupportVectorMachine
vii

Contents
1 Introduction 1
1.1 Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1
1.2 Ourcontribution . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2
1.2.1 ComparativeAnalysisofLSAandLDA . . . . . . . . . . . . . . . . . 2
1.2.2 Multi-corpusLDA . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3
1.2.3 LinkedLDA . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4
1.2.4 FastGibbssamplers . . . . . . . . . . . . . . . . . . . . . . . . . . . 5
1.2.5 WebSpamFiltering . . . . . . . . . . . . . . . . . . . . . . . . . . . 6
1.3 Organization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7
2 Documentmodelingandclassification 9
2.1 Preprocessing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12
2.2 Textrepresentation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13
2.2.1 Vectorspacemodel . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13
2.2.2 LatentSemanticAnalysis . . . . . . . . . . . . . . . . . . . . . . . . 14
2.3 Classifierlearning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16
2.3.1 BayesianNetworks . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16
2.3.2 SupportVectorMachine . . . . . . . . . . . . . . . . . . . . . . . . . 19
2.3.3 C4.5decisiontree . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20
2.3.4 Stackedgraphicallearning . . . . . . . . . . . . . . . . . . . . . . . . 22
2.4 Evaluation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23
3 LatentDirichletAllocation 27
3.1 HistoryofLDA . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27
3.2 BayesianInference . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29
3.2.1 Distributions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32
3.3 Latentvariablemodels . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33
ix
3.4 LDAmodelingframework . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37
3.4.1 BayesianNetworkofLDA . . . . . . . . . . . . . . . . . . . . . . . . 37
3.4.2 Modelinferenceandparameterestimation . . . . . . . . . . . . . . . . 39
3.4.3 UnseeninferenceforLDA . . . . . . . . . . . . . . . . . . . . . . . . 43
4 AComparativeAnalysisofLSAandLDA 45
4.1 Relatedresults . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 45
4.2 Experimentalsetup . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 46
4.2.1 Inputdata . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 46
4.2.2 Modelconstruction . . . . . . . . . . . . . . . . . . . . . . . . . . . . 47
4.3 Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 47
4.4 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 49
5 Multi-corpusLDA 51
5.1 MLDAmodel . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 51
5.1.1 ϑ-smoothingwithPersonalizedPageRank . . . . . . . . . . . . . . . . 53
5.2 Experimentalsetup . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 56
5.2.1 Termselection . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 56
5.2.2 LDAinference . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 58
5.3 Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 58
5.3.1 PlainMLDAvsLDA . . . . . . . . . . . . . . . . . . . . . . . . . . . 58
5.3.2 Vocabulariesandϑ-smoothing . . . . . . . . . . . . . . . . . . . . . . 59
5.3.3 Runningtimes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 60
5.4 Conclusionandfuturework . . . . . . . . . . . . . . . . . . . . . . . . . . . . 60
6 LinkedLDAandsparseGibbssamplers 63
6.1 Relatedresults . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 65
6.2 LinkedLDAmodel . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 66
6.3 FastGibbssamplingheuristics . . . . . . . . . . . . . . . . . . . . . . . . . . 68
6.4 Experimentalsetup . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 70
6.4.1 Thedatasetandclassifiers . . . . . . . . . . . . . . . . . . . . . . . . 70
6.4.2 Baselineclassifiers . . . . . . . . . . . . . . . . . . . . . . . . . . . . 71
6.4.3 LDAinference . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 71
6.5 Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 72
6.5.1 SpeedupwiththeGibbssamplingstrategies . . . . . . . . . . . . . . . 72
6.5.2 LikelihoodandconvergenceofLDAinference . . . . . . . . . . . . . 73
6.5.3 Comparisonwiththebaseline . . . . . . . . . . . . . . . . . . . . . . 74
6.6 Conclusionandfuturework . . . . . . . . . . . . . . . . . . . . . . . . . . . . 76