Explorer et apprendre à partir de collections de textes multilingues à l'aide des modèles ...

182 Pages·2017·2.87 MB·French

THÈSE
Pour obtenir le grade de
DOCTEUR DE LA COMMUNAUTÉ UNIVERSITÉ
GRENOBLE ALPES
Spécialité : Mathématiques et Informatique
Arrêté ministériel : 25 mai 2016
Présentée par
Georgios BALIKAS
Thèse dirigée par Massih-Reza AMINI, Professeur, Université
Grenoble Alpes
préparée au sein du Laboratoire d’Informatique de Grenoble dans
l'École Doctorale Mathématiques, Sciences et technologies de
l’information, Informatique
Explorer et apprendre à partir de
collections de textes multilingues à l'aide
des modèles probabilistes latents et des
réseaux profonds
Mining and Learning from Multilingual
Text Collections using Topic Models and
Word Embeddings
Thèse soutenue publiquement le 20 octobre 2017,
devant le jury composé de :
Monsieur MASSIH-REZA AMINI
PROFESSEUR, UNIVERSITE GRENOBLE ALPES, Directeur de these
Monsieur CYRIL GOUTTE
CHERCHEUR, CONSEIL NATIONAL DE RECHERCHES CANADA,
Rapporteur
Monsieur GAËL DIAS
PROFESSEUR, UNIVERSITE DE CAEN NORMANDIE, Rapporteur
Monsieur LAURENT BESACIER
PROFESSEUR, UNIVERSITE GRENOBLE ALPES, Examinateur, Président
du jury
Monsieur GUILLAUME VERNAT
DIRECTEUR GENERAL, SOCIETE COFFREO, Examinateur
Monsieur PATRICK GALLINARI
PROFESSEUR, UNIVERSITE PIERRE ET MARIE CURIE, Examinateur
Abstract
Text is one of the most pervasive and persistent sources of informa-
tion. Content analysis of text in its broad sense refers to methods for
studyingandretrievinginformationfromdocuments. Nowadays, with
theeverincreasingamountoftextbecomingavailableonlineinseveral
languages and writing styles, content analysis of text is of tremendous
importanceasitenablesavarietyofapplications. Tothisend,unsuper-
visedrepresentationlearningmethodsliketopicmodelsandwordem-
beddings constitute prominent tools. The goal of this thesis is to study
andaddresschallengingproblemsinthisarea,focusingonboththede-
sign of novel text mining algorithms and tools, as well as on studying
how these tools can be applied to text collections written in a single or
several languages.
In the first part of the thesis we focus on topic models and more pre-
ciselyonhowtoincorporatepriorinformationoftextstructuretothem.
Topic models are built on the premise of bag-of-words, and therefore
words are exchangeable. While this assumption benefits the calcula-
tions of the conditional probabilities it results in loss of information.
To overcome this limitation we propose two mechanisms that extend
topic models by integrating knowledge of text structure to them. To
this end, we begin by assuming that the documents are partitioned in
thematicallycoherenttextsegments. Then,thefirstmechanismassigns
thesametopictothewordsofasegment. Thesecond,capitalizesonthe
propertiesofcopulas,atoolmainlyusedinthefieldsofeconomicsand
risk management that is used to model the joint probability distribu-
tions of random variables while having access only to their marginals.
Through the use of copulas we propose flexible topic models that can
ff
modeldi erentdegreesofdependencebetweenthetopicsofasegment.
The second part of the thesis explores bilingual topic models for com-
parable corpora with explicit document alignments. Typically, a docu-
mentcollectionforsuchmodelsisintheformofcomparabledocument
ff
pairs. The documents of a pair are written in di erent languages and
are thematically similar. Unless translations, the documents of a pair
are similar to some extent only. Meanwhile, representative topic mod-
elsassumethatthedocumentshaveidenticaltopicdistributions,which
isastrongandlimitingassumption. Toovercomethelimitationsofthis
assumption we propose novel bilingual topic models that incorporate
the notion of cross-lingual similarity of the documents that constitute
the pairs in their generative and inference processes. Calculating this
cross-lingual document similarity is a task on itself, which we propose
to address using cross-lingual word embeddings.
The last part of the thesis concerns the use of word embeddings and
neural networks for three text mining applications. First, we discuss
polylingual document classification where we argue that translations
of a document can be used to enrich its representation. Using an auto-
encoder to obtain these robust document representations we demon-
strate improvements in the task of multi-class document classification.
Second, we explore multi-task sentiment classification of tweets argu-
ing that jointly training classification systems on correlated tasks can
improve the obtained performance. To this end we show how one can
achieve state-of-the-art performance on a sentiment classification task
using recurrent neural networks. The third application we explore is
cross-lingual information retrieval. Given a document written in one
language, the task consists in retrieving the most similar documents
from a pool of documents written in another language. In this line
of research, we demonstrate how adapting the transportation problem
forestimatingdocumentdistancesonecanachieveimportantimprove-
ments.
ii
List of Publications
The following publications are included in parts or in an extended version in this
thesis:
• Georgios Balikas, Ioannis Partalas, Eric Gaussier, Rohit Babbar, and Massih-
ffi
Reza Amini. E cient model selection for regularized classification by ex-
ploiting unlabeled data. In International Symposium on Intelligent Data Anal-
ysis, IDA 2015, pages 25–36. Springer, 2015.
• Georgios Balikas and Massih-Reza Amini. Multi-label, multi-class classifi-
cation using polylingual embeddings. In Advances in Information Retrieval
137-38thEuropeanConferenceonIRResearch,ECIR2016,Padua,Italy,March
20-23, 2016. pages 723–728. Springer, 2016.
• Georgios Balikas and Massih-Reza Amini. Twise at semeval-2016 task 4:
Twittersentimentclassification. InProceedingsofthe10thInternationalWork-
shop on Semantic Evaluation, SemEval@NAACL-HLT 2016, San Diego, CA,
USA, June 16-17, 2016, pages 85–91, 2016.
• Georgios Balikas, Massih-Reza Amini, and Marianne Clausel. On a topic
model for sentences. In Proceedings of the 39th International ACM SIGIR
conference on Research and Development in Information Retrieval, SIGIR 2016,
Pisa, Italy, July 17-21, 2016, pages 921–924, 2016.
• Georgios Balikas, Hesam Amoualian, Marianne Clausel, Éric Gaussier, and
Massih-RezaAmini. Modelingtopicdependenciesinsemanticallycoher-ent
text spans with copulas. In Proceedings of the 26th International Conference
on Computational Linguistics, COLING 2016, December 11-16, 2016, Osaka,
Japan, pages 1767–1776, 2016.
• Georgios Balikas, Simon Moura, and Massih-Reza Amini. Multitask learn-
ing for fine-grained twitter sentiment analysis. In Proceedings of the 40th
iii
International ACM SIGIR conference on Research and Development in Informa-
tion Retrieval, SIGIR 2017, Tokyo, Japan, August 7-11, 2017, 2017.
Inadditiontothetopicsstudiedinthismanuscriptwhicharementionedabove,
duringmythesisIworkedonseveralotherproblemsleadingtothefollowingpub-
lications:
• HesamAmoualian,WeiLu,EricGaussier,GeorgiosBalikas,MassihRAmini,
MarianneClausel: TopicalCoherenceinLDA-basedModelsthroughInduced
Segmentation,Proceedingsofthe55thAnnualMeetingoftheAssociationfor
Computational Linguistics, Vancouver, Canada, 2017.
• Georgios Balikas: TwiSe at SemEval-2017 Task 4: Five-point Twitter Senti-
ment Classification and Quantification , In Proceedings of the 10th Interna-
tional Workshop on Semantic Evaluation, SemEval@ACL 2017, Vancouver,
Canada, 2017.
• Georgios Balikas and Massih-Reza Amini: An empirical study on large scale
text classification with skip-gram embeddings, Neu-IR @ SIGIR 2016, Pisa,
Italy, 2016.
• Anil Goyal, Georgios Balikas, Prantapa Goswami, Massih-Reza Amini and
EricGaussier: TransferLearningforIRModelParameterTuning,Conférence
Francophone sur l’Apprentissage Automatique (CAp), Lille, 2015.
• Georgios Balikas and Massih-Reza Amini: Learning language-independent
sentence representations for multi-lingual, multi-document summarization,
ConférenceFrancophonesurl’ApprentissageAutomatique(CAp),Lille,2015.
iv
Acknowledgements
ff
Thisthesisistheresultofe ortsofaten-yearuniversitystudentperiod.
During this endeavor is was supported by several people, that I would
like to deeply thank.
First, I would like to express my sincere gratitude to my advisor Prof.
Massih-Reza Amini. I am grateful for his support and advice. Our
discussions and brainstorming during all these years have deeply in-
fluenced me. I believe I have become not only a better researcher but
also a better person due to our collaboration. Without his support and
continuous encouragement several of the results presented in this the-
sis would not have been obtained.
I would also like to express my gratitude to the members of my Ph.D.
thesiscommittee,Prof. GaelDias(UniversityofCaen),Dr. CyrilGoutte
(National Research Council Canada), Prof. Patrick Gallinari (Univer-
sityofPierre&MarieCurie/Paris6),Prof. LaurentBesacier(University
ff
of Grenoble-Alps) and Dr. Guillaume Vernat (Co reo). I was honored
to have them in my defense committee. In particular, I would like to
thank my thesis reviewers, Prof. Gael Dias and Dr. Cyril Goutte for
the time they invested, their valuable feedback on my work and their
detailed comments in their reports.
ff
Formygraduatestudies,IhadthehonortobefundedbyCo reo. First,
I would like to thank the two founders of the company, Emmanuel
CudryandDr. GuillaumeVernat,notonlyfortrustingmewiththerole
but also for the discussions we had and the valuable lessons I learned
ff
for leading a company. I would also like to thank everybody at Co reo
for making my visits at Clermont Ferrand interesting! I would partic-
ularly like to thank Dr. Pierre-Antoine Papon, Karim Sayabou and Dr.
Guenaelle De Julis, for the long discussions and our collaboration dur-
ing these years.
I am especially grateful to Prof. Eric Gaussier for the various interac-
tions we had over the last three years. I would like to thank him for
being a great research collaborator and for suggesting me interesting
directions for my work.
IhavebeentrulyluckytointeractwithmanybrilliantpeopleatUniver-
sity of Grenoble Alps and Grenoble in general. Among them, a special
thanks go to my friend Dr. Ioannis Partalas for his encouragement and
advice during my thesis. I would also like to thank all the members of
the AMA team, current and past, as I learned a lot of things from them
and I shared many beautiful moments.
I would also like to deeply thank Anthie. Living with me, caring for
andsupportingmychoicesforalmosteightyearsnowmademeabetter
person. Thank you for being patient and understanding all these years
tolerating my idiosyncrasies and habits.
Lastbutnotleast,Iwouldliketothankfromthedepthsofmyheartmy
family back in Greece: My parents Andreas and Olga and my brother
Stathis. Your continuous encouragement, patience, care and love are
giving me courage and confidence. Thank you for everything you did
and continue doing for me.
Georgios Balikas
Grenoble, Fall 2017
vi
Contents
1 Introduction 1
1.1 Thesis statement and overview of contributions . . . . . . . . . . . . 3
1.1.1 Extending Probabilistic Topic Models . . . . . . . . . . . . . 5
1.1.2 Word Embeddings and Text Mining Applications . . . . . . . 8
1.1.3 Quantification-based algorithmic tools . . . . . . . . . . . . . 9
1.2 Outline of the thesis . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9
2 PreliminariesforProbabilisticTopicModels 11
2.1 The Multinomial and Dirichlet Distributions . . . . . . . . . . . . . 13
2.1.1 The Multinomial Distribution . . . . . . . . . . . . . . . . . . 14
2.1.2 The Dirichlet Distribution . . . . . . . . . . . . . . . . . . . . 14
2.1.3 Conjugacy between the Multinomial and Dirichlet distribu-
tions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15
2.2 Fromtheterm-documentco-occurrencematrixtoprobabilistictopic
models . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16
2.2.1 Latent Semantic Analysis . . . . . . . . . . . . . . . . . . . . . 16
2.2.2 Probabilistic Latent Semantic Allocation . . . . . . . . . . . . 17
2.2.3 Latent Dirichlet Allocation . . . . . . . . . . . . . . . . . . . . 19
2.2.4 Distributional Semantics . . . . . . . . . . . . . . . . . . . . . 22
2.3 Multilingual Topic Models . . . . . . . . . . . . . . . . . . . . . . . . 23
2.3.1 Parallel and comparable corpora . . . . . . . . . . . . . . . . 23
2.3.2 Bilingual Latent Dirichlet Allocation . . . . . . . . . . . . . . 24
2.4 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25
3 PreliminariesforNeuralNetworks 27
3.1 Word Embeddings with shallow Neural Networks . . . . . . . . . . 28
3.1.1 The Skipgram Model . . . . . . . . . . . . . . . . . . . . . . . 29
3.1.2 The Continuous Bag-of-Words Model . . . . . . . . . . . . . . 32
vii
3.2 Text Representations using deep neural networks . . . . . . . . . . . 33
3.3 Cross-lingual Word Embeddings . . . . . . . . . . . . . . . . . . . . 35
3.3.1 Bilbowa . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37
3.3.2 Concept Net . . . . . . . . . . . . . . . . . . . . . . . . . . . . 38
3.4 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 39
4 IncorporatingPriorKnowledgeofTextStructuretoTopicModels 41
4.1 An overview of the relevant literature . . . . . . . . . . . . . . . . . . 43
4.2 The coherent text segments . . . . . . . . . . . . . . . . . . . . . . . . 44
4.3 Incorporating text structure to topic models . . . . . . . . . . . . . . 46
4.3.1 segmentLDA: Integrating segment boundaries to LDA . . . . 46
4.3.2 Copulas and random variables (intermezzo) . . . . . . . . . . 49
4.3.3 copulaLDA: Integrating segment boundaries to LDA using
copulas . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 52
4.4 The Experimental Evaluation . . . . . . . . . . . . . . . . . . . . . . 55
4.4.1 Intrinsic Evaluation . . . . . . . . . . . . . . . . . . . . . . . . 59
4.4.2 Extrinsic Evaluation . . . . . . . . . . . . . . . . . . . . . . . 66
4.5 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 69
5 ExtendingBilingualTopicModels 73
5.1 An overview of the relevant literature . . . . . . . . . . . . . . . . . . 76
5.2 Framework . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 79
5.2.1 The bilingual LDA . . . . . . . . . . . . . . . . . . . . . . . . 80
5.2.2 Incorporating text structure into bilingual topic models . . . 81
5.2.3 Extracting multilingual topics from comparable corpora . . . 84
5.2.4 Combining the two models . . . . . . . . . . . . . . . . . . . 86
5.3 Experimental Framework . . . . . . . . . . . . . . . . . . . . . . . . . 87
5.3.1 Intrinsic Evaluation . . . . . . . . . . . . . . . . . . . . . . . . 90
5.3.2 Extrinsic Evaluation . . . . . . . . . . . . . . . . . . . . . . . 93
5.4 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 96
6 Applicationsofwordembeddingstotextmining 101
6.1 Polylingual text classification . . . . . . . . . . . . . . . . . . . . . . 102
6.1.1 The model for learning the polylingual embeddings . . . . . 103
6.1.2 The Experimental Evaluation . . . . . . . . . . . . . . . . . . 105
6.1.3 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 109
6.2 Multitask Learning with Neural Networks . . . . . . . . . . . . . . . 109
viii
6.2.1 Multitask Learning for Sentiment Classification . . . . . . . . 111
6.2.2 The Experimental Framework . . . . . . . . . . . . . . . . . . 112
6.2.3 Experimental results . . . . . . . . . . . . . . . . . . . . . . . 119
6.2.4 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 121
6.3 Cross-lingual text retrieval . . . . . . . . . . . . . . . . . . . . . . . . 123
6.3.1 AWasserstein-alikedistanceforCross-lingualDocumentRe-
trieval . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 124
6.3.2 The Experimental Framework . . . . . . . . . . . . . . . . . . 126
6.3.3 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 130
6.4 Chapter Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 131
7 ConcludingRemarks 133
7.1 Summary of Contributions . . . . . . . . . . . . . . . . . . . . . . . . 133
7.2 Future Directions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 134
Bibliography 137
A Efficient Model Selection for Regularized Classification by Exploiting
UnlabeledData 157
A.1 Related Work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 158
A.2 Accuracy and Macro-F1 Quantification Bounds . . . . . . . . . . . . 159
A.3 Experimental Framework . . . . . . . . . . . . . . . . . . . . . . . . . 164
A.4 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 168
ix