UNIVERSITY OF CALIFORNIA, MERCED Statistical Methods for Aspect Level Sentiment Analysis ...

123 Pages·2015·1.82 MB·English

UC Merced
UC Merced Electronic Theses and Dissertations
Title
Statistical Models for Aspect-Level Sentiment Analysis
Permalink
https://escholarship.org/uc/item/2ks913br
Author
Farhadloo, Mohsen
Publication Date
2015
Copyright Information
This work is made available under the terms of a Creative Commons Attribution-
NonCommercial-NoDerivatives License, availalbe at
https://creativecommons.org/licenses/by-nc-nd/4.0/

Peer reviewed|Thesis/dissertation
eScholarship.org Powered by the California Digital Library
University of California
UNIVERSITYOFCALIFORNIA,MERCED
Statistical MethodsforAspectLevelSentimentAnalysis
Adissertationsubmittedinpartialsatisfactionofthe
requirements forthedegree
DoctorofPhilosophy
in
ElectricalEngineering&Computer Science
by
MohsenFarhadloo
Committee incharge:
ProfessorErikRolland,Chair
ProfessorPaulMaglio
ProfessorDavidNoelle
ProfessorRaymondPatterson
2015
Copyright
MohsenFarhadloo, 2015
Allrightsreserved.
The dissertation of Mohsen Farhadloo is approved, and it is
acceptable in quality and form for publication on microfilm
andelectronically:
Chair
UniversityofCalifornia, Merced
2015
iii
DEDICATION
I dedicate my dissertation work to my family. A special feeling of
gratitude to my loving parents, Maryam and Faramarz whose words of
encouragement and push for tenacity ring in my ears. My sisters Fateme,
Mina and Zahra and my brother Majid have never left my side and are
very special. I lost my loving grandparents while working on this
dissertation and they will be always part of my heart.
iv
TABLEOFCONTENTS
Dedication . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . i
TableofContents . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . iii
ListofFigures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . v
ListofTables . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . vii
Acknowledgements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . viii
VitaandPublications . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1
AbstractoftheDissertation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2
Chapter1 Sentimentanalysisandopinion mining . . . . . . . . . . . . . . . 5
1.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . 5
1.2 Problemdefinition . . . . . . . . . . . . . . . . . . . . . . 9
1.3 Challenges . . . . . . . . . . . . . . . . . . . . . . . . . . 11
1.3.1 Synonymyandpolysemy . . . . . . . . . . . . . . . 11
1.3.2 Sarcasm . . . . . . . . . . . . . . . . . . . . . . . . 13
1.3.3 Compoundsentences . . . . . . . . . . . . . . . . . 14
1.3.4 Unstructureddata . . . . . . . . . . . . . . . . . . . 14
1.3.5 Aspectidentification . . . . . . . . . . . . . . . . . 14
1.3.6 Sentimentidentification . . . . . . . . . . . . . . . 17
1.3.7 Topicmodelbasedapproaches . . . . . . . . . . . . 23
1.4 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . 31
Chapter2 Anautomatedframeworkforaspectlevelsentimentanalysis . . . 32
2.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . 32
2.2 Methodology . . . . . . . . . . . . . . . . . . . . . . . . . 35
2.2.1 Aspectidentification . . . . . . . . . . . . . . . . . 35
2.2.2 Sentimentidentification . . . . . . . . . . . . . . . 37
2.3 Experimentalresults . . . . . . . . . . . . . . . . . . . . . 41
2.4 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . 48
Chapter3 Modeling customer satisfaction from unstructured data using a
Bayesianapproach . . . . . . . . . . . . . . . . . . . . . . . . . 50
3.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . 50
3.2 Relatedwork . . . . . . . . . . . . . . . . . . . . . . . . . 51
3.3 Problemdefinition . . . . . . . . . . . . . . . . . . . . . . 54
3.4 Methods . . . . . . . . . . . . . . . . . . . . . . . . . . . . 55
v
3.4.1 Transforming the unstructured into semi-structured
data . . . . . . . . . . . . . . . . . . . . . . . . . . 55
3.4.2 Probabilisticmodelingofoverallsatisfaction . . . . 57
3.4.3 Modelparametersestimation . . . . . . . . . . . . . 59
3.4.4 Howtousethelearnedprobabilistic model . . . . . 61
3.5 Experimentalresults . . . . . . . . . . . . . . . . . . . . . 62
3.6 Extendedexperiment . . . . . . . . . . . . . . . . . . . . . 67
3.7 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . 70
3.8 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . 76
Chapter4 Futurework: Topicmodeling approach . . . . . . . . . . . . . . . 79
4.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . 79
4.1.1 Unigramlanguagemodel . . . . . . . . . . . . . . . 81
4.1.2 Mixtureofunigrams . . . . . . . . . . . . . . . . . 81
4.1.3 Probabilisticlatentsemanticanalysis . . . . . . . . 82
4.1.4 LatentDirichletallocation . . . . . . . . . . . . . . 83
4.2 Problemdefinition . . . . . . . . . . . . . . . . . . . . . . 83
4.3 NaiveapplicationofLDAtoSAOM . . . . . . . . . . . . . 84
4.4 SomemodificationstoLDA . . . . . . . . . . . . . . . . . 84
4.5 Ourmethodology . . . . . . . . . . . . . . . . . . . . . . . 88
4.5.1 Learningthemodel . . . . . . . . . . . . . . . . . . 94
Chapter5 Conclusionremarks . . . . . . . . . . . . . . . . . . . . . . . . . 99
Chapter6 Directionsforfurther research . . . . . . . . . . . . . . . . . . . 101
vi
LISTOFFIGURES
Figure1.1: An example parse tree. Each node is associated with e vector and
thevectorrepresentationofthenodesarecomputedbottom-up. . . 20
Figure1.2: LatentDirichletAllocationgraphicalmodel. Theshadednodeisan
observedrandomvariableandtheunshadedonesarehiddenrandom
variables. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24
Figure1.3: Topical N-grams graphical model. The shaded node is an observed
random variable and the unshaded ones are hidden random vari-
ables. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28
Figure1.4: Coherent Facet and Sentiments graphical model. The shaded node
is an observed random variable and the unshaded ones are hid-
den random variables. This model introduces two new variables:
ψ whichdeterminesthedependencybetweenwindowsandc that
d,x i
specifies to which category each word belongs: aspects, sentiments
orbackgroundcategory. . . . . . . . . . . . . . . . . . . . . . . . 29
Figure1.5: Phrase Latent Dirichlet Allocation graphical model. The shaded
node is an observed random variable and the unshaded ones are
hiddenrandomvariables. . . . . . . . . . . . . . . . . . . . . . . . 31
Figure2.1: System diagram of our methodology. In preprocessing step the vo-
cabulary lists (BOW and BON) and the scores of words are calcu-
lated. AspectsareidentifiedbyclusteringthesentencesusingBON
representation. Sentimentareidentifiedusingscorerepresentation. . 41
Figure2.2: Results of cross validating the penalty and kernel width parameters
in SVM. The accuracy of the classifier has been coded with colors.
The first two rows are for BOW representation and the third and
fourth rows for score representation. Using these figures the best
parametersareselectedandtheclassifiersaredesigned. . . . . . . . 46
Figure3.1: The system flowchart of transforming the raw input data (written
reviews)intosemi-structured data. . . . . . . . . . . . . . . . . . . 52
Figure3.2: Asamplereviewfromtripadvisor.com . . . . . . . . . . . . . . . . 53
Figure3.3: Aspect-levelframeworkof[FarhadlooandRolland(2013)]forsen-
timent analysis. Aspects are identified by clustering the sentences
usingBONrepresentation. Sentimentareidentifiedusingscorerep-
resentation. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 56
Figure4.1: The graphicalmodel of a mixture density. Giventhe iand x then t
i
variablesareconditionally independent. . . . . . . . . . . . . . . . 80
Figure4.2: The graphical model of probabilistic latent semantic analysis. For
each document d there is distribution over topics and topic assign-
mentz foreachwordisdrawnfromthatdistribution. . . . . . . . . 82
vii
Figure4.3: ThegraphicalmodelofseparatelatentDirichletallocation. . . . . . 85
Figure4.4: ThegraphicalmodelofseparatelatentDirichletallocation. . . . . . 85
Figure4.5: ThegraphicalmodelofdependentlatentDirichletallocation. . . . . 86
Figure4.6: ThegraphicalmodelofLakkarajuetal.(2011). . . . . . . . . . . . 87
Figure4.7: The graphical model of another model with one distribution over
ratingforeachaspect. . . . . . . . . . . . . . . . . . . . . . . . . 88
Figure4.8: The graphical model of focus-LDA. There is a hidden variable for
each document that determines the focus of the document. The
focus of the document tells which one of the multiple priors over
aspect-topicshouldbeused. . . . . . . . . . . . . . . . . . . . . . 90
Figure4.9: The graphical model of alternative focus LDA. In the alternative
model there is no category variable however the parameter space
extendssincethereareKa ×Kr topics. . . . . . . . . . . . . . . . 92
Figure4.10: Exampletopics. Thedistributionontheleftisanexampleforeither
aspect/rating topics. Some aspect/rating words have high proba-
bility and other words have low probability in those topics. The
distributionontherightisanexampleofatopicwhereaspectterms
andcorrespondingratingtermshavehighprobabilitynottherestof
thewordsinthevocabulary. . . . . . . . . . . . . . . . . . . . . . 93
viii
LISTOFTABLES
Table2.1: BOWrepresentation . . . . . . . . . . . . . . . . . . . . . . . . . . 36
Table2.2: BONrepresentation . . . . . . . . . . . . . . . . . . . . . . . . . . 37
Table2.3: Normalizedrecall(r˜)fordifferentdimensioninthelow-dimensional
spaceusingLSA. . . . . . . . . . . . . . . . . . . . . . . . . . . . 41
Table2.4: Aspect identification with clustering the sentences. The first row is
the desired list of aspects. The rest are the extracted representative
termsafterclusteringusingBOWandBONwith/without LSA. . . . 42
Table2.5: Normalizedrecall(r˜)fordifferentnumberofclusters. . . . . . . . . 43
Table2.6: Performance of sentiment classification with BOW as feature set.
The table compares the tf-idf weighting scheme with term occur-
rencescheme. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 44
Table2.7: Performance of sentiment classification with score representation as
feature set. The table compares the tf-idf weighting scheme with
termoccurrencescheme. . . . . . . . . . . . . . . . . . . . . . . . 44
Table2.8: Thecomputedscoresforsomeoftheterms. . . . . . . . . . . . . . 49
Table3.1: Datasetstatistics . . . . . . . . . . . . . . . . . . . . . . . . . . . 62
Table3.2: MLEstimatedofmixingcoefficients. . . . . . . . . . . . . . . . . . 63
Table3.3: Aspectimportanceusinggroundtruth sentiments. . . . . . . . . . . 64
Table3.4: Aspectimportanceusingpredictedsentiments. . . . . . . . . . . . . 64
Table3.5: Diagnostictestresults. . . . . . . . . . . . . . . . . . . . . . . . . 66
Table3.6: NumberofAll/UsableReviewsbyParkandRating. . . . . . . . . . 68
Table3.7: R2 analysisfortestparks. . . . . . . . . . . . . . . . . . . . . . . . 69
Table3.8: R2 analysisforextendedexperiments . . . . . . . . . . . . . . . . . 71
ix