Towards a science of human stories: using sentiment analysis and emotional arcs to understand ...

286 Pages·2017·30.11 MB·English

Towards a science of human stories: using
sentiment analysis and emotional arcs to
understand the building blocks of complex
social systems
A Dissertation Presented
by
Andrew J. Reagan
to
The Faculty of the Graduate College
of
The University of Vermont
In Partial Fulfillment of the Requirements
for the Degree of Doctor of Philosophy
Specializing in Mathematical Sciences
May, 2017
Defense Date: February 24, 2017
Dissertation Examination Committee:
Christopher M. Danforth, Ph.D., Advisor
Peter Sheridan Dodds, Ph.D., Advisor
Samuel V. Scarpino, Ph.D.
Jacques A. Bailly, Ph.D., Chairperson
Cynthia J. Forehand, Ph.D., Dean of Graduate College
Abstract
We can leverage data and complex systems science to better understand society and human
nature on a population scale through language — utilizing tools that include sentiment analysis,
machine learning, and data visualization. Data-driven science and the sociotechnical systems that
weuseeverydayareenablingatransformationfromhypothesis-driven,reductionistmethodologyto
complex systems sciences. Namely, the emergence and global adoption of social media has rendered
possible the real-time estimation of population-scale sentiment, with profound implications for our
understanding of human behavior. Advances in computing power, natural language processing, and
digitizationoftextnowmakeitpossibletostudyaculture’sevolutionthroughitstextsusinga“big
data” lens.
Giventhegrowingassortmentofsentimentmeasuringinstruments,itisimperativetounderstand
which aspects of sentiment dictionaries contribute to both their classification accuracy and their
ability to provide richer understanding of texts. Here, we perform detailed, quantitative tests and
qualitative assessments of 6 dictionary-based methods applied to 4 different corpora, and briefly
examine a further 20 methods. We show that while inappropriate for sentences, dictionary-based
methodsaregenerallyrobustintheirclassificationaccuracyforlongertexts. Mostimportantlythey
can aid understanding of texts with reliable and meaningful word shift graphs if (1) the dictionary
covers a sufficiently large enough portion of a given text’s lexicon when weighted by word usage
frequency; and (2) words are scored on a continuous scale.
Our ability to communicate relies in part upon a shared emotional experience, with stories
often following distinct emotional trajectories, forming patterns that are meaningful to us. By
classifying the emotional arcs for a filtered subset of 4,803 stories from Project Gutenberg’s fiction
collection,wefindasetofsixcoretrajectorieswhichformthebuildingblocksofcomplexnarratives.
We strengthen our findings by separately applying optimization, linear decomposition, supervised
learning,andunsupervisedlearning. Foreachofthesesixcoreemotionalarcs,weexaminetheclosest
characteristic stories in publication today and find that particular emotional arcs enjoy greater
success, as measured by downloads. Within stories lie the core values of social behavior, rich with
both strategies and proper protocol, which we can begin to study more broadly and systematically
as a true reflection of culture. Of profound scientific interest will be the degree to which we can
eventually understand the full landscape of human stories, and data driven approaches will play a
crucial role.
Finally, we utilize web-scale data from Twitter to study the limits of what social data can tell us
about public health, mental illness, discourse around the protest movement of #BlackLivesMatter,
discourse around climate change, and hidden networks. We conclude with a review of published
works in complex systems that separately analyze charitable donations, the happiness of words in
10 languages, 100 years of daily temperature data across the United States, and Australian Rules
Football games.
Citations
Material from this dissertation has been submitted for publication in EPJ Data Science on January
10, 2017, with the preprint available as follows:
Reagan, A. J., Danforth, C. M., Tivnan, B., Williams, J. R., Dodds, P. S.. (2016). Benchmarking
sentimentanalysismethodsforlarge-scaletexts: Acaseforusingcontinuum-scoredwordsandword
shift graphs. Preprint available online at https://arxiv.org/ abs/1512.00531.
AND
Material from this dissertation has been published in EPJ Data Science on November 4, 2016, in
the following form. Within a month of publication, our paper was the most shared of all papers
published in EPJ Data Science, as measured by Altmetric.
Reagan, A. J., Mitchell, L., Kiley, D., Danforth, C. M., Dodds, P. S.. (2016). The emotional arcs of
stories are dominated by six basic shapes. EPJ Data Science 5(1), 31.
ii
Table of Contents
1 Introduction and Literature Review 1
1.1 Introduction. . . . . . . . . . . . . . . . . . . . . . . . . . . 1
1.2 Sentiment analysis . . . . . . . . . . . . . . . . . . . . . . . . 2
1.2.1 Psychology of emotion . . . . . . . . . . . . . . . . . . . . . 3
1.2.2 Goals of sentiment analysis . . . . . . . . . . . . . . . . . . . 8
1.2.3 Publicly available annotated data . . . . . . . . . . . . . . . . . 8
1.2.4 Natural Language Processing techniques. . . . . . . . . . . . . . . 10
1.2.5 Building corpus-specific sentiment dictionaries . . . . . . . . . . . . 16
1.2.6 Visualization. . . . . . . . . . . . . . . . . . . . . . . . . 21
1.2.7 Benchmarking literature . . . . . . . . . . . . . . . . . . . . 25
1.3 Emotional arcs. . . . . . . . . . . . . . . . . . . . . . . . . . 28
1.3.1 Story graphs, plot diagrams, and inferring causality . . . . . . . . . . 29
1.3.2 Story generation . . . . . . . . . . . . . . . . . . . . . . . 31
1.3.3 Character Identification and Networks . . . . . . . . . . . . . . . 32
1.3.4 Frames for NLP . . . . . . . . . . . . . . . . . . . . . . . 33
1.3.5 Visualization. . . . . . . . . . . . . . . . . . . . . . . . . 35
1.3.6 Emotional arcs . . . . . . . . . . . . . . . . . . . . . . . . 36
1.3.7 Suzyhet and validation . . . . . . . . . . . . . . . . . . . . . 37
2 Benchmarking sentiment analysis methods for large-scale texts: A case for using
continuum-scored words and word shift graphs. 40
2.1 Introduction. . . . . . . . . . . . . . . . . . . . . . . . . . . 41
2.2 Sentiment Dictionaries, Corpora, and Word Shift Graphs . . . . . . . . . . 43
2.2.1 Sentiment Dictionaries . . . . . . . . . . . . . . . . . . . . . 43
2.2.2 Corpora Tested. . . . . . . . . . . . . . . . . . . . . . . . 47
2.2.3 Word Shift Graphs . . . . . . . . . . . . . . . . . . . . . . 48
2.3 Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . 49
2.3.1 New York Times Word Shift Analysis . . . . . . . . . . . . . . . 54
2.3.2 Movie Reviews Classification and Word Shift Graph Analysis . . . . . . . 58
2.3.3 Google Books Time Series and Word Shift Analysis . . . . . . . . . . 62
2.3.4 Twitter Time Series Analysis. . . . . . . . . . . . . . . . . . . 64
2.3.5 Brief Comparison to Machine Learning Methods. . . . . . . . . . . . 67
2.4 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . 68
2.5 References . . . . . . . . . . . . . . . . . . . . . . . . . . . 69
3 The emotional arcs of stories are dominated by six basic shapes 73
3.1 Introduction. . . . . . . . . . . . . . . . . . . . . . . . . . . 73
3.2 Methods . . . . . . . . . . . . . . . . . . . . . . . . . . . . 75
3.2.1 Emotional arc construction . . . . . . . . . . . . . . . . . . . 75
3.2.2 Project Gutenberg Corpus. . . . . . . . . . . . . . . . . . . . 76
3.2.3 Principal Component Analysis (SVD) . . . . . . . . . . . . . . . 78
3.2.4 Hierarchical Clustering . . . . . . . . . . . . . . . . . . . . . 79
3.2.5 Self Organizing Map (SOM) . . . . . . . . . . . . . . . . . . . 79
iii
3.3 Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . 80
3.3.1 Principal Component Analysis (SVD) . . . . . . . . . . . . . . . 80
3.3.2 Hierarchical Clustering . . . . . . . . . . . . . . . . . . . . . 84
3.3.3 Self Organizing Map (SOM) . . . . . . . . . . . . . . . . . . . 84
3.3.4 Null comparison . . . . . . . . . . . . . . . . . . . . . . . 86
3.3.5 The Success of Stories . . . . . . . . . . . . . . . . . . . . . 87
3.4 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . 88
3.5 References . . . . . . . . . . . . . . . . . . . . . . . . . . . 89
4 Selected contributions to published work 91
4.1 Collective Philanthropy: Describing and Modeling the Ecology of Giving. . . . . 92
4.1.1 Abstract . . . . . . . . . . . . . . . . . . . . . . . . . . 92
4.1.2 Contribution. . . . . . . . . . . . . . . . . . . . . . . . . 92
4.2 Shadow networks: Discovering hidden nodes with models of information flow . . . 97
4.2.1 Abstract . . . . . . . . . . . . . . . . . . . . . . . . . . 97
4.2.2 Contribution. . . . . . . . . . . . . . . . . . . . . . . . . 97
4.3 Human language reveals a universal positivity bias. . . . . . . . . . . . . 98
4.3.1 Abstract . . . . . . . . . . . . . . . . . . . . . . . . . . 98
4.3.2 Contribution. . . . . . . . . . . . . . . . . . . . . . . . . 98
4.4 Climate change sentiment on Twitter: An unsolicited public opinion poll. . . . . 99
4.4.1 Abstract . . . . . . . . . . . . . . . . . . . . . . . . . . 99
4.4.2 Contribution. . . . . . . . . . . . . . . . . . . . . . . . . 100
4.5 Reply to Garcia et al.: Common mistakes in measuring frequency dependent word
characteristics . . . . . . . . . . . . . . . . . . . . . . . . . . 100
4.5.1 Abstract . . . . . . . . . . . . . . . . . . . . . . . . . . 100
4.5.2 Contribution. . . . . . . . . . . . . . . . . . . . . . . . . 101
4.6 The game story space of professional sports: Australian Rules Football . . . . . 102
4.6.1 Abstract . . . . . . . . . . . . . . . . . . . . . . . . . . 102
4.6.2 Contribution. . . . . . . . . . . . . . . . . . . . . . . . . 102
4.7 The Lexicocalorimeter: Gauging public health through caloric input and output on
social media. . . . . . . . . . . . . . . . . . . . . . . . . . . 103
4.7.1 Abstract . . . . . . . . . . . . . . . . . . . . . . . . . . 103
4.7.2 Contribution. . . . . . . . . . . . . . . . . . . . . . . . . 103
4.8 Tracking the Teletherms: The spatiotemporal dynamics of the hottest and coldest
days of the year . . . . . . . . . . . . . . . . . . . . . . . . . 107
4.8.1 Abstract . . . . . . . . . . . . . . . . . . . . . . . . . . 107
4.8.2 Contribution. . . . . . . . . . . . . . . . . . . . . . . . . 107
4.9 DivergentDiscourseBetweenProtestsandCounter-Protests: #BlackLivesMatterand
#AllLivesMatter . . . . . . . . . . . . . . . . . . . . . . . . . 110
4.9.1 Abstract . . . . . . . . . . . . . . . . . . . . . . . . . . 110
4.9.2 Contribution. . . . . . . . . . . . . . . . . . . . . . . . . 110
5 Conclusion 112
5.1 Future directions . . . . . . . . . . . . . . . . . . . . . . . . . 112
5.1.1 Sentiment analysis . . . . . . . . . . . . . . . . . . . . . . 112
5.1.2 Emotional arcs . . . . . . . . . . . . . . . . . . . . . . . . 113
5.1.3 Other projects . . . . . . . . . . . . . . . . . . . . . . . . 114
5.2 Parting thoughts . . . . . . . . . . . . . . . . . . . . . . . . . 114
iv
Bibliography 116
Appendices 130
A Supplementary Material for Sentiment Dictionary Comparisons 131
A.1 S1 Appendix: Computational methods . . . . . . . . . . . . . . . . . 131
A.1.1 Stem matching . . . . . . . . . . . . . . . . . . . . . . . . 131
A.1.2 Regular expression parsing . . . . . . . . . . . . . . . . . . . 132
A.2 S2 Appendix: Continued individual comparisons . . . . . . . . . . . . . 133
A.3 S3 Appendix: Coverage for all corpuses. . . . . . . . . . . . . . . . . 137
A.4 S4 Appendix: Sorted New York Times rankings. . . . . . . . . . . . . . 139
A.5 S5 Appendix: Movie Review Distributions. . . . . . . . . . . . . . . . 141
A.6 S6 Appendix: Google Books correlations and word shifts . . . . . . . . . . 143
A.7 S7 Appendix: Additional Twitter time series, correlations, and shifts . . . . . . 148
A.8 S8 Appendix: Naive Bayes results and derivation . . . . . . . . . . . . . 154
A.9 S9 Appendix: Movie review benchmark of additional dictionaries . . . . . . . 159
A.10S10 Appendix: Coverage removal and binarization tests of labMT dictionary . . . 162
A.10.1Binarization . . . . . . . . . . . . . . . . . . . . . . . . . 162
A.10.2Reduced coverage . . . . . . . . . . . . . . . . . . . . . . . 167
B Supplementary Material for Emotional Arcs 179
B.1 Plot theories . . . . . . . . . . . . . . . . . . . . . . . . . . 179
B.2 Additional Figures . . . . . . . . . . . . . . . . . . . . . . . . 181
B.3 Emotional Arc Construction. . . . . . . . . . . . . . . . . . . . . 184
B.3.1 Null emotional arc construction. . . . . . . . . . . . . . . . . . 185
B.3.2 Further Gutenberg Processing . . . . . . . . . . . . . . . . . . 187
B.4 Book list . . . . . . . . . . . . . . . . . . . . . . . . . . . . 189
B.5 Principal Component Analysis (SVD) . . . . . . . . . . . . . . . . . 209
B.5.1 Additional details for 40 download threshold . . . . . . . . . . . . . 211
B.6 Additional Hierarchical Clustering Figures. . . . . . . . . . . . . . . . 248
B.7 Additional SOM Figures . . . . . . . . . . . . . . . . . . . . . . 257
B.8 Null comparison details. . . . . . . . . . . . . . . . . . . . . . . 258
B.8.1 Null SVD. . . . . . . . . . . . . . . . . . . . . . . . . . 259
B.8.2 Null Hierarchical Clustering . . . . . . . . . . . . . . . . . . . 263
B.8.3 Null Self Organizing Map (SOM) . . . . . . . . . . . . . . . . . 265
C labMTsimple: A Python Library for Sentiment Analysis 267
C.1 Getting Started . . . . . . . . . . . . . . . . . . . . . . . . . 267
C.1.1 Usage . . . . . . . . . . . . . . . . . . . . . . . . . . . 267
C.1.2 Installation . . . . . . . . . . . . . . . . . . . . . . . . . 268
C.1.3 Running tests . . . . . . . . . . . . . . . . . . . . . . . . 268
C.1.4 Developing with labMT-simple locally . . . . . . . . . . . . . . . 268
C.1.5 Building these docs . . . . . . . . . . . . . . . . . . . . . . 269
C.2 Detailed Examples . . . . . . . . . . . . . . . . . . . . . . . . 269
C.2.1 Preparing texts. . . . . . . . . . . . . . . . . . . . . . . . 269
C.2.2 Loading dictionaries . . . . . . . . . . . . . . . . . . . . . . 269
v
C.3 Making Wordshifts . . . . . . . . . . . . . . . . . . . . . . . . 270
C.3.1 Installing phantom-crowbar . . . . . . . . . . . . . . . . . . . 270
C.3.2 Installing inkscape . . . . . . . . . . . . . . . . . . . . . . 271
C.3.3 Installing rsvg . . . . . . . . . . . . . . . . . . . . . . . . 271
C.3.4 Installing labMTsimple . . . . . . . . . . . . . . . . . . . . . 271
C.3.5 Making your first shift . . . . . . . . . . . . . . . . . . . . . 271
C.3.6 Full Automation . . . . . . . . . . . . . . . . . . . . . . . 272
C.4 Advanced Usage . . . . . . . . . . . . . . . . . . . . . . . . . 272
C.4.1 About Tries . . . . . . . . . . . . . . . . . . . . . . . . . 272
C.4.2 Advanced Parsing. . . . . . . . . . . . . . . . . . . . . . . 272
D Code for VACC Twitter Database Keyword Searches 273
E Infrastructure of Hedonometer.org 276
F Online Code Repositories 278
vi
Chapter 1
Introduction and Literature Review
Epoch watch: Welcome to the Sociotechnocene.
-@peterdodds 10 Jan 2012
1.1 Introduction
Individualwordsencapsulateinformationandemotionasthebuildingblocksthatweusetocapture
experiences in stories. Beyond words, multi-word expressions (phrases), conceptual metaphor, and
complicatedgrammar(syntax)coalescetoprovideincredibleexpressivepower. Attemptstoquantify
semantic content build atop syntactic understand of language with the aim of transforming a model
of meaning that has proven useful to our own cognitive machinery into something more readily
applicable for another purpose (e.g., summarization by a computer). One such goal of semantic
understanding is to measure the sentiment expressed in written communication, which is broadly
knownassentimentanalysis. Thenextevolutionofnaturallanguagesystemswilltackletheharder-
yetproblemsofpragmatics,wherenarrativeunderstandingandgenerationcanenablecommon-sense
reasoning on par with human intuition.
In our work, we transfer the emotion of single, isolated words into a one-dimensional happiness
measure to build the Hedonometer. Leveraging the Hedonometer technology and modern compu-
tational power, we analyze digitized text with the ultimate goal of understanding stories. This
dissertation proceeds as follows: in this chapter we explore the foundations of sentiment analysis
and narrative structure. In Chapter 2 we benchmark and compare methods for sentiment analysis.
1
CHAPTER 1. INTRODUCTION AND LITERATURE REVIEW
In Chapter 3 we apply these methods and extract dominant emotional arcs from digitized text.
In Chapter 4, we discuss contributions made to published work in the broader science of complex
systems. Finally, in Chapter 5 we offer some concluding remarks.
Next, we examine prior work in natural language processing, sentiment analysis, and computa-
tional narrative understanding.
1.2 Sentiment analysis
ThefieldofNaturalLanguageProcessing(NLP)hasbeenaroundsincetheadventofcomputers,but
is growing rapidly alongside computational advances. While major advances have been made, there
remain many open problems. We focus here on a specific NLP problem, namely understanding the
emotional content of language. We refer to the emotional content in a written text broadly as the
sentiment. In addition to the summaries given in recent review articles (Giachanou and Crestani,
2016), the landscape of tools and technologies is expanding quickly and sentiment analysis systems
are deployed to tackle important challenges. As we will see, sentiment analysis is a sub-field of NLP
that can benefit from advancement in other realms of NLP as well (e.g., phrase partitioning).
Applications of sentiment analysis span academia, industry, and government. Just some of the
current uses include predicting elections (Tumasjan et al., 2010), product sales (Liu et al., 2007),
stock market movement (Bar-Haim et al., 2011), and tracking public opinion (Cody et al., 2015).
NLP and measures of sentiment are used to analyze consumption of information from the media,
and societal level decisions are driven by this flow of public opinion online. Beyond individual and
collective decisions, corporate success demands an understanding of the public sentiments directed
towards their products.
Advances in Artificial Intelligence (AI) have elucidated the distinction between problems that
are hard for computers and those that are hard for humans—a difference that is not obvious at the
outset. Determining sentiment is one such task: understanding the sentiment of our friends and
colleagues through informal text is relatively easy for us, but it is hard to codify in a computer
algorithm. As we will see, machine learning (often broadly referred to as AI) is finding uses in all
areas of Natural Language Processing (NLP), including advancing the state-of-the-art in sentiment
2
CHAPTER 1. INTRODUCTION AND LITERATURE REVIEW
classification and sentiment dictionary creation. While sentiment analysis benefits from machine
learning to create classifiers and sentiment dictionaries, the output of sentiment detection also aids
higher level approaches to language understanding.
1.2.1 Psychology of emotion
Withfewexceptions,currentsentimentanalysismethodsaimtodetectsentimentone-dimensionally,
givingascoreonarangefromnegativetopositivesentiment. Whilethispragmaticapproachproves
useful, Jack et al. (2014) conjectured that there are four basic emotions, Ekman (1992) names six,
and Plutchik (1991) identifies two additional basic emotions in humans. These theories are only the
mostwellknownclassifications,withatleast90suchclassificationsbeinggivenoverthepastcentury,
as noted by Plutchik (2001). Through the use of brain imaging and fMRI techniques, researchers
in neuroscience have also attempted to distinguish whether basic emotions are best captured as
discrete categories (anger, fear) or underlying dimensions (valence, arousal). Altogether they have
foundconsistentneurallocationsforbasicemotionsbutnoone-to-onemapping,andfurtherresearch
is still needed (Harrison et al., 2010; Hamann, 2012).
The widely acknowledged six basic emotions identified by Paul Eckman are:
• happy,
• surprised,
• afraid,
• disgusted,
• angry,
• and sad.
In Figure 1.1, a visualization of these six basic emotions is shown. As noted in the caption,
thesesixemotionsserveasabasisformorecomplexemotions. TheeightbasicemotionsofPlutchik
(1991) are shown as the variations along four dimensions in Figure 1.2. While we do not expect
thateachofthesixbasicemotionshaveorthogonalrepresentationsintheirembodimentinlanguage
(as orthogonality may be inferred from the Figures, is found in facial expression, and underlies the
theory), a basis of more than a single dimension is likely necessary to represent the full range of
emotion. Thebasicemotionstheoryrejectsthatallemotionscanberepresentedaseitherpositiveof
negative states, and this should extend to language. Indeed, attempts to cast the basic emotions as
3