Zipf's law is a consequence of coherent language production

1.3 MB·

by  Jake Ryland Williams

Zipf’s law is a consequence of coherent language production
Jake Ryland Williams,1,∗ James P. Bagrow,2,† Andrew J. Reagan,2,‡ Sharon
E. Alajajian,1,§ Christopher M. Danforth,2,¶ and Peter Sheridan Dodds2,∗∗
1School of Information, University of California,
Berkeley 102 South Hall #4600 Berkeley, CA 94720-4600.
2Department of Mathematics & Statistics, Vermont Complex Systems Center,
Computational Story Lab, & the Vermont Advanced Computing Core,
The University of Vermont, Burlington, VT 05401.
(Dated: August 9, 2016)
The task of text segmentation may be undertaken at many levels in text analysis—paragraphs,
sentences,words,orevenletters. Here,wefocusonarelativelyfinescaleofsegmentation,hypothe-
sizingittobeinaccordwithastochasticmodeloflanguagegeneration,asthesmallestscalewhere
independentunitsofmeaningareproduced. Ourgoalsinthisletterincludethedevelopmentofmeth-
6
odsforthesegmentationoftheseminimalindependentunits,whichproducefeature-representations
1
oftextsthatalignwiththeindependenceassumptionofthebag-of-termsmodel,commonlyusedfor
0
prediction and classification in computational text analysis. We also propose the measurement of
2
texts’association(withrespecttorealizedsegmentations)tothemodeloflanguagegeneration. We

g find(1)thatoursegmentationsofphrasesexhibitmuchbetterassociationstothegenerationmodel
u than words and (2), that texts which are well fit are generally topically homogeneous. Because
A our generative model produces Zipf’s law, our study further suggests that Zipf’s law may be a
consequence of homogeneity in language production.
5

PACSnumbers: 89.65.-s,89.75.-k,89.70.-a
]
L
C
Inourpreviousworkontextpartitioning[1],andagain writers who generate text for their meanings.
s. in our study on the effects of text mixing [2], we have Inourworkonstochastictextpartitioning[1],wepro-
c observed the relevance of the selection model proposed posed a simple mechanism for the fine-grained chunking
[
longagobySimon[3]. Wewillbuildonthismodelagain, of text into phrases. Under this framework, sentences

2 together with Zipf’s law [4, 5], as the basis for a model were broken down stochastically, splitting on whitespace
v of frequency data of text. with a uniform bond-breakage probability, q. While this
9
The defining aspects of Simon’s model for language (uniform) framework is clearly not the ideal mechanism
6
9 generation hold that as the terms of a text appear, they toisolatephrasesfor,say,dictionarylookups,wedidfind
7 do so independently, and in proportion to their historic that random partitioning of phrases (q = 1) produced
2
0 frequenciesofoccurrence. Thefirstassumption(ofmem- Zipf laws extending many orders of magnitude in rank
1. oryless term-term independence) is likewise the basis for beyond the standard version for words only.
0 the bag-of-terms model that is pervasive in the compu- Here, we wish to explore informed partitions, taking
6 tational text analysis community. It is evident that this into account empirical segmentation information. Some
1
assumptionfailsforstrictlydefinedwordswhenonecon- interesting options for estimating bond-breakage fre-
:
v sidersthedependenceoftermsthatarebound,likeNew, quencies exist—in language learning environments, sim-
i
X York, and City. As we have argued in [2], dependence ilar data are generated through exercises referred to as
can also be caused by mixing, where the rate of a term’s phrase-cuedtext,whichhavebeenshowntohelplearners
r
a occurrencevariesinacompositecorpusofmanytextsas developprosody[7](thepatternsofstressandintonation,
one transitions from document to document. considered to be one of the essential features of reading
The two types of dependence indicate that for the fluency [8]). This suggests value in the phrase-scale of
Simon model to apply, and consequently, for the bag- segmentation.
of-terms model to apply, texts should be analyzed on Toobtainphrases,surveys(essentiallyclassroomexer-
thehomogeneous(unmixed)scalesofproduction(i.e.,by cises) could be run that would benefit educators and
writer, oreventopic), andthatthetermsoftextsshould students, while developing a base of training data for
be segmented according to their local dependence (i.e., informed partitions. Ideally, such surveys would have
groupedtogetherintoirreducibleexpressions), rendering students learning a language asked to segment their own
distributions of terms of mixed sizes. The first criterion writing. However, this is unconventional in the class-
can be relatively easy to satisfy, and in our analyses will room, and poses a more difficult task to request. So, for
come for free by the curation of the Project Gutenberg now we will turn to expertly-segmented data.
eBooks[6]database. Thesecond,however,posesamuch Inthecomputationallinguistics(CL)communitythere
greater challenge, as the meanings that bind words into have been recent efforts to establish benchmarks and
meaning-irreducibleformsareonlyknownaprioritothe testing sets for multiword expressions (MWEs) (loosely
Typeset by REVTEX
2
defined as groups of “tokens in a sentence that cohere withtheresultingq-valuesfallingonthesamesideofq =
more strongly than ordinary syntactic combinations”) 0.5 for ∼ 65% of all unique word pairs (which later will
extraction [9]. While these data can be used to inform a be all that is necessary to ensure equivalent partitions),
partitioner, they are alone insufficient (narrowly focused andasmuchas75%forthosemostfrequentlyappearing.
on business reviews), and are better used for testing and While the two sources share coverage on about 20,000
benchmarking. Furthermore,itisnotclearifMWEsand word pairs (and disagree on a substantial number), we
their extraction (used for work in information retrieval notethattheWikipediahyperlinkssourceismuchlarger,
and other CL tasks) coincide with language generation with coverage on many proper nouns (names of places,
and distributional representations, which are the focus, people,etc.),whichwerefoundtobeimportantforMWE
here. We have also executed a companion piece to this identification in [10].
study[10],focusedonthetaskofMWEextraction,where In our original work on text partitioning [1], we con-
the text partitioning framework was tuned to exhibit its sidered stochastic text partitions, where bonds between
ability to perform as strongly, if not better than, the pairs words would either be preserved or broken at ran-
state-of-the-art [11], with very substantial increases in dom (uniformly). Despite our solving for the analyt-
speed, while being trained on less data. ic expectation across all possible partitions of phrase-
Fortraining,weturntothebond-breakageinformation partitionfrequencies,itbecamenecessarytoexecuteone-
held latently inside of the Wiktionary [12] with its trove offrandompartitionsforthestudyoftruerank-frequency
of larger-than-word entries and their example usages. distributions. HoweverforCLtasks,thesestochasticpar-
NotingthatexampleusagesofphrasesintheWiktionary titionshavelittlevalueonaccountoftheirvariation,and
are consistently coded in boldfaced text, we are able to whilethevariationinthecaseofinformedstochasticpar-
resolve isolated empirical bond-breakage frequency data. titionsisreduced, itisstillnotideal, asCLtasksrequire
For example, at the time of writing this letter, the entry consistency.
forthephrase“haveaball”wasrepresentedbytheusage: A perhaps unexpected bonus to the na¨ıvet´e of our
assumptions (specifically, defining the q’s to be indepen-
The kids had a ball playing in the fountain.
dentofoneanother)isthatthemaximumlikelihoodpar-
tition(MLP),ormostfrequentlyoccurringpartitionofa
which informed us that (kids,had) is a pair more like-
text, is easily computed. Accepting this partition (for a
ly split, and that (had,a) is a pair more likely bound.
text)isdeterministic,andaccomplishedsimplybybreak-
TheWiktionaryexampleshavetheaddedbonusofbeing
ing a bond whenever its bounding pair of words have a
trustworthy, on account of their being archetypal-usage
breakage probability of at least 0.5—note that this con-
examples, and will serve to inform our partitions for all
servativelysetstheill-determinedcaseofq =0.5tobond-
of the experiments in this letter. However, to make
breaking. Using this scheme, a text can be segmented
informed text partitions available in many other lan-
very rapidly based on the Wiktionary (with the same
guages(whereexamplesmaybelessprolificand/orcoded
computational complexity as simple word count), and
differently), we also construct bond-breakage frequencies
requiresnoglobalinformation. Forallofthesereasons—
in a similar manner, but from the embedded hyperlinks
speed, determinism, and maximum likelihood—we con-
on Wikipedia, which are ubiquitous and coded-for con-
tinue with the MLP in this work.
sistently across languages. From either source of data,
We now return to Simon’s model of language gen-
we define the partitioning probability, q(w ,w ), for an
(cid:96) r
eration, which analytically produces single-parameter
ordered pair of words, (w ,w ), according to the total
(cid:96) r
power-law rank-frequency distributions, and is our
number of known bond breaking appearances f (w ,w )
b (cid:96) r
hypothesis as the dominant productive mechanism for
and the total number of bond preserving appearances
Zipf’s law. Zipf’s law succinctly formulates a relation-
f (w ,w ):
p (cid:96) r
ship between the frequencies of occurrence of terms, and
f (w ,w ) their ranks by frequency (descending):
q(w ,w )= b (cid:96) r ,
(cid:96) r fp(w(cid:96),wr)+fb(w(cid:96),wr) f =C·r−θ, (1)
with a default value of q(w ,w ) = 1 for all pairs unob- for a scaling constant, C. Simon’s model has one tun-
(cid:96) r
servedinatrainingset. Notethatthismodelisstillquite ableparameter—therateoftermintroduction, α. When
na¨ıve, and may be improved by allowing values of q to language is generated by Simon’s model, Zipf’s scaling
vary, dependingontermsexistingfartherawayinatext. exponent is given by θ = 1−α, and in the case of an
However, training a more sophisticated partitioner like empiricaltext,thetermintroductionratecanbeinferred
this would be done best if the training data were fully as α = N/M, where N is the number of unique term
segmented (which might be accomplished with phrase- typesinthetext, andM isthetotalnumberofallterms
cued text). occurring in a text.
We note that both sources (examples and hyperlinks) The main caveat for the compatibility of Zipf’s empir-
haveareasonabledegreeofaccord(consideringEnglish), ical law with Simon’s theoretical model arises from the
3
goodnessoffitfortheZipf/Simonmodel, andasaresult
of the Simon model’s independence assumption, a mea-
sureofthequalityofatext’sbag-of-termsrepresentation.
Notethatthiswillalsogiveusanopportunitytoevaluate
the qualitative nature of language produced by Simon’s
model.
Since the connection of Simon’s model to the parame-
ter θ occurs naturally in the complimentary cumulative
distribution function (CCDF) of term frequencies, we
measure and regress all quantities along CCDFs, while
we present all results in the intuitive and familiar rank-
frequency (Zipf) representations.
In Fig. 1, we plot an example informed partition for
Herman Melville’s “Moby Dick” (black line, main axes),
and behind it, the spectrum of uniform stochastic par-
titions ranging from q = 0 (green) sentences/clauses, to
q =1(pink)words. Onecanseethedistributionssteepen
alongthegradientasqisincreased(whichisgenerallythe
trend throughout the eBooks), and as much can be seen
in the lower inset, where a regressed scaling parameter θ
is compared with θ = 1−α. Also within the main
mod
axesliestheZipf/Simonfitforthetheinformedpartition
(blue, dashed), which appears reasonable by inspection,
FIG. 1: The main axes show rank-frequency distributions
and outperforms all uniform partitions (including the
from Herman Melville’s “Moby Dick” for one-off uniform
q =1wordpartition)bygoodnessoffitoftheZipf/Simon
stochasticpartitionsforvaluesofq rangingfrom0(green)to
1(pink),togetherwithaone-off,dictionary-informedstochas- model(upperinset),wheretheinformedpartitionisindi-
ticpartition(black)anditsZipf/Simonmodel(blue,dashed). cated by the black point (in both insets). Note that this
In the upper inset we present the Zipf/Simon goodness of fit increased quality of fit by the Zipf/Simon model is also
asafunctionofq (withtheblackpointfortheinformedpar- observed readily in the lower inset, where θ and 1−α
tition positioned at the text’s average value of q), and in the
nearly align.
lower inset we present regressed scalings (θ) against those
Zooming out to the larger collection of (roughly
determined by the Zipf/Simon model (1−α). In the lower
inset,completeagreementoccursalongthe(bluedashed)line 20,000)EnglisheBooks,wefocusoncomparingtwopar-
1−α=θ, and for both insets, colors of points correspond to titions for each text—the q = 1 (word) partitions, and
those in the main axes. the informed (phrase) partitions. In Fig. 2, we compare
thegoodnessoffitbytheZipf/Simonmodelbetweenthe
two partition types. This comparison divides the collec-
fact that Simon’s model is only able to generate single- tionofbooksintotwosubsetsthatwewillrefertoasthe
scaling rank-frequency distributions with scaling param- word-based and phrase-based texts. Our demarcation
eters θ in [0,1). While many texts exhibit multiple scal- is given by the line R2 = R2 (red, dashed), and
q=1 MLP
ingsandvaluesofθlargerthan1,wehaveseenthatthese shows that most texts were better bag-models under the
anomalies are often the result of text mixing and term informed (phrase-based, R2 <R2 ) partition frame-
q=1 MLP
dependence [2]. Consequently, we view the degree to work. While a good number of texts (∼20%) are word-
which a particular Zipf/Simon model aligns with empir- based, we note that these texts tend to have weaker R2
ical rank-frequency data as a measure of the goodness values in general, which can be seen in Fig. 3 where we
of fit for the ‘bag-of-terms’ representation (tokenization) apply an R2 cutoff to the books and measure the por-
of a text (regardless of how the terms are defined, i.e., tion of the data set removed. This quantifies the num-
words or phrases). bers of phrase-and word-based texts that are strong fits
TheZipf/SimonfitforatextofN uniqueandM total (R2 > 0.6) to be in a proportion of more than 12:1,
words is defined as follows. Assuming a text was gener- respectively, indicating that a strongly-fit bag-of-terms
ated precisely according to Simon’s model, the constant representation of a text is better (and generally quite
word introduction rate is inferred by the text-wide aver- significantly) achieved by MLP phrases more than 90%
age: α=N/M, and an estimate of the scaling exponent of the time.
is then θ = 1−α. The exact form of the model’s From Fig. 3, we can see that R2 ∼0.6 appears near a
mod
fit is then obtained by computing the constant of pro- sharp drop for the cutoff, which, if applied as a thresh-
portionality: Cmod = Nθmod, whereupon we may take old for analysis, results in approximately a 20% loss (red
the coefficient of determination, or R2, as a measure of line/stars) for the dataset (largely due to the low-R2,
4
FIG. 2: We plot the goodness of fit (R2) of the Zipf/Simon FIG. 3: The percent of books in the Project Gutenberg
English eBooks database that are phrase-based (solid green
model,appliedtothetextsoftheEnglishProjectGutenberg
line,filledsquares),word-based(solidblueline,filledcircles),
eBooksdatabase,wherethedictionary-informedone-offparti-
andpoorlyfit(redline,stars),whenacutoffinR2 isapplied
tions(horizontalaxis)areplottedagainsttheq=1wordpar-
(right). A dropoff in R2 occurs near R2 ∼ 0.6, which is
titions (vertical axis). The discriminating line (red, dashed, MLP
R2 =R2 ) helps divide the collection into texts that are denoted by the vertical, black dotted line. We also present
MLP q=1 curvesthatindicatethepercentsofbooksremainingabovethe
word-based and phrase-based.
cutoffwhenblanketedusageofeitherwords(bluedashedline,
open circles) or informed phrases (green dashed line, open
squares) are applied.
word-based texts). By allowing the books to be either
phrase- or word-based (according to their R2 values), we
are then able to accommodate a strong (R2 ≥0.6) ‘bag’ Fig. 4, where we observe stark differences in the ranges
analysis for over 80% of the dataset, which is a large of box plots. As the majority of texts have an MLP
improvement over the conventional, blanket usage of representation that is relatively strong (R2 > 0.6),
MLP
words(dashedblueline/opencircles),whichonlyaccom- the medians for most classifications are high. Howev-
modates strong fits for approximately 37.5% of the col- er, it is clear that texts under classification A—general
lection. works, which includes dictionaries, encyclopedias, and
Whileonecanviewthe∼20%ofpoorly-fitbooksasa periodicals—are more often poorly-fit. This is of note,
loss to quality analysis, it is possible that refinement of as all other classifications, including textbook-reference
the partition function (through better data or modeling) classifications, such as medicine (R), law (K), and naval
would allow one to obtain better tokenizations/fits for (V) and military (U) science, exhibit medians substan-
all books. However, one can also look at these 20% in tially above 0.6, though we note that many classifica-
a different way—as potentially being unfit for the bag- tionsexhibitoutliersoflowR2 ,notablywithlanguage
MLP
of-terms framework. Sorting the eBooks according to and literature (P), which contains long, brief, and mixed
max{R2 ,R2 },wefindthetenpoorestfitstoinclude texts.
q=1 MLP
dictionaries,spellingbooks,andbooksofextremelysmall The main difference we note between classification A
size (often as placeholders for other media), indicating and the other reference classifications mentioned is that
that low R2 values appropriately identify texts unfit for thegeneralworksofAarefrequentlynottopicallyhomo-
analyses. geneous. This observation is echoed in the bottom of
To explore the qualitative nature of fit quality and Fig.4, whereweconsidertheeBooks’subjects. Here, we
the Zipf/Simon model, we also consider the variation of again see low values for dictionaries and periodicals, but
R2 (strictly, as R2 is so frequently outperformed) likewise for other heterogeneous subjects, such as ques-
MLP q=1
across the different Library of Congress classifications tions and answers, travel, and social life. We also find
(Fig. 4, top), and subjects (Fig. 4, bottom) provided that poetry, short stories, and science fiction all exhibit
in the eBooks collection. This variation can be seen in alargeproportionoflowR2 values. Thesesubjectsall
MLP
5
share the commonality of frequent brevity. This makes
some sense in relation to the Simon model, as the gener-
ation of a text may not have an opportunity to stabilize
(distributionally). We also note that the science fiction
subject may frequently use phrases that are outside of
the informed partitioner’s training.
Our results show that informed text partitioning is
capable of improving the basic methodologies of fea-
ture selection in text analysis on a broad scale, allow-
ing for better adherence to assumptions (specifically, for
the bag-of-terms model) that underpin a vast collection
of algorithms currently in practice throughout industry
and academia. Additionally, phrase-based text analysis
improves the independent interpretability of features—
in the soft sense, at the level of user experience. With
phrase-based text analysis, end-point users (e.g., poli-
cy makers or product users interpreting lists of phrase-
feature topics from a topic model readout) who may not
understand the workings of an algorithm will be better
able to interpret results, as phrases provide critical con-
text for interpretation. We have also proposed the mea-
surement of association to a model of language genera-
tion, which we have found here to be the likely mech-
anism for the production of topically-homogeneous lan-
guage, allowing us to assert new meaning for the pres-
ence of Zipf’s law in natural language—long observed,
thoughlargelyamystery. Wealsosuggestthepossibility
that as phrase-segmentation methods improve, associa-
FIG. 4: Box plots, showing variation of R2 across the
tion to the Zipf/Simon model might also be used as a MLP
primaryLibraryofCongressclassifications(top),andaselec-
metric for determination of topical breaks in text, since tionoftheProjectGutenbergeBookssubjects(bottom). All
R2 doesappeartoincreaseinthepresenceoftopically- boxes represent at least 50 texts.
MLP
homogeneous texts.
To make these tools both explorable and avail-
Dodds, CoRR (2015), http://arxiv.org/abs/1409.3870.
able to the computational text analysis community,
[3] H. A. Simon, Biometrika 42, 425 (1955).
we have with this letter developed a Python pack-
[4] G.K.Zipf,The Psycho-Biology of Language (Houghton-
age for text partitioning (for detailed information, Mifflin, 1935).
see https://github.com/jakerylandwilliams/partitioner), [5] G.K.Zipf,HumanBehaviourandthePrincipleofLeast-
which makes our tools available in 11 languages Effort (Addison-Wesley, 1949).
(English, German, Russian, Portuguese, Polish, Dutch, [6] Project Gutenberg (2010), http://www.gutenberg.org.
[7] M. J. Glavach, The Practical Teacher (2011).
Italian, French, Finnish, Spanish and Greek), and
[8] J.MillerandP.J.Schwanenflugel,JournalofEducation-
in addition present an explorable online appendix
al Psychology 98, 839 (2006).
(http://jakerylandwilliams.github.io/partitioner/).
[9] N. Schneider, S. Onuffer, N. Kazour, E. Danchik, M. T.
Mordowanec, H. Conrad, and N. A. Smith, in Proceed-
ings of the Ninth International Conference on Language
Resources and Evaluation (LREC’14) (European Lan-
guage Resources Association (ELRA), Reykjavik, Ice-
∗ Electronic address: [email protected] land, 2014), ISBN 978-2-9517408-8-4.
† Electronic address: [email protected] [10] J. R. Williams, CoRR (2016), URL http:
‡ Electronic address: [email protected] //people.ischool.berkeley.edu/~jakeryland/
§ Electronic address: [email protected] documents/williams2016b.pdf.
¶ Electronic address: [email protected] [11] N. Schneider and N. A. Smith, in Proceedings of the
∗∗ Electronic address: [email protected] 2015 Conference of the North American Chapter of the
[1] J. R. Williams, P. R. Lessard, S. Desu, E. M. Association for Computational Linguistics: Human Lan-
Clark, J. P. Bagrow, C. M. Danforth, and guage Technologies (Association for Computational Lin-
P. S. Dodds, CoRR abs/1406.5181 (2015), guistics, Denver, Colorado, 2015), pp. 1537–1547, URL
http://arxiv.org/abs/1406.5181. http://www.aclweb.org/anthology/N15-1177.
[2] J. R. Williams, J. P. Bagrow, C. M. Danforth, and P. S. [12] Wiktionary(2014),http://dumps.wikimedia.org/enwiktionary/2014.