Embodied Properties of Semantic Knowledge Acquired From Natural Language

108 Pages·2017·1.12 MB·English

UUnniivveerrssiittyy ooff WWiinnddssoorr
SScchhoollaarrsshhiipp aatt UUWWiinnddssoorr
Electronic Theses and Dissertations Theses, Dissertations, and Major Papers
2013
EEmmbbooddiieedd PPrrooppeerrttiieess ooff SSeemmaannttiicc KKnnoowwlleeddggee AAccqquuiirreedd FFrroomm
NNaattuurraall LLaanngguuaaggee
Kevin K. Durda
University of Windsor
Follow this and additional works at: https://scholar.uwindsor.ca/etd
RReeccoommmmeennddeedd CCiittaattiioonn
Durda, Kevin K., "Embodied Properties of Semantic Knowledge Acquired From Natural Language" (2013).
Electronic Theses and Dissertations. 4910.
https://scholar.uwindsor.ca/etd/4910
This online database contains the full-text of PhD dissertations and Masters’ theses of University of Windsor
students from 1954 forward. These documents are made available for personal study and research purposes only,
in accordance with the Canadian Copyright Act and the Creative Commons license—CC BY-NC-ND (Attribution,
Non-Commercial, No Derivative Works). Under this license, works must always be attributed to the copyright holder
(original author), cannot be used for any commercial purposes, and may not be altered. Any other use would
require the permission of the copyright holder. Students may inquire about withdrawing their dissertation and/or
thesis from this database. For additional inquiries, please contact the repository administrator via email
([email protected]) or by telephone at 519-253-3000ext. 3208.
E P S K
MBODIED ROPERTIES OF EMANTIC NOWLEDGE
A F N L
CQUIRED ROM ATURAL ANGUAGE
by
KevinDurda
ADissertation
SubmittedtotheFacultyofGraduateStudies
throughtheSchoolofComputerScience
inPartialFulfillmentoftheRequirementsfor
theDegreeofDoctorofPhilosophyatthe
UniversityofWindsor
Windsor,Ontario,Canada
2013
(cid:13)c 2013KevinDurda
Library and Archives Bibliothèque et
Canada Archives Canada
Published Heritage Direction du
Branch Patrimoine de l'édition
395 Wellington Street 395, rue Wellington
Ottawa ON K1A 0N4 Ottawa ON K1A 0N4
Canada Canada
Your file Votre référence
ISBN: 978-0-494-98350-8
Our file Notre référence
ISBN: 978-0-494-98350-8
NOTICE: AVIS:
The author has granted a non- L'auteur a accordé une licence non exclusive
exclusive license allowing Library and permettant à la Bibliothèque et Archives
Archives Canada to reproduce, Canada de reproduire, publier, archiver,
publish, archive, preserve, conserve, sauvegarder, conserver, transmettre au public
communicate to the public by par télécommunication ou par l'Internet, prêter,
telecommunication or on the Internet, distribuer et vendre des thèses partout dans le
loan, distrbute and sell theses monde, à des fins commerciales ou autres, sur
worldwide, for commercial or non- support microforme, papier, électronique et/ou
commercial purposes, in microform, autres formats.
paper, electronic and/or any other
formats.
The author retains copyright L'auteur conserve la propriété du droit d'auteur
ownership and moral rights in this et des droits moraux qui protege cette thèse. Ni
thesis. Neither the thesis nor la thèse ni des extraits substantiels de celle-ci
substantial extracts from it may be ne doivent être imprimés ou autrement
printed or otherwise reproduced reproduits sans son autorisation.
without the author's permission.
In compliance with the Canadian Conformément à la loi canadienne sur la
Privacy Act some supporting forms protection de la vie privée, quelques
may have been removed from this formulaires secondaires ont été enlevés de
thesis. cette thèse.
While these forms may be included Bien que ces formulaires aient inclus dans
in the document page count, their la pagination, il n'y aura aucun contenu
removal does not represent any loss manquant.
of content from the thesis.
EMBODIEDPROPERTIES OF SEMANTICKNOWLEDGEACQUIREDFROM NATURAL
LANGUAGEUSAGE
by
KevinDurda
APPROVEDBY:
D.Inkpen
SchoolofElectricalEngineeringandComputerScience
UniversityofOttawa
T.Traynor
Mathematics&Statistics
R.Gras
ComputerScience
L.Rueda
ComputerScience
L.Buchanan
Psychology
R.Caron
Mathematics&Statistics
March12th,2013
Author’sDeclarationofOriginality
IherebycertifythatIamthesoleauthorofthisthesisandthatnopartofthisthesis
hasbeenpublishedorsubmittedforpublication.
Icertifythat,tothebestofmyknowledge,mythesisdoesnotinfringeuponanyone’s
copyrightnorviolateanyproprietaryrightsandthatanyideas,techniques,quotations,
or any other material from the work of other people included in my thesis, published
orotherwise,arefullyacknowledgedinaccordancewiththestandardreferencingprac-
tices. Furthermore, to the extent that I have included copyrighted material that sur-
passes the bounds of fair dealing within the meaning of the Canada Copyright Act, I
certifythatIhaveobtainedawrittenpermissionfromthecopyrightowner(s)toinclude
suchmaterial(s)inmythesisandhaveincludedcopiesofsuchcopyrightclearancesto
myappendix.
I declare that this is a true copy of my thesis, including any final revisions, as ap-
provedbymythesiscommitteeandtheGraduateStudiesoffice,andthatthisthesishas
notbeensubmittedforahigherdegreetoanyotherUniversityorInstitution.
iii
Abstract
Thesymbolinterdependencyhypothesis(Louwerse,2007,2008)positsthatwordmean-
ingisdependentupontwosourcesofinformation: embodiedorgroundedknowledge,
obtainedfromobservationofandinteractionwiththephysicalworld,andsymbolicor
co-occurrenceinformation,gleanedfromexperiencewithhowwordsareusedtogether
inwrittenandspokenlanguage. Thistheoryassumesthatembodiedpropertiesofob-
jectsinfluencethestatisticalstructureoflanguagetosuchanextentthattheembodied
propertiesbecomepartiallyencodedwithinthestructureoflanguage.
Theworkpresentedinthisdissertationprovidessupportforthesymbolinterdepen-
dency hypothesis by demonstratingthat grounded knowledge (in the form of physical
and behavioural properties of living and non-living objects) can be identified by ana-
lyzing word usage in a large body of written text. An automated method of creating
high-dimensionalvector-basedsemanticrepresentationsispresented. Severaldemon-
strationsshowthattherepresentationscapturewordmeaninginawaythatalignswith
intuitionand are able to reproduce non-intuitiveresults of experiments from the psy-
cholinguisticliterature.
A feedforward neural network was trained to produce a list of physical and behav-
iouralpropertiesofanobjectinresponsetotheobject’shigh-dimensionalvectorrepre-
sentation. Theresultingnetworkwasabletoidentifyfeaturesoftheconceptsonwhich
itwastrainedwithnear-perfectaccuracyandwasabletogeneralizethisabilitytonovel
concepts and identify properties of concepts to which it was not previously exposed.
Theseresultsindicatethatthereissufficientinformationinwordusagetoidentifyem-
bodiedpropertiesofconcepts,afindingthatisconsistentwiththesymbolinterdepen-
dencyhypothesis.
iv
Acknowledgments
Firstandforemost, Iwould liketosaythankyou tomysupervisors,Dr. Richard Caron
andDr. LoriBuchanan. Iwouldnothavebeenabletocompletethiswithoutyourmo-
tivation, expertise, insight, and support. Thank you both for the opportunity to work
with you, for your guidance, both dissertation-related and otherwise, throughout the
years, and for your patience while I meandered through graduate school. Thank you
to my committee members, Dr. Diana Inkpen, Dr. Tim Traynor, Dr. Robin Gras, and
Dr. Luis Rueda, for your critiques and comments on my research. For their unwaver-
ingsupportandencouragement,thankyoutomyparents,PatandJohn,andmysister,
Krystelle. Thankyou to myfriendsandlabmates,ScottWisdom, ChrisDomen, Court-
neyHeffernan,TaraMcHugh,KareyWilson,GillianMcDonald,BruceKing,andDarren
Schmidt. Chris, thanks for all the times we spent studying, discussing research ideas,
andthecountlessroundsofgolf. ScottandCourtney,thankyouforthelate-nightstudy
sessions. Iwouldn’thavemadeitthroughgradschoolwithoutyouguystoworkandre-
laxwith. Finally,thankyou,LindsayMcNiff,forbeingsupportiveandinspiring,andfor
yourpatience,evenwhiletheAncientOnesstirredthreateninglyintheirslumber.
This work was made possible by the facilities of the Shared Hierarchical Academic
Research Computing Network (SHARCNET: ) and Com-
http://www.shar
net.
a/
pute/CalculCanada.
v
Table of Contents
Author’sDeclarationofOriginality iii
Abstract iv
Acknowledgments v
ListofFigures viii
ListofTables ix
1 Introduction 1
2 CreatingCo-occurrenceRepresentations 17
2.1 DescriptionofModel . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17
2.2 InputCorpus . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23
2.3 ImplementationDetails . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24
3 DemonstrationsofModel 26
3.1 IndependenceofFrequency . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26
3.2 MultidimensionalScalingResults . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26
3.2.1 CategoricalInformation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26
3.2.2 ProperNames . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32
3.2.3 PartsofSpeech. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 34
3.2.4 TemporalInformationandRelativeMagnitudes . . . . . . . . . . . . . . 36
3.3 SimulationsofBehaviouralResults . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 38
3.3.1 SemanticandAssociativePriming. . . . . . . . . . . . . . . . . . . . . . . . . 38
3.3.2 MediatedPriming . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 45
3.3.3 CategoryNorms . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 48
3.4 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 50
4 IdentifyingFeaturesinCo-occurrenceRepresentations 51
4.1 NetworkStructure. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 51
4.2 TrainingandTestingData . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 52
4.3 NetworkDynamics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 55
4.4 TheBackpropagationAlgorithm . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 57
4.5 MeasuringNetworkPerformance . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 61
4.6 DeterminingNetworkParameters . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 62
4.7 Experiments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 62
4.8 EffectofNumberofInputandOutputUnits. . . . . . . . . . . . . . . . . . . . . . . 70
5 SummaryandConclusions 75
5.1 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 75
5.2 ShortcomingsandFutureWork . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 80
5.3 SummaryofContributions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 83
vi
Bibliography 85
Appendices 95
AppendixA SubsetofRosch(1975)Norms 95
AppendixB Part-of-speechStimuli 96
VitaAuctoris 98
vii
List of Figures
1.1 Asemantichierarchy. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2
1.2 Asemanticnetwork. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3
3.1 Scatterplotofwordsimilarityagainstwordlog-frequency. . . . . . . . . . . . . 27
3.2 Multidimensionalscalingofanimals,bodyparts,geographicallocations,
andcities.. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28
3.3 Meansimilaritybetweenitemsinthecategoriesanimals,bodyparts,ge-
ographicallocations,andcities.. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29
3.4 MultidimensionalscalingofconceptsfromRosch(1975)norms.. . . . . . . . 30
3.5 Mean similaritybetween itemsineach categoryfrom Rosch (1975)cate-
gorynorms. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31
3.6 Multidimensionalscalingofcommonnouns,maleandfemalegivennames,
andsurnames. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33
3.7 Meansimilaritybetweenasampleofcommonandpropernouns. Proper
nounsweretypicallymale,typicallyfemale,orsurnames.. . . . . . . . . . . . . 34
3.8 ResultsofMDSperformedonwordsfromfourpart-of-speechcategories. . 36
3.9 Meansimilaritybetweenitemsineachpart-of-speechcategory. . . . . . . . . 37
3.10 ResultsofsimulationofChiarello,etal.1990. . . . . . . . . . . . . . . . . . . . . . . 40
3.11 ResultsofsimulationofFerrandandNew2003. . . . . . . . . . . . . . . . . . . . . 42
4.1 TheneuralnetworkusedinthesimulationsinChapter4. . . . . . . . . . . . . . 52
4.2 A scatter plot showing the relationship between number of active input
unitsandnumberofactiveoutputunits. . . . . . . . . . . . . . . . . . . . . . . . . . 53
4.3 Precision,recall,F,andcross-entropyerrorbyitemtypeandtrainingepoch. 65
4.4 Precision,recall,andF fortrainingitems. . . . . . . . . . . . . . . . . . . . . . . . . 65
4.5 Precision,recall,andF fortestingitems. . . . . . . . . . . . . . . . . . . . . . . . . . 66
4.6 Precision,recall,andF forrandomizedtestitems. . . . . . . . . . . . . . . . . . . 66
4.7 Cross-entropyerrorbyitemtype. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 67
viii