Table Of ContentNiladri Sekhar Dash · L. Ramamoorthy
Utility and
Application
of Language
Corpora
Utility and Application of Language Corpora
Niladri Sekhar Dash L. Ramamoorthy
(cid:129)
Utility and Application
of Language Corpora
123
Niladri Sekhar Dash L. Ramamoorthy
Linguistic Research Unit Linguistic DataConsortium-Indian
Indian Statistical Institute Languages
Kolkata, West Bengal, India Central Institute ofIndian Languages
Mysore, Karnataka,India
ISBN978-981-13-1800-9 ISBN978-981-13-1801-6 (eBook)
https://doi.org/10.1007/978-981-13-1801-6
LibraryofCongressControlNumber:2018949868
©SpringerNatureSingaporePteLtd.2019
Thisworkissubjecttocopyright.AllrightsarereservedbythePublisher,whetherthewholeorpart
of the material is concerned, specifically the rights of translation, reprinting, reuse of illustrations,
recitation, broadcasting, reproduction on microfilms or in any other physical way, and transmission
orinformationstorageandretrieval,electronicadaptation,computersoftware,orbysimilarordissimilar
methodologynowknownorhereafterdeveloped.
The use of general descriptive names, registered names, trademarks, service marks, etc. in this
publicationdoesnotimply,evenintheabsenceofaspecificstatement,thatsuchnamesareexemptfrom
therelevantprotectivelawsandregulationsandthereforefreeforgeneraluse.
The publisher, the authors and the editors are safe to assume that the advice and information in this
book are believed to be true and accurate at the date of publication. Neither the publisher nor the
authorsortheeditorsgiveawarranty,expressorimplied,withrespecttothematerialcontainedhereinor
for any errors or omissions that may have been made. The publisher remains neutral with regard to
jurisdictionalclaimsinpublishedmapsandinstitutionalaffiliations.
ThisSpringerimprintispublishedbytheregisteredcompanySpringerNatureSingaporePteLtd.
Theregisteredcompanyaddressis:152BeachRoad,#21-01/04GatewayEast,Singapore189721,
Singapore
To
Prof. N. Deiva Sundaram
and
The crescent moon floating on the Marina
beach
Preface
Betweenthedesire
Andthespasm
Betweenthepotency
Andtheexistence
Betweentheessence
Andthedescent
FallstheShadow
ForThineistheKingdom
T.S.Eliot:TheHollowMan
Inrecentyears,therehasbeenanotableupsurgeintheuseoflanguagecorpusasa
dependableresource inallmajordomainsoflinguistics.Thegrowing tendencyfor
using language corpus as a source of authentic empirical evidence is triggered due
to easy access to corpus data in a machine-readable form. Also, objective analysis
ofcorpusdatacontributesinamajorwaytotheunderstandingoftheintricateform
and function of language properties considered useful in various domains of lin-
guistics.Thischangeinapproachreflectsontheideologicalandtechnologicalshifts
the discipline has undergone over the last 60 years or more.
Theideologicalshifthasfocusedonempiricallanguagedataasthemosttrusted
linguistic evidence because our traditional intuitive assumptions are proved to be
not much convincing for drawing reliable, conclusive observations about various
aspectsofanaturallanguageonwhichprinciplesandtheoriesoflinguisticscanbe
formulated. This leads us to focus on empirical language data so that our under-
standings of languages and linguistics are reliable and trustworthy. The techno-
logical shift, on the other hand, has given us an opportunity to use a computer to
collect language data ina new device, access it in different platforms,process it in
different ways, analyze it from different perspectives, and utilize it in different
domains.Thus,computerbringsinanewshadeintheworldoflinguisticstolookat
the language from newer angles to make our studies authentic and reliable.
vii
viii Preface
In reality, thecomputer hasopened before us many new ways oflooking atthe
language data, interpreting it in a new light and utilizing it in various fields of
linguistics. This was hardly possible before the use of the computer in linguistics.
In the earlier years, scholars had to be content with handmade language data for
their works because they had no system under their access by which they could
assemble a large amount of language data from different spheres of language use
within a short span of time, interpret them, and utilize them. Now the scenario is
greatly changed. Now we can use a computer to collect language data of any size,
type, and variety as well as analyze it to find out new examples, new information,
and new evidence to furnish in our studies. The referential value of a corpus is
increasingdaybydaybecauseacorpushasthepotentialtocontributetonewtheory
formulation as well as modifying the existing ones. Corpus-based studies are
incorporating new insights to look into the cognitive areas of the human mind to
understand the mysteries operating behind the cognitive process like receiving,
processing, comprehending, and sharing linguistic signals (Winograd 1983: 18).
Nearlyaftersixdecadesofcorpususeinvariousfieldsofdescriptivelinguistics,
appliedlinguistics,andlanguagetechnology,itisclearthattheutilityofacorpusis
not limited to language teaching and dictionary compilation. It is now an open
resourcewhichcanbeusedinanydomainofsocialscienceswherelanguagedatais
an integral part, although scholars more often were found to depend on it in the
activitieslikelanguageprocessing,machinelearning,sentimentanalysis,dictionary
compilation, grammar writing, WordNet design, word-sense disambiguation,
translation, language documentation, and other areas of linguistics (Dash and
Arulmozi2018:Chap.14).Sincetheanalysisofdatainalanguagecorpussupplies
important authentic perspectives toward linguistic description and interpretation
(Biber 1996), information retrieved from a corpus can also be used, besides the
domains stated above, in several other linguistic works like diachronic lexical
semantics, pragmatic analysis of texts, sociolinguistic studies, and discourse anal-
ysis (Leech and Fligelstone 1992). From an informal pilot survey, we have found
that digital language corpora are used in more than 500 different types of devel-
opment and research works covering almost all the major areas of linguistics and
languagetechnology (Dash andChaudhuri 2003). Themultipurpose applicationof
language corpora is best visualized by Svartvik (1986):
…lexicography, lexicology, syntax, semantics, word-formation, parsing, question-answer
synthesis, software development, spelling checkers, speech synthesis and recognition,
text-to-speech conversion, pragmatics, text linguistics, language teaching and learning,
stylistics, machinetranslation, childlanguage,psycholinguistics, sociolinguistics, theoret-
icallinguistics,corpusclonesinotherlanguagessuchasArabicandSpanish—well,even
languageandsex.
We can summarize the importance of a language corpus in the following
manner.Therelevanceofcorpusisunderstandablewhenwerealizethatthegoalof
a corpus is to perform the following tasks (Leech 1992):
Preface ix
(a) Focus on linguistic performance rather than competence,
(b) Focus on linguistic description rather than linguistic universals,
(c) Focus on quantitative and qualitative models of language,
(d) Focus on an empiricist, rather than a rationalist, view of scientific inquiry of a
language.
Eachoftheargumentsstatedabovecontrastswith‘Chomskyanparadigmwhich
hasdominatedmuchoflinguistic thinkingsincethe1950’(Leech 1992:107).The
scopeofcorpususeisfurtherexpandedintheobservationsofAtkinsetal.(1992),
Leech and Fligelstone (1992), McEnery and Wilson (1996), Rundell (1996),
Barlow(1996),ThomasandShort(1996),Biber(1996),Biberetal.(1998),Teubert
(2000),Cheng(2011),CrawfordandCsomay(2015),McEneryandHardie(2011),
Vandelanotte et al. (2014), Weisser (2015), Dash (2005), Dash and Arulmozi
(2018), and many others. In essence, a language corpus is a valuable linguistic
resource for all areas of descriptive linguistics, language technology, applied lin-
guistics, discourse analysis, and cognitive linguistics. With the multidimensional
use of language data, corpus linguistics now emerges as a new approach toward
linguistics. It is also identified as a new way of studying and applying natural
language using the techniques of computer science (Landau 2001: 277).
The appreciation for language corpus in linguistics is a new thing. It has been
possible because the data and information obtained from a corpus have been of
muchuseindevelopingcorpus-basedgrammarandtextmaterialsforbothfirst-and
second-languagelearners.Atthesametime,thecreationofacorpus,inbothspoken
andwritten forms,hasbeenanewtaskofdifferent orientation thathascontributed
in two different ways:
(a) It has developed a new league of scholars who are very much skilled in the
generation of speech and text corpora as well as in processing them.
(b) It has developed a pool of data and information for catering the needs of
different kinds of linguistic study and application.
Thecreationofthiskind ofambiencefortheIndianlanguagesisanurgenttask
today. Here our goal is to develop different types of the corpora in such a manner
that they can contribute to new types of linguistic studies, research in written and
spokenlanguages,anddevelopnewtechnologythatcanservetherequirementsofa
multilingual and multicultural country like India. Although these corpora will be
used primarily for linguistics and allied disciplines, these will also function as a
source of data and information for various related disciplines. The corpora that we
trytoenvisagewilleventuallybeavailableonlineforeachmajorandminorIndian
languages and will be marked with genres, periods, times, and subjects—all being
ready to mirror the discourse varieties that range over various types of linguistic
interactions in which the members of the speech communities are engaged in.
From this observation, it becomes clear that in a multilingual and multicultural
country like India, the functional potential of language corpus is multidimensional
invariousinnovativeworksofdescriptivelinguistics,languagetechnology,applied
linguistics, cognitive linguistics, and discourse studies. Definitely, corpus-based
x Preface
approach to linguistic research and development will open up many new avenues
forthebenefitofthepeopleofthecountry.Wecanvisualizethatthecorporamade
from the Indian languages may be utilized in the following works: language
description and interpretation, first- and second-language education, generation of
terminology data banks, compiling lists of translational equivalents, compiling
dictionariesofvarioustypes,studyingtraitsofculturaldifferenceacrossspeechand
languagecommunities, analyzingcoursesofdiscourse andpragmatics inlinguistic
settings, analyzing dialogues and conversations in various settings and back-
grounds, studying grammar and syntax of the Indian languages, developing
grammar of different types in the Indian languages, developing systems and tools
for understanding ambiguities, understanding the nature and texture of social
psychology,analyzingsystemsandmethodsoflanguageacquisition,devisingtools
and systems for language technology, designing texts and course materials for
language education, exploring stylistic variations observed in different types of
texts, studying language variation across regional and social territories, language
revival and revitalization, translation across the Indian languages, and resource
generation for serving language impairments.
Sincearaw(i.e.,non-normalizedandnon-annotated)corpushaslimitedutilities
in analysisand description of a language or a variety, it has less value in works of
languagedescription,application,andtechnology.Therefore,therearisesanurgent
need for generating normalized and annotated corpora in the Indian languages for
bothspokenandwrittenvarieties.SincethereishardlyanyIndianlanguagecorpus
thatistaggedatvariouslevels(i.e.,orthographic,grammatical,syntactic,semantic,
discourse,anaphoric,etymological,andfigurative)ofatext,thereisanurgentneed
for generating normalized and annotated corpora, which will be used to contribute
toward devising advanced resources and tools for the works of applied linguistics
andlanguagetechnology.Inordertoachievethisgoal,weneedtodesignelaborate
tagsets, which will encompass all linguistic properties of written and spoken texts
with a proper focus on sounds, segments, characters, words, morphemes, com-
pounds, collocations, multiword units, phrases, sentences, idioms, proverbs, and
other linguistic properties. There is also a need for describing different methods of
text annotation for the corpora of Indian languages based on which subsequent
works of corpus processing and utilization can be fruitfully executed (all these
issues will be discussed in some details in the next volume).
Besidestaggingtheselinguisticelements,properties,andinformationvisibleina
corpus, we also need a corpus tagged with various invisible linguistic information
suchasmeaning,anaphora,discourse,andpragmaticssothatthesecorporabecome
useful and accessible for various applied linguistic works relating to dictionary
compilation, language teaching, word-sense disambiguation, machine translation,
etc. In this context, the tools and techniques that we require for all the Indian
language corpora are the following: frequency counting, lexical sorting, sentence
breaking, lexical decomposition, compound decomposition, spelling checking,
word concordance, lexical collocation, keyword searching, local word grouping,
lemmatization, morphological processing, morphological generation, grammatical
annotation, sentence annotation, multiword extraction, sentence parsing, Sandhi
Preface xi
splitting, speech-to-text conversion, and transliteration. Although these tools and
techniques are absolutely necessary for the Indian language corpora, it is not so
easy to develop these tools and resources without the support of actual Indian
language corpora and active involvement of a large group of skilled people
endowed with advanced knowledge of Indian languages, corpus linguistics, and
computer science. This appears to be a Himalayan task before we can make a real
breakthrough in the areas of corpus linguistics and language technology for the
Indian languages. We have addressed some of the issues and needs mentioned
above in this volume.
The present book is an outcome of our corpus-based studies on the Indian
languages. In this book, we have tried to address some of the issues of using data
and information from the corpus in some basic areas of language application. We
have tried to show how language corpus may be utilized for developing quality
language teaching textbooks, course books, and reference materials; compiling
usage-based dictionaries; generating database for translational equivalents; deci-
phering contextual information for understanding sense variations in words;
understanding unique linguistic properties and features of dialects; and developing
usefulsystems,tools,andresourcesfortranslation.Inessence,withclosereference
tothecorpus,inthisbook,wehavetriedtoperformtheroleofaharbingertomake
peopleawareoffunctionalandreferentialbenefitsofthecorpusinvariousworksof
descriptive linguistics, applied linguistics, andlanguage technology. The academic
relevance of this book may be attested in its direct focus on the application of
languagedataanditssincereappealforredirectingthefocusoflinguiststowardthis
new method of language study for the benefit of the entire discipline as well as its
sister disciplines. Following are the notable contributions of this book:
[1] Itdiscussestheuseofcorpusinseveralimportantareasoflinguistics,namely
language teaching, dictionary making, dialect study, word-sense disam-
biguation, TermBank compilation, machine translation, lexicology, soci-
olinguistics, lexical semantics, psycholinguistics, stylistics.
[2] The observation, information, and arguments furnished in this book are
based on the analysis of language corpus of various types.
[3] From the perspective of application of corpus in the areas stated above, we
havetriedtoshowhowlinguisticinformationandexamplesobtainedfroma
corpus can contribute toward the growth, maturity, and advancement of
linguistics.
[4] Discussionspresentedinthisbookareprimarilybuiltonlinguisticresources
collected from the real use of language in normal life. These are authentic
and reliable than any experimental results and intuitive speculations.
[5] The research works presented in this book focus on the qualitative and
functionalinterpretationofcorpusdatatounderstandtheintricaciesnotedin
the internal and external textures of a natural language.
