Computational Methods for Corpus Annotation and Analysis

192 Pages·2014·3.23 MB·English

by  Xiaofei Lu (auth.)

Computational Methods for Corpus Annotation
and Analysis
Xiaofei Lu
Computational Methods
for Corpus Annotation
and Analysis
1 3
Xiaofei Lu
Department of Applied Linguistics
The Pennsylvania State University
University Park
Pennsylvania
USA
ISBN 978-94-017-8644-7 ISBN 978-94-017-8645-4 (eBook)
DOI 10.1007/978-94-017-8645-4
Springer New York Heidelberg Dordrecht London
Library of Congress Control Number: 2014931404
© Springer Science+Business Media Dordrecht 2014
This work is subject to copyright. All rights are reserved by the Publisher, whether the whole or part of
the material is concerned, specifically the rights of translation, reprinting, reuse of illustrations, recita-
tion, broadcasting, reproduction on microfilms or in any other physical way, and transmission or infor-
mation storage and retrieval, electronic adaptation, computer software, or by similar or dissimilar meth-
odology now known or hereafter developed. Exempted from this legal reservation are brief excerpts in
connection with reviews or scholarly analysis or material supplied specifically for the purpose of being
entered and executed on a computer system, for exclusive use by the purchaser of the work. Duplica-
tion of this publication or parts thereof is permitted only under the provisions of the Copyright Law of
the Publisher’s location, in its current version, and permission for use must always be obtained from
Springer. Permissions for use may be obtained through RightsLink at the Copyright Clearance Center.
Violations are liable to prosecution under the respective Copyright Law.
The use of general descriptive names, registered names, trademarks, service marks, etc. in this publica-
tion does not imply, even in the absence of a specific statement, that such names are exempt from the
relevant protective laws and regulations and therefore free for general use.
While the advice and information in this book are believed to be true and accurate at the date of publica-
tion, neither the authors nor the editors nor the publisher can accept any legal responsibility for any errors
or omissions that may be made. The publisher makes no warranty, express or implied, with respect to
the material contained herein.
Printed on acid-free paper
Springer is part of Springer Science+Business Media (www.springer.com)
Dedicated to my wife, Xiaomeng, and our
daughter, Jasmine.
v
Preface
This book grew out of sets of lecture notes for a graduate course on computational
methods for corpus annotation and analysis that I have taught in the Department of
Applied Linguistics at The Pennsylvania State University since 2006. After several
iterations of the course, my students and I realized that while there is an abundance
of introductory sources on the fundamentals of corpus linguistics, most of them do
not provide the types of detailed and systematic instructions that are necessary to
help language and linguistics researches get off the ground with using computa-
tional tools other than concordancing programs for automatic corpus annotation and
analysis. A large proportion of the students taking the course were not yet ready to
embark on learning to program, and to them the introductory sources on program-
ming for linguistics, natural language processing, and computational linguistics
appeared overwhelming. What seemed to be lacking was something in the middle
ground, something that enables novice language and linguistics researchers to use
more sophisticated and powerful corpus annotation and analysis tools than con-
cordancing programs and yet still does not require programming. This book was
written with the aim to provide that middle ground.
I owe a special thanks to all the students who have taken the course with me at The
Pennsylvania State University. This book could not have been written without their
inspiration. In particular, I want to thank Brody Bluemel and Ben Pin-Yun Wang for
providing very detailed feedback on earlier drafts of several chapters; Edie Furniss,
Qianwen Li, and many others for pointing me to various stylistic issues in the book;
Haiyang Ai, Brody Bluemel, Tracy Davis, Alissa Hartig, Shibamouli Lahiri, Kwan-
ghyun Park, Jie Zhang, and Xian Zhang for numerous discussions about the lecture
notes that the book grew out of while taking and/or co-teaching the course with me.
It would be difficult to thank all the people who have influenced the ideas behind
this book. I am deeply indebted to Detmar Meurers and Chris Brew, who first intro-
duced me to the field of computational linguistics. I have also learned tremendously
from a large number of other colleagues, directly or indirectly. To name just a few:
Gabriella Appel, Stacey Bailey, Douglas Biber, Donna Byron, Marjorie Chan, Rich-
ard Crouch, Markus Dickinson, Nick Ellis, Anna Feldman, Eric Fosler-Lussier,
ZhaoHong Han, Jirka Hana, Erhard Hinrichs, Tracy Holloway King, James Lantolf,
vii
viii Preface
Michael McCarthy, Lourdes Ortega, Richard Sproat, Hongyin Tao, Steven Thorne,
Mike White, Richard Xiao, and many others.
Last but not least, I would like to sincerely thank Jolanda Voogd at Springer for
her vision, enthusiasm and patience, Helen van der Stelt at Springer for her con-
tinued support, and the anonymous reviewers for their insightful and constructive
comments.
Contents
1 Introduction ............................................................................................... 1
1.1 Objectives and Rationale of the Book ................................................ 1
1.2 Why Do We Need to Go Beyond Raw Corpora ................................. 3
1.3 What Is Corpus Annotation ................................................................ 5
1.4 Organization of the Book ................................................................... 6
References ................................................................................................... 7
2 Text Processing with the Command Line Interface ............................... 9
2.1 The Command Line Interface ............................................................ 9
2.2 Basic Commands ................................................................................ 11
2.2.1 Notational Conventions .......................................................... 11
2.2.2 Printing the Current Working Directory ................................. 11
2.2.3 Listing Files and Subdirectories ............................................. 12
2.2.4 Making New Directories ........................................................ 12
2.2.5 Changing Directory Locations ............................................... 13
2.2.6 Creating and Editing Text Files with UTF-8 Encoding ......... 14
2.2.7 V iewing, Renaming, Moving, Copying, and
Removing Files ...................................................................... 16
2.2.8 Copying, Moving, and Removing Directories ....................... 20
2.2.9 Using Shell Meta-Characters for File Matching .................... 21
2.2.10 Manual Pages, Command History, and Command
Line Completion ..................................................................... 21
2.3 Tools for Text Processing ................................................................... 22
2.3.1 Searching for a String with ........................................ 22
egrep
2.3.2 Regular Expressions ............................................................... 24
2.3.3 Character Translation with ................................................ 29
tr
2.3.4 Editing Files from the Command Line with ................... 30
sed
2.3.5 Data Filtering and Manipulation Using .......................... 31
awk
2.3.6 Task Decomposition and Pipes .............................................. 35
2.4 Summary ............................................................................................ 38
References ................................................................................................... 38
ix
x Contents
3 Lexical Annotation .................................................................................. 39
3.1 Part-of-Speech Tagging .................................................................... 39
3.1.1 What is Part-of-Speech Tagging ........................................... 39
3.1.2 Understanding Part-of-Speech Tagsets ................................ 42
3.1.3 The Stanford Part-of-Speech Tagger .................................... 46
3.2 Lemmatization .................................................................................. 54
3.2.1 What is Lemmatization and Why is it Useful ...................... 54
3.2.2 T he TreeTagger .................................................................. 55
3.3 Additional Tools ............................................................................... 58
3.3.1 T he Stanford Tokenizer ........................................................ 58
3.3.2 The Stanford Word Segmenter for Arabic and Chinese ....... 59
3.3.3 T he CLAWS Tagger for English .......................................... 61
3.3.4 T he Morpha Lemmatizer for English ................................... 61
3.4 Summary .......................................................................................... 64
References ................................................................................................. 64
4 Lexical Analysis ....................................................................................... 67
4.1 Frequency Lists ................................................................................ 67
4.1.1 Working with Output Files from the TreeTagger ................. 68
4.1.2 Working with Output Files from the Stanford POS
Tagger and Morpha .............................................................. 72
4.1.3 Analyzing Frequency Lists with Text Processing Tools ...... 73
4.2 N-Grams ........................................................................................... 76
4.3 Lexical Richness .............................................................................. 80
4.3.1 Lexical Density .................................................................... 80
4.3.2 Lexical Variation .................................................................. 82
4.3.3 Lexical Sophistication .......................................................... 84
4.3.4 T ools for Lexical Richness Analysis .................................... 84
4.4 Summary .......................................................................................... 90
References ................................................................................................. 91
5 Syntactic Annotation ............................................................................... 95
5.1 S yntactic Parsing Overview ............................................................. 95
5.1.1 What is Syntactic Parsing and Why is it Useful? ................. 95
5.1.2 Phrase Structure Grammars .................................................. 96
5.1.3 Dependency Grammars ........................................................ 102
5.2 Syntactic Parsers .............................................................................. 106
5.2.1 The Stanford Parser .............................................................. 106
5.2.2 Collins’ Parser ...................................................................... 110
5.3 Summary .......................................................................................... 112
References ................................................................................................. 113
Contents xi
6 Syntactic Analysis .................................................................................... 115
6.1 Querying Syntactically Parsed Corpora ........................................... 115
6.1.1 Tree Relationships ................................................................ 115
6.1.2 Tregex ................................................................................... 121
6.2 Syntactic Complexity Analysis ........................................................ 130
6.2.1 Measures of Syntactic Complexity ...................................... 130
6.2.2 Syntactic Complexity Analyzers .......................................... 136
6.3 Summary .......................................................................................... 142
References ................................................................................................. 142
7 Semantic, Pragmatic and Discourse Analysis ....................................... 147
7.1 Semantic Field Analysis ................................................................... 147
7.1.1 The UCREL Semantic Analysis System .............................. 147
7.1.2 Profile in Semantics-Lexical in Computerized Profiling ..... 152
7.2 Analysis of Propositions ................................................................... 154
7.2.1 Computerized Propositional Idea Density Rater .................. 154
7.2.2 Analysis of Propositions in Computerized Profiling ............ 157
7.3 Conversational Act Analysis in Computerized Profiling ................. 158
7.4 Coherence and Cohesion Analysis in Coh-Metrix ........................... 160
7.4.1 Referential Cohesion Features ............................................. 160
7.4.2 Features Based on Latent Semantic Analysis ....................... 161
7.4.3 Features Based on Connectives ............................................ 162
7.4.4 Situation Model Features ..................................................... 163
7.4.5 Word Information Features .................................................. 164
7.5 T ext Structure Analysis .................................................................... 164
7.6 Summary .......................................................................................... 169
References ................................................................................................. 170
8 Summary and Outlook ........................................................................... 175
8.1 Summary of the Book ...................................................................... 175
8.2 Future Directions in Computational Corpus Analysis ..................... 177
8.2.1 Computational Analysis of Language Meaning and Use ..... 178
8.2.2 Computational Analysis of Learner Language ..................... 178
8.2.3 Computational Analysis Based on Specific
Language Theories ............................................................... 180
References ................................................................................................. 182
Appendix ........................................................................................................ 185