Table Of Contentwww.it-ebooks.info
Taming Text
www.it-ebooks.info
www.it-ebooks.info
Taming Text
HOW TO FIND, ORGANIZE, AND MANIPULATE IT
GRANT S. INGERSOLL
THOMAS S. MORTON
ANDREW L. FARRIS
MANNING
SHELTER ISLAND
www.it-ebooks.info
For online information and ordering of this and other Manning books, please visit
www.manning.com. The publisher offers discounts on this book when ordered in quantity.
For more information, please contact
Special Sales Department
Manning Publications Co.
20 Baldwin Road
PO Box 261
Shelter Island, NY 11964
Email: [email protected]
©2013 by Manning Publications Co. All rights reserved.
No part of this publication may be reproduced, stored in a retrieval system, or transmitted, in
any form or by means electronic, mechanical, photocopying, or otherwise, without prior written
permission of the publisher.
Many of the designations used by manufacturers and sellers to distinguish their products are
claimed as trademarks. Where those designations appear in the book, and Manning
Publications was aware of a trademark claim, the designations have been printed in initial caps
or all caps.
Recognizing the importance of preserving what has been written, it is Manning’s policy to have
the books we publish printed on acid-free paper, and we exert our best efforts to that end.
Recognizing also our responsibility to conserve the resources of our planet, Manning books
are printed on paper that is at least 15 percent recycled and processed without the use of
elemental chlorine.
Manning Publications Co. Development editor: Jeff Bleiel
20 Baldwin Road Technical proofreader: Steven Rowe
PO Box 261 Copyeditor: Benjamin Berg
Shelter Island, NY 11964 Proofreader: Katie Tennant
Typesetter: Dottie Marsico
Cover designer: Marija Tudor
ISBN 9781933988382
Printed in the United States of America
1 2 3 4 5 6 7 8 9 10 – MAL – 18 17 16 15 14 13
www.it-ebooks.info
brief contents
1 ■ Getting started taming text 1
2 ■ Foundations of taming text 16
3 ■ Searching 37
4 ■ Fuzzy string matching 84
5 ■ Identifying people, places, and things 115
6 ■ Clustering text 140
7 ■ Classification, categorization, and tagging 175
8 ■ Building an example question answering system 240
9 ■ Untamed text: exploring the next frontier 260
v
www.it-ebooks.info
www.it-ebooks.info
contents
foreword xiii
preface xiv
acknowledgments xvii
about this book xix
about the cover illustration xxii
1 Getting started taming text 1
1.1 Why taming text is important 2
1.2 Preview: A fact-based question answering system 4
Hello, Dr. Frankenstein 5
1.3 Understanding text is hard 8
1.4 Text, tamed 10
1.5 Text and the intelligent app: search and beyond 11
Searching and matching 12 ■ Extracting information 13
Grouping information 13 ■ An intelligent application 14
1.6 Summary 14
1.7 Resources 14
2 Foundations of taming text 16
2.1 Foundations of language 17
Words and their categories 18 ■ Phrases and clauses 19
Morphology 20
vii
www.it-ebooks.info
viii CONTENTS
2.2 Common tools for text processing 21
String manipulation tools 21 ■ Tokens and tokenization 22
Part of speech assignment 24 ■ Stemming 25 ■ Sentence
detection 27 ■ Parsing and grammar 28 ■ Sequence
modeling 30
2.3 Preprocessing and extracting content from common file
formats 31
The importance of preprocessing 31 ■ Extracting content using
Apache Tika 33
2.4 Summary 36
2.5 Resources 36
3 Searching 37
3.1 Search and faceting example: Amazon.com 38
3.2 Introduction to search concepts 40
Indexing content 41 ■ User input 43 ■ Ranking documents
with the vector space model 46 ■ Results display 49
3.3 Introducing the Apache Solr search server 52
Running Solr for the first time 52 ■ Understanding Solr
concepts 54
3.4 Indexing content with Apache Solr 57
Indexing using XML 58 ■ Extracting and indexing content
using Solr and Apache Tika 59
3.5 Searching content with Apache Solr 63
Solr query input parameters 64 ■ Faceting on extracted
content 67
3.6 Understanding search performance factors 69
Judging quality 69 ■ Judging quantity 73
3.7 Improving search performance 74
Hardware improvements 74 ■ Analysis improvements 75
Query performance improvements 76 ■ Alternative scoring
models 79 ■ Techniques for improving Solr performance 80
3.8 Search alternatives 82
3.9 Summary 83
3.10 Resources 83
www.it-ebooks.info
CONTENTS ix
4 Fuzzy string matching 84
4.1 Approaches to fuzzy string matching 86
Character overlap measures 86 ■ Edit distance measures 89
N-gram edit distance 92
4.2 Finding fuzzy string matches 94
Using prefixes for matching with Solr 94 ■ Using a trie for
prefix matching 95 ■ Using n-grams for matching 99
4.3 Building fuzzy string matching applications 100
Adding type-ahead to search 101 ■ Query spell-checking for
search 105 ■ Record matching 109
4.4 Summary 114
4.5 Resources 114
5 Identifying people, places, and things 115
5.1 Approaches to named-entity recognition 117
Using rules to identify names 117 ■ Using statistical
classifiers to identify names 118
5.2 Basic entity identification with OpenNLP 119
Finding names with OpenNLP 120 ■ Interpreting names
identified by OpenNLP 121 ■ Filtering names based on
probability 122
5.3 In-depth entity identification with OpenNLP 123
Identifying multiple entity types with OpenNLP 123
Under the hood: how OpenNLP identifies names 126
5.4 Performance of OpenNLP 128
Quality of results 129 ■ Runtime performance 130
Memory usage in OpenNLP 131
5.5 Customizing OpenNLP entity identification
for a new domain 132
The whys and hows of training a model 132 ■ Training
an OpenNLP model 133 ■ Altering modeling inputs 134
A new way to model names 136
5.6 Summary 138
5.7 Further reading 139
www.it-ebooks.info
