Table Of ContentIntelligent Document Retrieval
THESPRINGER INTERNATIONALSERIES
ON INFORMATION RETRIEVAL
Series Editor:
W. Bruce Croft
University of Massachusetts, Amherst
Also in the Series:
INFORMATION RETRIEVALSYSTEMS:Theory and Implementation,by Gerald
Kowalski;ISBN: 0-7923-9926-9
CROSS-LANGUAGE INFORMATION RETRIEVAL,edited by Gregory Grefenstette;
ISBN: 0-7923-8122-X
TEXTRETRIEVALAND FILTERING:Analytic Models of Performance,by Robert M. Losee;
ISBN: 0-7923-8177-7
INFORMATION RETRIEVAL: UNCERTAINTYAND LOGICS:Advanced Models for the
Representation and Retrieval of Information,by Fabio Crestani, Mounia Lalmas, and
Cornelis Joost van Rijsbergen;ISBN: 0-7923-8302-8
DOCUMENTCOMPUTING:Technologies for Managing Electronic Document Collections,
by Ross Wilkinson, Timothy Arnold-Moore, Michael Fuller, Ron Sacks-Davis, James
Thom, and Justin Zobel;ISBN: 0-7923-8357-5
AUTOMATIC INDEXINGAND ABSTRACTINGOFDOCUMENTTEXTS,by Marie-
Francine Moens; ISBN 0-7923-7793-1
ADVANCES IN INFORMATIONALRETRIEVAL:Recent Research from the Center for
Intelligent Information Retrieval,by W. Bruce Croft; ISBN 0-7923-7812-1
INFORMATION RETRIEVALSYSTEMS:Theory and Implementation, Second Edition,
by Gerald J. Kowalski and Mark T. Maybury; ISBN: 0-7923-7924-1
PERSPECTIVES ON CONTENT-BASED MULTIMEDIASYSTEMS,by Jian Kang Wu;
Mohan S. Kankanhalli;Joo-Hwee Lim;Dezhong Hong; ISBN: 0-7923-7944-6
MININGTHEWORLDWIDEWEB:An Information Search Approach, by George Chang,
Marcus J. Healey, James A. M. McHugh, Jason T. L. Wang; ISBN: 0-7923-7349-9
INTEGRATED REGION-BASED IMAGE RETRIEVAL, by James Z. Wang;
ISBN: 0-7923-7350-2
TOPIC DETECTION AND TRACKING: Event-based Information Organization,
edited by James Allan;ISBN: 0-7923-7664-1
LANGUAGE MODELINGFOR INFORMATION RETRIEVAL,edited by W. Bruce Croft,
John Lafferty; ISBN: 1-4020-12160-0
MACHINE LEARNINGANDSTATISTICALMODELINGAPPROACHES TO IMAGE
RETRIEVAL, by Yixin Chen, Jia Li, James Z. Wang;ISBN: 1-4020-8034-4
INFORMATION RETRIEVAL:Algorithms and Heuristics,by David A. Grossman, Ophir Frieder;
ISBN: 1-4020-3004-5
INFORMATION RETRIEVAL:Algorithms and Heuristics,by David A. Grossman, Ophir Frieder;
ISBN: 1-4020-3003-7
CHARTINGANEWCOURSE:NATURALLANGUAGE PROCESSINGAND INFORMATION
RETRIEVAL:Essays in Honour of Karen Sparck Jones,edited by John I. Tait;
ISBN: 1-4020-3343-5
Intelligent Document Retrieval
Exploiting Markup Structure
by
Udo Kruschwitz
University of Essex,
Colchester, U.K.
AC.I.P. Catalogue record for this book is available from the Library of Congress.
ISBN-10 1-4020-3767-8 (HB)
ISBN-13 978-1-4020-3767-2 (HB)
ISBN-10 1-4020-3768-6 (e-book)
ISBN-13 978-1-4020-3768-9 (e-book)
Published by Springer,
P.O. Box 17, 3300 AADordrecht, The Netherlands.
www.springeronline.com
Printed on acid-free paper
All Rights Reserved
© 2005 Springer
No part of this work may be reproduced, stored in a retrieval system, or transmitted
in any form or by any means, electronic, mechanical, photocopying, microfilming, recording
or otherwise, without written permission from the Publisher, with the exception
of any material supplied specifically for the purpose of being entered
and executed on a computer system, for exclusive use by the purchaser of the work.
Printed in the Netherlands.
Contents
Foreword ...................................................... IX
Preface ........................................................ XI
List of Figures.................................................XIII
List of Tables ..................................................XV
1 Introduction............................................... 1
1.1 Introductory Examples................................... 4
1.2 Using Markup to Extract Knowledge....................... 8
1.3 Applying the Extracted Knowledge ........................ 15
1.4 Structure of the Book.................................... 17
Part I The Model
2 Related Work ............................................. 23
2.1 Information Retrieval .................................... 24
2.2 Information Extraction................................... 26
2.3 Clustering .............................................. 27
2.4 Classification ........................................... 29
2.5 Web Search Techniques .................................. 31
2.6 Ontologies.............................................. 34
2.7 Layout Analysis ......................................... 36
2.8 Web Search Studies...................................... 36
2.9 Navigating Concept Hierarchies ........................... 38
2.10 Dialogue Systems........................................ 41
2.11 Usability Issues ......................................... 42
2.12 Concluding Remarks on Related Work ..................... 43
VI Contents
3 Data Analysis and Domain Model Construction ........... 45
3.1 Documents ............................................. 45
3.2 Concepts ............................................... 47
3.3 A Domain Model Based on Concepts....................... 51
3.4 Model Structure......................................... 53
3.5 Model Construction...................................... 54
3.6 Using the Model for Query Modification.................... 58
3.7 Implementational Issues .................................. 60
4 Incorporating Additional Knowledge....................... 63
4.1 Internal Knowledge ...................................... 63
4.2 External Knowledge ..................................... 67
5 A Dialogue System for Partially Structured Data.......... 69
5.1 Dialogue as Movement in Space ........................... 70
5.2 Dialogue Example ....................................... 71
5.3 Static vs. Dynamic Clusters............................... 73
5.4 Real User Queries ....................................... 73
5.5 Properties .............................................. 75
5.5.1 Document Properties .............................. 76
5.5.2 System Properties ................................. 76
5.5.3 Goal Description .................................. 77
5.6 Dialogue ............................................... 78
5.6.1 High Level Dialogue States ......................... 78
5.6.2 Low Level Dialogue States.......................... 80
5.6.3 Constructing Potential Choices...................... 85
5.6.4 Dialogue Strategies ................................ 89
5.6.5 Customization .................................... 89
Part II Practical Applications
6 UKSearch - Intelligent Web Search ........................ 93
6.1 Indexing Web Pages ..................................... 94
6.2 The UKSearch System ................................... 98
6.2.1 Indexing and Model Construction ...................100
6.2.2 Dialogue Strategy .................................102
6.3 Sample Domain 1: Essex University........................107
6.3.1 Index Tables......................................108
6.3.2 Domain Model ....................................109
6.3.3 Concepts vs. Real User Queries .....................111
6.4 Sample Domain 2: BBC News.............................112
6.4.1 Index Tables......................................115
6.4.2 Domain Model ....................................116
6.4.3 Adjusted Dialogue Strategy.........................117
Contents VII
6.5 Implementational Issues ..................................117
7 UKSearch - Evaluation and Discussion.....................121
7.1 Log Analysis............................................121
7.1.1 System Setup .....................................122
7.1.2 Results...........................................124
7.1.3 Discussion........................................125
7.2 Investigating Domain Model Relations .....................125
7.2.1 Task and Setup ...................................125
7.2.2 Results...........................................127
7.2.3 Discussion........................................128
7.3 Task-Based Evaluation: Essex University ...................129
7.3.1 Search Tasks......................................129
7.3.2 Experimental Setup ...............................133
7.3.3 Procedure ........................................134
7.3.4 Results...........................................134
7.3.5 Discussion........................................140
7.4 Task-Based Evaluation: BBC News ........................141
7.4.1 Search Tasks......................................142
7.4.2 Experimental Setup and Procedure ..................143
7.4.3 Results...........................................143
7.4.4 Discussion........................................151
8 YPA - Searching Classified Directories.....................157
8.1 System Overview........................................158
8.2 Indexing Classified Advertisements ........................159
8.2.1 Structure of the Backend ...........................160
8.2.2 Domain Model Construction ........................161
8.3 Dialogue Strategy in the YPA.............................162
8.3.1 Properties ........................................165
8.3.2 Dialogue Setup....................................166
8.3.3 Dialogue Function .................................168
8.3.4 Calculation of Potential Choices.....................168
8.4 Implementational Issues ..................................171
9 Future Directions and Conclusions.........................173
9.1 Towards Evolving Domain Models .........................173
9.2 Dialogue Management ...................................176
9.3 An Outlook on Future Evaluations ........................177
9.4 Conclusions.............................................178
References.....................................................181
Index..........................................................193
Foreword
UdoKruschwitz’sbook,basedonhisPhDthesis,arguesthatforGoogle-type
websearchesonlimiteddomains,oronsite-specificintranets,performancecan
be enhanced by making use of a domain model of the entities and relations
characteristic of that site. He shows how to use document structure mark-
up and lexical co-occurrences within and across documents to construct such
domain models automatically. Users are then able to engage in a dialogue in
whichcuesare provided,basedonthedomain model,toenablethemtorelax
or to refine their search until they find what they are looking for.
Themethodhasbeenimplementedandembodiedintwodifferentpractical
applications,andthesehavebeenevaluatedinusertrials.Thesetrialsprovide
some evidence that the technique is effective in helping users.
TheresearchcarriedoutbyUdoKruschwitzandreportedinthisbookisa
model of how to combine computational linguistics and information retrieval
techniques in a theoretically motivated - but practical - application, which
has also been fielded and empirically tested. Anyone working in the fields
of computational linguistics, information retrieval, document summarisation,
websearchingorquestionansweringwillfindsomethingofvalueinthisbook.
Oxford, Stephen Pulman
23rd March 2005 Professor of General Linguistics
Oxford University
Preface
Thanks to everyone who helped me with this book.
I wish to thank Sam Steel, Anne De Roeck, Massimo Poesio, Mounia
Lalmas, Thomas R¨olleke, Nick Webb, Paul Scott, Ray Turner, Maria Fasli,
Stephen Pulman and Bill Black in particular as well as all the students and
colleagues who volunteered to help with the evaluations. Some of the evalua-
tionworkdescribedinherewasjointworkwithHalaAl-Bakoursupportedby
EPSRC grant GR/R92813/01, who I would like to thank as well. Thanks to
Doug Arnold for suggesting a book title. Furthermore, I am very grateful to
Robbert van Berckelaer at Springer for his assistance in preparing this book.
Finally, special thanks to my family over there in Germany, the German
Society and the Horse & Groom.
Wivenhoe, Udo Kruschwitz
4th April 2005
