Table Of ContentNAT'L INST. OF STAND & TECf
MIST
PUBLICATIONS
NIST Special Publication 500-249
Information Technology:
The Ninth Text REtrieval
Conference (TREC-9)
E. M. Voorhees
and
D. K. Harman
Editors
Nisr
National Institute of Standards and Technology
NIST CENTENNIAL
Technology Administration, U.S. Department of Commerce
rhe National Institute of Standards and Technology was established in 1988 by Congress to "assist industry in
the development of technology . . . n eeded to improve product quality, to modernize manufacturing processes,
to ensure product reliability . . . a nd to facilitate rapid commercialization ... of products based on new scientific
discoveries."
NIST, originally founded as the National Bureau of Standards in 1901, works to strengthen U.S. industry's
competitiveness; advance science and engineering; and improve public health, safety, and the environment. One
of the agency's basic functions is to develop, maintain, and retain custody of the national standards of
measurement, and provide the means and methods for comparing standards used in science, engineering,
manufacturing, commerce, industry, and education with the standards adopted or recognized by the Federal
Government.
As an agency of the U.S. Commerce Department's Technology Administration, NIST conducts basic and
applied research in the physical sciences and engineering, and develops measurement techniques, test
methods, standards, and related services. The Institute does generic and precompetitive work on new and
advanced technologies. NIST's research facilities are located at Gaithersburg, MD 20899, and at Boulder, CO 80303.
Major technical operating units and their principal activities are listed below. For more information contact the
Publications and Program Inquiries Desk, 301-975-3058.
Office of the Director Chemical Science and Technology
Laboratory
• National Quality Program • Biotechnology
• International and Academic Affairs
• Physical and Chemical Properties2
Technology Services • Analytical Chemistry
• Standards Services • Process Measurements
• Technology Partnerships • Surface and Microanalysis Science
• Measurement Services
• Information Services Physics Laboratory
• Electron and Optical Physics
• Atomic Physics
Advanced Technology Program
• Optical Technology
• Economic Assessment
• Ionizing Radiation
• Information Technology and Applications
• Chemistry and Life Sciences • Time and Frequency'
• Materials and Manufacturing Technology • Quantum Physics'
• Electronics and Photonics Technology
Manufacturing Engineering
Laboratory
Manufacturing Extension Partnership • Precision Engineering
Program • Manufacturing Metrology
• Regional Programs • Intelligent Systems
• National Programs • Fabrication Technology
• Program Development
• Manufacturing Systems Integration
Electronics and Electrical Engineering
Building and Fire Research
Laboratory Laboratory
• Microelectronics
• Applied Economics
• Law Enforcement Standards • Structures
• Electricity • Building Materials
• Semiconductor Electronics
• Building Environment
• Radio-Frequency Technology' • Fire Safety Engineering
• Fire Science
• Electromagnetic Technology1
• Optoelectronics'
Information Technology Laboratory
Materials Science and Engineering
• Mathematical and Computational Sciences2
Laboratory • Advanced Network Technologies
• Intelligent Processing of Materials • Computer Security
• Ceramics • Information Access
• Materials Reliability1 • Convergent Information Systems
• Polymers • Information Services and Computing
• Metallurgy • Software Diagnostics and Conformance Testing
• NIST Center for Neutron Research • Statistical Engineering
'At Boulder, CO 80303.
2Some elements at Boulder, CO.
NIST Special Publication 500-249
Information Technology:
The Ninth Text REtrieval
Conference (TREC-9)
E.M. Voorhees and
D.K. Harman
Editors
Information Technology Laboratory
National Institute of Standards and Technology
Gaithersburg, MD 20890-8980
October 2001
U.S. Department of Commerce
Donald L. Evans, Secretary
Technology Administration
Karen H. Brown, Acting Under Secretary of Commerce for Technologx
National Institute of Standards and Technology
Karen H. Brown, Acting Director
Reports on Information Technology
The Information Technology Laboratory (ITL) at the National Institute of Standards and
Technology (NIST) stimulates U.S. economic growth and industrial competitiveness through technical
leadership and collaborative research in critical infrastructure technology, including tests, test methods,
reference data, and forward-looking standards, to advance the development and productive use of
information technology. To overcome barriers to usability, scalability, interoperability, and security in
information systems and networks, ITL programs focus on a broad range of networking, security, and
advanced information technologies, as well as the mathematical, statistical, and computational sciences,
This Special Publication 500-series reports on ITL's research in tests and test methods for information
technology, and its collaborative activities with industry, government, and academic organizations.
Certain commercial entities, equipment, or materials may be identified in this
document in order to describe an experimental procedure or concept adequately. Such
identification is not intended to imply recommendation or endorsement by the
National Institute of Standards and Technology, nor is it intended to imply that the
entities, materials, or eauipment are necessarily the best available for the purpose.
National Institute of Standards and Technology
Special Publication 500-249
Natl. Inst. Stand. Technol.
Spec. Publ. 500-249
946 pages (October 2001)
CODEN: NSPUE2
U.S. GOVERNMENT PRINTING OFFICE
WASHINGTON: 2001
For sale by the Superintendent of Documents, U.S. Government Printing Office
Internet: bookstore.gpo.gov — Phone: (202) 5 12-1800 — Fax: (202) 5 12-2250
Mail: Stop SSOP, Washington, DC 20402-0001
Foreword
This report constitutes the proceedings of the ninth Text REtrieval Conference (TREC-9)
held in Gaithersburg, Maryland, November 13-16, 2000. The conference was co-sponsored by
the National Institute of Standards and Technology (NIST), the Defense Advanced Research
Projects Agency (DARPA), and the Advanced Research and Development Agency (ARDA).
Approximately 175 people attended the conference, including representatives from seventeen
different countries. The conference was the ninth in an on-going series of workshops to
evaluate new technologies for text retrieval and related information-seeking tasks. Sixty-
nine groups submitted retrieval results to one or more of the workshop's tracks.
The workshop included plenary sessions, discussion groups, a poster session, and demonstra-
tions. Because the participants in the workshop drew on their personal experiences, they
sometimes cited specific vendors and commercial products. The inclusion or omission of
a particular company or product implies neither endorsement nor criticism by NIST. Any
opinions, findings, and conclusions or recommendations expressed in the individual papers
are the authors' own and do not necessarily relect those of the sponsors.
The sponsorship of the U.S. Department of Defense is gratefully acknowledged, as is the
tremendous work of the program committee and the track coordinators.
Ellen Voorhees,
Donna Harman
October 3, 2001
TREC-9 Program Committee
Ellen Voorhees, NIST, chair
James Allan, University of Massachusetts at Amherst
Nick Belkin, Rutgers University
Chris Buckley, Sabir Research, Inc.
Jamie Callan, Carnegie Mellon University
Susan Dumais, Microsoft
Donna Harman, NIST
David Hawking, CSIRO
Bill Hersh, Oregon Health Sciences Institute
Darryl Howard, U.S. Department of Defense
David Hull, WhizzBang Labs
John Prange, U.S. Department of Defense
Steve Robertson, Microsoft
Amit Singhal, AT&T Labs-Research
Karen Sparck Jones, University of Cambridge, UK
Tomek Strzalkowski, State University of New York, Albany
Ross Wilkinson, CSIRO
iii
TABLE OF CONTENTS
Index of TREC-9 Papers by Task/Track xi
Abstract xix
PAPERS
Overview of the Ninth Text Retrieval Conference (TREC-9) 1
E. Voorhees, D. Harman, National Institute of Standards and Technology (NIST)
TREC-9 Cross-Language Information Retrieval (English- Chinese) Overview 15
F. Gey, A. Chen, University of California, Berkeley
The TREC-9 Filtering Track Final Report 25
S. Robertson, Microsoft Research
D. A. Hull, WhizzBang Labs
The TREC-9 Interactive Track Report 41
W. Hersh, Oregon Health Sciences University
Paul Over, National Institute of Standards and Technology
Query Expansion Seen Through Return Order of Relevant Documents 51
W. Liggett, NIST
C. Buckley, SablR Research, Inc.
Overview of the TREC-9 Question Answering Track 71
E. Voorhees, NIST
The TREC-9 Query Track 81
Chris Buckley, Sabir Research, Inc.
Overview of the TREC-9 Web Track 87
D. Hawking, CSIRO Mathematical and Information Sciences
AT&T at TREC-9 103
A.Singhal, M. Kaszkiel, AT&T Labs-Research
TREC-9 Cross-lingual Retrieval at BBN 106
J. Xu, R. Weischedel, BBN Technologies
Spoken Document Retrieval for TREC-9 at Cambridge University 117
S.E. Johnson, P. Jourlin, K. Sparck Jones, P.C. Woodland,
Cambridge University
kNN at TREC-9 127
T. Ault, Y. Yang, Carnegie Mellon University
v
YFilter at TREC-9 135
Y. Zhang, J. Callan, Carnegie Mellon University
Passive Feedback Collection- An Attempt to Debunk the Myth of
Clickthroughs 141
C. Vogt, Chapman University
TREC-9 CLIR at CUHK: Disambiguation by Similarity Values Between
Adjacent Words 151
H. Jin, K-F Wong, The Chinese University of Hong Kong
Syntactic Clues and Lexical Resources in Question-Answering 157
K.C. Litkowski, CL Research
ACSys/CSIRO TREC-9 Experiments 167
D. Hawking, CSIRO Mathematics and Information Sciences
The Mirror DBMS at TREC-9 171
A.P. de Vries, CWI, Amsterdam, The Netherlands
Dublin City University Experiments in Connectivity Analysis for
TREC-9 179
C. Gurrin, A.F. Smeaton, Dublin City University
FDU at TREC-9: CLIR, Filtering and QA Tasks 189
L. Wu, X-j Huang, Y. Guo, B. Liu, Y. Zhang, Fudan University
Fujitsu Laboratories TREC-9 Report 203
I. Namba, Fujitsu Laboratories, Ltd.
Hummingbird's Fulcrum SearchServer at TREC-9 209
S. Tomlinson, T. Blackwell, Hummingbird
English-Chinese Information Retrieval at IBM 223
M. Franz, J.S. McCarley, W-J Zhu, IBM T.J. Watson Research Center
IBM's Statistical Question Answering System 229
A. Ittycheriah, M. Franz, W-J Zhu, A. Ratnaparkhi,
IBM T.J. Watson Research Center
R.J. Mammone, Rutgers University
One Search Engine or Two for Question-Answering 235
J. Prager, E. Brown, IBM T.J. Watson Research Center
D. R. Radev, University of Michigan
K. Czuba, Carnegie-Mellon University
ITT TREC-9 - E ntity Based Feedback with Fusion 241
A. Chowdhury, S. Beitzel, E. Jensen, M. Sai-lee, D. Grossman,
O. Frieder, Illinois Institute of Technology
M.C. McCabe, U.S. Government
D. Holmes, NCR Corporation
VI
A Simple Question Answering System 249
R. J. Cooper, S.M. Ruger, Imperial College of Science,
Technology and Medicine
T raining Context-Sensitive Neural Networks with Few Relevant
Examples for the TREC-9 Routing 257
M. Strieker, Informatique-CDC and ESPCI
F. Vichot, F. Wolinski, Informatique-CDC
G. Dreyfus, ESPCI
Mercure at trec9: Web and Filtering tasks 263
M. Abchiche, M. Boughanem, T. Dkaki, J. Mothe, C. Soule Dupuy,
M. Tmar, IRIT-SIG
The HAIRCUT System at TREC-9 273
P. McNamee, J. Mayfield, C. Piatko, The Johns Hopkins University, APL
Reflections on "Aboutness" TREC-9 Evaluation Experiments
at Justsystem 281
S. Fujita, Justsystem Corporation
Kasetsart University TREC-9 Experiments 289
P. Narasetsathapom, A. Rungsawang, Kasetsart University,
Bangkok, Thailand
Experiments on the TREC-9 Filtering Track 295
K. Hoashi, K. Matsumoto, N. Inoue, K. Hashimoto, KDD R&D
Laboratories, Inc.
T. Hasegawa, K. Shirai, Waseda University
TREC-9 Experiments at KAIST: QA, CLIR and Batch Filtering 303
K-S Lee, J-H Oh, JX Huang, J-H Kim, K-S Choi, Korea Advanced
Institute of Science and Technology
Question Answering Considering Semantic Categories and
Co-Occurrence Density 317
S-M Kim, D-H Baek, S-B Kim H-C Rim Korea University
QALC-The Question-Answering System of LIMSI-CNRS 325
O. Ferret, B. Grau, M. Hurault-Plantet, G. Dlouz, C. Jacquemin, LIMSI-CNRS
N. Masson, P. Lecuyer, Bertin Technologies
The LIMSI SDR System for TREC-9 335
J.-L. Gauvain, L. Lamel, C. Barras, G. Adda, Y. de Kercardio, LIMSI-CNRS
TREC-9 CLIR Experiments at MSRCN 343
J. Gao, E. Xun, M. Zhou, C. Huang, Microsoft Research China
J-Y Nie, Universite de Montreal
J. Zhang, Y. Su, Tsinghua University China
vii
Question Answering Using a Large NLP System 355
D. Elworthy, Microsoft Research Ltd.
Microsoft Cambridge at TREC-9: Filtering Track 361
S.E. Robertson, S. Walker, Microsoft Research Ltd., UK
Another Sys Called Qanda 369
E. Breck, J. Burger, L. Ferro, W. Greiff, M. Light, I. Mani,
J. Rennie, The MITRE Corporation
CINDOR TREC-9 English-Chinese Evaluation 379
M.E. Ruiz, S. Rowe, M. Forrester, P. Sheridan, MNIS-TextWise Labs
Description of NTU QA and CLIR Systems in TREC-9 389
C-J Lin, W-C Lin, H-H Chen, National Taiwan University
NTT DATA TREC-9 Question Answering Track Report 399
T. Takaki, NTT Data Corporation
Further Analysis of Whether Batch and User Evaluations Give
the Same Results with a Question-Answering Task 407
W. Hersh, A. Turpin, L. Sacherek, D. Olson,
S. Price, B. Chan, D. Kraemer, Oregon Health Sciences University
TREC-9 Cross Language, Web and Question-Answering Track
Experiments Using PIRCS 417
K.L. Kwok, L. Grunfeld, N. Dinstl, M. Chan, Queens College, CUNY
Structuring and Expanding Queries in the Probabilistic Model 427
Y. Ogawa, H. Mano, M. Narita, S. Honma, RICOH Co., Ltd.
Melbourne TREC-9 Experiments 437
D. D'Souza, M. Fuller, J. Thorn, P. Vines, J. Zobel, RMIT University
O. de Kretser, University of Melbourne
R. Wilkinson, M. Wu, CSIRO, Division of Mathematics and Information Science
Logical Analysis of Data in the TREC-9 Filtering Track 453
E. Boros, P.B. Kantor, D.J. Neu, Rutgers University
Support for Question- Answering in Interactive Information
Retrieval: Rutgers' TREC-9 Interactive Track Experience 463
N.J. Belkin, A. Keller, D. Kelly, J. Perez-Carballo,
C. Sikora, Y. Sun, Rutgers University
Sabir Research at TREC-9 475
C. Buckley, J. Walz, Sabir Research
FALCON: Boosting Knowledge for Answer Engines 479
S. Harabagiu, D. Moldovan, M. Pasca, R. Mihalcea,
M. Surdeanu, R. Bunescu, R. Girju, V. Rus, P. Morarescu,
Southern Methodist University
viii
