Table Of ContentLazy Learning
Edited by
David W. Aha
Navy Center for Applied Research in Artijiciallntelligence,
Naval Research Laboratory, Washington D.C., USA
Reprinted from Artificial Intelligence Review
Volume 11, Nos. 1-5, 1997
Springer-Science+Business Media, B.V.
A C.I.P. catalogue record for this book is available from the Library of Congress.
ISBN 978-90-481-4860-8 ISBN 978-94-017-2053-3 (eBook)
DOI 10.1007/978-94-017-2053-3
Printed on acid-free paper.
AII Rights Reserved
© 1997 Springer Sciencet-Business Media Dordrecht
Originally published by Kluwer Academic Publishers in 1997
Softcover reprint of the hardcover 1s t edition 1997
No part of the material protected by this copyright notice may be reproduced or
utilized in any form or by any means, electronic or mechanical,
including photocopying, recording or by any information storage and
retrieval system, without written permission from the copyright owner.
Table of Contents
About the Authors 1
DAVID W. AHA / Editorial 7
CHRISTOPHER G. ATKESON, ANDREW W. MOORE and STEFAN
SCHAAL / Locally Weighted Learning 11
CHRISTOPHER G. ATKESON, ANDREW W. MOORE and STEFAN
SCHAAL / Locally Weighted Learning for Control 75
ETHEM ALPAYDIN / Voting over Multiple Condensed Nearest
Neighbors 115
MARCOS SALGANICOFF / Tolerating Concept and Sampling Shift
in Lazy Learning Using Prediction Error Context Switching 133
KAI MING TING / Discretisation in Lazy Learning Algorithms 157
JIANPING ZHANG, YEE-SAT YIM and JUNMING YANG / Intelli-
gent Selection of Instances for Prediction Functions in Lazy Learn-
ing Algorithms 175
ODED MARON and ANDREW W. MOORE I The Racing Algorithm:
Model Selection for Lazy Learners 193
PEDRO DOMINGOS I Context-Sensitive Feature Selection for Lazy
Learners 227
CHARLES X. LING and HANDONG WANG I Computing Optimal
Attribute Weight Settings for Nearest Neighbor Algorithms 255
DIETRICHWETTSCHERECK, DAVID W. AHA and TAKAO MOHR!
/ A Review and Empirical Evaluation of Feature Weighting Meth-
ods for a Class of Lazy Learning Algorithms 273
PAT LANGLEY, KARL PFLEGER and MEHRAN SAHAMl / Lazy
Acquisition of Place Knowledge 315
JOHN W. SHEPPARD and STEVEN L. SALZBERG I A Teaching
Strategy for Memory-Based Control 343
IV TABLE OF CONTENTS
DANIEL BORRAJO and MANUELA VELOSO / Lazy Incremental
Learning of Control Knowledge for Efficiently Obtaining Quality
Plans 371
WALTER DAELEMANS, ANTAL VAN DEN BOSCH and TON
WEUTERS / IGTree: Using Trees for Compression and Classi-
fication in Lazy Learning Algorithms 407
Artificial Intelligence Review 11: 1-6, 1997. 1
About the Authors
David W. Aha (Ue Irvine 1990) joined Department and a member of the MIT
the Naval Research Laboratory's Navy Artificial Intelligence Laboratory. His
Center for Applied Research in Artificial research focuses on numerical approach
Intelligence in 1993, having held post es to machine learning, and uses robot
doctoral fellowships at the Turing Insti ics as a domain in which to explore
tute, the Johns Hopkins University, and the behavior of learning algorithms. His
the University of Ottawa. His research early work was on model-based learn
interests lie at the intersection of machine ing. Recent work has explored task level
learning (ML) and case-based reason learning and memory-based learning.
ing (CBR). He serves both communities
as a frequent conference program com Daniel Borrajo is an Associate Profes
mittee member, editing board member sor of Computer Science at the Univer
(Machine Learning, Journal ofA rtificial sidad Carlos III de Madrid. He received
Intelligence Research, Applied Intelli his Ph.D. in Computer Science from the
gence), workshop (co-)organizer (AAAI- Universidad Politecnica de Madrid in
94 Workshop on eRR, ICML-95 Work 1990. Dr. Borrajo leads several machine
shop on Applying ML in Practice, AAAI- learning and problem solving projects at
95 Fall Symposium on Adaptation of his research institution. His recent inter
Knowledge for Reuse), and WWW page ests span across several areas, including
maiptainer. machine learning, planning, game theo
ry, and scientific discovery. His research
Ethem Alpaydin received his Ph.D. explores, among other things, the inte
in Computer Science from the Swiss gration of: analytical learning, induc
Federal Institute of Technology, Lau tive learning, and genetic programming;
sanne in 1990. He was a postdoc at the high-level planning and reactive plan
International Computer Science Institute, ning; and search strategies and learn
Berkeley in 1991 and a visiting scholar ing mechanisms. He is also interested
at the Department of Brain and Cogni in theory-based discovery, as well as the
tive Sciences, MIT in 1994. Since 1991 development of more efficient planners.
he has been teaching at the Department of Dr. Borrajo is the author of a book on
Computer Engineering, Bogazici Univer Artificial Intelligence.
sity, Istanbul, Turkey. His research is on
statistical techniques for machine learn Antal van den Bosch received his
ing and their application to pattern recog (Dutch equivalent of the) M.A. degree
nition, especially optical and pen-based in Computational Linguistics in 1992,
handwritten character recognition. from Tilburg University, The Nether
lands. After having worked as a research
Chris Atkeson recently joined the Geor assistant at the Institute for Language
gia Tech College of Computing facul Technology and AI and at the depart
ty. Previously, he was on the faculty of ment of Psychology at Tilburg Univer
the MIT Brain and Cognitive Sciences sity (1993), he is currently working on
2 ABOUT THE AUTHORS
a four-year Ph.D. project on machine cal University of Lisbon, where he has
learning of natural language, especial also been a teaching and research assis
ly morpho-phonology, at the Department tant. His main research interests are in
of Computer Science at the Universi machine learning and data mining. Pre
ty of Limburg, Maastricht, The Nether viously, he has worked in real-time rea
lands (since 1994). His research inter soning and computer graphics. He is the
ests include symbolic and connectionist author of six journal articles and twenty
inductive learning, and new methods in conference publications.
natural language processing.
Pat Langley has published widely on the
Walter Daelemans (1960, Antwerp) topics of machine learning and scientific
studied linguistics and psycholinguistics discovery, including the recent text Ele
at the Universities of Antwerp and Leu ments of Machine Learning. He is well
ven. He worked as a research assistant at known as an advocate of experimental
the University of Nijmegen in a project studies in artificial intelligence, and his
on the development of a dialogue system research crosses the areas of planning,
and author environment for Dutch, and at natural language, diagnosis, vision, and
the Artificial Intelligence Laboratory in control. Dr. Langley is an editor of the
Brussels, where he was responsible for journal Machine Learning and edits the
an ESPRIT project on office automation. Morgan Kaufmann series on that topic.
In 1987 he earned a Ph.D. (University of He holds a research position at Stanford
Leuven) with an object-oriented model of University and serves as Director of the
Dutch morphology and phonology and its Institute for the Study of Learning and
applications in language technology. He Expertise.
is presently affiliated as an associate pro
fessor to the Computational Linguistics Charles X. Ling obtained his B.Sc. from
group of Tilburg University and to the Shanghai Jiao Tong University in Chi
Linguistics Department of the Universi na in 1985, and an M.Sc. and a Ph.D.
ty of Antwerp (UIA), teaching Compu from the University of Pennsylvania in
tational Linguistics and Artificial Intelli 1989. Since 1989 he has been a facul
gence courses. His current research inter ty member in the Department of Com
ests are in Machine Learning of Natural puter Science at the University of West
Language, and knowledge representation ern Ontario (UWO). He is currently
techniques for natural language process an Associate Professor. Currently he is
ing. He has published on intelligent text also a faculty member at the Universi
processing, lexical database design, intel ty of Hong Kong. He has done exten
ligent tutoring systems, speech synthe sive research in computational model
sis, linguistic knowledge representation, ing of landmark cognitive learning tasks.
machine learning of natural language, He has also worked in several areas
and office automation. of machine learning, including Induc
tive Logic Programming (lLP), inductive
Pedro Domingos is a Ph.D. candidate learning from examples, artificial neural
in Information and Computer Science at networks, and real-world application of
the University of California, Irvine. He machine learning. His Web home page,
holds an M.S. in Information and Com http://www.csd.uwo.calfaculty/ling.
puter Science from U.C. Irvine, and an contains details of his research areas and
M.S. and B.S. in Electrical Engineering publications.
and Computer Science from the Techni-
3
ABOUT THE AUTHORS
Oded Maron is a graduate student at agent architectures. He has strong inter
the M.I.T. Artificial Intelligence Lab. He disciplinary interests in the long-term
received his undergraduate degree from goals of artificial intelligence and cogni
Brown University. He has done work tive science. His current research focuses
in machine learning, model selection, on learning and abstraction, specifical
and path planning. He has applied his lyon hierarchical compositional struc
research in the financial industry, creat ture and chunking.learning in sequences,
ing several automated trading systems for context effects in the integration of
an international investment firm. bottom-up and top-down processing, and
lossless data compression.
Takao Mohri received his Ph.D. from
the University of Tokyo in 1995 and spent Mehran Sahami is a doctoral candi
a year there as a Postdoctoral Fellow. date in the Computer Science Depart
He is now a researcher at Fujitsu Labo ment at Stanford University. He received
ratories Ltd. His interests lie mainly in both his B.S. and M.S. in Comput
artificial intelligence, especially in the er Science from Stanford in 1992 and
areas of inductive reasoning. case-based 1993, respectively. His research inter
and memory-based reasoning. Recently ests include machine learning, neural net
he is working on the application of artifi works, adaptive agents, and probabilis
cial intelligence techniques to the Inter tic reasoning. He is currently completing
net including WWW and netnews. a dissertation on probabilistic methods
for information retrieval and data min
Andrew Moore has worked in the area ing. He is also an instructor at Stan
of machine learning, locally weighted ford, teaching classes on programming
learning and reinforcement learning, for methodology and the ethical implications
seven years. He has published over 25 of technology, for which he received the
papers in the area, and given invit George Forsythe Memorial Award for
ed talks at the International Confer Excellence in student teaching.
ence on Machine Learning, the World
Congress on Neural Nets, and numer Marcos Salganicoff received his
ous industrial and academic departments. B.S.E.E. from Case Western Reserve
His funding includes an NSF CAREER University in 1985, was a member of the
award, an NSF Research Initiation Award technical staff at the Jet Propulsion Lab
and gifts from the 3M corporation and in the Applied Robotics Laboratory from
a food processing company. He has 1985 to 1987, and received his Ph.D.
applied these machine learning methods in Computer and Information Science
to processes in the power-distribution, from the Moore School of the Universi
automotive, food-manufacture, and tex ty of Pennsylvania in 1992. He is cur
tile industries. He is a co-founder of a rently Director of Algorithm Research
Pittsburgh-based AI start-up company: and Development for Sensar Inc., a sub
Schenley Park Research, Inc. sidiary of the David Sarnoff Research
Center, in Moorestown, New Jersey. His
Karl Pfleger is currently a Ph.D. stu research interests include machine learn
dent in Computer Science at Stanford ing for vision and action, computation
University, having received a B.S.E. al neuroscience, and real-time vision
from Princeton University in 1992. He processing applications.
has done research in machine learning,
neural networks, mobile robotics, and Steven Salzberg received the B.A.
4 ABOUT THE AUTHORS
degree in English and the M.S. and John Sheppard is a Principal Research
M.Phil. degrees in computer science Analyst with ARINC Incorporated. He
from Yale University in 1980, 1982, and holds a B.S. in Computer Science from
1984 respectively. He received the Ph.D. Southern Methodist University and an
degree in computer science from Har M.S. in Computer Science from Johns
vard University. From 1985 to 1987, he Hopkins University. Currently, he is a
was a research scientist with Applied Ph.D. candidate in Computer Science
Expert Systems of Cambridge, Massa at Johns Hopkins where he is doing
chusetts. In 1988 and 1989 he was research in reinforcement learning and
a Research Associate at the Harvard multiagent systems. At ARINC, he is
Business School, where he worked on responsible for research and develop
advanced manufacturing systems. In ment in intelligent diagnostic systems
1989 he joined the faculty of the Depart and holds a patent for a method and appa
ment of Computer Science at Johns ratus for intelligent diagnostic testing. He
Hopkins University in Baltimore, Mary has published over 60 papers in test, diag
land, where he is currently an Asso nosis, artificial intelligence, and machine
ciate Professor. He holds joint faculty learning and is co-author of the first book
appointments in the Departments of Bio on system test and diagnosis. In addition
medical Information Sciences and Cog to his research activities, he is actively
nitive Science. His research interests involved in IEEE and IEC standardiza
include machine learning and computa tion efforts in test and design automation.
tional biology, and he has authored more
than 50 papers in these areas. Kai Ming Ting received his undergrad
uate degree in Electrical Engineering
Stefan Schaal was a postdoctoral fellow (1986) from the University of Technol
at the Department of Brain and Cogni ogy, Malaysia, his master's degree in
tive Sciences and the Artificial Intelli Computer Science (1992) from the Uni
gence Laboratory at MIT, after receiving versity of Malaya, and his Ph.D. (1996)
his Ph.D. from the Technical University from the University of Sydney, Aus
of Munich in 1992. Currently, he is an tralia. He was a practicing engineer at
Adjunct Assistant Professor at the Geor the National Electricity Board, Malaysia
gia Institute of Technology, the Head from 1986 to 1992. He joined the Depart
of the Computational Learning group of ment of Computer Science at the Univer
the ERATO Neural Computation Project, sity ofWaikato, New Zealand in 1995, as
an Adjunct Assistant Professor at the a Post-Doctoral FellowlPart-Time Lec
Pennsylvania State University, and he turer. His main areas of research are
holds a part-time affiliation with the ATR Machine Learning and other aspects of
Human Information Processing Research Artificial Intelligence.
Laboratories in Japan. Stefan's interests
include topics such as statistical learn Manuela M. Veloso is Finmeccanica
ing, neural networks, nonlinear dynam Assistant Professor of Computer Sci
ics, and nonlinear control theory, applied ence at Carnegie Mellon University. She
to research on artificial and biological received her Ph.D. in Computer Sci
motor control and motor learning. His ence from Carnegie Mellon in 1992.
research approach focuses on both the Dr. Veloso's main research interest con
oretical investigations and experiments sists of the development of experience
with human subjects and anthropomor based intelligent agents that combine
phic robot equipment. high-level planning, low-level execution,
5
ABOUT THE AUTHORS
and learning. She investigates differ ment, focusing on the applicability of
ent planning algorithms, analogical/case machine learning techniques to linguis
based reasoning learning strategies, and tic domains and the development of new
the integration of analytical and induc machine-learning algorithms.
tive learning methods applied to plan
ning. She also researches on methods in Dietrich Wettschereck received his
which perception and learning are com M.Sc. and Ph.D. degrees in comput
bined to address jointly high-level and er science from Oregon State Universi
low-level reasoning tasks. Dr. Veloso is ty in 1990 and 1994, respectively. He
interested in strategy planning and learn has been a SYLFF (Sasakawa Young
ing in the context of multiple experience Leaders Fellowship Fund Program) fel
based agents, in collaborative and adver low since 1993. He has been a Post
sarial environments, such as robotic soc doctoral Fellow at the German National
cer. She is the author of a book on Plan Research Center for Information Tech
ning and Analogical Reasoning. nology (GMD) since September 1994.
He has published on the topics of neural
Handong Wang received the B.Sc. networks, instance-based learning algo
degree in precision machinery & instru rithms for propositional and relational
ments from the National University of representations, and data mining. He also
Defense Science and Technology, China, participates in projects revolving around
in 1991, and the M.S. degree in computer applied machine learning, data mining,
science from the University of Electron and inductive logic programming.
ic Science and Technology of China in
1994. From Oct. 1994 to Sept. 1995 he Junming Yang obtained the B.S. degree
was a research associate in the Depart in Mathematics from Shandong Univer
ment of Computer Science at the Univer sity in Jinan China in 1983 and a Ph.D. in
sity of Western Ontario, London, Cana Statistics from the Utah State Universi
da. He is currently a graduate student in ty in 1996. His primary research interests
the Department of Computer Science at are in statistical inference, statistical sim
the University of Western Ontario. His ulation, and machine learning. He is now
research interests include software engi a graduate student in Computer Science
neering, machine learning, and image at the Utah State University.
processing.
Yeesat Yim completed her B.S. degree in
Ton Weijters received his Ph.D. in Lan Applied Mathematics from the Nation
guage Philosophy from the University al Chung Hsing University in Taiwan in
of Nijmegen, The Netherlands, in 1989. 1988. She then worked for the Informa
In his Ph.D. thesis, entitled 'Denota tion Division of the National Museum of
tion in Discourse: Analysis and Algo Natural Science as a System Programmer
rithm', a denotation resolution algorithm and Research Assistant for several years.
was developed, specifying how a dis She obtained her MS degree in Comput
course representation for a given text er Science from Utah State University in
can be constructed. At present he is 1995.
assistant professor at the Department of
Computer Science at the University of
Jianping Zhang is Associate Profes
Limburg in Maastricht, The Netherlands.
sor of Computer Science at Utah State
He is involved in the research of the
University. He received his Ph.D. in
machine learning group of his depart-
Computer Science from the Universi-
