Decision Theory Models for Applications in Artificial Intelligence: Concepts and Solutions

445 Pages·2011·8.93 MB·English

by  L. Enrique Sucar

Decision Theory Models
for Applications in
Artificial Intelligence:
Concepts and Solutions
L. Enrique Sucar
National Institute for Astrophysics, Optics and Electronics, Mexico
Eduardo F. Morales
National Institute for Astrophysics, Optics and Electronics, Mexico
Jesse Hoey
University of Waterloo, Canada
Managing Director: Lindsay Johnston
Senior Editorial Director: Heather Probst
Book Production Manager: Sean Woznicki
Development Manager: Joel Gamon
Development Editor: Joel Gamon
Acquisitions Editor: Erika Gallagher
Typesetters: Mackenzie Snader
Print Coordinator: Jamie Snavely
Cover Design: Nick Newcomer, Greg Snader
Published in the United States of America by
Information Science Reference (an imprint of IGI Global)
701 E. Chocolate Avenue
Hershey PA 17033
Tel: [phone]
Fax: [phone]
E-mail: [email protected]
Web site: http://www.igi-global.com
Copyright © 2012 by IGI Global. All rights reserved. No part of this publication may be reproduced, stored or distributed in
any form or by any means, electronic or mechanical, including photocopying, without written permission from the publisher.
Product or company names used in this set are for identification purposes only. Inclusion of the names of the products or
companies does not indicate a claim of ownership by IGI Global of the trademark or registered trademark.
Library of Congress Cataloging-in-Publication Data
Decision theory models for applications in artificial intelligence: concepts
and solutions / L. Enrique Sucar, Eduardo F. Morales and Jesse Hoey, editors.
p. cm.
Summary: “This book provides an introduction to different types of decision theory techniques, including MDPs, POM-
DPs, Influence Diagrams, and Reinforcement Learning, and illustrates their application in artificial intelligence”-- Provided
by publisher.
Includes bibliographical references and index.
ISBN 978-1-60960-165-2 (hardcover) -- ISBN 978-1-60960-167-6 (ebook) 1. Artificial intelligence--Statistical methods.
2. Bayesian statistical decision theory. I. Sucar, L. Enrique, 1957- II. Morales, Eduardo F. III. Hoey, Jesse.
Q335.D43 2011
006.3--dc22
2010054421
British Cataloguing in Publication Data
A Cataloguing in Publication record for this book is available from the British Library.
All work contributed to this book is new, previously-unpublished material. The views expressed in this book are those of the
authors, but not necessarily of the publisher.
List of Reviewers
Concha Bielza, Universidad Politécnica de Madrid, Spain
Fabio Cozman, University of Sao Paulo, Brazil
Boris Defourny, University of Liege, Belgium
Javier Díez, Universidad Nacional de Educación a Distancia, Spain
Pantelis Elinas, Australian Centre for Field Robotics, Australia
Nando de Freitas, University of British Columbia, Canada
Marcel van Gerven, Radboud University Nijmegen, Netherlands
Phan H. Giang, George Mason University, USA
Robby Goetschalckx, University of Waterloo, Canada
Kevin Grant, University of Lethbridge, Canada
Jesse Hoey, University of Waterloo, Canada
Finn V. Jensen, Aalborg University, Denmark
Omar Zia Khan, University of Waterloo, Canada
Eduardo F. Morales, INAOE, Mexico
Rubén Morales-Menendez, Tecnológico de Monterrey - Campus Monterrey, Mexico
Abdel-Illah Mouaddib, University of Caen, France
Kasia Muldner, University of British Columbia, Canada
Julieta Noguez, Tecnológico de Monterrey - Campus Ciudad de México, Mexico
Alberto Reyes, Instituto de Investigaciones Eléctricas, Mexico
Claude Sammut, University of New South Wales, Australia
Eugene Santos, Dartmouth College, USA
Matthijs Spaan, Delft University of Technology, Netherlands
L. Enrique Sucar, INAOE, Mexico
Jason Williams, AT&T Research, USA
Table of Contents
Foreword ..............................................................................................................................................xii
Preface .................................................................................................................................................xiii
Acknowledgment ................................................................................................................................xiv
Section 1
Fundamentals
Chapter 1
Introduction .............................................................................................................................................1
L. Enrique Sucar, National Institute for Astrophysics, Optics and Electronics, Mexico
Eduardo Morales, National Institute for Astrophysics, Optics and Electronics, Mexico
Jesse Hoey, University of Waterloo, Canada
Chapter 2
Introduction to Bayesian Networks and Influence Diagrams .................................................................9
Luis Enrique Sucar, National Institute for Astrophysics, Optics and Electronics, Mexico
Chapter 3
An Introduction to Fully and Partially Observable Markov Decision Processes .................................33
Pascal Poupart, University of Waterloo, Canada
Chapter 4
An Introduction to Reinforcement Learning .........................................................................................63
Eduardo F. Morales, National Institute for Astrophysics, Optics and Electronics, México
Julio H. Zaragoza, National Institute for Astrophysics, Optics and Electronics, México
Section 2
Concepts
Chapter 5
Inference Strategies for Solving Semi-Markov Decision Processes .....................................................82
Matthew Hoffman, University of British Columbia, Canada
Nando de Freitas, University of British Columbia, Canada
Chapter 6
Multistage Stochastic Programming: A Scenario Tree Based Approach to Planning
under Uncertainty ..................................................................................................................................97
Boris Defourny, University of Liège, Belgium
Damien Ernst, University of Liège, Belgium
Louis Wehenkel, University of Liège, Belgium
Chapter 7
Automatically Generated Explanations for Markov Decision Processes ...........................................144
Omar Zia Khan, University of Waterloo, Canada
Pascal Poupart, University of Waterloo, Canada
James P. Black, University of Waterloo, Canada
Chapter 8
Dynamic LIMIDS ...............................................................................................................................164
Francisco J. Díez, UNED, Spain
Marcel A. J. van Gerven, Radboud University Nijmegen, The Netherlands
Chapter 9
Relational Representations and Traces for Efficient Reinforcement Learning ...................................190
Eduardo F. Morales, National Institute for Astrophysics, Optics and Electronics, México
Julio H. Zaragoza, National Institute for Astrophysics, Optics and Electronics, México
Section 3
Solutions
Chapter 10
A Decision-Theoretic Tutor for Analogical Problem Solving ............................................................219
Kasia Muldner, Arizona State University, USA
Cristina Conati, University of British Columbia, Canada
Chapter 11
Dynamic Decision Networks Applications in Active Learning Simulators ........................................248
Julieta Noguez, Tecnológico de Monterrey, Mexico
Karla Muñoz, University of Ulster, Northern Ireland
Luis Neri, Tecnológico de Monterrey, Mexico
Víctor Robledo-Rella, Tecnológico de Monterrey, Mexico
Gerardo Aguilar, Tecnológico de Monterrey, Mexico
Chapter 12
An Intelligent Assistant for Power Plant Operation and Training Based on Decision-
Theoretic Planning ..............................................................................................................................271
Alberto Reyes, Instituto de Investigaciones Eléctricas, México
Francisco Elizalde, Instituto de Investigaciones Eléctricas, México
Chapter 13
POMDP Models for Assistive Technology .........................................................................................294
Jesse Hoey, University of Waterloo, Canada
Pascal Poupart, University of Waterloo, Canada
Craig Boutilier, University of Toronto, Canada
Alex Mihailidis, University of Toronto, Canada
Chapter 14
A Case Study of Applying Decision Theory in the Real World: POMDPs and
Spoken Dialog Systems ......................................................................................................................315
Jason D. Williams, AT&T Labs, USA
Chapter 15
Task Coordination for Service Robots Based on Multiple Markov Decision Processes ....................343
Elva Corona, National Institute for Astrophysics, Optics and Electronics, Mexico
L. Enrique Sucar, National Institute for Astrophysics, Optics and Electronics, Mexico
Chapter 16
Applications of DEC-MDPs in Multi-Robot Systems ........................................................................361
Aurélie Beynier, University Pierre and Marie Curie, France
Abdel-Illah Mouaddib, University of Caen, France
Compilation of References ...............................................................................................................385
About the Contributors ....................................................................................................................417
Index ...................................................................................................................................................424
Detailed Table of Contents
Foreword ..............................................................................................................................................xii
Preface .................................................................................................................................................xiii
Acknowledgment ................................................................................................................................xiv
Section 1
Fundamentals
Chapter 1
Introduction .............................................................................................................................................1
L. Enrique Sucar, National Institute for Astrophysics, Optics and Electronics, Mexico
Eduardo Morales, National Institute for Astrophysics, Optics and Electronics, Mexico
Jesse Hoey, University of Waterloo, Canada
This chapter gives a general introduction to decision-theoretic models in artificial intelligence and an
overview of the book.
Chapter 2
Introduction to Bayesian Networks and Influence Diagrams .................................................................9
Luis Enrique Sucar, National Institute for Astrophysics, Optics and Electronics, Mexico
This chapter covers the fundamentals of probabilistic graphical models, in particular: (i) Bayesian
networks, (ii) Dynamic Bayesian networks and (iii) Influence diagrams. For each it describes the rep-
resentation and main inference techniques. For Bayesian networks and dynamic Bayesian networks it
includes an overview of structure and parameter learning.
Chapter 3
An Introduction to Fully and Partially Observable Markov Decision Processes .................................33
Pascal Poupart, University of Waterloo, Canada
This chapter provides a gentle introduction to Markov decision processes as a framework for sequential
decision making under uncertainty. It reviews fully and partially observable Markov decision processes,
describes basic algorithms to find good policies and discusses modeling and computational issues that
arise in practice.
Chapter 4
An Introduction to Reinforcement Learning .........................................................................................63
Eduardo F. Morales, National Institute for Astrophysics, Optics and Electronics, México
Julio H. Zaragoza, National Institute for Astrophysics, Optics and Electronics, México
This chapter provides a concise and updated introduction to Reinforcement Learning from a machine
learning prespective. It gives the require background to undersand the chapters related to reinforcement
learning in this book, and includes an overview of some of the latest trends in the area.
Section 2
Concepts
Chapter 5
Inference Strategies for Solving Semi-Markov Decision Processes .....................................................82
Matthew Hoffman, University of British Columbia, Canada
Nando de Freitas, University of British Columbia, Canada
Semi-Markov decision processes are used to formulate many control problems and play a key role in
hierarchical reinforcement learning. This chapter shows how to translate the decision making problem
into a form that can instead be solved by inference and learning techniques. It establishes a formal con-
nection between planning in semi-MDPs and inference in probabilistic graphical models.
Chapter 6
Multistage Stochastic Programming: A Scenario Tree Based Approach to Planning
under Uncertainty ..................................................................................................................................97
Boris Defourny, University of Liège, Belgium
Damien Ernst, University of Liège, Belgium
Louis Wehenkel, University of Liège, Belgium
This chapter presents the multistage stochastic programming framework for sequential decision making
under uncertainty. It describes the standard technique for solving approximately multistage stochastic
problems, which is based on a discretization of the disturbance space called scenario tree. It also shows
how supervised learning techniques can be used to evaluate reliably the quality of an approximation.
Chapter 7
Automatically Generated Explanations for Markov Decision Processes ...........................................144
Omar Zia Khan, University of Waterloo, Canada
Pascal Poupart, University of Waterloo, Canada
James P. Black, University of Waterloo, Canada
It presents a technique to explain policies for factored MDPs by populating a set of domain-independent
templates, and a mechanism to determine a minimal set of templates that, viewed together, completely
justify the policy. The technique is demonstrated using the problems of advising undergraduate students
in their course selection and assisting people with dementia in completing the task of handwashing.
Chapter 8
Dynamic LIMIDS ...............................................................................................................................164
Francisco J. Díez, UNED, Spain
Marcel A. J. van Gerven, Radboud University Nijmegen, The Netherlands
Dynamic limited-memory influence diagrams (DLIMIDs) are a new type of decision-support model. Its
main difference with other models is the restriction of limited memory, which means that the decision
maker must make a choice based only on recent observations. This chapter presents several algorithms
for evaluating DLIMIDs, shows a real-world model for a medical problem, and compares DLIMIDs
with related formalisms.
Chapter 9
Relational Representations and Traces for Efficient Reinforcement Learning ...................................190
Eduardo F. Morales, National Institute for Astrophysics, Optics and Electronics, México
Julio H. Zaragoza, National Institute for Astrophysics, Optics and Electronics, México
This chapter introduces an approach for reinforcement learning based on a relational representation. The
underlying idea is to represent states as set of first order relations, actions in terms of these relations, and
policies over those generalized representations. The effectiveness of the approach is tested on a flight
simulator and on a mobile robot.
Section 3
Solutions
Chapter 10
A Decision-Theoretic Tutor for Analogical Problem Solving ............................................................219
Kasia Muldner, Arizona State University, USA
Cristina Conati, University of British Columbia, Canada
A decision-theoretic tutor that helps students learn from Analogical Problem Solving is described. The
tutor incorporates an innovative example-selection mechanism that tailors the choice of example to
a given student. An empirical evaluation shows that this selection mechanism is more effective than
standard selection approaches for fostering learning.
Chapter 11
Dynamic Decision Networks Applications in Active Learning Simulators ........................................248
Julieta Noguez, Tecnológico de Monterrey, Mexico
Karla Muñoz, University of Ulster, Northern Ireland
Luis Neri, Tecnológico de Monterrey, Mexico
Víctor Robledo-Rella, Tecnológico de Monterrey, Mexico
Gerardo Aguilar, Tecnológico de Monterrey, Mexico
This chapter describes how an intelligent tutor based on Dynamic Decision Networks is applied in an
undergraduate Physics scenario, where the aim is to adapt the learning experience to suit the learners’
needs. It employs Probabilistic Relational Models to facilitate the construction of the model. With this
representation, the tutor can be easily adapted to different experiments, domains, and student levels.