Table Of ContentACADEMIC PRESS RAPID MANUSCRIPT REPRODUCTION
Proceedings of the
International Conference on
Dynamic Programming and Its Applications
University of British Columbia
Vancouver, British Columbia, Canada
April 14-16,1977
DYNAMIC PROGRAMMING
AND
ITS APPLICATIONS
edited by
Martin L. Puterman
FACULTY OF COMMERCE AND BUSINESS
ADMINISTRATION
THE UNIVERSITY OF BRITISH COLUMBIA
VANCOUVER, B.C., CANADA
ACADEMIC PRESS New York San Francisco London 1978
A Subsidiary of Harcourt Brace Jovanovich, Publishers
COPYRIGHT © 1978, BY ACADEMIC PRESS, INC.
ALL RIGHTS RESERVED.
NO PART OF THIS PUBLICATION MAY BE REPRODUCED OR
TRANSMITTED IN ANY FORM OR BY ANY MEANS. ELECTRONIC
OR MECHANICAL, INCLUDING PHOTOCOPY, RECORDING, OR ANY
INFORMATION STORAGE AND RETRIEVAL SYSTEM, WITHOUT
PERMISSION IN WRITING FROM THE PUBLISHER.
ACADEMIC PRESS, INC.
Ill Fifth Avenue, New York, New York 10003
United Kingdom Edition published by
ACADEMIC PRESS, INC. (LONDON) LTD.
24/28 Oval Road, London NW1 7DX
Library of Congress Cataloging in Publication Data
International Conference on Dynamic Programming and
its Applications, University of British Columbia, 1977.
Dynamic programming and its applications.
"Proceedings of the International Conference on
Dynamic Programming and its Applications, University of
British Columbia, Vancouver, British Columbia, Canada,
April 14-16, 1977."
1. Dynamic programming—Congresses. I. Puterman,
Martin L. II. Title.
T57.83.157 1977 19.7Ό3 78-21621
ISBN 0-12-568150-X
PRINTED IN THE UNITED STATES OF AMERICA
78 79 80 81 82 9 8 7 6 5 4 3 2 1
To Karel deLeeuw
(1930-1978)
CONTRIBUTORS
Numbers in parentheses indicate the pages on which authors' contributions begin.
Dimitri P. Bertsekas (115), Decision and Control Laboratory, Department of
Electrical Engineering and Coordinated Science Laboratory, University
of Illinois, Urbana, Illinois 61801
Shelby L. Brumelle (91), Faculty of Commerce and Business Administration,
University of British Columbia, Vancouver, British Columbia V6T
1W5, Canada
Eric V. Denardo (249, 255, 393), School of Organization and Management,
Yale University, New Haven, Connecticut 06520
Cyrus Derman (163), Division of Mathematical Methods in Engineering and
Operations Research, Columbia University, New York, New York
10027
Bharat T. Doshi (269), Department of Statistics, Hill Center for the Mathe
matical Sciences, Rutgers, The State University of New Jersey, Busch
Campus, New Brunswick, New Jersey 08903
Am Federgruen (3, 23), Foundation Mathematics Centrum, 2e Boer-
haavestraat 49, Amsterdam, The Netherlands
JamesFlynn (173), Apt. 303,2116 Pauline Blvd., Ann Arbor, Michigan 48103
Bennett L. Fox (249), Département d'Informatique, Université de Mon
tréal, C.P. 6128 — Succursale "A", Montréal, P.Q. H3C 3J7 Canada
Karl Hinderer (289, 389), Institut fur Mathematische Statistik, der Universität
Karlsruhe, Englerstrasse 2, 75 Karlsruhe 1, West Germany
Arie Hordijk (3), Rijksuniversiteit te Leiden, Insti tuut voor Toegepaste
Wiskunde en Informacia, Wassenaarseweg 80, Postbus 9512, 2300 Ra
Leiden, The Netherlands
Ronald A. Howard (201), Department of Engineering - Economic Systems,
Stanford University, Stanford, California 94305
Seiichi Iwamoto (319), Department of Mathematics, Faculty of Integrated
Arts and Science, Hiroshima University, Hiroshima, 730, Japan
Mark R. Lembersky (207), Leader, Allocations and Systems Group, Weyer-
hauser Company, Tacoma, Washington 99401
xi
XU Contributors
Gerald J. Lieberman (163), Department of Operations Research, Stanford
University, Stanford, California 94305
Thomas L. Morin (53), School of Industrial Engineering, Purdue University,
West Lafayette, Indiana 47907
A. John Petkau (221), Department of Mathematics, University of British Co
lumbia, Vancouver, British Columbia V6T 1W5, Canada
Stanley R. Pliska (335), Department of Industrial Engineering and Manage
ment Sciences, Northwestern University, Evanston, Illinois 60201
Martin L. Puterman (91), Faculty of Commerce and Business Administration,
University of British Columbia, Vancouver, British Columbia V6T
1W5, Canada
Sheldon M. Ross (163), Department of Operations Research, University of
California, Berkeley, California 94720
Uriel G. Rothblum (255), School of Organization and Management, Yale Uni
versity, New Haven, Connecticut 06520
Manfred Schäl (351), der Universität Bonn, Institut für Angewandte
Mathematik, Wegelerstrasse 6, 53 Bonn, West Germany
Paul Schweitzer (23), The Graduate School of Management, The University of
Rochester, Rochester, New York 14627
Steven E. Shreve (115), Department of Statistics, University of California,
Berkeley, California 94720
Henk C. Tijms (3), Interfaculteit der Actuariele Wetenschappen en
Econometrie, der Vrije Universiteit, De Boelelaan 1081, Amsterdam,
The Netherlands
Arthur F. Veinott, Jr. (397), Department of Operations Research, Stanford
University, Stanford, California 94305
Carl J. Walters (233), Institute of Animal Resource Ecology, University of
British Columbia, Vancouver, British Columbia V6T 1W5, Canada
DJ. White (131), Faculty of Economics and Social Studies, Department of
Decision Theory, University of Manchester, Manchester M13 9PL,
England
Jacob Wijngaard (369), Eindhoven University of Technology, P.O. Box 513,
Eindhoven, The Netherlands
PREFACE
Research and interest in dynamic programming have evolved rapidly since
publication over twenty years ago of the initial papers in the field. Scholars
have investigated the theory and application of dynamic programming at
many centres throughout the world, and I and my colleagues at the University
of British Columbia felt that an international meeting of prominent research
ers would be timely and worthwhile. This volume presents the proceedings of
the International Conference on Dynamic Programming and Its Applications
held at the University of British Columbia, Vancouver, British Columbia,
Canada, April 14-16,1977.
These proceedings consist of twenty papers and an edited summary of a
panel discussion on the state of the art and future directions for dynamic pro
gramming. All of the papers have been reviewed by me and, in most cases, by
external referees. The papers have been divided into three categories: surveys,
theory, and applications. Some of the papers could have been classified differ
ently, but I felt that the three groupings accurately represent the main focus of
each paper. Two of the papers included for publication (the ones by Lember-
sky and White) were not presented at the conference because the authors were
unable to attend.
The survey papers summarize and unify work in several of the main areas of
dynamic programming research. The paper of Federgruen, Hordijk, and
Tijms presents recurrence conditions for countable state Markov decision
problems which ensure that the optimal average reward exists and satisfies the
functional equation of dynamic programming. Federgruen and Schweitzer
give an extensive analysis of the theory of successive approximation for
Markov decision problems including many of the authors' recent results for
the undiscounted case. Morin summarizes, unifies, and comments on compu
tational methods for deterministic, finite horizon problems and includes an ex
tensive bibliography. Puterman and Brumelle survey work on policy iteration
and present and extend their recent work on the relationship between policy
iteration and Newton's method. Shreve and Bertsekas present a unified and
extremely insightful presentation of several foundational questions. White
Xlll
XIV Preface
summarizes and unifies the theory of action elimination procedures for
Markov decision problems.
The application papers present practical and theoretical applications of dy
namic programming methodology. Derman, Lieberman, and Ross outline
their recent work on the structure of optimal policies for replacement prob
lems. Flynn discusses the theory and use of steady-state policies and their ap
plicability to optimal economic growth problems. Howard gives an anecdotal
presentation of his original work on Markov decision problems and comments
on why these methods have not been applied; a brief summary of questions
and answers following his conference presentation is included. Lembersky
summarizes applications of Markov decision process techniques to forest
stand management and also presents some related theoretical results. Pet kau
shows how backward induction and optimal stochastic control methodology
can be used to solve statistical decision problems arising from sequential medi
cal trials. Walters discusses recent formulation and computational results for
fisheries management problems.
The theory section contains several original papers. Denardo and Fox give
an extended summary of their new computational methods for shortest route
problems. Denardo and Rothblum detail their work on problems with affine
reward and transition structure. Doshi presents new results on the structure
and characterization of optimal policies from a problem in controlled diffu
sions. Hinderer analyzes methods for approximating dynamic programs
including several potentially useful bounds. Iwamoto extends his work on in
verse dynamic programming to infinite horizon problems and gives some inter
esting examples. Pliska generalizes Veinott's and Hordijk's results on transient
policies to problems with general state spaces and discusses the applicability of
these generalized results to a problem in the control of multitype branching
processes. Schäl studies negative dynamic programming in an abstract setting
and among other results gives conditions that ensure that value iteration con
verges to the optimal return function. Wijngaard studies Markov decision
problems with average reward criterion and shows how an unbounded reward
structure can be exploited to prove the existence of optimal policies.
Professors Denardo, Hinderer, and Veinott gave short presentations in a
panel discussion during the concluding session of the conference. Transcribed,
edited, and annotated copies of their comments are included in the final sec
tion of these proceedings. These thought-provoking remarks survey the cur
rent vista and highlight some areas that researchers will be studying in the near
future.
I hope that this volume illustrates the richness and scope of dynamic pro
gramming research of the past quarter-century and helps to inspire further
work in this field.
This conference would not have been possible without the generous finan
cial assistance of the National Research Council of Canada, the Faculty of
Commerce of the University of British Columbia, and the Management
Preface xv
Science Research Centre of the University of British Columbia. Special thanks
are due to the Faculty of Commerce for use of the E.D. MacPhee Conference
Centre. I also would like to thank the following people for their conscientious
refereeing of papers in these proceedings: Dmitri Bertsekas, Shelby Brumelle,
Bharat Doshi, Bennett Fox, Frieda Granot, Arie Hordijk, Thomas Morin,
Stanley Pliska, Manfred Schäl, Moon Shin, Bruce Sinclair and Henk Tijms.
Finally, I am extremely grateful to Mrs. Maryse Ellis for her exceptional typ
ing of this publication and to my wife, Dorothea Katzenstein, for editorial
assistance.
MARTIN L. PUTERMAN
Dynamic Programming and Its Applications
RECURRENCE CONDITIONS IN DENUMERABLE STATE
MARKOV DECISION PROCESSES
A. Federgruen
Mathematisch Centrum
Amsterdam
A. Hordijk
Universiteit van Leiden
Leiden
H.C. Tijms
Vrije Universiteit
Amsterdam
This paper considers an undiscounted semi-Markov decision
problem with denumerable state space and compact metric action
spaces. Recurrence conditions on the transition probability
matrices associated with the stationary policies are consider
ed and relations between these conditions are established.
Also it is shown that under each of these conditions the
optimality equation for the average costs has a bounded
solution.
I. INTRODUCTION
In this paper we consider an undiscounted semi-Markov
decision model specified by five objects (I, A(i), ρ^j(a),
c(i,a), i(i,a)). We are concerned with a dynamic system which
at decision epochs beginning with epoch 0 is observed to be in
one of the states of the denumerable state space I. After
observing the state of the system, an action must be chosen.
Received September 1977 - Revised March 1978
Copyright © 1979 by Academic Press, Inc.
3 All rights of reproduction in any form reserved.
ISBN 0-12-568150-X
