Evaluating Derivatives: Principles and Techniques of Algorithmic Differentiation (Frontiers in Applied Mathematics)

394 Pages·1987·20.54 MB·English

by  Andreas Griewank

Evaluating Derivatives
This page intentionally left blank
FRONTIERS IN
APPLIED MATHEMATICS
The SIAM series on Frontiers in Applied Mathematics publishes monographs dealing
with creative work in a substantive field involving applied mathematics or scientific
computation. All works focus on emerging or rapidly developing research areas that
report on new techniques to solve mainstream problems in science or engineering.
The goal of the series is to promote, through short, inexpensive, expertly written
monographs, cutting edge research poised to have a substantial impact on the solutions
of problems that advance science and technology. The volumes encompass a broad
spectrum of topics important to the applied mathematical areas of education,
government, and industry.
EDITORIAL BOARD
H.T. Banks, Editor-in-Chief, North Carolina State University
Richard Albanese, U.S. Air Force Research Laboratory, Brooks AFB
Carlos Castillo Chavez, Cornell University
Doina Cioranescu, Universite Pierre et Marie Curie (Paris VI)
Pat Hagan, NumeriX, New York
Matthias Heinkenschloss, Rice University
Belinda King, Virginia Polytechnic Institute and State University
Jeffrey Sachs, Merck Research Laboratories, Merck and Co., Inc.
Ralph Smith, North Carolina State University
Anna Tsao, Institute for Defense Analyses, Center for Computing Sciences
BOOKS PUBLISHED IN FRONTIERS
IN APPLIED MATHEMATICS
Griewank, Andreas, Evaluating Derivatives: Principles and Techniques of Algorithmic
Differentiation
Kelley, C.T., Iterative Methods for Optimization
Greenbaum, Anne, Iterative Methods for Solving Linear Systems
Kelley, C.T., Iterative Methods for Linear and Nonlinear Equations
Bank, Randolph E., PLTMG:A Software Package for Solving Elliptic Partial Differential Equations.
Users'Guide 7.0
More, Jorge J. and Wright, Stephen J., Optimization Software Guide
Rude, Ulrich, Mathematical and Computational Techniques for Multilevel Adaptive Methods
Cook, L. Pamela, Transonic Aerodynamics: Problems in Asymptotic Theory
Banks, H.T., Control and Estimation in Distributed Parameter Systems
Van Loan, Charles, Computational Frameworks for the Fast Fourier Transform
Van Huffel, Sabine and Vandewalle.Joos, The Total Least Squares Problem: Computational
Aspects and Analysis
Castillo, Jose E., Mathematical Aspects of Numerical Grid Generation
Bank, R. E., PLTMG: A Software Package for Solving Elliptic Partial Differential Equations.
Users'Guide 6.0
McCormick, Stephen R, Multilevel Adaptive Methods for Partial Differential Equations
Grossman, Robert, Symbolic Computation: Applications to Scientific Computing
Coleman, Thomas R and Van Loan, Charles, Handbook for Matrix Computations
McCormick, Stephen R, Multigrid Methods
Buckmaster, John D., The Mathematics of Combustion
Ewing, Richard E., The Mathematics of Reservoir Simulation
Evaluating Derivatives
Principles and Techniques of Algorithmic Differentiation
Andreas Griewank
Institute of Scientific Computing
Technical Institute of Dresden
Dresden, Germany
Society for Industrial and Applied Mathematics
Philadelphia
Copyright © 2000 by the Society for Industrial and Applied Mathematics
1098765432 I
All rights reserved. Printed in the United States of America. No part of this book may be
reproduced, stored, or transmitted in any manner without the written permission of the
Publisher. For information, write the Society for Industrial and Applied Mathematics, 3600
University City Science Center, Philadelphia, PA 19104-2688.
Library of Congress Cataloging-in-Publication Data
Griewank, Andreas.
Evaluating derivatives: principles and techniques of algorithmic differentiation /
Andreas Griewank.
p. cm.— (Frontiers in applied mathematics ; 19)
Includes bibliographical references and index.
ISBN 0-89871-451-6 (pbk.)
I. Differential calculus—Data processing. I. Title. II. Series.
QA304.G76 2000
5l5'.33--dc2l 99-052587
is a registered trademark.
Acknowledgments
The original plan for writing this book was hatched with Bruce
Christianson, who contributed most of the Introduction, wrote much
of Chapter 5 and taught me the word bough. The book was com-
pleted during a sabbatical at INRIA, Sophia Antipolis, aided by the
financial and intellectual resources of project Tropics. My graduate
students Uwe Naumann, Andrea Walther, Hristo Mitev, and Olaf
Vogel generously provided their insights, numerical results, figures,
and proofreading expertise. George Corliss and several other col-
leagues from the automatic differentiation community helped with
comments and suggestions to make the book more readable. The
completion of this book would have been impossible without the un-
tiring efforts of Mrs. Sigrid Eckstein in composing and revising the
LAT X source. Gail Pieper and the SIAM staff corrected the En-
E
glish. My wife and children put up with an often absent-minded
and occasionally frantic husband and father. Thanks to all of them;
it probably won't happen again.
I am indebted to various farsighted individuals and agencies who
have supported my research for the last 12 years, while I was do-
ing mostly derivatives. A short list includes Jorge More of Argonne
National Laboratory, Charles Holland of the U.S. Department of De-
fense, the late Fred Howes of the Department of Energy, John Burns
of Virginia Tech, Hubert Schwetlick of the TU Dresden, and Paul
Konigs of the Deutsche Forschungsgemeinschaft. Finally, I would
like to express my gratitude to Christian Bischof, Martin Berz and
several other close colleagues whose contributions to the field I may
not have fully appreciated and appropriately recognized at one time
or another.
My author's royalties will be donated to the Transnational Foun-
dation for Peace and Future Research (www.transnational.org) to
further the cause of international cooperation and understanding.
vii
This page intentionally left blank
Contents
Rules xiii
Preface xv
Prologue xix
Mathematical Symbols xxiii
1 Introduction 1
I Tangents and Gradients 13
2 A Framework for Evaluating Functions 15
2.1 The Lighthouse Example 16
2.2 Three-Part Evaluation Procedures 18
2.3 Elemental Differentiability 24
2.4 Memory Allocation and Overwrites 26
2.5 A Temporal Complexity Model 29
2.6 Examples and Exercises 36
3 Fundamentals of Forward and Reverse 37
3.1 Forward Propagation of Tangents 38
3.2 Complexity of Tangent Propagation 43
3.3 Reverse Propagation of Gradients 45
3.4 Complexity of Gradient Propagation 56
3.5 Summary and Discussion 60
3.6 Examples and Exercises 65
4 Repeating and Extending Reverse 69
4.1 Adjoining Iterative Assignments 71
4.2 Selective Saves and Restores 76
4.3 Adjoints of Adjoints 79
4.4 Tangents of Adjoints 82
4.5 Complexity of Second-Order Adjoints 84
4.6 Examples and Exercises 87
ix