Table Of ContentWorld. Scientific Series in Computer Science Vol. 33
Mathematical
Foundations
of Parallel
Computing
WORLD SCIENTIFIC SERIES IN COMPUTER SCIENCE
Published
11: Control Theory of Robotic Systems (J M Skowronski)
12: An Introduction to Chinese, Japanese, and Korean Computing
(J KT Huang & T D Huang)
13: Mathematical Logic lor Computer Science (Z W Lu)
14: Computer Vision and Shape Recognition (Eds. A Krzyzak, T Kasvand & C Y Suen)
15: Stochastic Complexity in Statistical Inquiry (J Rissanen)
16: A Perspective in Theoretical Computer Science — Commemorative Volume
for Gift Siromoney (Ed. R Narasimnan)
17: Computer Transformation of Digital Images and Patterns (Z C Li, T D Bui,
Y Y Tang &CY Suen)
18: Array Grammars. Patterns and Recognizers (Ed. P S P Wang)
19: Structural Pattern Analysis (Eds. R Mohr, Th Paviidis & A Sanleliu)
20: A Computational Model of First Language Acquisition {N Satake)
21: The Design and Implementation of ConcurrentSmalltalk (V Yokole)
22: From Humans to Computers — Cognition Through Visual Perception
(V V Alexandrov & N D Gorsky)
23: Introduction to Theoretical Computer Science (Ma Xiwen)
24: A Digital Optical Cellular Image Processor — Theory, Architecture and
Implementation (K-S Huang)
25: Computer Epistemology — A Treatise on the Feasibility of the Unfeasible or Old
Ideas Brewed New (7" Vatnos)
26: Applications of Learning and Planning Methods {Ed. N G Bourbakis)
27: Advances in Artificial Intelligence — Applications and Theory (Ed. J Bezdek)
28: Introduction to Database and Knowledge-Base Systems (S Krishna)
29: Pattern Recognition: Architectures, Algorithms and Applications
(Eds R Plamondon & H D Cheng)
30: Character and Handwriting Recognition — Expanding Frontiers (Ed. P S P Wang)
31: Software Science and Engineering — Selected Papers from the Kyoto Symposia
(Eds. / Nakata & M Hagiya)
33: Mathematical Foundations of Parallel Computing {V V Voevodin)
Forthcoming
32: Advances in Machine Vision — Strategies and Applications
(Eds. C Archibald S E Petriu)
34: Language Architectures and Programming Environments (Eds. T Ichikawa &
H Tsubolam)
For a complete list of published Mies in the series, please write in to the publisher.
World Scientific Series in Computer Science Vol. 33
Mathematical
Foundations
of Parallel
Computing
Valentin V.
Voevodin
Russian Academy of Sciences
Vfe World Scientific
Singapore • New Jersey • London • Hong Kong
Published by
World Scientific Publishing Co. Pie. Lid.
POBox 128. FarrerRoad, Singapore 9128
USA office: Suite IB, 1060 Main Street, River Edge, NI07661
UK office: 73 Lynton Mead, Toiteridge. London N20 SDH
MATHEMATICAL FOUNDATIONS OF PARALLEL COMPUTING
Copyright © 1992 by World Scientific Publishing Co. Pte. Ltd.
All rights reserved. This book, or parts thereof, may not be reproduced in any form
or by any means,electronic or mechanical, including photocopying,recording or any
information storage and retrieval system now known or to be invented, without
written permission from the Publisher.
ISBN 981-02-0820-0
Printed in Singapore by Genera! Printing Services Pte. Ltd.
V
To my faithful, kind, and loving wife Sima
This page is intentionally left blank
vii
Contents
Preface
Chapter I. Algorithm and I ts Graph 1
§ 1. General notion of algorithm 2
§ 2. Algorithm notations 6
§ 3. Graph of algorithm - 17
§ 4. Topological sorting 26
§ 5. Schedules and graph machine -- — 34
§ 6. Examples ... 43
Chapter 2. Algorithm Execution Time 55
§ 7. Vector properties of schedules 57
§ 8. Number semirings and other sets 67
§ 9. Minimax properties of schedules 80
§ 10. Optimal and high-speed schedules 89
§ 11. Examples - 100
Chapter 3. Algorithms and Computer Memory — 107
§ 12. Examples - • 107
§ 13. Total required memory size 120
§ 14. Hierarchical memory 135
§ 15. Sectioning of memory 143
§ 16. Decomposition of algorithm and of i ts graph 148
Chapter 4. Matrix Investigation of Algorithm Structure -- 155
§ 17. Graphs and matrices - -— - - 156
§ 18. Recovering the linear functional 164
§ 19. Computing gradient and derivative • 169
§ 20. Roundoff error analysis 174
§ 21. Examples 186
Chapter 5. Functional Investigation of Algorithm Structure 193
§ 22. Space-time schedules - 193
§ 23. Regular graphs 203
viii
215
§ 24. Passage to the limit
231
§ 25. Data streams - -
§ 26. Examples --- -2 A0
Chapter 6. Algorithm Graph and Schedules Building 252
§ 27. Some statistics — 253
§ 28. Order relation - — 256
§ 29. Notation particularities 267
§ 30, Guidelines for algorithm graph building 275
§ 31. Splitting the algorithm graph 282
§ 32. Linear index expressions 292
§ 33. Branching — - 301
§ 34. Linear information closure 308
§ 35. Examples 313
Afterword — - - 327
References 332
Index 340
ix
Preface
The subject matter of this book is nontraditlonal for mathemati
cians. A lot of things still feel unfamiliar, in spite of the fact that
the objects of our examination are well known: we will study algo
rithms. However, our interest in algorithms is of a rather specific
nature. We focus on studying the structure of algorithms and on partic
ularities of their implementation on parallel computers. It is this as
pect of algorithm investigation that is nontraditlonal and poorly known
to mathematicians. The most nontraditlonal part of it Is research
methodology.
Before the reader proceeds with the book, the author's viewpoint
should be made clear. Hopefully both the research goals and the ways to
achieve them shall be easily understood after that.
For several decades, the author's research interests are in the
field of computational mathematics and numerical software. The way
things went, more than a dozen different computers had to be learned
over that period. One and the same problem persisted every time the
programs were transported to a new computer: changes in the source code
had to be made, although all programs were written in strict accordance
with the specifications of machine-independent algorithmic languages.
This state of things is highly abnormal from the point of view of
a mathematician for whom a computer is merely a tool he uses in his
work. Recall that algorithmic languages were originally Intended to
spare algorithm designers the necessity to scrutinize the peculiarities
of various computers. Operating systems together with compilers were
supposed to handle all activities required to adjust application pro
grams to current machine settings.
This initial goal has not been achieved yet, and it is unlikely
that it will be reached in near future. One does not have to be an ex
pert in algorithmic languages to come to this conclusion. Just try to
name a language that guarantees that programs in it will be implemented
efficiently on different computers without any changes. You will pretty
soon be convinced that no such languages exist.
X
Of course, we may blame language, compiler, and system developers
for poor appreciation of algorithm designers' needs. The rebuke is
partly justified as the algorithm designers' vested interests have cer
tainly got lost among a lot of other interests and are no longer dom
inant. It must also be owned however that the designers' problems are
far from being simple.
The accuracy of obtained results is one of the most important
characteristics of computational processes. It is well known that this
accuracy is influenced by both the overall scheme of an algorithm and
such issues as number representation and mode of rounding-off. The lat
ter factors are machine-dependent. The related information is not re
flected In most algorithmic language notations. Strictly speaking, such
languages are not machine-independent, as different computers may re
present numbers and round them off in different ways. In practice this
eventually brings about substantial diversity of the results. Then we
are left with a bundle of questions, as: "What do we actually mean by a
machine-independent language? What rules should we follow while design
ing algorithms and developing programs to make sure that they can be
viewed as machine-independent (maybe with certain reservations)? How
should the results be treated from the point of view of accuracy?" And
so on.
Huge amounts of effort have gone into finding the answers to the
above questions. Standards were adopted on number representation and on
round-off modes. Error bounds were obtained for many algorithms. On the
whole, however, the results are not spectacular. The forward and back
ward error analysis approaches are not constructive, and interval ana
lysis is too time-consuming. No other approaches currently exist. It is
possible that the chief achievement is the full realization of the fact
that roundoff error influence cannot be ignored during numerical soft
ware development.
For a long time it seemed that the accuracy problem was the only
fundamental difficulty concerning the implementation of algorithms on
different computers. Though its easy solution could not be counted on,
certain prospects nonetheless came into view. Gradually the process
went on of building up the confidence that the development of machine-
