Algorithms for Parallel Processing

372 Pages·1999·13.141 MB·English

by  Vijaya Ramachandran (auth.), Michael T. Heath, Abhiram Ranade, Robert S. Schreiber (eds.)

The IMA Volumes
in Mathematics
and its Applications
Volume 105
Series Editors
Avner Friedman Robert Gulliver
Springer Science+Business Media, LLC
Institute for Mathematics and
its Applications
IMA
The Institute for Mathematics and its Applications was estab
lished by a grant from the National Science Foundation to the University
of Minnesota in 1982. The IMA seeks to encourage the development and
study of fresh mathematical concepts and questions of concern to the other
sciences by bringing together mathematicians and scientists from diverse
fields in an atmosphere that will stimulate discussion and collaboration.
The IMA Volumes are intended to involve the broader scientific com
munity in this process.
Willard Miller, Jr., Professor and Director
* * * * * * * * * *
IMA ANNUAL PROGRAMS
1982-1983 Statistical and Continuum Approaches to Phase Transition
1983-1984 Mathematical Models for the Economics of Decentralized
Resource Allocation
1984-1985 Continuum Physics and Partial Differential Equations
1985-1986 Stochastic Differential Equations and Their Applications
1986-1987 Scientific Computation
1987-1988 Applied Combinatorics
1988-1989 Nonlinear Waves
1989-1990 Dynamical Systems and Their Applications
1990-1991 Phase Transitions and Free Boundaries
1991-1992 Applied Linear Algebra
1992-1993 Control Theory and its Applications
1993-1994 Emerging Applications of Probability
1994-1995 Waves and Scattering
1995-1996 Mathematical Methods in Material Science
1996-1997 Mathematics of High Performance Computing
1997-1998 Emerging Applications of Dynamical Systems
1998-1999 Mathematics in Biology
1999-2000 Reactive Flows and Transport Phenomena
2000-2001 Mathematics in Multi-Media
Continued at the back
Michael T. Heath Abhiram Ranade
Robert S. Schreiber
Editors
Algorithms for
Parallel Processing
With 129 Illustrations
Springer
Michael T. Heath Abhiram Ranade
Department of Computer Science Department of Computer Science
University of Illinois at Urbana and Engineering
Urbana, IL 61801-2987, USA Indian Institute of Technology
Powai, Mumbai 400076, India
Robert S. Schreiber
Hewlett-Packard, Inc.
Palo Alto, CA 94305-1126, USA
Series Editors:
Avner Friedman
Robert Gulliver
Institute for Mathematics and its
Applications
University of Minnesota
Minneapolis, MN 55455, USA
Mathematics Subject Classifications (1991): 65Y05, 6806
CIP Data AvaiJable.
Printed on acid-free paper.
© 1999 Springer Science+Business Media New York
Originally published by Springer-Verlag New York, Inc. in 1999
Softcover reprinl of the hardcover 1s i edition 1999
AII rights reserved. This work may not be translated or copied in whole or in part without the wrilten
permission of the publisher, Springer Science+Business Media, LLC
except for brief excerpts in connection with reviews or scholarly analysis. Vse in connection with
any form of information storage and retrieval, electronic adaptation, computer software, or by similar or
dissimilar methodology now known or hereafter developed is forbidden.
The use of general descriptive names, trade names, trademarks, etc., in this publication, even if the former
are not especially identified, is not to be taken as a sign that such names, as understood by the Trade Marks
and Merchandise Marks Act, may accordingly be used freely by anyone.
Authorization to photocopy items for internal or personal use, or the internal or personal use of specific cli
ents, is granted by Springer Science+Business Media, LLC, provided that the appropriate fee is paid dir
ectly to Copyright Clearance Center, 222 J{osewood Urive, Uanvers, MA 01923, USA (Telephone:
[phone]), stating ilie ISBN number, the title of the book, and the first and last page numbers of
each article copied. The copyright owner's consent does not include copying for general distribution, pro
motion, new works, or resale. In these cases, specific wrilten permission must first be obtained from the
publisher.
Production managed by Alian Abrams; manufacturing supervised by Jeffrey Taub.
Camera-ready copy prepared by the IMA.
981654321
ISBN 978-1-4612-7175-8 ISBN 978-1-4612-1516-5 (eBook)
DOI 10.1007/978-1-4612-1516-5
FOREWORD
This IMA Volume in Mathematics and its Applications
ALGORITHMS FOR PARALLEL PROCESSING
is based on the proceedings of a workshop that was an integral part of the
1996-97 IMA program on "MATHEMATICS IN HIGH-PERFORMANCE
COMPUTING." The workshop brought together algorithm developers from
theory, combinatorics, and scientific computing. The topics ranged over
models, linear algebra, sorting, randomization, and graph algorithms and
their analysis.
We thank Michael T. Heath of University of lllinois at Urbana (Com
puter Science), Abhiram Ranade of the Indian Institute of Technology
(Computer Science and Engineering), and Robert S. Schreiber of Hewlett
Packard Laboratories for their excellent work in organizing the workshop
and editing the proceedings.
We also take this opportunity to thank the National Science Founda
tion (NSF) and the Army Research Office (ARO), whose financial support
made the workshop possible.
A vner Friedman
Robert Gulliver
v
PREFACE
The Workshop on Algorithms for Parallel Processing was held at the
IMA September 16 - 20, 1996; it was the first workshop of the IMA year
dedicated to the mathematics of high performance computing. The work
shop organizers were Abhiram Ranade of The Indian Institute of Tech
nology, Bombay, Michael Heath of the University of Illinois, and Robert
Schreiber of Hewlett Packard Laboratories. Our idea was to bring together
researchers who do innovative, exciting, parallel algorithms research on a
wide range of topics, and by sharing insights, problems, tools, and methods
to learn something of value from one another. As the remarkable ensemble
of papers contained herein should show, we seem to have succeeded in creat
ing ample opportunity for such exchanges. The papers offer a wide-ranging
tour of recent developments in the very rapidly growing and changing field
of parallel algorithms. They cover the following general areas:
• models and mechanisms for parallel machines (Chapters 1-4),
• discrete and combinatorial algorithms (Chapters 5-7),
• mathematical issues in parallelizing compilers (Chapter 8),
• parallel algorithms for matrix computation, differential equations,
random number generation, and Fourier methods (Chapters 9-14),
• new parallel computer systems and software (Chapters 15-16).
We hope that readers will find this collection as enjoyable and infor
mative as we have.
Michael T. Heath
Abhiram Ranade
Robert S. Schreiber
vii
CONTENTS
Foreword ............................................................. v
Preface ............................................................. vii
MODELS AND MECHANISMS IN PARALLEL COMPUTATION
A general-purpose shared-memory model
for parallel computation. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. 1
Vi jay a Ramachandran
Supporting a coherent shared address space across
SMP nodes: An application-driven investigation. . . . . . . . . . . . . . . . . . . . .. 19
Angelos Bilas, Liviu Iftode, Rudrajit Sam ant a,
and laswinder Pal Singh
Single-message vs. batch communication ............................. 61
Mark W. Goudreau and Satish B. Rao
Some simple and practical strategies for parallelism ................... 75
Nicholas Carriero and David Gelernter
DISCRETE AND COMBINATORIAL ALGORITHMS
Locality in computing connected components. . . . . . . . . . . . . . . . . . . . . . . .. 89
Abhiram Ranade
Routing in optical and wireless networks ............................ 101
Eric 1. Schwabe
Transparent parallel transactions on
replicated autonomous databases. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. 117
Rekha Goel and Gautam M. Shroff
MATHEMATICS OF PARALLELIZING COMPILERS
Mathematical tools for loop transformations: From systems
of uniform recurrence equations to the polytope model .............. 147
Alain Darte
ix
x CONTENTS
NUMERICAL ALGORITHMS
The scalability of mesh improvement algorithms. . . . . . . . . . . . . . . . . . . .. 185
Lori A. Freitag, Mark T. Jones, and Paul E. Plassmann
Data parallel performance optimizations using array aliasing ......... 213
Y. Charlie Hu and S. Lennart Johnsson
Coarsening, sampling, and smoothing: Elements of the
multilevel method ................................................... 247
Shang-Hua Teng
Some methods of parallel pseudorandom number generation. . . . . . . .. 277
Michael M ascagni
Performance of parallel sparse triangular solution .................... 289
Michael T. Heath and Padma Raghavan
Determining an out-of-core FFT decomposition strategy
for parallel disks by dynamic programming. . . . . . . . . . . . . . . . . . . . . . . . .. 307
Thomas H. Cormen
PARALLEL COMPUTER SYSTEMS AND SOFTWARE
Enabling department-scale supercomputing .......................... 321
David S. Greenberg, William E. Hart, and Cynthia A. Phillips
Providing uniform dynamic access to numerical software. . . . . . . . . . . .. 345
Henri Casanova and Jack Dongarra
A GENERAL-PURPOSE SHARED-MEMORY MODEL
FOR PARALLEL COMPUTATION*
VIJAYA RAMACHANDRANt
Abstract. We describe a general-purpose shared-memory model for parallel com
putation, called the QSM [22], which provides a high-level shared-memory abstraction for
parallel algorithm design, as well as the ability to be emulated in an effective manner
on the BSP, a lower-level, distributed-memory model. We present new emulation results
that show that very little generality is lost by not having a 'gap parameter' at memory.
1. Introduction. The design of general-purpose models of parallel
computation has been an important topic of study in recent years. How
ever, due to the diversity of architectures among parallel machines, this
has also proved to be a very challenging task. The challenge here has been
to find a model that is general enough to encompass the wide variety of
parallel machines available, while retaining enough of the essential features
of these diverse machines in order to serve as a reasonably faithful model
of them.
Until recently there have been two approaches taken towards modeling
parallel machines for the purpose of algorithm design. The more popular
of the two approaches has been to design parallel algorithms on the PRAM,
which is a synchronous, shared-memory model in which each processor can
perform a local computation or access a shared memory location in a unit
time step, and there is global synchronization after each step. As a simple
model at a high level of abstraction, the PRAM has served an important
role, and most of the basic paradigms for parallel algorithm design as well
as the basic ideas underlying the parallel algorithms for many problems
have been developed on this model (see, e.g., [24, 28, 42]).
The other approach that has been used to design parallel algorithms
has been to consider distributed-memory models, and tailor the parallel
algorithm to a specific interconnection network that connects the proces
sors and memory, e.g., mesh, hypercube, shuffle-exchange, cube-connected
cycles, etc. There are several results known on embedding one of these
networks, the source network, on to another, the target network (see, e.g.,
[31]), so that an efficient algorithm on the source network results in an
efficient algorithm on the target network.
Neither of the above approaches has been very satisfactory. On the
one hand, the PRAM is too high-level a model, and it ignores completely
the latency and bandwidth limitations of real parallel machines. On the
other hand, algorithms developed for a specific interconnection network
are tailored to certain standard, regular networks, as are the embedding
"This work was supported in part by NSF grant CCR/GER-90-23059.
tDept. of Computer Sciences, University of Texas at Austin, Austin, TX 78712.
Email: [email protected].
1
M. T. Heath et al. (eds.), Algorithms for Parallel Processing
© Springer-Verlag New York, Inc. 1999
2 VIJAYA RAMACHANDRAN
results, and hence are not truly general-purpose. Thus it is not surprising
that a variety of other models have been proposed in the literature, (e.g.,
[2, 5-7, 9, 13, 15, 18, 23, 29, 32, 35, 36, 39, 41, 46, 47]) to address specific
drawbacks of the PRAM although none of these are general-purpose models
in the sense of being able to model a wide variety of parallel machines with
reasonable accuracy .
In recent years, distributed-memory models that characterize the inter
connection network abstractly by parameters that capture its performance
have gained much attention. An early work along these lines is the CTA
[43]. More recently, the BSP (Bulk Synchronous ParalleQ model [44, 45]
and the LogP model [14] have gained wide acceptance as general-purpose
models of parallel computation. In these models the parallel machine is
abstracted as a collection of processor-memory units with no global shared
memory. The processors are interconnected by a network whose perfor
mance is characterized by a latency parameter L and a gap parameter g.
The latency of the network is the time needed to transmit a message from
one processor to another. The gap parameter g indicates that a processor
can send no more than one message every g steps. This parameter reflects
the bandwidth of the network - the higher the bandwidth, the lower is the
value of g. The models may have some additional parameters, such as the
overhead in sending messages, and the time for synchronization (in a model
that is not asynchronous). In contrast to earlier fixed interconnection net
work models, the BSP and LOgP models do not take into consideration the
exact topology of the interconnection network.
The BSP and LOgP models have gained popularity in recent years, and
many algorithms have been designed and analyzed on these models and
their extensions (see, e.g., [4, 8, 17, 25, 27, 37, 48]). However, algorithms
designed for these models tend to have rather complicated performance
analyses, because of the number of parameters in the model as well as the
need to keep track of the exact memory partition across the processors at
each step.
Very recently, in Gibbons, Matias & Ramachandran [22] the issue of
whether there is merit in developing a general-purpose model of paral
lel computation starting with a shared-memory framework, was explored.
Certainly, shared-memory has been a widely-supported abstraction in par
allel programming [30]. Additionally, the architectures of many parallel
machines are either intrinsically shared-memory or support it using suit
able hardware. The main issues addressed in [22] are the enhancements
to be made to a simple shared-memory model such as the PRAM, and the
effectiveness of the resulting model in capturing the essential features of
parallel machines along the lines of the BSP and the LogP models.
The work reported in [22] builds on earlier results by the same authors
in [19] where a simple variant of the PRAM model is described in which the
read-write steps are required to be queuing; this model is called the QRQW
PRAM (Queue-Read Queue-Write PRAM). Prior to this work there were a