Parallel Programming with Co-Arrays

220 Pages·2019·19.66 MB·English

by  Robert W. Numrich

Parallel Programming
with Co-arrays

Parallel Programming
with Co-arrays
Robert W. Numrich
CRC Press
Taylor & Francis Group
6000 Broken Sound Parkway NW, Suite 300
Boca Raton, FL 33487-2742
© 2019 by Taylor & Francis Group, LLC
CRC Press is an imprint of Taylor & Francis Group, an Informa business
No claim to original U.S. Government works
Printed on acid-free paper
Version Date: 20180716
International Standard Book Number-13: 978-1-4398-4004-7 (Hardback)
International Standard Book Number-13: 978-0-4297-9327-1 (Paperback)
This book contains information obtained from authentic and highly regarded sources. Reasonable
efforts have been made to publish reliable data and information, but the author and publisher cannot
assume responsibility for the validity of all materials or the consequences of their use. The authors and
publishers have attempted to trace the copyright holders of all material reproduced in this publication
and apologize to copyright holders if permission to publish in this form has not been obtained. If any
copyright material has not been acknowledged please write and let us know so we may rectify in any
future reprint.
Except as permitted under U.S. Copyright Law, no part of this book may be reprinted, reproduced,
transmitted, or utilized in any form by any electronic, mechanical, or other means, now known or
hereafter invented, including photocopying, microfilming, and recording, or in any information
storage or retrieval system, without written permission from the publishers.
For permission to photocopy or use material electronically from this work, please access
www.copyright.com (http://www.copyright.com/) or contact the Copyright Clearance Center, Inc.
(CCC), 222 Rosewood Drive, Danvers, MA 01923, 978-750-8400. CCC is a not-for-profit organization
that provides licenses and registration for a variety of users. For organizations that have been granted
a photocopy license by the CCC, a separate system of payment has been arranged.
Trademark Notice: Product or corporate names may be trademarks or registered trademarks, and
are used only for identification and explanation without intent to infringe.
Library of Congress Cataloging-in-Publication Data
Names: Numrich, Robert W., author.
Title: Parallel programming with co-arrays / Robert W. Numrich.
Description: First edition. | Boca Raton, FL : CRC Press/Taylor & Francis
Group, 2018. | Series: Chapman & Hall/CRC computational science ; 33 |
Includes bibliographical references and index.
Identifiers: LCCN 2018024855 | ISBN 9781439840047 (hardback : acid-free paper)
Subjects: LCSH: Parallel processing (Electronic computers)
Classification: LCC QA76.642 .N86 2018 | DDC 004/.35--dc23
LC record available at https://lccn.loc.gov/2018024855
Visit the Taylor & Francis Web site at
http://www.taylorandfrancis.com
and the CRC Press Web site at
http://www.crcpress.com
Contents
Preface ix
1 Prologue 1
2 The Co-array Programming Model 3
2.1 A co-array program . . . . . . . . . . . . . . . . . . . . . . . 3
2.2 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7
3 Partition Operators 9
3.1 Uniform partitions . . . . . . . . . . . . . . . . . . . . . . . . 9
3.2 Non-uniform partitions . . . . . . . . . . . . . . . . . . . . . 12
3.3 Row-partitioned matrix-vector multiplication . . . . . . . . . 14
3.4 Input/output in the co-array model . . . . . . . . . . . . . . 16
3.5 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17
4 Reverse Partition Operators 19
4.1 The partition of unity . . . . . . . . . . . . . . . . . . . . . . 19
4.2 Column-partitioned matrix-vector multiplication . . . . . . . 21
4.3 The dot-product operation . . . . . . . . . . . . . . . . . . . 23
4.4 Extended definition of partition operators . . . . . . . . . . . 24
4.5 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25
5 Collective Operations 27
5.1 Reduction to root . . . . . . . . . . . . . . . . . . . . . . . . 27
5.2 Broadcast from root . . . . . . . . . . . . . . . . . . . . . . . 30
5.3 The sum-to-all operation . . . . . . . . . . . . . . . . . . . . 31
5.4 The max-to-all and min-to-all operations . . . . . . . . . . . 32
5.5 Vector norms . . . . . . . . . . . . . . . . . . . . . . . . . . . 33
5.6 Collectives with array arguments . . . . . . . . . . . . . . . . 33
5.7 The scatter and gather operations . . . . . . . . . . . . . . . 34
5.8 A cautionary note about functions with side effects . . . . . 37
5.9 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37
6 Performance Modeling 39
6.1 Execution time for the sum-to-all operation . . . . . . . . . . 40
6.2 Execution time for the dot-product operation . . . . . . . . . 41
6.3 Speedup and efficiency . . . . . . . . . . . . . . . . . . . . . 43
v
vi Contents
6.4 Strong scaling under a fixed-size constraint . . . . . . . . . . 43
6.5 Weak scaling under a fixed-time constraint . . . . . . . . . . 47
6.6 Weak scaling under a fixed-work constraint . . . . . . . . . . 49
6.7 Weak scaling under a fixed-efficiency constraint . . . . . . . 50
6.8 Some remarks on computer performance modeling . . . . . . 52
6.9 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 54
7 Partitioned Matrix Classes 57
7.1 The abstract matrix class . . . . . . . . . . . . . . . . . . . . 57
7.2 Sparse matrix classes . . . . . . . . . . . . . . . . . . . . . . 59
7.3 The compressed-sparse-row matrix class . . . . . . . . . . . . 61
7.4 Matrix-vector multiplication for a CSR matrix . . . . . . . . 64
7.5 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 66
8 Iterative Solvers for Sparse Matrices 67
8.1 The conjugate gradient algorithm . . . . . . . . . . . . . . . 67
8.2 Other Krylov solvers . . . . . . . . . . . . . . . . . . . . . . 70
8.3 Performance analysis for the conjugate gradient algorithm . 71
8.4 Strong scaling . . . . . . . . . . . . . . . . . . . . . . . . . . 74
8.5 Weak scaling . . . . . . . . . . . . . . . . . . . . . . . . . . . 76
8.6 Iso-efficiency . . . . . . . . . . . . . . . . . . . . . . . . . . . 78
8.7 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 78
9 Blocked Matrices 79
9.1 Partitioned dense matrices . . . . . . . . . . . . . . . . . . . 80
9.2 An abstract class for dense matrices . . . . . . . . . . . . . . 80
9.3 The dense matrix class . . . . . . . . . . . . . . . . . . . . . 81
9.4 Matrix-matrix multiplication . . . . . . . . . . . . . . . . . . 84
9.5 LU decomposition . . . . . . . . . . . . . . . . . . . . . . . . 87
9.6 Partial pivoting . . . . . . . . . . . . . . . . . . . . . . . . . 92
9.7 Solving triangular systems of equations . . . . . . . . . . . . 94
9.8 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 98
10 The Matrix Transpose Operation 99
10.1 The transpose operation . . . . . . . . . . . . . . . . . . . . 99
10.2 A row-partitioned matrix transposed to a row-partitioned ma-
trix . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 102
10.3 The Fast Fourier Transform . . . . . . . . . . . . . . . . . . 105
10.4 Performance analysis . . . . . . . . . . . . . . . . . . . . . . 106
10.5 Strong scaling . . . . . . . . . . . . . . . . . . . . . . . . . . 108
10.6 Weak scaling . . . . . . . . . . . . . . . . . . . . . . . . . . . 109
10.7 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 110
Contents vii
11 The Halo Exchange Operation 111
11.1 Finite difference methods . . . . . . . . . . . . . . . . . . . . 111
11.2 Partitioned finite difference methods . . . . . . . . . . . . . . 114
11.3 The halo-exchange subroutine . . . . . . . . . . . . . . . . . 116
11.4 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 118
12 Subpartition Operators 119
12.1 Subpartition operators . . . . . . . . . . . . . . . . . . . . . 119
12.2 Assigning blocks to images . . . . . . . . . . . . . . . . . . . 121
12.3 Combined effect of the two partition operations . . . . . . . 122
12.4 Permuted distributions . . . . . . . . . . . . . . . . . . . . . 122
12.5 The cyclic distribution . . . . . . . . . . . . . . . . . . . . . 124
12.6 Load balancing . . . . . . . . . . . . . . . . . . . . . . . . . . 125
12.7 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 126
13 Blocked Linear Algebra 127
13.1 Blocked matrices . . . . . . . . . . . . . . . . . . . . . . . . . 127
13.2 The block matrix class . . . . . . . . . . . . . . . . . . . . . 130
13.3 Optimization of the LU-decomposition algorithm . . . . . . . 136
13.4 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 138
14 The Finite Element Method 141
14.1 Basic ideas from finite element analysis . . . . . . . . . . . . 142
14.2 Nodes, elements and basis functions . . . . . . . . . . . . . . 143
14.3 Mesh partition operators . . . . . . . . . . . . . . . . . . . . 147
14.4 The mesh class . . . . . . . . . . . . . . . . . . . . . . . . . . 150
14.5 Integrating the heat equation . . . . . . . . . . . . . . . . . . 154
14.6 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 156
15 Graph Algorithms 159
15.1 Graphs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 159
15.2 The breadth-first search . . . . . . . . . . . . . . . . . . . . . 161
15.3 The graph class . . . . . . . . . . . . . . . . . . . . . . . . . 164
15.4 A parallel breadth-first-search algorithm . . . . . . . . . . . 165
15.5 The Graph 500 benchmark . . . . . . . . . . . . . . . . . . . 167
15.6 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 169
16 Epilogue 171
A A Brief Reference Manual for the Co-array Model 173
A.1 The image index . . . . . . . . . . . . . . . . . . . . . . . . . 174
A.2 Co-arrays and co-dimensions . . . . . . . . . . . . . . . . . . 175
A.3 Relative co-dimension indices . . . . . . . . . . . . . . . . . . 177
A.4 Co-array variables with multiple co-dimensions . . . . . . . . 178
A.5 Co-array variables of derived type . . . . . . . . . . . . . . . 179
A.6 Allocatable co-array variables . . . . . . . . . . . . . . . . . 182
viii Contents
A.7 Pointers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 182
A.8 Procedure interfaces . . . . . . . . . . . . . . . . . . . . . . . 183
A.9 Execution control . . . . . . . . . . . . . . . . . . . . . . . . 183
A.10Full barriers . . . . . . . . . . . . . . . . . . . . . . . . . . . 184
A.11Partial barriers . . . . . . . . . . . . . . . . . . . . . . . . . . 185
A.12Critical segments and locks . . . . . . . . . . . . . . . . . . . 187
A.13Input/output . . . . . . . . . . . . . . . . . . . . . . . . . . . 189
A.14Command line arguments . . . . . . . . . . . . . . . . . . . . 190
A.15Program termination . . . . . . . . . . . . . . . . . . . . . . 191
A.16Inquiry functions . . . . . . . . . . . . . . . . . . . . . . . . . 191
A.17Image index functions . . . . . . . . . . . . . . . . . . . . . . 192
A.18Execution control statements . . . . . . . . . . . . . . . . . . 192
Bibliography 197
Index 207
Preface
This book describes basic parallel algorithms encountered in scientific and
engineering applications. It implements them in modern Fortran using the
co-array programming model and it analyzes their performance.
Thebook’sintendedaudienceincludesgraduateandadvancedundergrad-
uate students who want an introduction to basic techniques. It also includes
experienced researchers who want to understand the relationships among
seemingly disparate techniques. The book assumes that the reader knows the
modern Fortran language, basic linear algebra, and basic techniques for solv-
ing partial differential equations.
The book contains many code examples. Each example has been tested,
butthereisnowarrantythatthecodeisfreeoferrors.Noristhereanywarrant
that a particular example represents the best implementation. Reading code
without writing code, however, is not enough. The only way to learn parallel
algorithms is to write code and to test it. Modifying the sample codes is a
good way to experiment with alternative implementations.
The entries in the Bibliography provide a guide to the development of
parallel programming techniques over the last few decades. It is not exhaus-
tive,anditdoesnotprovideanaccuratehistoricalrecordofhowtheyevolved
or who first formulated them. They simply provide a guide to more detailed
discussions of topics covered in the book.
ManythankstoJohnReidwhoseencouragementandsupporthelpedmake
this book possible. He worked through many of the technical details of the
co-array model and shepherded it through the rigorous standardization pro-
cess to make it part of the Fortran language. He read several versions of the
manuscript, and I greatly appreciate his suggestions for improvement.
Robert Numrich
Minneapolis, Minnesota
ix