Table Of ContentScheduling and Automatic Parallelization
Alain Darte Yves Robert Frederic Vivien
Scheduling and
Automatic Parallelization
With 94 Figures
Springer Science+Business Media, LLC
Alain Darte Frederic Vivien
Yves Robert Laboratoire ICPS
Laboratoire LIP Universite Louis Pasteur
Ecole Normale Superieure F67400 Illkich
de Lyon France
69364 Lyon Cedex 07
France
Library of Congress Cataloging-in-Publication Data
Darte, Alain.
Scheduling and automatic parallelization / Alain Darte, Yves Robert, Frederic Vivien.
p. cm.
Includes bibliographical references.
ISBN 978-1-4612-7113-0 ISBN 978-1-4612-1362-8 (eBook)
DOI 10.1007/978-1-4612-1362-8
1. Compiling (Electronic computers) 2. Parallel processing (Electronic computers)
I. Robert, Yves, 1958- 11. Vivien, Frederic. Ill. Title.
QA76.76.C65 D37 2000
004'.35-dc21 99-046585
AMS Subject Classifications: 10.15.10, 10.10.20, 10.10
Printed on acid-free paper.
© 2000 Springer Science+Business Media New York
Originally published by Birkhauser Boston in 2000
Soficover reprint of the hardcover 18t edition 2000
All rights reserved. This work may not be translated or copied in whole or in part without the written
permission of the publisher Springer Science+Business Media, LLC,
except for brief excerpts in connection with reviews or scholarly analysis.
Use in connection with any form of information storage and retrieval, electronic adaptation, computer
software, or by similar or dissimilar methodology now known or hereafter developed is forbidden.
The use of general descriptive names, trade names, trademarks, etc., in this publication, even if the
former are not especially identified, is not to be taken as a sign that such names, as understood by the
Trade Marks and Merchandise Marks Act, may accordingly be used freely by anyone.
ISBN 978-1-4612-7113-0 SPIN 10733875
Typeset by the authors in TeX.
9 8 7 6 5 432 1
Contents
Preface ix
Introduction xi
PartI Unidimensional Problems 1
1 SchedulingDAGswithoutCommunications 3
1.1 Introduction . 3
1.2 WhereDoTaskGraphsComeFrom? 3
1.3 SchedulingDAGs 6
1.4 SolvingPb(00) . . . . . . . . . . . . 10
1.5 SolvingPb(P) . 14
1.5.1 NP-CompletenessofPb(p) 14
1.5.2 ListHeuristics . . . . . . . . 15
1.5.3 Implementinga ListSchedule. 18
1.5.4 Critical PathScheduling . 20
1.6 Conclusion........ 22
1.7 Bibliographical Notes. 22
1.8 Exercises......... 22
2 SchedulingDAGswith Communications 25
2.1 Introduction . 25
2.2 A Model withCommunicationCosts . . . . . 25
2.3 NP-CompletenessofPb(00) . . . . . 27
2.4 AGuaranteed Heuristic for Pb(00) . . 31
2.4.1 FavoriteSuccessors . 32
2.4.2 Hanenand Munier's Heuristic 32
v
vi Contents
2.5 ListHeuristics for Pb(p) . 35
2.5.1 NaiveCritical Path . 36
2.5.2 Modified Critical Path . . . . . . . 37
2.5.3 Hints for Comparison . . . . . . . . . . 38
2.6 Two-StepClustering Heuristics . . . . . . . . . . . . . . . . . 39
2.6.1 Heuristics for theClusteringPhase . . . . . . . . . . . 40
2.6.2 FromClustering toSchedulingwithResources. 49
2.6.3 Clustering Epilogue . . . . .. 51
2.7 LinearClustering . . . . ..... 51
2.8 Conclusion....... 57
2.9 Bibliographical Notes. 57
2.10 Exercises . . . . 57
3 Cyclic Scheduling 61
3.1 Introduction . 61
3.2 Problem Formulation . 62
3.2.1 AnExample .. 62
3.2.2 AverageCycleTime . . . . . 64
3.2.3 Playingwiththe Example . . . . . 65
3.2.4 Problem Formulation:Summary ... 66
3.2.5 LowerBoundsfor the AverageCycleTime 68
3.3 SolvingBCS(00) . . . . . . . . . . . . . . . . . . 69
3.3.1 SchedulingPotentialGraphs . 69
3.3.2 The Bellman-Ford Algorithm . 71
3.3.3 OptimalSchedulefor Unlimited Resources 74
3.4 SolvingBCS(p) . 79
3.4.1 NP-CompletenessofBCS(p) . 79
3.4.2 LoopCompaction. . . . . . . . . 81
3.4.3 LoopShifting . . . . . . . . . . . . . . . . . 82
3.4.4 The Leiserson-SaxeRetiming Algorithm. . . . . 85
3.4.5 Minimizing the NumberofConstraints inA(Gr) .•• 88
3.5 Bibliographical Notes .... 92
3.6 Exercises.............................. 92
PartII Multidimensional Problems 97
4 SystemsofUniform Recurrence Equations 99
4.1 Introduction . 99
4.2 ComputabilityofUniformRecurrence Equations . 100
4.2.1 Definitionofa SURE . . . . . . . . . . . . . 100
4.2.2 Computability: Definitionand Properties . 101
Contents Vll
4.3 UREand LinearScheduling . . . . . . . . . . . . . . . . 107
4.3.1 Introduction..................... 107
4.3.2 ConstructionofDependence Paths . . . . . . . . 109
4.3.3 ComputabilityCriterionfor a Single Equation . 113
4.3.4 Optimal LinearSchedules. . . . . . . . . . . . 115
4.3.5 Conclusion 123
4.4 SUREand MultidimensionalScheduling. . . . . . . . 124
4.4.1 Introduction............... 124
4.4.2 DetectingZero-WeightCycles. . . . . . . . . . 126
4.4.3 Constructionand PropertiesofG' .. . . . . . 134
4.4.4 Optimal MultidimensionalSchedulefor aSURE . 137
4.4.5 Conclusion 142
4.5 Bibliographical Notes. . . . . . . . . . . . . . . . . . . . . .. 143
4.6 Exercises . . . . . . . . . . . . . . . . . . . . . 144
4.6.1 Computability.............. 144
4.6.2 UREand LinearScheduling. . . . . . 144
4.6.3 SUREand MultidimensionalScheduling 145
5 Parallelism Detection in Nested Loops 149
5.1 Introduction....................... 149
5.2 DependenceAnalysis and DependenceAbstraction 152
5.2.1 Nested Loopsand DependenceRelations . . 152
5.2.2 DependenceAnalysis 157
5.2.3 ApproximationsofDistanceSets . . . 162
5.3 Allenand Kennedy's Algorithm .. 170
5.3.1 LoopDistribution. . . . . . . 171
5.3.2 BasicAlgorithm. . . . . . . . 172
5.4 UnimodularTransformations . . . . 176
5.4.1 Definitionand Validity. . . . . . . . . 177
5.4.2 The HyperplaneMethod (Uniform LoopNests) 180
5.4.3 The HyperplaneMethod (DirectionVectors) . . 183
5.4.4 Wolfand Lam'sAlgorithm . . . . . . . 185
5.5 Darteand Vivien's Algorithm . . . . . . . . . . . .. 193
5.5.1 Motivation........... . . . . . . . .. 193
5.5.2 Uniformization...... . . . .. 196
5.5.3 Computabilityofa PRDG . . . . . . . . . . .. 201
5.5.4 Schedulinga PRDG . . . .. 206
5.5.5 ExtensiontoMedium-GrainParallelization . 214
5.5.6 LinkwithALLEN-KENNEDY and WOLF-LAM 215
5.6 Feautrier's Algorithm. . . . . . . . . . . . . . . . . . . 216
5.6.1 MonodimensionalAlgorithm. . . . . . . . . 217
5.6.2 Extensionto MultidimensionalScheduling . . 222
viii Contents
5.6.3 ExtensiontoOtherDependenceRepresentations . 223
5.6.4 Comparisonwith DARTE-VIVIEN . 224
5.7 Optimality . 226
5.7.1 TheDifficulty ofDefiningan "Optimality" Notion .. 226
5.7.2 A Formal Definition . . . . . . . . . 231
5.7.3 Optimality ofALLEN-KENNEDY . 235
5.7.4 Optimality ofWOLF-LAM. . . . 239
5.7.5 OptimalityofDARTE-VIVIEN 239
5.7.6 SuboptimalityofFEAUTRIER 242
5.7.7 Conclusion .. 243
5.8 Bibliographical Notes. 243
5.9 Exercises . 244
Bibliography 249
Index 260
Preface
Readership
This book is devoted to the study of compiler transformations that are
needed toexpose the parallelism hiddenina program. This bookis notan
introductory book to parallel processing, nor is it an introductory book to
parallelizingcompilers.Weassumethatreadersare familiar withthebooks
High PerformanceCompilers for Parallel ComputingbyWolfe [121] and Super
compilers for Paralleland VectorComputers byZimaandChapman[125],and
thattheywant to know moreaboutscheduling transformations.
In this book we describe both task graph scheduling and loop nest
scheduling.Taskgraphschedulingaimsatexecutingtaskslinkedbyprece
denceconstraints; itis a run-time activity. Loop nestschedulingaims atex
ecutingstatementinstanceslinked bydatadependences;itisacompile-time
activity. Weare mostlyinterested inloop nestscheduling,butwealsodeal
with task graph scheduling for two main reasons: (i) Beautiful algorithms
and heuristics havebeenreported intheliteraturerecently;and (ii) Several
techniques used intaskgraphscheduling, likelistscheduling,are thebasis
ofthe loop transformations implemented inloop nestscheduling.
As for loop nest scheduling our goal is to capture in a single place
the fantastic developments of the last decade or so. Dozens of loop trans
formations have been introduced (loop interchange, skewing, fusion, dis
tribution, etc.) before a unifying theory emerged. The theory builds upon
the pioneering papers of Karp, Miller, and Winograd [65] and of Lam
port [75], and it relies on sophisticated mathematical tools (unimodular
transformations, parametric integer linear programming, Hermite decom
position,Smithdecomposition,etc.).
Thisbookisintendedforgraduateorpostgraduatestudentswhowant
to understand the foundations of automatic parallelization techniques. It
ix
x Preface
will be very useful to researchers interested inscheduling, compilers, and
programtransformations.
Thebookisself-contained,inthatall proofsareincluded.Readerswill
need some basic mathematical skills and some familiarity with standard
graphand linearprogrammingalgorithms,allofwhichcanbe found inIn
troduction toAlgorithmsbyCormen,Leiserson,and Rivest[26]and inTheory
ofLinearandInteger ProgrammingbySchrijver [110].
Acknowledgments
This book grew out of several lectures at the DEA Informatique de Lyon,
theDEAAlgorithmiquedeParis,andCS594oftheUniversityofTennessee
at Knoxville. Several people have deeply influenced ourview ofthe field.
Even though the following list cannot be exhaustive, we would like to
thank:
• our colleagues Paul Feautrier, Jose Fortes, Fran~ois Irigoin, Cather
ineMongenet,PatriceQuinton,Guy-RenePerrin,SanjayRajopadhye,
and RobSchreiber.
• our former and current PhD students Pierre Boulet, Pierre-Yves Cal
land, MicheleDion,CyrilRandriamaro, FabriceRastello,TanguyRis
set,and Georges-AndreSilber.
Lyon,France AlainDarte
May1999 YvesRobert
FredericVivien
Introduction
Motivation
Scheduling theory originated from operation research. In a computer sci
enceframework,schedulingtasks(orsubroutines,processes,orthreads) to
minimize someoptimizationcriteria (suchas the total execution time) is a
fundamental problem. See the classic book Computer and Job-Shop Schedu
ling Theory byCoffman [24], the recent book Task Scheduling in Parallel and
Distributed Systems by EI-Rewini, Lewis and Ali [37], and the researchori
ented survey Scheduling Theory and Its Applications, edited by Chretienne,
Coffman, Lenstra,and Liu [22].
Tounderstandwhata tasksystemis,considerthefollowing toyexam
ple:
DOi=l, n
TaskTi,i: x(i) =b(i) / a(i,i)
DOj =i+l, n
TaskTi,j: b(j) =b(j) -a(j,i) *x(i)
ENDDO
ENDDO
This example will be processed in full detail in Chapter 1. Here we only
sketchthe techniques thatwillbeexplained throughoutthebook. Building
the task graph is not very difficult here. TaskTi,i, which computes Xi and
writes itin memory, must precede taskTi,j, which reads this value, for all
j > i. Also, theupdatingatstepi ofb(j) mustprecedetheupdatingatstep
i +1, which means that taskTi,j precedes taskTi+l,j for j ~ i +1. Ifsuch
precedence constraints are notenforced, we cannot guarantee that the se
mantics of the original program are preserved. We are led to the graphof
Figure 1. Nodesare tasks, and edges represent precedenceconstraints. We
xi
