Parallel algorithms and efficient implementation techniques for finite element approximations

152 Pages·2013·2.31 MB·French

by  Popescu, Radu

Parallel algorithms and efficient implementation techniques
for finite element approximations
THÈSE NO 5980 (2013)
PRÉSENTÉE LE 18 NOVEMBRE 2013
À LA FACULTÉ DES SCIENCES DE BASE
CHAIRE DE MODÉLISATION ET CALCUL SCIENTIFIQUE
PROGRAMME DOCTORAL EN MATHÉMATIQUES
ÉCOLE POLYTECHNIQUE FÉDÉRALE DE LAUSANNE
POUR L'OBTENTION DU GRADE DE DOCTEUR ÈS SCIENCES
PAR
Radu POPESCU
acceptée sur proposition du jury:
Prof. K. Hess Bellwald, présidente du jury
Prof. A. Quarteroni, Dr S. Deparis, directeurs de thèse
Prof. L. Formaggia, rapporteur
Prof. M. A. Heroux, rapporteur
Prof. M. Picasso, rapporteur
Suisse
2013

Putanicequotehere.Plentyoftimetodecide...

Support
ThisworkhasbeenfinanciallysupportedbytheSwissHighPerformanceandHighProductivity
Computing(HP2C)1initiativeandbytheChairofModellingandScientificComputing(CMCS)
atÉcolePolytechniqueFédéraledeLausanne(EPFL)2.
IwouldliketoacknowledgetheCenterforAdvancedModellingScience(CADMOS)3andthe
SwissCenterforScientificComputing(CSCS)4forprovidingtheinvaluablecomputational
resourcesneededfortheworkdescribedinthisthesisandforthemanyhoursofhelpand
supportprovidedbytheirstaff.
FinallyIwouldliketoacknowledgeourcollaboratorsMichaelHeroux,SivaRajamanickam
andErikBomanfromtheTrilinosProject5 atSandiaNationalLaboratoriesinNewMexico,
USA.Iamtrulygratefulforourcollaborationandforthehospitalitytheyextendedduringmy
internshipatSandiaNationalLabs.
Lausanne,October22,2013 R.P.
1HP2C-http://www.hp2c.ch
2CMCS-EPFL-http://cmcs.epfl.ch
3CADMOS-http://www.cadmos.org
4CSCS-http://www.cscs.ch
5TrilinosProject-http://trilinos.sandia.gov
v

Acknowledgements
IhavereachedtheendofthistripandverysoonIwillstartanewone.It’stimeformetolook
backatthelastfouryearsofmylifeandtrytounderstandwhichwerethekeypointsand
peoplewhohelpedmeonmyway.
ComingtoSwitzerlandtoworkinthegrouphereatEPFLwasdefinitelyalifechangingdecision
forme.It’sthebestdecisionI’veevermadeanditwouldn’thavecometopasswithoutAlfio
Quarteroni,whoagreedtohiremeandwelcomedmeintohisveryclose-knitgroup.Although,
duetothetopicofmywork,weweren’treallyabletoworktogetherdirectly,knowingthat
he was there keeping an eye on me and directing me when needed has always been very
reassuringforme.
ThenthereisSimoneDeparis,myotheradvisor,withwhomI’vespentcountlessdays(accu-
mulated,notatonce)discussingandplanningmywork. Iamgratefulforhissupport,his
adviceandhisknowledgeandifIamtolookbackandtrytoidentifymyfondestmemorywith
him,itisdefinitelythemanyhourswepassedtogethersolvingproblemsrelatedtotheLifeV
softwareproject.Itwasreallyfunputtingthetoolstogetherandkeepingthemworking.I’m
retiringmySSHkeyssoon,IhopethenextonetodothisworkwillenjoyitasmuchasIdid.
IamveryluckytohavebeenpartoftheCMCSgroup:reallycompetentandfriendlypeople,
pastandpresent,arelaxingandwelcomingworkingenvironment,oneisluckytofindthisat
hisworkplace.
IwouldalsoliketoacknowledgeMikeHeroux.I’mreallyhappytohavestartedworkingwith
himandIthinkhe’sanexample:extremelycompetentandinsightful,butwithsuchacalm
andkinddemeanorthatmakesitreallyconfortabletoworkwithhim.Also,hehasreallycrazy
timemanagementskills!Idon’tknowhowheandAlfiocanbothkeepgoinglikethisbutit’s
reallyimpressive.
Ofcourse,Iwouldn’tcallthisperiodlifechangingifworkwastheonlygoodpartofit. I’ve
mademanygoodfriendshere.Paolo(T),Elena,Claudia,Gosia,Jacopo,Ale:Ithankyoufor
yourfriendshipandknowthatIreallycherishthetimewespenttogethersofar.Ithasreally
changedmeandyouallknowthisbest.Gilles,IlovedourdinnersinBucharestandtalkingto
youintheeveningonthecomputer,usuallybeforeyouneededtoputAnatobed...Iwould
alsoliketorememberthefriendswhoarenolongerwithusnow... PaoloCrosetto(he’sin
Germany),Ricardo(he’satUNIL).Thenthere’sKellyandDanny(theyareinAustria).Kelly,I
wanttothankyoulast:beingfriendswithyouhasdefinitelybroughtmanycrazyandweird
experiencestomylife,buttheyaresomeofthebrightestmemoriesfromtheseyears.Know
thatIwillnotbeabletolookatMorgesthesamewayasbefore.
vii
Acknowledgements
Thenthereisanunexpectedsourceofdistraction(thegoodkind),supportandaffectionthat
hasmadeitswayintomylifeandmadeitbetter.Helene,youmadewritingthisthesissomuch
easier.Youkeepmewarmandlevel-headedandI’vegrowntorelyonyou.
Last,butnotleast,thereismyfamily:myparents,OmamaandmysistersAncaandLidiawith
theirownfamilies. Ithankyouallformakingme(whereapplicable),raisingme,lovingme
andsupportingmealways.Iamtrulyluckytohavesuchawonderfulfamily.
AndthisiswhereI’dalsothankmycat,ifIhadone... HypotheticalMittens,atthispointI
wouldhavebeenverygratefulforwhatIassumehadbeenyourenduringaffectionthroughout
allthis.
Lausanne,October22,2013 R.P.
viii
Abstract
In this thesis we study the efficient implementation of the finite element method for the
numericalsolutionofpartialdifferentialequations(PDE)onmodernparallelcomputerarchi-
tectures,suchasCrayandIBMsupercomputers.Thedomain-decomposition(DD)method
representsthebasisofparallelfiniteelementsoftwareandisgenerallyimplementedsuch
thatthenumberofsubdomainsisequaltothenumberofMPIprocesses.Weareinterested
inbreakingthisparadigmbyintroducingasecondlevelofparallelism. Eachsubdomainis
assignedtomorethanoneprocessorandeitherMPIprocessesormultiplethreadsareused
toimplementtheparallelismonthesecondlevel. Thethesisisdevotedtothestudyofthis
secondlevelofparallelismandincludesthestagesdescribedbelow.
ThealgebraicadditiveSchwarz(AAS)domain-decompositionpreconditionerisanintegral
partofthesolutionprocess.Weseektounderstanditsperformanceontheparallelcomputers
whichwetargetandweintroduceanimprovedconstructionapproachfortheparallelprecon-
ditioner.WeexamineanovelstrategyforsolvingtheAASsubdomainproblems,usingmultiple
MPI processes. At the subdomain level, this is represented by the ShyLU preconditioner.
Webringimprovementstoitsalgorithmintheformofanovelinexactsolverbasedonan
incompleteQR(IQR)factorization.Theperformanceofthenewpreconditionerframeworkis
studiedforLaplacianandadvection-diffusion-reaction(ADR)problemsandforNavier-Stokes
problems,asacomponentwithinalargerframeworkofspecializedpreconditioners.
Thepartitioningofthecomputationalmeshcomeswithconsiderablememorylimitations,
whendoneatruntimeonparallelcomputers,duetothelowamountofavailablememoryper
processor. Wedescribeandimplementasolutiontothisproblem,basedonoffloadingthe
partitioningprocesstoapreliminaryofflinestageofthesimulationprocess.Wealsopresent
theefficientimplementation,basedonparallelMPIcollectiveinstructions,oftheroutines
whichloadthemeshpartsduringthesimulation.
Wediscussanalternativeparallelimplementationofthefiniteelementsystemassemblybased
onmulti-threading. Thisnewapproachisusedtosupplementtheexistingonebasedon
MPIparallelism,insituationswhereMPIalonecannotmakeuseofalltheavailableparallel
hardwareresources.
Theworkpresentedinthethesishasbeendoneintheframeworkoftwosoftwareprojects:the
TrilinosprojectandtheLifeVparallelfiniteelementmodelinglibrary. Allthenewdevelop-
mentshavebeencontributedbacktotherespectiveprojects,tobeusedfreelyinsubsequent
publicreleasesofthesoftware.
Keywords:finiteelementmethod,parallelpreconditioners,MPI,multi-threading
ix