Array-based Hierarchical Mesh Generation in Parallel

13 Pages·2015·2.46 MB·English

by  Navamita Ray

Available online at www.sciencedirect.com
ScienceDirect
Procedia Engineering 124 ( 2015 ) 291 – 303
24thInternationalMeshingRoundtable(IMR24)
Array-Based Hierarchical Mesh Generation in Parallel
NavamitaRaya∗,IulianGrindeanua,XinglinZhaob,VijayMahadevana,XiangminJiaob
aMathematicsandComputerScience,ArgonneNationalLaboratory,Argonne,IL60439,USA
bDepartmentofAppliedMathematicsandStatistics,StonyBrookUniversity,StonyBrook,NY11794,USA.
Abstract
In this paper, we describe an array-based hierarchical mesh generation capability through uniform refinement of unstructured
meshesforefficientsolutionofPDE’susingfiniteelementmethodsandmultigridsolvers.Amulti-degree,multi-dimensionaland
multi-levelframeworkisdesignedtogeneratethenestedhierarchiesfromaninitialmeshthatcanbeusedforanumberofpurposes
suchasmulti-levelmethodstogeneratinglargemeshes. Thecapabilityisdevelopedundertheparallelmeshframework“Mesh
OrienteddAtaBase”a.k.aMOAB[16].Wedescribetheunderlyingdatastructuresandalgorithmstogeneratesuchhierarchiesand
presentnumericalresultsforcomputationalefficiencyandmeshquality.Wealsopresentresultstodemonstratetheapplicabilityof
thedevelopedcapabilitytoamultigridfinite-elementsolver.
(cid:2)©c 22001155 PTubhleisAheudth boyr sE.lsPeuvbielirs Lhetdd. bTyhiEs lisse avni eorpLentd a.ccess article under the CC BY-NC-ND license
P(hetetpr-:r//ecvrieeawtivuencodmermreosnpso.onrsgi/bliicleitnyseosf/boyrg-nacn-inzdin/4g.0c/o).mmitteeofthe24thInternationalMeshingRoundtable(IMR24).
Peer-review under responsibility of organizing committee of the 24th International Meshing Roundtable (IMR24)
Keywords: uniformmeshrefinement,hierarchicalmeshes,multigridmethods,parallelcomputation,half-facet
1. Introduction
In the numerical solution of complex partial differential equations (PDE’s) using finite element methods for un-
structured meshes, the two most computationally intensive steps are mesh generation and linear solvers. An initial
coarsemeshrepresentingthecomputationaldomainmightnotbeofsufficientresolutiontogetmeaningfulresultsout
ofthediscretizationsforphysicalscalesthatmightbepresent.Asaresult,thecapabilitytorefineameshisanessen-
tialpartofanysimulationprocess. Ontheotherhand, itiswellknownthatmulti-levelmethodssuchasgeometric
multigridmethods(GMG)cantheoreticallydeliveroptimaltimecomplexityforsolvingsparselinearsystemsfrom
PDE discretizations. Thus, it would be advantageous to use nested multi-level i.e., hierarchical meshes to achieve
accuracyandcomputationalefficiency,especiallyinthecontextoflarge-scaleparallelcomputing,asboththenumber
of processors and the mesh resolution increase. Uniform mesh refinement (UMR) provides a simple and efficient
waytogeneratesuchhierarchiesviasuccessiverefinementofthemeshatapreviouslevel. Italsoprovidesanatural
hierarchy via parent and child type of relationship between entities of meshes at different levels. While UMR is a
relativelysimpleprocess,itisbynomeanstrivial. Notableissuesincludemeshquality,multi-levelandmulti-degree
refinement,anddatastructureandsoftwaredesign.
∗ Correspondingauthor.Email:[email protected].
1877-7058 © 2015 Published by Elsevier Ltd. This is an open access article under the CC BY-NC-ND license
(http://creativecommons.org/licenses/by-nc-nd/4.0/).
Peer-review under responsibility of organizing committee of the 24th International Meshing Roundtable (IMR24)
doi: 10.1016/j.proeng.2015.10.140
292 Navamita Ray et al. / Procedia Engineering 124 ( 2015 ) 291 – 303
Inthispaper,wedevelopparalleluniformrefinement-basedalgorithmstogeneratemulti-degree,multi-dimensional
andmulti-levelmeshesfromcoarseunstructuredmeshes. Thegeneratedmeshhierarchiescanbeusedforavariety
of purposes such as convergence studies, multilevel methods, generating large meshes in parallel to overcome IO
bottlenecks,etc. Whilethemulti-degreerefinementallowsachievinggreaterresolutionfaster,themulti-dimensional
refinement preserves the hierarchy over explicit lower dimensional entities such as curves in surfaces, or surfaces
embeddedinvolumes.
The key contributions of the paper include: 1) developing a template-based refinement strategy for subdividing
eachentityintosmallerentitiestosupportmulti-degreerefinementpatterns, 2)extendingthearray-basedhalf-facet
(AHF) data structure [3] to support hierarchy generation and efficient mesh traversals, and 3) developing efficient
parallelcommunicationstrategiestoresolvesharedentitiesalongprocessorboundariesafterrefinement. Wedevelop
the capability under the parallel array-based mesh framework “Mesh Oriented dAtaBase” a.k.a MOAB[16]. The
developedmeshhierarchygenerationsupports1D(edges),2D(triangles,quadrilaterals),3D(tetrahedral,hexahedral)
meshes and mixed-dimensional meshes. We present results to demonstrate the computational efficiency, memory
requirementsandeffectonmeshqualityduetotemplate-basedUMR.Wealsodemonstratetheeffectivenessofthe
hierarchicalmeshesthroughamultigridapplication.
Theremainderofthepaperisorganizedasfollows. Section2reviewssomebackgroundknowledgeandrelated
meshdatastructures. Section3describestherefinementtemplates,underlyingmeshhierarchystorageandextended
half-facet data structures. Section 4 describes the algorithms for the parallel communication and mesh hierarchy
generation. Section5presentsnumericalresultsforcomputationalefficiency,memoryrequirements,effectonmesh
qualityandanexamplemultigridapplication.Section6concludesthepaperwithadiscussion.
2. Background
Meshdatastructuresarefundamentaltomeshingalgorithmsandmesh-basednumericalmethods. Theunderlying
datastructurestronglyinfluencestheoverallperformanceofthealgorithmsorsimulationssinceitisusedtoperform
allthemesh-basedcombinatorialoperationsandasaresulthavebeeninvestigatedsincetheinceptionofmeshgen-
eration and computational geometry. We review some terminology before describing our data structures and mesh
frameworks. Wesayameshisamanifoldornon-manifoldifitsgeometricrealizationisamanifoldornon-manifold,
respectively. In a d-dimensional mesh, we refer to the d-dimensionalentities as elements, and refer to the (d−1)-
dimensionalsub-entitiesasitsfacets.Eachfacetina2-Dor3-Dmeshhasanorientationwithrespecttothecontaining
element.Forexample,eachedgeofatrianglehasadirection,andalltheedgesformanorientedloop.Thus,itmakes
sensetocallthefacetshalf-facets.Eachfacetmayhavemultipleincidentelements,especiallyfornon-manifoldenti-
ties. Werefertoallsuchhalf-facetsassiblinghalf-facets. Ameshissaidtobeconformalifthepairwiseintersection
ofanytwoentitiesiseitheranotherentity(lower-dimensional)orisempty.Inthispaper,weconsideronlyconformal
meshes,whichmaybemanifoldornon-manifold. Insomeengineeringapplications,especiallyincoupledormulti-
componentsystems,thedomainofinterestmaybecomposedofaunionoftopologically1-D,2-D,and3-Dobjects,
suchasamixtureofcables,thin-shells,andsolids.Werefertosuchadomainanditsmeshasmixed-dimensional.We
refertoasubsetofthemeshcorrespondingtoa1-D,2-D,and3-Dobjectinthedomainasasub-mesh.
2.1. Array-BasedHalf-Facet(AHF)DataStructure
Thereare a number of mesh data structures such as entity-based, boundary representations, corner table, radial-
edge,winged,half-edge/face,incidencegraphs,etc.,thatareusedformeshrepresentationandqueries.Thetwodata
structuresthatarerelevantinourcontextarethehalf-edgeandhalf-facedatastructures. Thehalf-edgedatastructure
is for 2D and surface meshes. It uses edge as the core object where the edge within each face is called a directed
or half-edge. Typical implementations (e.g., CGAL [4,9], OpenMesh [2] and Surface_Mesh [15]) store mappings
fromeachhalf-edgetoitsoppositehalf-edge,itspreviousandnexthalf-edgewithinitsface,itsvertices,itsincident
face,aswellasthemappingfromeachvertexandeachfacetoanincidenthalf-edge. Morecompactrepresentations,
suchas[1], canbeobtainedbystoringonlythemappingbetweenoppositehalf-edge, optionallythemappingfrom
eachvertextoanincidenthalf-edge,alongwiththeelementconnectivity. Thehalf-edgeconceptwasgeneralizedto
half-faces(e.g.,[1]and[11])forvolumemesheswherehalf-facesrefertotheorientedfaceswithinacell.Thesebasic
Navamita Ray et al. / Procedia Engineering 124 ( 2015 ) 291 – 303 293
(cid:5)
(cid:4) element connectivity
(cid:4)
1 1 2 3
(cid:1) (cid:2) (cid:3) 2 2 4 3
(cid:1)
(cid:3) (cid:2) 3 2 5 3 sibhf sibhe
4 2 3 6 Cells Exp-Faces
(cid:6) v2hf
v2he
vertextohalf-edges faces
siblinghalf-edges vertices v2he cells
element sibhes 1 (cid:3)1,0(cid:4)
edges
1 (cid:3)0,0(cid:4) (cid:3)2,2(cid:4) (cid:3)0,0(cid:4) 2 (cid:3)3,0(cid:4) Exp-Edges Vertices
v2hv
2 (cid:3)0,0(cid:4) (cid:3)0,0(cid:4) (cid:3)3,2(cid:4) 3 (cid:3)4,1(cid:4) sibhv
3 (cid:3)0,0(cid:4) (cid:3)0,0(cid:4) (cid:3)4,0(cid:4) 4 (cid:3)2,1(cid:4)
4 (cid:3)1,1(cid:4) (cid:3)0,0(cid:4) (cid:3)0,0(cid:4) 5 (cid:3)3,1(cid:4) (b)AHFmapsformixed-dimensionalmeshes.
6 (cid:3)4,2(cid:4)
(a)AHFmapsforanon-manifoldmesh.
Figure1:(a)AnexampleoftheAHFmapsstoredforanon-manifoldtriangularmesh.(b)TypicalAHFformixed-dimensionalmeshesiscomposed
ofhalf-vertex(black,forexplicitedgesonly),half-edge(blue,forexplicitfacesonly),andhalf-face(red)datastructures.
half-edge and half-face data structures are simple and are restricted to oriented, manifold meshes (with or without
boundary)in2-Dand3-D,respectively.
In[3],anefficient,compactandgeneralarray-basedhalf-facet(AHF)meshdatastructurewithsupportformixed-
dimensional meshes, which may be non-manifold and/or non-oriented was proposed. The core object of AHF is
half-facet as defined previously and is represented as an implicit entity. The concept of sibling half-facets unifies
the half-vertex, half-edge, and half-face data structures for 1-D, 2-D, and 3-D meshes, which may be manifold or
non-manifoldwithboundary.TheAHFdatastructureconsistsoftwokeymaps:
• siblinghalf-facets(sibhfs):Themappingbetweenthesiblinghalf-facetsusingacycliclinkedlist.
• vertextohalf-facet(v2hf): Themapofeachvertextoitsincidenthalf-facet. Thismapprovidesananchorfor
eachvertextoitslocalityinthemesh.
AnexampleoftheAHFmapsforanon-manifoldmeshisillustratedinFigure1a.Ford≥2,AHFprovidesacompact
representation,sincetheintermediatedimensionalentitiesarenotstoredbutreferencedimplicitly. Thisdatamodel
stores the above two maps for each dimension in a modular and self-contained manner and hence supports mixed-
dimensionalmeshes, whichmay be composed of sub-meshes of 1-D, 2-D, and 3-D. Figure 1bshows a diagram of
a typical half-facet data structure, where the half-vertices and half-edges are only required for explicit edges and
faces in the mesh, respectively. In addition, this data model can also be used for meshes with high-order elements
(suchassix-nodetrianglesor10-nodetetrahedra),wherethemid-edge,mid-faceormid-cellnodesdonotaffectthe
definitionandidentificationofthehalf-facets.WeusetheAHFastheunderlyingmeshdatastructurefordeveloping
therefinementalgorithms.
2.2. MeshOrienteddatABase(MOAB)
Tobeofanypracticalusetonumericalsimulationworkflows,themeshframeworkneedstosupportawiderange
offunctionalitiessuchasefficientlocalmeshtraversalsformatrixassemblies,boundaryextractionforboundarycon-
ditions, representation of mesh data, etc., in a parallel setting. Over the past years, a number of such frameworks
havebeendevelopede.g.,FMDB[13],MSTK[5],libMesh[10],etc. Insuchimplementations,theentitiesarerep-
resentedas“objects”explicitly,andpointers(orhandles)areusedtorefertotheseexplicitobjects. Inourwork,we
294 Navamita Ray et al. / Procedia Engineering 124 ( 2015 ) 291 – 303
choosetousearray-based,pointer-freeimplementationsforanumberofreasons.First,usingarrayscanleadtofaster
memoryaccess,fewercachemissesandhencebetterefficiency. Secondly,inanarray-basedimplementation,wecan
treatintermediatedimensionalentities(suchashalf-facets)asimplicitentities,andreferencethemwithoutforming
explicitobjects. Thiscanleadtosignificantsavingsinstorage,especiallyoncomputerswith64-bitpointers. Inad-
dition,array-basedimplementationsalsoofferbetterinteroperabilityacrossapplicationcodes,differentprogramming
languages,anddifferenthardwareplatforms(suchasbetweenGPUsandCPUs).
One such array-based parallel mesh framework is the Mesh Oriented datABase a.k.a MOAB [16] which is a
mesh data representation designed to support a range of mesh related operations, such as memory efficient mesh
representation, meshqueryingandrepresentationofapplicationspecificdata. Internallyitusesarray-basedstorage
forfine-graineddata,whichinmanycasesprovidesmoreefficientaccess,especiallyforlargeportionsofmeshand
associateddata.TheAHFdatastructurewasimplementedinMOABtosupportquery-intensivealgorithmsanditwas
foundtobeextremelyefficient(insomecasesovertwoorderofmagnitudeimprovement[3,12])overtheexisting
data structures. MOAB is also a parallel framework and supports spatial domain-decomposed view of a parallel
meshwhereeachsubdomainisassignedtoaprocessor,lower-dimensionalentitiesoninterfacesbetweensubdomains
are shared between processors, and ghost entities can be exchanged with neighboring processors. We develop our
hierarchicalmeshgenerationalgorithmsontopofMOAB’sparallelframework. Inthenextsection,wedescribethe
threekeycomponentsoftheproposedhierarchicalmeshgenerationalgorithm.
3. Multi-Degree,Multi-dimensionalandMulti-LevelHierarchicalMeshes
Theuniformrefinementbasedmeshhierarchygenerationhasthreekeydesigncomponents. First,entitytypeand
degree-specificrefinementtemplatesaredefinedthatareusedtosubdivideanentityintoitschildren. Thetemplates
are also used to update the underlying data structures for all the new children. The second key design component
istheuseofarray-basedhalf-facetdatastructuresforefficientmeshtraversalduringrefinement. Finally,thenewly
createdmeshesarestoredinlevel-wisecontiguousarraytoprovidememorycompactness.
3.1. Multi-degreeRefinementTemplates
Thestandardrefinementstrategydivideseachddimensionalentityto1 →2dsubentites.However,adesiredmesh
resolutioncanbereachedmuchfasterifahigherdegreeofrefinementisused. Thelengthofthemeshhierarchiesis
alsosmallinsuchcaseswhichmightbepreferablefromsomemultigridmethodssuchasGMGs.Anothermotivation
tousehighdegreerefinementpatternsisthatitallowsstraightforwardextensiontohigh-orderentitiesandthussupport
hp-refinements. By multi-degree refinement we mean a refinement pattern following vertex positions analogous to
high-order (degree p where p ≥ 2 ) Lagrangian elements. Thus, for a d dimensional entity, a degree p refinement
dividestheentityto1 → pd subentites. Wesupportprimenumberdegreesasanyhigherdegreerefinementscanbe
obtained by applying a sequence of such prime degree refinements. Table(1) lists the currently supported degrees
foreachdimension. Wenotethatinuniformrefinementthesamemodificationoperationofdividinganentityintoa
specificnumberofentitiesisappliedtoeachentityofthemesh.Thusdefiningstatictemplatesw.r.tareferenceentity
thatcontaincertainentitytypeanddegreespecificinformationcouldaidtherefinementprocess.Thetemplatemainly
storesthefollowinginformationforeachentitytypeandsupporteddegreeofrefinement:
• Numbering convention of new vertices: The new vertices are assigned local indices through which they are
uniquelyidentifiedwithinareferenceentity.
• Connectivity of new child entities: The connectivity of the new entities using the local indices of the new
vertices.Eachsuchchildisalsouniquelyidentifiablebyalocalidw.r.treference.
• Half-facet maps for new child entities: The local sibling half-facet (sibhfs) and vertex to incident half-facet
(v2hf)mapsarestored.
Figure2 shows the templates for triangle and quadrilateral reference entities for degree 2 and 3. The numbering
conventionofthereferenceentityfollowstheMOABcanonicalnumbering.Apartfromthesethreepiecesofdata,the
templatealsostoressomeotherauxiliaryinformationtoaidtrackingofnewverticesintroducedontheentityboundary
Navamita Ray et al. / Procedia Engineering 124 ( 2015 ) 291 – 303 295
(cid:3)
(cid:10)
(cid:5) (cid:8) (cid:7)
(cid:6) (cid:5) (cid:7) (cid:8) (cid:9)
(cid:9) (cid:6)
(cid:3) (cid:10)
(cid:2) (cid:4) (cid:2) (cid:3) (cid:4) (cid:5) (cid:6)
(cid:1) (cid:2) (cid:4) (cid:4) (cid:5)
(cid:11)(cid:12)(cid:13)(cid:12)(cid:14)(cid:12)(cid:15)(cid:16)(cid:12)(cid:17)(cid:18)(cid:15)(cid:19)(cid:20)(cid:19)(cid:21) (cid:22)(cid:12)(cid:23)(cid:14)(cid:12)(cid:12)(cid:17)(cid:3) (cid:22)(cid:12)(cid:23)(cid:14)(cid:12)(cid:12)(cid:17)(cid:4)
(cid:4) (cid:3) (cid:7) (cid:10) (cid:9)
(cid:8) (cid:9) (cid:10)
(cid:4) (cid:5) (cid:2)(cid:1) (cid:8)
(cid:2)(cid:6) (cid:2)(cid:5)
(cid:8) (cid:6) (cid:5) (cid:6) (cid:7)
(cid:9)
(cid:2)(cid:2) (cid:7)
(cid:2) (cid:3) (cid:2)(cid:3) (cid:2)(cid:4)
(cid:2) (cid:3) (cid:4)
(cid:1) (cid:2) (cid:5) (cid:5) (cid:6)
Figure2:Therefinementtemplatesdefinedfordegree2and3refinementsoverareferencetriangleandquadrilateralentity.Thelocalvertexorder
ofthereferenceentitiesfollowMOABconventions[17]. Thenewverticestobeintroducedduringrefinementareassignedlocalidswrttothe
reference.Similarly,thechildrenentitiesareassignedlocalorder.TheAHFmapsarecreatedfortherefinedentitiesintermsoftheirlocalids.
toavoidvertexduplicationduringrefinement. Anexampleofadegree3refinementtemplateforareferencetriangle
isillustratedintheTable(2).
Except for tetrahedron entity, the templates are static for all other entities (triangle, quads, hexes) in the sense
thattheydonotneedtorefertothephysicalentityinordertorefinetheparententity. Therefinementschemesfor
atetrahedroninvolvesdivisionintosmallercongruenttetrahedronsandoctahedronswhicharesubsequentlydivided
intofourtetrahedrons.Figure3showshowrefiningatetrahedronleadstocongruentsmallertetrahedraandoctahedra.
Therightsideofthefigureshowsthethreepossiblediagonalchoicesforeachsuchoctahedrontobedividedintofour
tetrahedrons. Todeliveragoodmeshquality, eachsuchoctahedraisdividedintosub-tetrahedrabyconnectingthe
shortestdiagonal[19].Thus,forhigher-degreerefinements,eachoctahedronhastofindtheshortestdiagonalusingthe
physicalentity. However,wenotethatthoughthesmallertetrahedra’sobtainedbytessellatinganoctahedronarenot
congruenttotheparenttetrahedron,alltheoctahedra’sarecongruenttoeachother. Therefore,theshortestdiagonal
isuniqueandhenceeachoctahedronwouldbedividedusingthesamepattern. Weusethisfacttodefinethreestatic
templatesforeachdegree. Duringrefinement,eachtetrahedronmakesachoiceoftheappropriatetemplatebasedon
thesmallestdiagonalofthephysicaltetrahedron. Currently,thetemplategenerationisnotautomatic. Effortisbeing
madetomakeitautomaticsothathigher-degreessuchas7,11,etc.canalsobesupported.
3.2. Multi-dimensionalMeshDataStructures
Duringrefinement,oneofthekeytasksistoavoidintroducingduplicateverticesforsharedboundariesbetween
entities. Thisrequiresfrequentcallstoadjacencyroutinestogetentitiesconnectedthroughentityboundaries. Since
AHFstoresthehalf-facetmapsbetweenentities,thesetypesofqueriesarethemostefficientonesitcansupport. As
aresult,itisnaturaltouseAHFastheunderlyingdatastructure. Aseachlevelofthemeshhierarchyisgeneratedby
refinementofthepreviouslevel,itisnecessarytoupdatetheappropriateAHFmapsforthenewlevelsothatthenew
meshcanbequeriedlater. TheprocessofupdatingtheAHFmapsforthenewlevelisaidedbothbytherefinement
templates (defined in the previous subsection) and also by the fact that the topology of the domain doesn’t change
duringrefinement. Thus,childrenofnon-manifoldentitiesremainsoforalllevelsandnonewnon-manifoldentities
areintroduced.TheAHFmapsareupdatedintwostages:
296 Navamita Ray et al. / Procedia Engineering 124 ( 2015 ) 291 – 303
(cid:6) (cid:6) (cid:6) (cid:6)
(cid:4) (cid:5)
(cid:7)(cid:8)(cid:9)(cid:10)(cid:11)(cid:12)(cid:9)(cid:13)(cid:14)(cid:1)(cid:15)(cid:6) (cid:2) (cid:3) (cid:3) (cid:5) (cid:4) (cid:2)
(cid:6) (cid:1) (cid:1) (cid:1) (cid:1)
(cid:5) (cid:6) (cid:6) (cid:5) (cid:4) (cid:4)
(cid:2) (cid:4) (cid:7)(cid:8)(cid:9)(cid:10)(cid:11)(cid:12)(cid:9)(cid:13)(cid:14)(cid:2)(cid:15)(cid:4) (cid:4) (cid:5) (cid:4) (cid:2) (cid:2) (cid:3)
(cid:3) (cid:2) (cid:3) (cid:2) (cid:1) (cid:1)
(cid:1) (cid:6) (cid:6) (cid:5) (cid:5) (cid:4)
(cid:2)
(cid:7)(cid:8)(cid:9)(cid:10)(cid:11)(cid:12)(cid:9)(cid:13)(cid:14)(cid:3)(cid:15)(cid:5) (cid:5) (cid:5) (cid:4) (cid:3) (cid:3)
(cid:2)
(cid:3) (cid:3) (cid:1) (cid:1)
Figure3: Left: Atetrahedronisdividedintocongruentsmallertetrahedronsandoctahedrons. Middle: Areferenceoctahedron. Right: Each
octahedroncanbefurthersubdividedintofourtetrahedronsinthreepossiblewaysdependingonwhichdiagonalischosen.
Table1:Thedegreesofrefinementcurrentlysupportedand Table2: Anexampleforthedegree3templateofatrianglereference
thecorrespondingnumberofchildren. entity. Childrenentityconnectivityandthetemplatev2hf and sibhfs
maps.
Dimension Degrees #Children
1 2,3,5 2,3,5 ChildEntityConnectivity sibhfs v v2hf
1 (cid:3)0,3,8(cid:4) (cid:3)0,0(cid:4) (cid:3)2,2(cid:4) (cid:3)0,0(cid:4) 0 (cid:3)1,0(cid:4)
2 2,3,5 4,9,25
2 (cid:3)3,9,8(cid:4) (cid:3)3,2(cid:4) (cid:3)6,0(cid:4) (cid:3)1,1(cid:4) 1 (cid:3)5,1(cid:4)
3 2,3 8,27
3 (cid:3)3,4,9(cid:4) (cid:3)0,0(cid:4) (cid:3)4,2(cid:4) (cid:3)2,1(cid:4) 2 (cid:3)9,2(cid:4)
4 (cid:3)4,5,9(cid:4) (cid:3)5,2(cid:4) (cid:3)8,0(cid:4) (cid:3)3,1(cid:4) 3 (cid:3)3,0(cid:4)
5 (cid:3)4,1,5(cid:4) (cid:3)0,0(cid:4) (cid:3)0,0(cid:4) (cid:3)4,0(cid:4) 4 (cid:3)5,0(cid:4)
6 (cid:3)8,9,7(cid:4) (cid:3)2,1(cid:4) (cid:3)7,2(cid:4) (cid:3)0,0(cid:4) 5 (cid:3)8,1(cid:4)
7 (cid:3)9,6,7(cid:4) (cid:3)8,2(cid:4) (cid:3)9,0(cid:4) (cid:3)6,1(cid:4) 6 (cid:3)9,1(cid:4)
8 (cid:3)9,5,6(cid:4) (cid:3)4,1(cid:4) (cid:3)0,0(cid:4) (cid:3)7,0(cid:4) 7 (cid:3)6,2(cid:4)
9 (cid:3)7,6,2(cid:4) (cid:3)7,1(cid:4) (cid:3)0,0(cid:4) (cid:3)0,0(cid:4) 8 (cid:3)1,2(cid:4)
9 (cid:3)8,0(cid:4)
1. Updatemapsforchildrenofanentity: Afteranentityisrefined, thesiblinghalf-facet(sibhfs)andvertex-to-
incidenthalf-facet(v2hf)mapsareupdatedonlyforthechildentitiesoftheworkingentitybyusingthesame
fromtherefinementtemplates.
2. Updatemapsbetweenchildrenofparentsiblings: Afteralltheentitiesofthemeshhavebeenrefined,themaps
arenowupdatedtoconnectthechildentitiesincidentonboundariesoftheparents.
Figure4illustratestheabovetwostepsstepsduringrefinementofatrianglemeshwithtwoentities.Algorithms1and
2outlinetheprocedureforupdatinglocalandglobalAHFmaps. Formixed-dimensionalmeshes,themapsforeach
lowerdimensionalsubmeshareupdatedinasimilarmanner.
3.3. Multi-levelMeshStorage
Anarray-basedmeshstorageleadstoincreasedefficiency,howeveritrequirescarefulconsiderationtomaintainit
foroperationsinvolvingachangeinthecontiguityofthememoryspace. Byvirtueoftheuniformrefinementofthe
mesh,itispossibletoestimatethetotalnumberofentitiesthatwillbecreatedforagivendegreeofrefinement.Table3
showstheestimatesfornewentitiescreatedafterasinglelevelofrefinementfora3Dmeshwithembeddedcurves
andsurfaces.Oncethestoragerequirementforanewlevelisestimated,thememoryisallocatedincontiguousblocks.
Duringrefinement,aseachentityissubdivided,thenewentitiesarestoredaccordingtothelocalorderspecifiedinthe
Navamita Ray et al. / Procedia Engineering 124 ( 2015 ) 291 – 303 297
Algorithm1UpdatingAHFmapsforchildrenofanentity
Input: : childEnts: orderedchildentitiesofaparent,childVerts: orderednewvertexindicesofchildEnts,refTem-
plate:refinementtemplateforentitytypeanddegree
Output: sibhfs:updatesiblinghalf-facetsforchildEnts,v2hf:updatevertextohalf-facetforchildVerts
1: foreachvertexvinchildVertsdo
2: ifv2hf(v)=0then
3: cid←childidfromrefTemplate→v2hf;
4: lid←half-facetidfromrefTemplate→v2hf;
5: v2hf(v)=(childEnts[cid],lid);
6: endif
7: endfor
8: foreachchildcinchildEntsdo
9: foreachfacet f incdo
10: ifsibhfs(f isnotsetthen
11: cid←siblingchildidfromrefTemplate→sibhfs;
12: lid←siblinghalf-facetidfromrefTemplate→sibhfs;
13: sibhfs(c, f)=(childEnts[cid],lid);
14: sibhfs(childEnts[cid],lid)=(c, f);
15: endif
16: endfor
17: endfor
Algorithm2UpdatingAHFmapsbetweenchildrenofsiblingparentssharingafacet.
Input: : PE:entitiesofpreviouslevel,CE:entitiesofcurrentlevel,PV:vertexindicesofPE,CV:vertexindicesof
CE,refTemplate:refinementtemplateforentitytypeanddegreeentitytypeanddegree
Output: sibhfs:updatesiblinghalf-facetsforCE,v2hf:updatevertextohalf-facetforduplicatesofPVinCV
1: foreachvertexvinPVdo
2: lvid←localidofvin pid;
3: pid←parententityfromincidenthalf-facetv2hf(v);
4: cid←childidfromrefTemplate→v2hf;
5: lid←half-facetidfromrefTemplate→v2hf;
6: curvid←currentidofvinCV;
7: v2hf(curvid)=(CE[cid],lid);
8: endfor
9: foreachentityeinPEdo
10: foreachfacet f inedo
11: childs←childrenentitiesofeincidenton f;
12: sibeids←siblingentitiesfromsibhfs(e, f);
13: siblids←siblingfacetidsfromsibhfs(e, f);
14: foreachsiblingseidinsibeidsdo
15: sibchilds←childrenentitiesofseidincidentonsiblids[seid];
16: findorientationof f andsiblids[seid];
17: updatesibhfs(childs,f)withmatching(sibeids,siblids);
18: endfor
19: endfor
20: endfor
298 Navamita Ray et al. / Procedia Engineering 124 ( 2015 ) 291 – 303
(cid:13)(cid:14)(cid:10)(cid:15)(cid:16)(cid:5)(cid:9)(cid:17)(cid:18)(cid:19)(cid:9)(cid:20)(cid:15)(cid:14)(cid:21)(cid:9)(cid:6)(cid:22)(cid:23)(cid:9)(cid:24)(cid:25)(cid:7)(cid:26)(cid:10)(cid:9)(cid:5)(cid:8)(cid:16)(cid:7)(cid:16)(cid:7)(cid:5)(cid:21)
(cid:4)(cid:5)(cid:6)(cid:7)(cid:8)(cid:5)(cid:9)(cid:10)(cid:5)(cid:11)(cid:9)(cid:12)
(cid:1)(cid:2)
(cid:13)(cid:14)(cid:10)(cid:15)(cid:16)(cid:5)(cid:9)(cid:17)(cid:18)(cid:19)(cid:9)(cid:20)(cid:15)(cid:14)(cid:21)(cid:9)(cid:27)(cid:5)(cid:16)(cid:28)(cid:5)(cid:5)(cid:8)
(cid:24)(cid:25)(cid:7)(cid:26)(cid:10)(cid:23)(cid:5)(cid:8)(cid:9)(cid:22)(cid:8)(cid:9)(cid:21)(cid:25)(cid:15)(cid:23)(cid:5)(cid:10)(cid:9)(cid:6)(cid:15)(cid:24)(cid:5)(cid:16)(cid:9)(cid:22)(cid:6)(cid:9)(cid:14)(cid:15)(cid:23)(cid:5)(cid:8)(cid:16)(cid:21)
(cid:1)(cid:3)
(cid:4)(cid:5)(cid:6)(cid:7)(cid:8)(cid:5)(cid:9)(cid:10)(cid:5)(cid:11)(cid:9)(cid:12)
(cid:13)(cid:14)(cid:10)(cid:15)(cid:16)(cid:5)(cid:9)(cid:17)(cid:18)(cid:19)(cid:9)(cid:20)(cid:15)(cid:14)(cid:21)(cid:9)(cid:6)(cid:22)(cid:23)(cid:9)(cid:24)(cid:25)(cid:7)(cid:26)(cid:10)(cid:9)(cid:5)(cid:8)(cid:16)(cid:7)(cid:16)(cid:7)(cid:5)(cid:21)
Figure4:UpdatingtheAHFmapsfortherefinedmeshtakesplaceintwostages.First,mapsforthechildrenofeachtriangleareupdated.After,
boththetriangleshavebeenrefined,themapsarenowupdatedtoconnectthechildrenentitiessharingaparentfacet.
(cid:13)(cid:2)(cid:14)(cid:15)(cid:5)(cid:25)(cid:11)(cid:26)(cid:20)(cid:12)(cid:19)(cid:2)
(cid:1)(cid:2)(cid:3)(cid:2)(cid:4)(cid:5) (cid:18)(cid:2)(cid:19)(cid:20)(cid:2)(cid:2)
(cid:6)(cid:7)(cid:8)(cid:9)(cid:10)(cid:11)(cid:10)(cid:12)(cid:4)(cid:5)(cid:13)(cid:2)(cid:14)(cid:15) (cid:21)(cid:7)(cid:22)
(cid:16) (cid:17)
(cid:17) (cid:23)
(cid:1) (cid:24)
Figure5:Thedatalayoutofmeshesateachlevel.Forexample,startingwiththefirstquadofthespheremesh,aftereachlevelofrefinement,the
childrenarestoredinapre-decidedorderasdefinedintherefinementtemplates.Thislocalorderuniquelyidentifieseachchildoftheparent.
Table3:Estimatesfornewentitiesthatwillbecreatedafterarefinementofdegreep.Theseestimatesarefora3Dmeshwithexplicitcurvesand
surfaces.Here|·|denotesthenumberofentitiesinaparticularentityset.EpandFparethetotalnumberofedgesandfacesinthe3Dmeshatthe
previouslevel,respectively.
#Entitiesatpreviouslevel #Entitiesafterrefinementofdegreep
EExxppVllieiccriittticEFedascgeess (cid:2)(cid:2)(cid:2)(cid:2)(cid:2)(cid:2)EF|Veeppxxppp|(cid:2)(cid:2)(cid:2)(cid:2)(cid:2)(cid:2) |V|=|Vp|+n(cid:2)(cid:2)(cid:2)v(cid:2)(cid:2)(cid:2)FEeee∗xxpp|(cid:2)(cid:2)(cid:2)E(cid:2)(cid:2)(cid:2)==p|p+p2∗n∗v(cid:2)(cid:2)(cid:2)(cid:2)(cid:2)(cid:2)EfFep∗epxxp|pF(cid:2)(cid:2)(cid:2)(cid:2)(cid:2)(cid:2)p|+nvc∗|C|
Cells |C| |C|=p3∗|Cp|
refinementtemplate.Thus,childrenofthefirstentityinpreviouslevelarestoredfirst,thenthechildrenofthesecond
andsoon. Thisdatalayoutsupportsstraightforwardindexbasedinter-level(i.e.,parent-to-childorchild-to-parent)
querieswhicharevitaltomulti-levelmethods.Figure5illustratesthismemorylayout.Toprovidemeshindependence
ateachlevel,weduplicateverticesfromthepreviousleveltocreatethenewmeshalongwithnewvertices.Asaresult,
themeshhierarchygenerationwouldreturnasequenceofmeshesthatareindependentofeachotherwhileproviding
inter-andintra-levelmeshaccess. Ifnecessary,theverticesataspecificlevelcanberenumberedtomaintainsmall
bandwidthsfortheassembledmatrices.
Navamita Ray et al. / Procedia Engineering 124 ( 2015 ) 291 – 303 299
4. HierarchicalMeshGenerationinParallel
Generating mesh hierarchies using uniform refinement in parallel may seem to be a relatively straightforward
processsinceeachprocessorrefinesitslocalmesh.Iftheinitialmeshdistributedisbalanced,thenuniformrefinement
would not introduce any additional imbalance. Thus, there is no need to move mesh entities between processors.
However, generation of each level of the hierarchy also introduces new entities on the shared interface between
processors. Unlessthesenewentitiesareresolved,thegeneratedhierarchiesareonlyusefulforlocaloperationsand
morecomplicatedalgorithmsrequiringsharedinformationsuchasexchangingghost-layerswouldbreakdown.Inthis
section,wediscussaparallelcommunicationalgorithmthatresolvessuchnewentitiesonthesharedinterfacebetween
processors. We also describe the refinement algorithm to generate multi-degree, multi-dimensional hierarchies of
unstructuredmeshesinparallel.
4.1. ParallelCommunicationtoResolveSharedEntities
The MOAB library implemented with array-based data-structures have been designed to be scalable in memory
accessandthereareseveraloptimizedone-sidedcommunicationalgorithmsthatmakeuseofstrategiestominimize
totaldatatransferredbetweenprocessors. Usingthisparallelframework,themeshhierarchygenerationcanbeper-
formedinaseriesofoptimalsteps. Onceeachprocessorloadsapartofthedistributedcoarsemesh,localrefinement
foralltheentitiescanbeperformed. Sincetherefinementisbasedonpre-definedtemplates,theco-ordinatesforthe
newverticesandentitiesonthesharedinterfacebetweenprocessorswillbethesame. Weutilizethisinformationto
designthecommunicationalgorithmtoresolvethenewlycreatedentitiesinpreparationforsynchronizationofghost
layersandexchangeofmeta-data(MOABtags).
This parallel merge of the interface mesh algorithm first matches mesh vertices based on geometric proximity
andthenusesconnectivitymatchingalgorithmtodecipherthecorrespondingentityinthelocalmesh. Themerging
algorithmderivesitsmotivationfromthevertex-matchingalgorithmdescribedin[6,18]. Thealgorithmproceedsby
first partitioning the geometric bounding box of all vertices over processors, with each processor responsible for a
distinct geometric region (plus a small epsilon layer whose thickness is twice the distance tolerance of the merge).
Then,eachprocessorretrievestheverticesontheskinofthelocalmeshandassemblesatuplelist[18]thatholdsthe
coordinatepositionsoftheverticesandthedestinationprocessor. Finally,thehigher-dimensionalentitiesontheskin
areresolvedusingtheconnectivities.
Alternately, we can design a scheme for assigning consistent GLOBAL_ID’s across processors for the newly
created entities, which can yield a significantly faster parallel merging algorithm due to lower communication and
local search costs. In the current work, we only present the scalability results for the co-ordinate based merging
algorithmandtheoptimizedGLOBAL_IDmergealgorithmalongwithcomparisonswillbepursuedseparately.
4.2. RefinementAlgorithm
Figure6showstheflowchartoftherefinementalgorithmtogenerateahierarchyofunstructuredmeshesinparallel.
Akeyaspectoftherefinementalgorithmisthepositioningofthenewverticesasentitiesarerefined. Inthiswork,
weusealinearpointprojectionschemeforthenewvertices. However,thiswouldcompromisetheaccuracyofthe
geometry and in turn that of the finite element solver. To address this issue, we are currently exploring integration
withgeometricmodels,whenavailable,ortouseapolynomialbasedhigh-orderboundaryreconstruction[7,8]. We
havealsodevelopedatoolontopofthisfunctionalityinMOABwhichcanbeusedtoreadinamesh,generatethe
hierarchiesinparallelforagivennumberoflevelsandasequenceofdegreeofrefinements.
5. NumericalResults
Wepresentsomenumericalresultstodemonstratetheeffectivenessofuniformmeshrefinement,intermsofmesh
quality,computationalefficiencyoftheparallelframeworkanditsapplicationtomultigridandefficiencyofmultigrid
methods,andmostimportantly,theaccuracyoffiniteelementmethods.
300 Navamita Ray et al. / Procedia Engineering 124 ( 2015 ) 291 – 303
Figure6:Theflowchartforgenerationofmeshhierarchywithlengthnum_levels.
50% 50%
initial mesh initial mesh
1st refinement deg2 refinement
40% 2nd refinement 40% deg3 refinement
ge30% ge30%
nta nta
Perce20% Perce20%
10% 10%
0% 0%
0−0.17 0.17−0.330.33−0.50.5−0.670.67−0.83 0.83−1 0−0.17 0.17−0.330.33−0.50.5−0.670.67−0.83 0.83−1
Aspect Ratios Aspect Ratios
Figure7:Aspectratiodistributionswithtwo-leveldegrees2+2(left)anddegrees2+3(right)refinements.
5.1. MeshQualityforHierarchicalMeshes
For tetrahedral meshes, uniform mesh refinement does not produce congruent sub-tetrahedra. It is interesting
to note that the average mesh quality actually improves during UMR with the shortest-diagonal approach[19]. To
demonstrateit,wegenerateaninitialcoarsetetrahedralmeshusingTetGen[14],withapproximately5Kverticesand
2.5Ktetrahedra.Weuniformlyrefinetwicewithtwostrategies.First,werefineitusingtwodegree-2refinements.The
resultingfinestmeshhasabout280Kverticesand1.6Mtetrahedra. Second,werefineitusingonedegree-2followed
byadegree-3refinement. Wemeasurethemeshqualityusingtheaspectratio,computedasthreetimestheinscribed
radius(IR)dividedbythecircumsphereradius(CR),i.e.,3×IR/CR. Largeraspectratio(closerto1)correspondsto
betterelementquality.ThetwographsinFigure7showthedistributionsoftheaspectratiosoneachlevelforthetwo
strategies, respectively. In both cases, the overall mesh quality improved, and degree-3 refinement delivers similar
andevenslightlybetterqualityimprovementbecausetherearemoreintermediateoctahedraindegree-3refinement.