Multivariate predictions of local reduced-order-model errors and dimensions

0.67 MB·

by  Azam Moosavi

page0
Computer Science Technical Report CSTR-1
January 16, 2017
7
1
Azam Moosavi, Razvan Stefanescu, Adrian Sandu
0
2

n
a
J

3
1

]
A
N
“Multivariate predictions of local
.
s
c
reduced-order-model errors and dimensions ”
[

1
v
0
2
7
3
0
.
1
0
7 ComputationalScienceLaboratory
1
ComputerScienceDepartment
:
v
VirginiaPolytechnicInstituteandStateUniversity
i
X Blacksburg,VA24060
r Phone: [phone]
a
Fax: [phone]
Email: [email protected]
Web: http://csl.cs.vt.edu
InnovativeComputationalSolutions
1
Abstract
Thispaperintroducesmultivariateinput-outputmodelstopredicttheerrorsandbasesdimensionsoflocalparametricProperOrthog-
onalDecompositionreduced-ordermodels. WerefertothesemultivariatemappingsastheMP-LROMmodels. WeemployGaussian
ProcessesandArtificialNeuralNetworkstoconstructapproximationsofthesemultivariatemappings. Numericalresultswithaviscous
BurgersmodelillustratetheperformanceandpotentialofthemachinelearningbasedregressionMP-LROMmodelstoapproximatethe
characteristics of parametric local reduced-order models. The predicted reduced-order models errors are compared against the multi-
fidelitycorrectionandreducedordermodelerrorsurrogatesmethodspredictions, whereasthepredictedreduced-orderdimensionsare
testedagainstthestandardmethodbasedonthespectrumofsnapshotsmatrix.SincetheMP-LROMmodelsincorporatemorefeaturesand
elementstoconstructtheprobabilisticmappingstheyachievemoreaccurateresults. However,forhigh-dimensionalparametricspaces,
theMP-LROMmodelsmightsufferfromthecurseofdimensionality.ScalabilitychallengesofMP-LROMmodelsandthefeasibleways
ofaddressingthemarealsodiscussedinthisstudy.
keywords Localreduced-ordermodels,ProperOrthogonalDecomposition,regressionmachinelearningtechniques.
1 Introduction
Manyphysicalphenomenaaredescribedmathematicallybypartialdifferentialequations(PDEs),and,afterapplyingsuitablediscretiza-
tionschemes,aresimulatedonacomputer. PDE-basedmodelsfrequentlyrequirecalibrationandparametertuninginordertoprovide
realisticsimulationresults. Recentdevelopmentsinthefieldofuncertaintyquantification[42,71,27,13]providethenecessarytools
forvalidationofsuchmodelseveninthecontextofvariabilityandlackofknowledgeoftheinputparameters. Techniquestopropagate
uncertaintiesthroughmodelsincludedirectevaluationforlinearlyparametricmodels,samplingmethodssuchasMonteCarlo[66],Latin
hypercube[32]andquasi-MonteCarlotechniques[43],perturbationmethods[10,11,12]andspectralrepresentation[42,21,1]. While
stochastic Galerkin methods [42] are intrusive in nature, Monte Carlo sampling methods [66] and stochastic collocations [21] do not
requirethemodificationofexistingcodesandhencetheyarenon-intrusive. Whileuncertaintypropagationtechniquescanmeasurethe
impactofuncertainparametersonsomequantitiesofinterest,theyoftenbecomeinfeasibleduetothelargenumberofmodelrealizations
requirement. SimilardifficultiesareencounteredwhensolvingBayesianinferenceproblemssincesamplingfromposteriordistribution
isrequired.
Theneedforcomputationalefficiencymotivatedthedevelopmentofsurrogatemodelssuchasresponsesurfaces,lowresolution,and
reduced-order models. Data fitting or response surface models [71] are data-driven models. The underlying physics remain unknown
andonlytheinput-outputbehaviorofthemodelisconsidered. Datafittingcanusetechniquessuchasregression,interpolation,radial
basisfunction, GaussianProcesses, ArtificialNeuralNetworksandothersupervisedmachine-learningmethods. Thelattertechniques
can automatically detect patterns in data, and one can use them to predict future data under uncertainty in a probabilistic framework
[48]. While easy to implement due to the non-intrusive nature, the prediction abilities may suffer since the governing physics are not
specificallyaccountedfor.
Low-fidelity models attempt to reduce the computational burden of the high-fidelity models by neglecting some of the physical
aspects (e.g., replacing Navier-Stokes and Large Eddy Simulations with inviscid Euler’s equations and Reynolds-Averaged Navier-
Stokes[25,63,78],ordecreasingthespatialresolution[15,74]). Theadditionalapproximations,however,mayconsiderablydegrade
thephysicalsolutionwithonlyamodestdecreaseofthecomputationalload.
Reducedbasis[53,5,26,61,18]andProperOrthogonalDecomposition[39,45,36,46,47]aretwoofthepopularreduced-order
modeling (ROM)strategies availablein theliterature. Data analysisis conducted toextract basisfunctions from experimentaldata or
detailed simulations of high-dimensional systems (method of snapshots [67, 68, 69]), for subsequent use in Galerkin projections that
yieldlowdimensionaldynamicalmodels.Whilethesetypeofmodelsarephysics-basedandthereforerequireintrusiveimplementations,
theyareusuallymorerobustthandatafittingandlow-fidelitymodels. However,sincesurrogatemodelrobustnessdependsheavilyon
theproblem,itmustbecarefullyanalyzedespeciallyforlarge-scalenonlineardynamicalsystems.
ROM robustness in a parametric setting can be achieved by constructing a global basis [34, 54], but this strategy generates large
dimensionalbasesthatmayleadtoslowreduced-ordermodels. Localapproacheshavebeendesignedforparametricortimedomains
generatinglocalbasesforboththestatevariables[56,17]andnon-linearterms[20,52]. Arecentsurveyofstate-of-the-artmethodsin
projection-basedparametricmodelreductionisavailablein[6].
Inthisstudy,weproposemultivariatedatafittingmodelstopredictthelocalparametricProperOrthogonalDecompositionreduced-
ordermodelserrorsandbasesdimensions. WerefertothemasMP-LROMmodels. Letusconsideralocalparametricreduced-order
modelofdimensionK constructedusingahigh-fidelitysolutionassociatedwiththeparameterconfigurationµ .
POD p
Our first MP-LROM model consists in the mapping {µ,µ ,K } (cid:55)→ logεHF , where εHF is the error of the
p POD µ,µp,KPOD µ,µp,KPOD
local reduced-order model solution with respect to the high-fidelity solution for a viscosity parameter configuration µ. Our proposed
approachisinspiredfromthemulti-fidelitycorrection(MFC)[2]andreducedordermodelerrorsurrogatesmethod(ROMES)[19].MFC
[2,22,25,37]hasbeendevelopedforlow-fidelitymodelsinthecontextofoptimization. TheMFCmodelsimulatestheinput-output
relationµ(cid:55)→εHF,whereεHF isthelow-fidelitymodelerrordependingonaglobalreducedbasiswithaconstantreduced-ordermodel
µ µ
dimension. The ROMES method [19] introduced the concept of error indicators for global reduced-order models and generalized the
MFCframeworkbyapproximatingthemappingρ(µ)(cid:55)→logεHF. Theerrorindicatorsρ(µ)includerigorouserrorboundsandreduced-
µ
orderresidualnorms. Novariationofthereducedbasisdimensionwastakenintoaccount. Byestimatingthelogofthereduced-order
2
modelerrorinsteadoftheerroritself,theinput-outputmapexhibitsalowervarianceasshownbyournumericalexperimentsaswellas
thosein[19].
The second proposed MP-LROM model addresses the issue of a-priori selection of the reduced basis dimension for a prescribed
accuracyofthereducedsolution. Thestandardapproachistoanalyzethespectrumofthesnapshotsmatrix,andusethelargestsingular
valueremovedfromtheexpansiontoestimatetheaccuracylevel[75]. Toalsotakeintoaccounttheerrorduetothefull-order-model
equationsprojectioninthereducedspace,hereweproposethemapping{µ ,logεHF }(cid:55)→K topredictthedimensionofa
p µp,µp,KPOD POD
localparametricreduced-ordermodelgivenaprescribederrorthreshold.
To approximate the mappings {µ,µ ,K } (cid:55)→ logεHF and {µ ,logεHF } (cid:55)→ K , we propose regression
p POD µ,µp,KPOD p µp,µp,KPOD POD
models constructed using Gaussian Processes (GP) [70, 44] and Artificial Neural Networks (ANN). In the case of one dimensional
Burgersmodel,theresultedMP-LROMerrormodelsareaccurateandtheirpredictionsarecomparedagainstthoseobtainedbytheMFC
andROMESmodels.Thepredicteddimensionsoflocalreduced-ordermodelsusingourproposedMP-LROMmodelsaremoreaccurate
thanthosederivedusingthestandardmethodbasedonthespectrumofsnapshotsmatrix.
Theremainderofthepaperisorganizedasfollows. Section2reviewsthereduced-ordermodelingparametricframework. TheMP-
LROMmodelsandtheregressionmachinelearningmethodsusedinthisstudytoapproximatetheMP-LROMmappingsaredescribed
indetailsinSection3. Section4describestheviscous1D-BurgersmodelandcomparestheperformancesoftheMP-LROMandstate
oftheartmodels. ConclusionsaredrawninSection5.
2 Parametric reduced-order modeling
Proper Orthogonal Decomposition has been successfully applied in numerous applications such as compressible flow [? ] and com-
putational fluid dynamics [40, 60, 79], to mention a few. It can be thought of as a Galerkin approximation in the state variable built
fromfunctionscorrespondingtothesolutionofthephysicalsystematspecifiedtimeinstances. AsystemreductionstrategyforGalerkin
modelsoffluidflowsbasedonapartitioninslow,dominant,andfastmodes,hasbeenproposedin[51].Closuremodelsandstabilization
strategiesforPODofturbulentflowshavebeeninvestigatedin[64,77].
In this paper we consider discrete inner products (Euclidean dot product), though continuous products may be employed as well.
Generally,anunsteadyproblemcanbewritteninsemi-discreteformasaninitialvalueproblem;i.e.,asasystemofnonlinearordinary
differentialequations
dx(µ,t)
=F(x,t,µ), x(µ,0)=x ∈RNstate, µ∈P. (1)
dt 0
Theinput-parameterµtypicallycharacterizesthephysicalpropertiesoftheflow.ByP wedenotetheinput-parameterspace.Foragiven
pthaeratomtaeltenrucmonbfiergoufradtiisocnreµtepmwoedseellevcatriaanbleenss,eamndblNeof∈NNt∗t.imTeheinPsOtaDncmeseothfothdechflooowsexs(aµnp,otr1th)o,.n.o.r,mxa(lµbpa,stiNstU)∈=RN[ustaµtpe,w··h·erueµNpstate]i∈s
t µp 1 KPOD
RNstate×KPOD, such that the mean square error between x(µp,ti) and the POD expansion xµPOpD(ti) = Uµpx˜µp(µ,ti), x˜µp(µ,ti) =
UµTpx(µp,ti) ∈ RKPOD,isminimizedonaverage. ThePODspacedimensionKPOD (cid:28) Nstate isappropriatelychosentocapturethe
dynamicsoftheflow. Algorithm1describesthereduced-orderbasisconstructionprocedure[72].
Algorithm1PODbasisconstruction
1: Compute the singular value decomposition for the snapshots matrix [x(µp,t1) ··· x(µp,tNt)] = U¯µpΣµpV¯µTp, with the singular
vectorsmatrixU¯ =[uµp ··· uµp].
µp 1 Nt
2: Usingthesingular-valuesλ1 ≥λ2 ≥...≥λNt ≥0storedinthediagonalmatrixΣµp,defineI(m)=(cid:80)mi=1λ2i/((cid:80)it=Nt1λ2i).
3: Choose KPOD, the dimension of the POD basis, such that KPOD = argminm{I(m) : I(m) ≥ γ} where 0 ≤ γ ≤ 1 is the
pUercecnotnasgiestosfotfotthaelifinrfsotrKmationccoaplutumrnedsobfyU¯the.reducedspaceXKPOD = range(Uµp). Itiscommontoselectγ = 0.99. Thebasis
µp POD µp
Next,aGalerkinprojectionofthefullmodelstate(1)ontothespaceXKPOD spannedbythePODbasiselementsisusedtoobtain
thereduced-ordermodel
dx˜ (µ,t) (cid:18) (cid:19)
µp =UT F U x˜ (µ,t),t,µ , x˜ (µ,0)=UT x . (2)
dt µp µp µp µp µp 0
Thenotationx˜ (µ,t)expressesthesolutiondependenceonthevaryingparameterµandalsoonµ theconfigurationwhoseassociated
µp p
high-fidelity trajectory was employed to generate the POD basis. While being accurate for µ = µ , the reduced model (2) may lose
p
accuracy when moving away from the initial setting. Several strategies have been proposed to derive a basis that spans the entire
parameterspace. Theseincludethereducedbasismethodcombinedwiththeuseoferrorestimates[61,55,54],globalPOD[73,65],
Krylov-based sampling methods [16, 76], and greedy techniques [29, 50]. The fundamental assumption used by these approaches is
thatasmoothlow-dimensionalglobalmanifoldcharacterizesthemodelsolutionsovertheentireparameterdomain. Thepurposeofour
paperistoestimatethesolutionerroranddimensionofthereduced-ordermodel(2)thatcanbesubsequentlyusedtogenerateaglobal
basisfortheparameterspace.
3
3 Multivariate prediction of local reduced-order models characteristics (MP-LROM)
Weproposemultivariateinput-outputmodels
φ:z(cid:55)→y, (3)
z∈Rr,topredictcharacteristicsy ∈Roflocalparametricreduced-ordermodels(2).
3.1 ErrorModel
InspiredfromtheMFCandROMESmethodologiesweintroduceaninput-outputmodeltopredicttheleveloferrorεHF ,where
µ,µp,KPOD
εHF =
µ,µp,KPOD (4)
(cid:107)x(µ,t )−U x˜ (µ,t ) x(µ,t )−U x˜ (µ,t ) ··· x(µ,t )−U x˜ (µ,t )(cid:107) .
1 µp µp 1 2 µp µp 2 Nt µp µp Nt F
Here(cid:107)·(cid:107) denotestheFrobeniusnorm,andK isthedimensionofthereduced-ordermodel. IncontrastwithROMESandMFC
F POD
models that predict the error of global reduced-order models with fixed dimensions, using univariate functions, here we propose a
multivariatemodel
φe :{µ,µ ,K }(cid:55)→logεHF (5)
MP−LROM p POD µ,µp,KPOD
topredicttheerroroflocalparametricreduced-ordermodels(2)ofvariousdimensions. Sincethedimensionofbasisusuallyinfluences
theleveloferrorweincludeitamongtheinputvariables.Todesignmodelswithreducedvarianceswelooktoapproximatethelogarithm
oftheerrorassuggestedin[19].
Forhigh-dimensionalparametricspaces,ROMESmethodhandleswellthecurseofdimensionalitywiththeirproposingunivariate
models. Incombinationwithactivesubspacemethod[14],wecanreducethenumberofinputvariablesincasetheamountofvariability
intheparametricspaceismild. Thiswillincreaseourerrormodelfeasibilityevenforhigh-dimensionalparametricspace.
3.2 Dimensionofthereducedbasis
The basis dimension represents one of the most important characteristic of a reduced-order model. The reduced manifold dimension
directlyaffectsboththeon-linecomputationalcomplexityofthereduced-ordermodelanditsaccuracy[41,33,23]. Byincreasingthe
dimensionofthebasis,theprojectionerrorusuallydecreasesandtheaccuracyofthereduced-ordermodelisenhanced. Howeverthisis
notnecessarilyvalidasseenin[59,Section5]. Neverthelessthespectrumofthesnapshotsmatrixoffersguidanceregardingthechoice
ofthereducedbasisdimensionwhensomeprescribedreduced-ordermodelerrorisdesired. Howevertheaccuracydependsalsoonthe
‘in-plane’error,whichisduetothefactthatthefull-order-modelequationsareprojectedonthereducedsubspace[58,35].
Weseektopredictthedimensionofthelocalparametricreduced-ordermodel(2)byaccountingforboththeorthogonalprojection
errorontothesubspace,whichiscomputablebythesumofsquaresofsingularvalues,andthe‘in-plane’error. Assuchweproposeto
modelthemapping
φd :{µ ,logεHF }(cid:55)→K . (6)
MP−LROM p µp,µp,KPOD POD
Oncesuchmodelisavailable,givenapositivethresholdε¯andaparametricconfigurationµ ,wewillbeabletopredictthedimension
p
K ofthebasisU ,suchthatthereduced-ordermodelerrorsatisfies
POD µp
(cid:107)x(µ ,t )−U x˜ (µ ,t ) x(µ ,t )−U x˜ (µ ,t ) ··· x(µ ,t )−U x˜ (µ ,t )(cid:107) ≈ε¯. (7)
p 1 µp µp p 1 p 2 µp µp p 2 p Nt µp µp p Nt F
3.3 SupervisedMachineLearningTechniques
Inordertoestimatethelevelofreduced-ordermodelsolutionerrorεHF (4)andthereducedbasisdimensionK , wewill
µ,µpj,KPOD POD
useregressionmachinelearningmethodstoapproximatethemapsφe andφd describedin(5)and(6).
MP−LROM MP−LROM
ArtificialNeuralNetworksandGaussianProcessesareusedtobuildaprobabilisticmodelφ : z (cid:55)→ yˆ,whereφisatransformation
function that learns through the input features z to estimate the deterministic output y [48]. As such, these probabilistic models are
approximations of the mappings introduced in (3). The input features z can be either categorical or ordinal. The real-valued random
variable yˆis expected to have a low variance and reduced bias. The features of z should be descriptive of the underlying problem at
hand[8]. TheaccuracyandstabilityofestimationsareassessedusingtheK-foldcross-validationtechnique. Thesamplesaresplitinto
Ksubsets(“folds”),wheretypically3≤K ≤10.ThemodelistrainedonK−1setsandtestedontheK-thsetinaround-robinfashion
[48]. Eachfoldinducesaspecificerrorquantifiedastheaverageoftheabsolutevaluesofthedifferencesbetweenthepredictedandthe
K-thsetvalues
(cid:80)N |yˆi−yi| (cid:80)N (cid:0)yˆi−E (cid:1)2
E = i=1 , VAR = i=1 fold , fold=1,2,...,K, (8a)
fold N fold N −1
whereN isthenumberoftestsamplesinthefold. Theerroristhenaveragedoverallfolds:
(cid:80)K E (cid:80)K (E −E)2
E= fold=1 fold, VAR= fold=1 fold . (8b)
K K−1
4
3.3 SupervisedMachineLearningTechniques
Thevarianceofthepredictionresults(8a)accountsforthesensitivityofthemodeltotheparticularchoiceofdataset. Itquantifies
thestabilityofthemodelinresponsetothenewtrainingsamples. Asmallervarianceindicatesmorestablepredictions, however, this
sometimestranslatesintoalargerbiasofthemodel. Modelswithsmallvarianceandhighbiasmakestrongassumptionsaboutthedata
andtendtounderfitthetruth, whilemodelswithhighvarianceandlowbiastendtooverfitthetruth[49]. Thetrade-offbetweenbias
andvarianceinlearningalgorithmsisusuallycontrolledviatechniquessuchasregularizationorbaggingandboosting[8].
InwhatfollowswebrieflyreviewtheGaussianProcessandArtificialNeuralNetworkstechniques.
3.3.1 Gaussianprocesskernelmethod
AGaussianprocessisacollectionofrandomvariables,anyfinitenumberofwhichhaveajointGaussiandistribution[57]. AGaussian
processisfullydescribedbyitsmeanandcovariancefunctions
(cid:0) (cid:1)
φ(z)∼gp m(z),K) , (9)
wherem(z)=E[φ(z)],andKisthecovariancematrixwithentriesK =E(cid:2)(cid:0)φ(zi)−m(zi)(cid:1)(cid:0)φ(zj)−m(zj)(cid:1)(cid:3)[57].
i,j
Inthisworkweemploythecommonlyusedsquared-exponential-covarianceGaussiankernelwith
(cid:32) (cid:13) (cid:13) (cid:33)
(cid:13)zi−zj(cid:13)
k :Rr×Rr →R, k(zi,zj)=σ2 exp − +σ2δ , (10)
φ 2(cid:125)2 n i,j
andK = k(zi,zj)[57],wherezi andzj arethepairsofdatapointsintrainingortestsamples,δ istheKroneckerdeltasymboland
ij
(cid:107)·(cid:107) is some appropriate norm. The model (10) has three hyper-parameters. The length-scale (cid:125) governs the correlation among data
points. Thesignalvarianceσ2 ∈Randthenoisevarianceσ2 ∈Rgoverntheprecisionofvarianceandnoise,respectively.
φ n
ConsiderasetoftrainingdatapointsZ=[z1z2 ··· zn]∈Rr×nandthecorrespondingnoisyobservationsy=[y1y2 ··· yn]∈
R1×n,
yi =φ(zi)+(cid:15) , (cid:15) ∼N (cid:0)0,σ2(cid:1), i=1,...,n. (11)
i i n
ConsideralsothesetoftestpointsZ∗ =[z∗1z∗2 ··· z∗m]∈Rr×mandthepredictionsyˆ =[yˆ1yˆ2 ··· yˆm]∈R1×m,
yˆi =φ(cid:0)z∗i(cid:1), i=1,...,m. (12)
ForaGaussianpriorthejointdistributionoftrainingoutputsyandtestoutputsyˆis
(cid:20)yT(cid:21) (cid:18)(cid:20)m(Z)T (cid:21) (cid:20) K K∗(cid:21)(cid:19)
∼N , , (13)
yˆT m(Z∗)T K∗T K∗∗
where
m(Z)=[m(z1)m(z2) ··· m(zn)]∈R1×n, m(Z∗)=[m(z∗1)m(z∗2) ··· m(z∗m)]∈R1×m,
K∗ =(K∗) =k(zi,zj∗)andK∗∗ =(K∗∗) =k(zi∗,zj∗).
ij i=1,...,n;j=1,...,m ij i=1,...,m;j=1,...,m
Thepredictivedistributionrepresentstheposteriorafterobservingthedata[8]andisgivenby
p(yˆ|Z,y,Z∗)∼N (cid:0)K∗TK−1y, K∗∗−K∗TK−1K∗(cid:1), (14)
wheresuperscriptT denotesthetransposeoperation.
ThepredictionofGaussianprocesswilldependonthechoiceofthemeanandcovariancefunctions,andontheirhyperparameters
(cid:125),σ2 andσ2 whichcanbeinferredfromthedata
φ n
θ∗ =[(cid:125),σ2,σ2]=argmin L(θ),
φ n
θ
byminimizingthemarginalnegativelog-likelihoodfunction
1 1 n
L(θ)=−log p(y|Z,θ)= logdet(K)+ (y−m(Z))K−1(y−m(Z))T + log(2π).
2 2 2
5
3.3.2 ArtificialNeuralNetworks
The study of Artificial Neural Networks begins in the 1910s in order to imitate human brain’s biological structure. Pioneering work
was carried out by Rosenblatt, who proposed a three-layered network structure, the perceptron [30] . ANN detect the pattern of data
by discovering the input–output relationships. Applications include the approximation of functions, regression analysis, time series
prediction,patternrecognition,andspeechsynthesisandrecognition[38,4].
ANNconsistofneuronsandconnectionsbetweentheneurons(weights). Neuronsareorganizedinlayers,whereatleastthreelayers
ofneurons(aninputlayer,ahiddenlayer,andanoutputlayer)arerequiredforconstructionofaneuralnetwork.
The input layer distributes input signals z = [z z ··· z ] to the first hidden layer. For a neural network with L hidden layers
1 2 r
and m(cid:96) neurons in each hidden layer, let yˆ(cid:96) = [yˆ(cid:96) yˆ(cid:96) ··· yˆ(cid:96) ] be the vector of outputs from layer (cid:96), b(cid:96) = [b(cid:96) b(cid:96) ··· b(cid:96) ] the
1 2 m(cid:96) 1 2 m(cid:96)
biasesatlayer(cid:96),andw(cid:96) =[w(cid:96) w(cid:96) ··· wl ]theweightsconnectingtheneuronjtotheinputofthatlayer(outputofpreviouslayer).
j j1 j2 jml
Thevectorsyˆ(cid:96)andw(cid:96) sharethesamedimensionwhichvariesalongthelayersdependingonthenumberofinputfeatures,neuronsand
j
outputs. Thenthefeed-forwardoperationis
x(cid:96)+1 =w(cid:96)+1Tyˆ(cid:96)+b(cid:96)+1, yˆ0 =z, j =1,...,m(cid:96).
j j j
yˆ(cid:96)+1 =ϕ(cid:0)x(cid:96)+1(cid:1), (cid:96)=0,1,...,L−1.
j
Allproductsofpreviouslayeroutputwithcurrentlayerneuronweightswillbesummedandthebiasvalueofeachneuronwillbeadded
toobtainthevectorx(cid:96) = [x(cid:96) x(cid:96) ··· x(cid:96) ]. Thenthefinaloutputofeachlayerwillbeobtainedbypassingthevectorx(cid:96) throughthe
1 2 m(cid:96)
transferfunctionϕ,whichisadifferentiablefunctionandcanbelog-sigmoid,hyperbolictangentsigmoid,orlineartransferfunction.
ThetrainingprocessofANNadjuststheweightsandthebiasesinordertoreproducethedesiredoutputswhenfedthegiveninputs.
The training process via the back propagation algorithm [62] uses a gradient descent method to modify weights and thresholds such
that the error between the desired output and the output signal of the network is minimized [24]. In supervised learning the network
isprovidedwithsamplesfromwhichitdiscoverstherelationsofinputsandoutputs. Theoutputofthenetworkiscomparedwiththe
desiredoutput,andtheerrorisback-propagatedthroughthenetworkandtheweightswillbeadjusted. Thisprocessisrepeatedduring
severaliterations,untilthenetworkoutputisclosetothedesiredoutput[31].
4 Numerical experiments
WeillustratetheapplicationoftheproposedMP-LROMmodelstopredicttheerroranddimensionofthelocalparametricreduced-order
modelsforaone-dimensionalBurgersmodel. The1D-Burgersmodelproposedhereinischaracterizedbytheviscositycoefficient. To
assesstheperformanceoftheMP-LROMmodelsconstructedusingGaussianProcessandArtificialNeuralNetworks,weemployvarious
cross-validationtests. Thedimensionsofthetrainingandtestingdatasetsarechosenempiricallybasedonthenumberofsamples. For
ArtificialNeuralNetworksmodelsthenumberofhiddenlayersandneuronsineachhiddenlayervaryforeachtypeofproblemsunder
study. Thesquared-exponential-covariancekernel(10)isusedforGaussianProcessmodels.
TheapproximatedMP-LROMerrormodelsarecomparedagainsttheROMESandmulti-fidelitycorrectionmodels,whereastheMP-
LROMmodelsthatpredictthedimensionofthereduced-ordermodelsareverifiedagainstthestandardapproachbasedonthespectrum
ofsnapshotsmatrix.
4.1 One-dimensionalBurgers’equation
Burgers’equationisanimportantpartialdifferentialequationfromfluidmechanics[9].Theevolutionofthevelocityuofafluidevolves
accordingto
∂u ∂u ∂2u
+u =µ , x∈[0,L], t∈(0,t], (15)
∂t ∂x ∂x2 f
witht =1andL=1. Hereµistheviscositycoefficient.
f
ThemodelhashomogeneousDirichletboundaryconditionsu(0,t) = u(L,t) = 0,t ∈ (0,t]. Fortheinitialconditions,weuseda
f
seventhorderpolynomialconstructedusingtheleast-squaremethodandthedataset{(0,0); (0.2,1); (0.4,0.5); (0.6,1); (0.8,0.2);
(0.9,0.1); (0.95,0.05); (1,0)}. WeemployedthepolyfitfunctioninMatlabandthepolynomialisshowninFigure1.
ThediscretizationusesaspatialmeshofN equidistantpointson[0,L],with∆x = L/(N −1). Auniformtemporalmeshwith
s s
Nt pointscoverstheinterval[0,tf],with∆t = tf/(Nt−1). Thediscretevelocityvectorisu(tj) ≈ [u(xi,tj)]i=1,2,...,Nstate ∈ RNstate,
j =1,2,...,N ,whereN =N −2(theknownboundariesareremoved). Thesemi-discreteversionofthemodel(15)is
t state s
u(cid:48) =−u(cid:12)A u+µA u, (16)
x xx
whereu(cid:48)isthetimederivativeofu,andAx,Axx ∈RNstate×Nstate arethecentraldifferencefirst-orderandsecond-orderspacederivative
operators, respectively, whichtakeintoaccounttheboundaryconditions, too. ThemodelisimplementedinMatlabandthebackward
Eulermethodisemployedfortimediscretization. ThenonlinearalgebraicsystemsaresolvedusingtheNewton-Raphsonmethodand
theallowednumberofNewtoniterationspereachtimestepissetto50.ThesolutionisconsideredtohaveconvergedwhentheEuclidean
normoftheresidualislessthen10−10.
6
4.2 Multivariatepredictionoflocalreduced-ordermodelscharacteristics(MP-LROM)usingregressionmachinelearningmethods
TheviscosityparameterspaceP issettotheinterval[0.01,1]. Smallervaluesofµcorrespondtosharpergradientsinthesolution,
andleadtodynamicsmoredifficulttoaccuratelyapproximateusingreduced-ordermodels.
3.5
s
n 3
o
diti2.5
n
o
c
al 2
niti
s i1.5
r
e
g 1
r
u
B
D 0.5
1
0
0 0.2 0.4 0.6 0.8 1
Space discretization
Figure1: Seventhorderpolynomialusedasinitialconditionsfor1DBurgersmodel.
Thereduced-ordermodelsareconstructedusingPODmethodwhereasthequadraticnonlinearitiesarecomputedviatensorialPOD
[72]forefficiency. AfloatingpointoperationsanalysisoftensorialPOD,PODandPOD/DEIMforpth orderpolynomialnonlinearities
isavailablein[72]. ThecomputationalefficiencyofthetensorialPOD1DBurgersmodelcanbenoticedinFigure2. Bothon-lineand
off-linecomputationalcostsareshown. Hereweselectedµ=µ =0.7,N =301,PODdimensionK =9,andweletthenumber
p t POD
ofspacepointsN tovary. ForN = 201and701, thetensorialPODmodelis5.17×and61.12×timesfasterthanthehigh-fidelity
s s
version. TherestofournumericalexperimentsusesN =201andN =301.
s t
101
High-Fidelity
ROM offline
ROM online
100
e
m
it U10-1
P
C
10-2
10-3
100 200 300 400 500 600 700
Number of space points
Figure2: ComputationalefficiencyofthetensorialPOD1DBurgersmodel. CPUtimeisgiveninseconds.
4.2 Multivariate prediction of local reduced-order models characteristics (MP-LROM) using regression
machinelearningmethods
4.2.1 ErrorestimationoflocalROMsolutions
Here, we will use GP and ANN to approximate the MP-LROM error model introduced in (5). The approximated models have the
followingform
HF
φeMP−LROM :{µ,µp,KPOD}(cid:55)→l(cid:100)ogεµ,µp,KPOD, (17)
wheretheinputfeaturesincludeaviscosityparametervalueµ,aparametervalueµ associatedwiththefullmodelrunthatgenerated
p
thebasisU , andthedimensionofthereducedmanifold K . Thetargetis theestimatedlogarithmoferrorofthereduced-order
µp POD
modelsolutionatµusingthebasisU andthecorrespondingreducedoperatorscomputedusingtheFrobeniusnorm
µp
logεHF =
(cid:32)µ,µp,KPOD (cid:33)
(18)
log (cid:107)x(µ,t )−U x˜ (µ,t ) x(µ,t )−U x˜ (µ,t ) ··· x(µ,t )−U x˜ (µ,t )(cid:107) .
1 µp µp 1 2 µp µp 2 Nt µp µp Nt F
7
4.2 Multivariatepredictionoflocalreduced-ordermodelscharacteristics(MP-LROM)usingregressionmachinelearningmethods
Theprobabilisticmodelsdescribedgenericallyinequation(17)arejustapproximationsoftheMP-LROMmodel(5)andhaveerrors.
For our experiments, the data set includes 10 and 100 equally distributed values of µ and µ over the entire parameter region; i.e.,
p
µ ∈ {0.1,0.2,...,1} and µ ∈ {0.01,...,1}, 12 reduced basis dimensions K spanning the interval {4,5,...,14,15} and the
p POD
reduced-ordermodellogarithmoferrorslogεHF .
µ,µp,KPOD
The entire data set contains 12000 samples, and for each 12 samples a high-fidelity model solution is calculated. Only one high-
fidelitymodelsimulationisrequiredforcomputingthereducedsolutionserrorsfortheparametricconfigurationµusingreduced-order
modelsofvariousK constructedbasedonasinglehigh-fidelitytrajectorydescribedbyparameterµ . Assuch,1000high-fidelity
POD p
simulationswereneededtoconstructtheentiredataset. High-fidelitysimulationsareusedtoaccuratelycalculatetheerrorsassociated
withtheexistingreduced-ordermodelsforparametricconfigurationsµ.
Figure3showsisocontoursoftheerrorεHF andlogεHF ofthereduced-ordermodelsolutionforvariousviscosity
µ,µp,KPOD µ,µp,KPOD
parametervaluesµandPODbasisdimensions.Thedesignofthereduced-ordermodelsreliesonthehigh-fidelitytrajectoryforµ =0.8.
p
The target values εHF vary over a wide range (from 300 to 10−6) motivating the choice of implementing models that target
µ,µp,KPOD
logεHF todecreasethevarianceofthepredictedresults.
µ,µp,KPOD
s dimension 11102466104620100.27.40.10.05 00.1.050.010.005 0.010.0050.00.00010050.0001 0.00050.00.0105 s dimension 1110242.51.50.5 −0.51−2.5 −−24.5 −5 −6 −4−7 −5−6−9−8−−107−10−9−8 −5−6−7
K basiPOD 68 104106 0.07.42 00.7.040..015 0.400..105 K basiPOD 68 2.51.5 −10.5 −−21.5 −−−421.5
4 2 0.7 1.5 0.5
4 106 4 6 42 4 2.5 1.5 1.5
0.2 0.4 0.6 0.8 0.2 0.4 0.6 0.8
Viscosity parameter µ Viscosity parameter µ
(a) IsocontoursoftheerrorsεHF (b) Isocontours for the logarithms of the errors
µ,µp,KPOD logεHF
µ,µp,KPOD
Figure3: IsocontoursofthereducedmodelerrorsfordifferentPODbasisdimensionsandparametersµ. Thereduced-ordermodeluses
abasisconstructedfromthefullordersimulationwithparametervalueµ =0.8.
p
Amoredetailedanalysis,comparingmodels
φe :{µ,µ ,K }(cid:55)→εˆHF (19)
MP−LROM p POD µ,µp,KPOD
thattargetnoscaleddataandmodel(17)isgiveninthefollowing.
TheapproximatedMP-LROMmodelsforestimatingthelocalparametricreduced-ordermodelerrorsareconstructedusingaGaus-
sian Process with a squared-exponential covariance kernel (10) and a neural network with six hidden layers and hyperbolic tangent
sigmoidactivationfunctionineachlayer. Tables1and2showtheaveragesandvariancesoferrorsinpredictionofMP-LROMmodels
fordifferentsamplesizes. Everysubsetofsamplesisselectedrandomlyfromashuffledoriginaldataset. Themisfitiscomputedusing
thesameformulaspresentedin(8a)toevaluatethepredictionerrors. Table1showsthepredictionerrorsof(19)computedviaequation
(8a)withy = εHF andyˆ= εˆHF ;i.e.,nodatascaling;thepredictionshavealargevarianceandalowaccuracy. Scaling
µ,µp,KPOD µ,µp,KPOD
the data and targeting logεHF results using (17), reduce the variance of the predictions, and increase the accuracy, as shown
µ,µp,KPOD
HF
in Table 2. The same formula (8a) with y = logεHF and yˆ = l(cid:100)ogε was applied. We notice that, for increasing
µ,µp,KPOD µ,µp,KPOD
samplesizeslessorequalthan700andforscaleddata,thevariancesofGPandANNpredictionsarenotnecessarilydecreasing. This
behaviorchangesandthevariancesofbothregressionmodelsdecreaseforincreasingsamplesizeslargerthan700asseeninTable2.
TheperformanceoftheANNandGPishighlydependentonthenumberofsamplesinthedataset. Asthenumberofdatapointsgrows,
theaccuracyincreasesandthevariancedecreases. TheresultsshowthatGPoutperformsANNforsmallnumbersofsamples≤ 1000
whereas,forlargerdatasets,ANNismoreaccuratethanGP.
8
4.2 Multivariatepredictionoflocalreduced-ordermodelscharacteristics(MP-LROM)usingregressionmachinelearningmethods
GPMP-LROM ANNMP-LROM
Samplesize E VAR E VAR
fold fold fold fold
100 13.4519 5.2372 12.5189 25.0337
400 6.8003 31.0974 6.9210 26.1814
700 5.6273 14.3949 7.2325 19.9312
1000 3.7148 13.8102 5.6067 14.6488
3000 0.5468 0.0030 1.2858 1.2705
5000 6.0563 22.7761 3.8819 23.9059
Table1: AverageandvarianceoferrorinpredictionsofMP-LROMmodels(19)constructedviaANNandGPusingerrorsεHF
µ,µp,KPOD
intrainingdatafordifferentsamplesizes.
GPMP-LROM ANNMP-LROM
Samplesize E VAR E VAR
fold fold fold fold
100 0.5319 0.0118 1.2177 0.1834
400 0.3906 0.0007 0.8988 0.2593
700 0.3322 0.0018 0.7320 0.5602
1000 0.2693 0.0002 0.5866 0.4084
3000 0.1558 0.5535×10−4 0.01202 0.2744×10−4
5000 0.0775 0.4085×10−5 0.0075 0.3812×10−5
Table2:AverageandvarianceoferrorinpredictionsofMP-LROMmodels(19)constructedviaANNandGPusinglogarithmsoferrors
logεHF intrainingdatafordifferentsamplesizes.
µ,µp,KPOD
Figures4and5showthecorrespondinghistogramoftheerrorsinpredictionofMP-LROMmodels(17)and(19)using100and1000
trainingsamplesforANNandGPmethods,respectively. ThehistogramsshowninFigure5canassessthevalidityofGPassumptions
(9),(11),(13). ThedifferencebetweenthetrueandestimatedvaluesshouldbehaveassamplesfromthedistributionN(0,σ2)[19]. In
n
ourcasetheyarehardlynormallydistributedandthisindicatesthatthedatasetsarenotfromGaussiandistributions.
ScalingthedataandtargetinglogεHF errorsclearlyimprovetheperformanceoftheMP-LROMmodels. Consequentlyfor
µ,µp,KPOD
therestofthemanuscriptwewillonlyusemodel(17).
To asses the quality of the MP-LROM models, we also implemented a five-fold cross-validation test over the entire dataset. The
resultscomputedusingformula(8b)areshowninTable3. ANNoutperformsGPandestimatestheerrorsmoreaccurately. Italsohas
lessvariancethantheGaussianProcesswhichindicatesithasmorestablepredictions.
E VAR
ANNMP-LROM 0.004004 2.16×10−6
GPMP-LROM 0.092352 1.32×10−5
Table3: MP-LROMstatisticalresultsoverfive-foldcross-validation.
Figure 6 illustrates the average of errors in prediction of five different errors models computed using ANN and GP regression
methods. The error models were constructed using a training set formed by 80% randomly selected data of the entire data set. The
predictionsweremadeusingafixedtestsetrandomlyselectedfromtheentiredatasetandcontainsvariousvaluesofµ,K andµ
POD p
showninthex-axesofFigure6. BuildingdifferentGPandANNMP-LROMerrormodels,eachtrainedondifferentpartofthedataset
andthentestingthemwiththesamefixedtestset, reducesthebiasinprediction. Again, ANNoutperformsGPhavingmoreaccurate
errorsestimates.
WealsocomparedtheMP-LROMmodelswiththoseobtainedbyimplementingROMESmethod[19]andMFCtechnique[2]. The
ROMESmethodconstructsunivariatemodels
φ :logρ(µ)(cid:55)→logεHF, (20)
ROMES µ
wheretheinputρ(µ)consistsoferrorindicators. Examplesofindicatorsincluderesidualnorms,dual-weightedresidualsandothererror
bounds. MFCimplementsinput-outputmodels
φ :µ(cid:55)→logεHF, (21)
MFC µ
wheretheinputoferrormodelsistheviscosityparameterµ. BothROMESandMFCmethodsuseaglobalreduced-ordermodelwitha
fixeddimensionincontrasttoourmethodthatemployslocalreduced-ordermodelswithvariousdimensions. ROMESandMFCmodels
areunivariatewhereastheMP-LROMmodelsaremultivariate.
9
4.2 Multivariatepredictionoflocalreduced-ordermodelscharacteristics(MP-LROM)usingregressionmachinelearningmethods
15 8
7
6
10
y y5
c c
n n
ue ue4
q q
e e
Fr Fr3
5
2
1
0 0
−1.5 −1 −0.5 0 0.5 1 −1.5 −1 −0.5 0 0.5 1
error in prediction error in prediction
(a) logεHF −l(cid:91)ogεHF -100samples (b) εHF −εˆHF -100samples
µ,µp,KPOD µ,µp,KPOD µ,µp,KPOD µ,µp,KPOD
200 120
100
150
80
y y
c c
n n
ue100 ue 60
q q
e e
Fr Fr
40
50
20
0 0
−1.5 −1 −0.5 0 0.5 1 −1.5 −1 −0.5 0 0.5 1
error in prediction error in prediction
(c) logεHF −l(cid:91)ogεHF -1000samples (d) εHF −εˆHF -1000samples
µ,µp,KPOD µ,µp,KPOD µ,µp,KPOD µ,µp,KPOD
Figure4: HistogramoferrorsinpredictionusingANNMP-LROM.
ToaccommodateourdatasettotherequirementsoftheROMESandMFCmethods,weseparatedthedatasetintomultiplesubsets.
Eachofthesesubsetshas100samplescorrespondingtoasingleµ andK and100valuesofparameterµ ∈ {0.01,0.02...,1},
p POD
so 100 high-fidelity simulations are required. For each subset we constructed ANN and GP models to approximate the input-output
models defined in (20) and (21) using the same training set. In the case of ROMES method we employed the logarithms of residuals
norms as inputs. We first computed the corresponding reduced-order solution and then the associated logarithm of residual norm by
usingtheprojectedreducedordersolutionintothehigh-fidelitymodelforparameterµ. TheoutputofbothROMESandMFCmodels
approximatesthelogarithmoftheFrobeniusnormofthereduced-order-modelerrors.
Figures7-10showstheisocontoursoftheE andVAR computedusing(8a)fordifferentK andµ usingROMES,MFC,
fold fold POD p
andMP-LROMmodelsconstructedusingGPandANNmethods. Intotalthereare12×10configurationscorrespondingtodifferent
K andµ andasmanyROMESandMFCmodels. TheMP-LROMmodelsareglobalinnatureandthetrainingsetisthewhole
POD p
originaldataset. Thetestingsetisthesameforallthecomparedmodelsanddiffersfromthetrainingsets. WecanseethatMP-LROM
modelsaremoreaccuratethanthoseobtainedviaROMESandMFCmodels.IncludingmoresamplesassociatedwithvariousPODbasis
sizesandµ isbenefic. Wealsotrainedandtestedallthemodelsusingfive-foldcross-validation. Theaverageerrorandvarianceofall
p
120E sandVAR sarecomparedagainstthoseobtainedusingMP-LROMerrormodelsandaresummarizedintables4and5. This
fold fold
showsthattheMFCmodelsoutperformtheROMESones,forourexperiment,andtheMP-LROMmodelsarethemostaccurate. The
MP-LROMmodelsperformbettersincetheyemploymorefeaturesandsamplesthantheothermodelswhichhelptheerrormodelstune
theparametersbetter. WealsonoticetheefficiencyoftheMFCmodelsfromaccuracypointofviewconsideringthattheyuseveryfew
samples. InthecaseoflargeparametricdomainstheMP-LROMerrormodelsmayrequireaverylargedatasetwithalotoffeatures.
Byusingonlysubsetsofthewholedatasetnearthevicinityoftheparametersofinterestandapplyingtheactivesubsetmethod[14]can
helppreventthepotentialcurseofdimensionalityproblemthatMP-LROMmightsuffer.
10