Online Structure Learning for Sum-Product Networks with Gaussian Leaves

0.55 MB·

by  Wilson Hsu

Online Structure Learning for Sum-Product Networks with
Gaussian Leaves
WilsonHsu,AgastyaKalra&PascalPoupart
DavidR.CheritonSchoolofComputerScience
UniversityofWaterloo
Waterloo,Ontario,Canada
7
1 {wwhsu,a6kalra,ppoupart}@uwaterloo.ca
0
2

n
Abstract
a
J Sum-productnetworkshaverecentlyemergedasanattractiverepresentationduetotheirdualviewasaspecial

9 type of deep neural network with clear semantics and a special type of probabilistic graphical model for which
1 inferenceisalwaystractable.Thosepropertiesfollowfromsomeconditions(i.e.,completenessanddecomposability)
thatmustberespectedbythestructureofthenetwork.Asaresult,itisnoteasytospecifyavalidsum-productnetwork

] byhandandthereforestructurelearningtechniquesaretypicallyusedinpractice.Thispaperdescribesthefirstonline
L
structure learning technique for continuous SPNs with Gaussian leaves. We also introduce an accompanying new
M
parameterlearningtechnique.
.
t
a
t 1 Introduction
s
[

Sum-productnetworks(SPNs)werefirstintroducedbyPoonandDomingos[2011]asanewtypeofdeeprepresenta-
1
tion. Theydistinguishthemselvesfromothertypesofneuralnetworksbyseveraldesirableproperties:
v
5
1. Thequantitiescomputedbyeachnodecanbeclearlyinterpretedas(un-normalized)probabilities.
6
2
2. SPNsareequivalenttoBayesianandMarkovnetworks[Zhaoetal.,2015]whileensuringthatexactinference
5
haslinearcomplexitywithrespecttothesizeofthenetwork.
0
.
1 3. Theyrepresentgenerativemodelsthatnaturallyhandlearbitraryquerieswithmissingdatawhilechangingwhich
0
variablesaretreatedasinputsandoutputs.
7
1
Thereisacatch:thesenicepropertiesariseonlywhenthestructureofthenetworksatisfiescertainconditions(i.e.,de-
:
v composabilityandcompleteness)[PoonandDomingos,2011]. Hence,itisnoteasytospecifysum-productnetworks
i byhand. Inparticular,fullyconnectednetworkstypicallyviolatethoseconditions. Similarly,mostsparsestructures
X
thatarehandcraftedbypractitionerstocomputespecifictypesoffeaturesorembeddingsalsoviolatethoseconditions.
r
a Whilethismayseemlikeamajordrawback,thebenefitisthatresearchershavebeenforcedtodevelopstructurelearn-
ing techniques to obtain valid SPNs that satisfy those conditions [Dennis and Ventura, 2012, Gens and Domingos,
2013,Peharzetal.,2013,Leeetal.,2013,RooshenasandLowd,2014,Adeletal.,2015,Vergarietal.,2015,Rahman
and Gogate, 2016, Melibari et al., 2016]. At the moment, the search for good network structures in other types of
neural networks is typically done by hand based on intuitions as well as trial and error. However the expectation is
thatautomatedstructurelearningtechniqueswilleventuallydominate. Forthistohappen,weneedstructurelearning
techniquesthatcanscaleeasilytolargeamountsofdata.
Tothateffect,weproposethefirstonlinestructurelearningtechniqueforSPNswithGaussianleaves.Theapproach
startswithanetworkstructurethatassumesthatallvariablesareindependent. Thisnetworkstructureisthenupdated
as a stream of data points is processed. Whenever a statistically significant correlation is detected between some
variables, acorrelationisintroducedinthenetworkintheformofamultivariateGaussianoramixturedistribution.
This is done while ensuring that the resulting network structure is necessarily valid. The approach is evaluated on
severallargebenchmarkdatasets.
1
Thepaperisstructuredasfollows. Section2providessomebackgroundaboutsum-productnetworks. Section3
describesouronlinestructurelearningtechniqueforSPNswithGaussianleaves. Section4evaluatestheperformance
of our structure learning technique on several large benchmark datasets. Finally, Section 5 concludes the paper and
discussespossibledirectionsforfuturework.
2 Background
Sum-productnetworks(SPNs)werefirstproposedbyPoonandDomingos[2011]asanewtypeofdeeparchitecture
consisting of a rooted acyclic directed graph with interior nodes that are sums and products while the leaves are
tractabledistributions, includingBernoullidistributionsfordiscreteSPNsandGaussiandistributionsforcontinuous
SPNs. Theedgesemanatingfromsumnodesarelabeledwithnon-negativeweightsw. AnSPNencodesafunction
f(X=x)thattakesasinputavariableassignmentX=xandproducesanoutputatitsroot. Thisfunctionisdefined
recursivelyateachnodenasfollows:

Pr(X =x ) ifisLeaf(n)
 n n
(cid:80)
f (X=x)= w f (x) ifisSum(n) (1)
n  (cid:81)if i childi((nx)) ifisProduct(n)
i childi(n)
Here, X = x denotes the variable assignment restricted to the variables contained in the leaf n. If none of the
n n
variables in leaf n are instantiated by X = x then Pr(X = x ) = Pr(∅) = 1. Note also that if leaf n contains
n n
continuousvariables,thenPr(X =x )shouldbeinterpretedaspdf(X =x ).
n n n n
AnSPNisaneuralnetworkinthesensethateachinteriornodecanbeinterpretedascomputingalinearcombina-
tionofitschildrenfollowedbyapotentiallynon-linearactivationfunction. Withoutlossofgenerality,assumethatthe
SPNisorganizedinalternatinglayersofsumsandproductnodes.1 Itiseasytoseethatsum-nodescomputealinear
combinationoftheirchildren. Productnodescanbeinterpretedasthesumofitschildreninthelogdomain. Hence
sum-productnetworkscanbeviewedasneuralnetworkswithlogarithmicandexponentialactivationfunctions.
An SPN can also be viewed as encoding a joint distribution over the random variables in its leaves when the
networkstructuresatisfiescertainconditions. Theseconditionsareoftendefinedintermsofthenotionofscope.
Definition1(Scope). Thescope(n)ofanodenisthesetofvariablesthataredescendantsofn.
Asufficientsetofconditionstoensureavalidjointdistributionincludes:
Definition2(Completeness[PoonandDomingos,2011]). AnSPNiscompleteifallchildrenofthesamesumnode
havethesamescope.
Definition 3 (Decomposability [Poon and Domingos, 2011]). An SPN is decomposable if all children of the same
productnodehavedisjointscopes.
Here decomposability allows us to interpret product nodes as computing factored distributions with respect to
disjoint sets of variables, which ensures that the product is a valid distribution over the union of the scopes of the
children.Similarly,completenessallowsustointerpretsumnodesascomputingamixtureofthedistributionsencoded
by the children since they all have the same scope. Each child is a mixture component with mixture probability
proportional to its weight. Hence, in complete and decomposable SPNs, the sub-SPN rooted at each node can be
interpreted as encoding an (un-normalized) joint distribution over its scope. We can use the function f to answer
inferencequerieswithrespecttothejointdistributionencodedbytheentireSPNasfollows:
• Marginalqueries: Pr(X=x)= froot(X=x)
froot(∅)
• Conditionalqueries: Pr(X=x|Y =y)= froot(X=x,Y=y)
froot(Y=y)
Unlikemostneuralnetworksthatcanansweronlyquerieswithfixedinputsandoutputs,SPNscananswerconditional
inference queries with varying inputs and outputs simply by changing the set of variables that are queried (outputs)
andconditionedon(inputs). Furthermore,SPNscanbeusedtogeneratedatabysamplingfromthejointdistributions
1Consecutivesumnodescanalwaysbemergedintoasinglesumnode.Similarly,consecutiveproductnodescanalwaysbemergedintoasingle
productnode.
2
theyencode. Thisisachievedbyatop-downpassthroughthenetwork. Startingattheroot, eachchildofaproduct
nodeisfollowed,asinglechildofasumnodeissampledaccordingtotheunnormalizeddistributionencodedbythe
weightsofthesumnodeandavariableassignmentissampledineachleafthatisreached. Thisisparticularlyuseful
innaturallanguagegenerationtasksandimagecompletiontasks[PoonandDomingos,2011].
Notealsothatinferencequeriescanbeansweredexactlyinlineartimewithrespecttothesizeofthenetworksince
eachqueryrequirestwoevaluationsofthenetworkfunctionf andeachevaluationisperformedinabottom-uppass
throughthenetwork. ThismeansthatSPNscanalsobeviewedasaspecialtypeoftractableprobabilisticgraphical
model, incontrasttoBayesianandMarkovnetworksforwhichinferenceis#P-hard[Roth,1996]. AnySPNcanbe
convertedintoanequivalentbipartiteBayesiannetworkwithoutanyexponentialblowup,whileBayesianandMarkov
networkscanbeconvertedintoequivalentSPNsattheriskofanexponentialblowup[Zhaoetal.,2015].
2.1 ParameterLearning
The weights of an SPN are its parameters. They can be estimated by maximizing the likelihood of a dataset (gen-
erativetraining)[PoonandDomingos,2011]ortheconditionallikelihoodofsomeoutputfeaturesgivensomeinput
features (discriminative training) by Stochastic Gradient Descent (SGD) [Gens and Domingos, 2012]. Since SPNs
are generative probabilistic models where the sum nodes can be interpreted as hidden variables that induce a mix-
ture, the parameters can also be estimated by Expectation Maximization (EM) [Poon and Domingos, 2011, Peharz,
2015]. ZhaoandPoupart[2016]providesaunifyingframeworkthatexplainshowlikelihoodmaximizationinSPNs
corresponds to a signomial optimization problem where SGD is a first order procedure, one can also consider a se-
quentialmonomialapproximationandEMcorrespondstoaconcave-convexprocedurethatconvergesfasterthanthe
othertechniques.SinceSPNsaredeeparchitectures,SGDandEMsufferfromvanishingupdatesandtherefore”hard”
variantshavebeenproposedtoremedytothisproblem[PoonandDomingos,2011,GensandDomingos,2012]. By
replacing all sum nodes by max nodes in an SPN, we obtain a max-product network where the gradient is constant
(hardSGD)andlatentvariablesbecomedeterministic(hardEM).ItisalsopossibletotrainSPNsinanonlinefashion
based on streaming data [Lee et al., 2013, Rashwan et al., 2016, Zhao et al., 2016, Jaini et al., 2016]. In particular,
it was shown that online Bayesian moment matching [Rashwan et al., 2016, Jaini et al., 2016] and online collapsed
variationalBayes[Zhaoetal.,2016]performmuchbetterthanSGDandonlineEM.
2.2 StructureLearning
SinceitisdifficulttospecifynetworkstructuresforSPNsthatsatisfythedecomposabilityandcompletenessproperties,
severalautomatedstructurelearningtechniqueshavebeenproposed[DennisandVentura,2012,GensandDomingos,
2013,Peharzetal.,2013,Leeetal.,2013,RooshenasandLowd,2014,Adeletal.,2015,Vergarietal.,2015,Rahman
and Gogate, 2016, Melibari et al., 2016]. The first two structure learning techniques [Dennis and Ventura, 2012,
Gens and Domingos, 2013] are top down approaches that alternate between instance clustering to construct sum
nodes and variable partitioning to construct product nodes. We can also combine instance clustering and variable
partitioning in one step with a rank-one submatrix extraction by performing a singular value decomposition [Adel
et al., 2015]. Alternatively, we can learn the structure of SPNs in a bottom-up fashion by incrementally clustering
correlatedvariables[Peharzetal.,2013]. ThesealgorithmsalllearnSPNswithatreestructureandunivariateleaves.
It is possible to learn SPNs with multivariate leaves by using a hybrid technique that learns an SPN in a top down
fashion, but stops early and constructs multivariate leaves by fitting a tractable probabilistic graphical model over
the variables in each leaf [Rooshenas and Lowd, 2014, Vergari et al., 2015]. It is also possible to merge similar
subtrees into directed acyclic graphs in a post-processing step to reduce the size of the resulting SPN [Rahman and
Gogate,2016]. Furthermore,Melibarietal.[2016]proposeddynamicSPNsforvariablelengthdataanddescribeda
search-and-scorestructurelearningtechniquethatdoesalocalsearchoverthespaceofnetworkstructures.
Sofar,allthesestructurelearningalgorithmsarebatchtechniquesthatassumethatthefulldatasetisavailableand
canbescannedmultipletimes. Leeetal.[2013]describesanonlinestructurelearningtechniquethatgraduallygrows
anetworkstructurebasedonmini-batches.ThealgorithmisavariantofLearnSPN[GensandDomingos,2013]where
the clustering step is modified to use online clustering. As a result, sum nodes can be extended with more children
whenthealgorithmencountersamini-batchthatisbetterclusteredwithadditionalclusters. Productnodesarenever
modifiedaftertheircreation.
SinceexistingstructurelearningtechniqueshaveallbeendesignedfordiscreteSPNsandhaveyettobeextended
tocontinuousSPNssuchasGaussianSPNs,thestateoftheartforcontinuous(andlargescale)datasetsistogenerate
3
a random network structure that satisfies decomposability and completeness after which the weights are learned by
a scalable online learning technique [Jaini et al., 2016]. We advance the state of the art by proposing a first online
structurelearningtechniqueforGaussianSPNs.
3 Proposed Algorithm
In this work, we assume that the leaf nodes all have Gaussian distributions. A leaf node may have more than one
variableinitsscope,inwhichcaseitfollowsamultivariateGaussiandistribution.
Suppose we want to model a probability distribution over a d-dimensional space. The algorithm starts with a
fullyfactorizedjointprobabilitydistributionoverallvariables,p(x)=p(x ,x ,...,x )=p (x )p (x )···p (x ).
1 2 d 1 1 2 2 d d
This distribution is represented by a product node with d children, the ith of which is a univariate distribution over
thevariablex . Therefore,initiallyweassumethatthevariablesareindependent,andthealgorithmwillupdatethis
i
probabilitydistributionasnewdatapointsareprocessed.
Given a mini-batch of data points, the algorithm passes the points through the network from the root to the leaf
nodesandupdateseachnodealongtheway. Thisupdateincludestwoparts:
• updatingtheparametersoftheSPN,and
• updatingthestructureofthenetwork.
3.1 Parameterupdate
Theparametersareupdatedbykeepingtrackofrunningsufficientstatistics. Therearetwotypesofparametersinthe
model: weightsonthebranchesunderasumnode,andparametersfortheGaussiandistributioninaleafnode.
Weproposeanewonlinealgorithmforparameterlearningthatissimplewhileensuringthataftereachupdate,the
likelihoodofthelastprocesseddatapointisincreased(similartostochasticgradientascent). Algorithm1describes
thepseudocodeofthisprocedure. Everynodeinthenetworkhasacount, n ,initializedto1. Whenadatapointis
c
received,thelikelihoodofthisdatapointiscomputedateachnode. Thentheparametersofthenetworkareupdatedin
arecursivetop-downfashionbystartingattherootnode.Whenasumnodeistraversed,itscountisincreasedby1and
thecountofthechildwiththehighestlikelihoodisincreasedby1. Thiseffectivelyincreasestheweightofthechild
withthehighestlikelihoodwhiledecreasingtheweightsoftheremainingchildren. Asaresult,theoveralllikelihood
atthesumnodewillincrease. Theweightw ofabranchbetweenasumnodesandoneofitschildrenccanthenbe
s,c
estimatedas
n
w = c (2)
s,c n
s
wheren isthecountofthesumnodeandn isthecountofthechildnode. Wealsorecursivelyupdatethesubtreeof
s c
thechildwiththehighestlikelihood. Inthecaseofties,wesimplychooseoneofthechildrenwithhighestlikelihood
atrandomtobeupdated.
Sincetherearenoparametersassociatedwithaproductnode,theonlywaytoincreaseitslikelihoodistoincrease
thelikelihoodateachofitschildren. Weincrementthecountateachchildofaproductnodeandrecursivelyupdate
thesubtreesrootedateachchild.
SinceeachleafnoderepresentsaGaussiandistribution,itkeepstrackoftheempiricalmeanvectorµandempirical
covariancematrixΣforthevariablesinitsscope. Whenaleafnodewithacurrentcountofnreceivesabatchofm
datapointsx(1),x(2),...,x(m),theempiricalmeanandempiricalcovarianceareupdatedaccordingtotheequations:
(cid:32) m (cid:33)
µ(cid:48) = 1 nµ +(cid:88)x(k) (3)
i n+m i i
k=1
and
(cid:34) m (cid:35)
Σ(cid:48) = 1 nΣ +(cid:88)(cid:16)x(k)−µ (cid:17)(cid:16)x(k)−µ (cid:17) −(µ(cid:48) −µ )(µ(cid:48) −µ ) (4)
i,j n+m i,j i i j j i i j j
k=1
whereiandj indexthevariablesintheleafnode’sscope, andµ(cid:48) andΣ(cid:48) arethenewmeanandcovarianceafterthe
update.
4
Algorithm1parameterUpdate(root(SPN),data)
Input: SPNandmdatapoints
Output: SPNwithupdatedparameters
n ←n +m
root root
ifisProduct(root)then
foreachchildofrootdo
parameterUpdate(child,data)
endfor
elseifisSum(root)then
foreachchildofrootdo
subset←{x∈data|likelihood(child,x)≥likelihood(child(cid:48),x)∀child(cid:48) ofroot}
parameterUpdate(child,subset)
wroot,child ← nroont+ch#ilcdh+il1dren
endfor
elseifisLeaf(root)then
updatemeanµ(root)basedonEq.3
updatecovariancematrixΣ(root)basedonEq.4
endif
This parameter update technique is related to, but different from hard SGD and hard EM used in [Poon and
Domingos,2011,GensandDomingos,2012,Leeetal.,2013]. HardSGDandhardEMalsokeeptrackofacountfor
thechildofeachsumnodeandincrementthosecountseachtimeadatapointreachesthischild. However,todecide
whenachildisreachedbyadatapoint,theyreplacealldescendantsumnodesbymaxnodesandevaluatetheresulting
max-productnetwork. Incontrast,weretainthedescendantsumnodesandevaluatetheoriginalsum-productnetwork
asitis. Thisevaluatesmorefaithfullytheprobabilitythatadatapointisgeneratedbyachild.
Alg. 1 does a single pass through the data. The complexity of updating the parameters after each data point is
linearinthesizeofthenetwork(i.e., #ofedges)sinceittakesonebottomuppasstocomputethelikelihoodofthe
data point at each node and one top-down pass to update the sufficient statistics and the weights. The update of the
sufficientstatisticscanbeseenaslocallymaximizingthelikelihoodofthedata. Theempiricalmeanandcovariance
oftheGaussianleaveslocallyincreasethelikelihoodofthedatathatreachthatleaf. Similarly,thecountratiosusedto
settheweightsunderasumnodelocallyincreasethelikelihoodofthedatathatreacheachchild. Weprovethisresult
below.
Theorem1. Letθ bethesetofparametersofanSPNs,andletf (·|θ )betheprobabilitydensityfunctionoftheSPN.
s s s
Givenanobservationx, supposetheparametersareupdatedtoθ(cid:48) basedontherunningaverageupdateprocedure,
s
thenwehavef (x|θ(cid:48))≥f (x|θ ).
s s s s
Proof. We will prove the theorem by induction. First suppose the SPN is just one leaf node. In this case, the
parametersaretheempiricalmeanandcovariance,whichisthemaximumlikelihoodestimatorforGaussiandistribu-
tion. Supposeθconsistsoftheparameterslearnedusingndatapointsx(1),...,x(n),andθ(cid:48)consistsoftheparameters
learnedusingthesamendatapointsandanadditionalobservationx. Thenwehave
n n n
(cid:89) (cid:89) (cid:89)
f (x|θ(cid:48)) f(x(i)|θ(cid:48))≥f (x|θ ) f (x(i)|θ )≥f (x|θ ) f (x(i)|θ(cid:48)) (5)
s s s s s s s s s s s
i=1 i=1 i=1
Thuswegetf (x|θ(cid:48))≥f (x|θ ).
s s s s
NowsupposewehaveanSPNswhereeachchildSPNtsatisfiesthepropertyf (x|θ(cid:48)) ≥ f (x|θ ). Iftherootof
t t t t
sisaproductnode,thenf (x|θ(cid:48))=(cid:81) f (x|θ(cid:48))≥(cid:81) f (x|θ )=f (x|θ ).
s s t t t t t t s s
Nowsupposetherootofsisasumnode. Letn bethecountofchildt,andletu = argmax f (x|θ ). Thenwe
t t t t
5
Figure1: Depictionofhowcorrelationsbetweenvariablesareintroducedinthemodel. Left: originalproductnode
withthreechildren. Middle:combineChild1andChild2intoamultivariateleafnode(Alg.4).Right:createamixture
tomodelthecorrelation(Alg.3).
have
(cid:32) (cid:33)
1 (cid:88)
f (x|θ(cid:48))= f (x|θ(cid:48))+ n f (x|θ(cid:48))
s s n+1 u u t t t
t
(cid:32) (cid:33)
1 (cid:88)
≥ f (x|θ )+ n f (x|θ ) byinductivehypothesis
n+1 u u t t t
t
(cid:32) (cid:33)
≥ 1 (cid:88)ntf (x|θ )+(cid:88)n f (x|θ )
n+1 n t t t t t
t t
1 (cid:88)
= n f (x|θ )
n t t t
t
=f (x|θ )
s s
3.2 Structureupdate
The simple online parameter learning described above can be easily extended to enable online structure learning.
Algorithm 2 describes the pseudocode of the resulting procedure called oSLRAU (online Structure Learning with
RunningAverageUpdate). Similartoleafnodes,eachproductnodealsokeepstrackoftheempiricalmeanvectorand
empiricalcovariancematrixofthevariablesinitsscope. Theseareupdatedinthesamewayastheleafnodes.
Initially,whenaproductnodeiscreated,allvariablesinthescopeareassumedindependent(seeAlgorithm5). As
new data points arrive at a product node, the covariance matrix is updated, and if the absolute value of the Pearson
correlationcoefficientbetweentwovariablesareaboveacertainthreshold,thealgorithmupdatesthestructuresothat
thetwovariablesbecomecorrelatedinthemodel.
We correlate two variables in the model by combining the child nodes whose scopes contain the two variables.
Thealgorithmemploystwoapproachestocombinethetwochildnodes:
• createamultivariateleafnode(Algorithm4),or
• createamixtureoftwocomponentsoverthevariables(Algorithm3).
ThesetwoprocessesaredepictedinFigure1. Ontheleft,aproductnodewithscopex ,...,x originallyhasthree
1 5
children. The product node keeps track of the empirical mean and empirical covariance for these five variables.
Supposeitreceivesamini-batchofdataandupdatesthestatistics. Asaresultofthisupdate, x andx nowhavea
1 3
correlationabovethethreshold.
Figure1illustratesthetwoapproachestomodelthiscorrelation.InthemiddleofFigure1,thealgorithmcombines
thetwochildnodesthathavex andx intheirscope,andturnsthemintoamultivariateleafnode. Sincetheproduct
1 3
node already keeps track of the mean and covariance of these variables, we can simply use those statistics as the
parametersforthenewleafnode.
6
Another way to correlate x and x is to create a mixture, as shown in the right part of Figure 1. The mixture
1 3
hastwocomponents. Thefirstcomponentcontainstheoriginalchildrenoftheproductnodethatcontainx andx .
1 3
Thesecondcomponentisanewproductnode,whichisagaininitializedtohaveafullyfactorizeddistributionoverits
scope(Alg.5). Themini-batchofdatapointsarethenpasseddownthenewmixturetoupdateitsparameters.
Notethatalthoughthechildrenaredrawnlikeleafnodesinthediagrams,theycaninfactbeentiresubtrees. Since
theprocessdoesnotinvolvetheparametersinachild,itworksthesamewayifsomeofthechildrenaretreesinstead
ofsinglenodes.
The technique chosen to induce a correlation depends on the number of variables in the scope. The algorithm
createsamultivariateleafnodewhenthecombinedscopeofthetwochildnodeshasanumberofvariablesthatdoes
notexceedsomethresholdandifthetotalnumberofvariablesintheproblemisgreaterthanthisthreshold,otherwise
it creates a mixture. Since the number of parameters in multivariate Gaussian leaves grows at a quadratic rate with
respecttothenumberofvariables,itisnotadvisedtoconsidermultivariateleaveswithtoomanyvariables.Incontrast,
themixtureconstructionincreasesthenumberofparametersatalinearrate, whichislesspronetooverfittingwhen
manyvariablesarecorrelated.
Tosimplifythestructure, ifaproductnodeendsupwithonlyonechild, itisremovedfromthenetwork, andits
onlychildisjoinedwithitsparent. Similarly,ifasumnodeendsupbeingachildofanothersumnode,thenthechild
sumnodecanberemoved,andallitschildrenarepromotedonelayerup.
Notethatthethisstructurelearningtechniquedoesasinglepassthroughthedataandthereforeisentirelyonline.
Thetimeandspacecomplexityofupdatingthestructureaftereachdatapointislinearinthesizeofthenetwork(i.e.,#
ofedges)andquadraticinthenumberoffeatures(sinceproductnodesstoreacovariancematrixthatisquadraticinthe
sizeoftheirscope). Thealgorithmalsoensuresthatthedecomposabilityandcompletenesspropertiesarepreserved
aftereachupdate.
Ouralgorithm(oSLRAU)isrelatedto,butdifferentfromtheonlinestructurelearningtechniqueproposedbyLee
etal.[2013]. Leeetal.’stechniquewasappliedtodiscretedatasetswhileoSLRAUlearnsSPNswithGaussianleaves
basedonreal-valueddata.Furthermore,Leeetal.’stechniqueincrementallyconstructsanetworkinatopdownfashion
by adding children to sum nodes by online clustering. Once a product node is constructed, it is never modified. In
contrast,oSLRAUincrementallyconstructsanetworkinabottomupfashionbydetectingcorrelationsandmodifying
productnodestorepresentthesecorrelations.Finally,Leeetal.’stechniqueupdatestheparametersbyhardEM(which
implicitly works with a max-product network) while oSLRAU updates the parameters by Alg. 1 (which retains the
originalsum-productnetwork)asexplainedintheprevioussection.
4 Experiments
Thissectiondiscussesourexperimentstoevaluatetheperformanceofourstructurelearningtechnique.2
4.1 Toydataset
Asaproofofconcept,wefirsttestthealgorithmonatoysyntheticdataset. Wegeneratedatafromthe3-dimensional
distribution
p(x ,x ,x )=[0.25N(x |1,1)N(x |2,2)+0.25N(x |11,1)N(x |12,2)
1 2 3 1 2 1 2
+0.25N(x |21,1)N(x |22,2)+0.25N(x |31,1)N(x |32,2)]N(x |3,3),
1 2 1 2 3
whereN(·|µ,σ2)isthenormaldistributionwithmeanµandvarianceσ2.
Therefore,thefirsttwodimensionsx andx aregeneratedfromaGaussianmixturewithfourcomponents,and
1 2
x isindependentfromtheothertwovariables.
3
Starting from a fully factorized distribution, we would expect x to remain factorized after learning from data.
3
Furthermore, thealgorithmshouldgeneratenewcomponentsalongthefirsttwodimensionsasmoredatapointsare
receivedsincex andx arecorrelated.
1 2
Thisisindeedwhathappens. Figure2showsthestructurelearnedafter200and500datapoints. Thevariablex
3
remainsfactorizedregardlessofthenumberofdatapointsseen,whereasmorecomponentsarecreatedforx andx
1 2
asmoredatapointsareprocessed.
2Thesourcecodeforouralgorithmisavailableatgithub.com/whsu/spn.
7
Algorithm2oSLRAU(root(SPN),data)
Input: SPNandmdatapoints
Output: SPNwithupdatedparameters
n ←n +m
root root
ifisProduct(root)then
updatecovariancematrixΣ(root)basedonEq.4
highestCorrelation←0
foreachc,c(cid:48) ∈children(root)wherec(cid:54)=c(cid:48)do
|Σ(root)|
correlation ←max ij
c,c(cid:48) i∈scope(c),j∈scope(c(cid:48)) (cid:113)
Σ(root)Σ(root)
ii jj
ifcorrelation >highestCorrelationthen
c,c(cid:48)
highestCorrelation←correlation
c,c(cid:48)
child ←c
1
child ←c(cid:48)
2
endif
endfor
ifhighest≥thresholdthen
if|scope(child )∪scope(child )|≥nVarsthen
1 2
createMixture(root,child ,child )
1 2
else
createMultivariateGaussian(root,child ,child )
1 2
endif
endif
foreachchildofrootdo
oSLRAU(child,data)
endfor
elseifisSum(root)then
foreachchildofrootdo
subset←{x∈data|likelihood(child,x)≥likelihood(child(cid:48),x)∀child(cid:48) ofroot}
oSLRAU(child,subset)
wroot,child ← nroont+ch#ilcdh+il1dren
endfor
elseifisLeaf(root)then
updatemeanµ(root)basedonEq.3
updatecovariancematrixΣ(root)basedonEq.4
endif
Figure3showsthedatapointsalongthefirsttwodimensionsandtheGaussiancomponentslearned. Wecansee
thatthealgorithmgeneratesnewcomponentstomodelthecorrelationbetweenx andx asitprocessesmoredata.
1 2
4.2 ComparisontootherAlgorithms
In a second experiment, we compare our algorithm to several alternatives on the same datasets used by Jaini et al.
[2016]. We use 0.1 as the correlation threshold in all experiments, and we use mini-batch sizes of 1 for the three
datasetswithfewestinstances(Quake,Banknote,Abalone),8forthetwoslightlylargerones(Kinematics,CA),and
256forthetwodatasetswithmostinstances(FlowSize,Sensorless).
Theexperimentalresultsforouralgorithmcalledonlinestructurelearningwithrunningaverageupdate(oSLRAU)
arelistedinTable1alongwithresultsreproducedfromJainietal.[2016]. Thetablereportstheaveragetestloglike-
lihoodswithstandarderroron10-foldcrossvalidation. oSLRAUachievedbetterloglikelihoodsthanonlineBayesian
moment matching (oBMM) [Jaini et al., 2016] and online expectation maximization (oEM) [Cappe´ and Moulines,
2009] with network structures generated at random or corresponding to Gaussian mixture models (GMMs). This
highlightsthemainadvantageofoSLRAU:learningastructurethatmodelsthedata. StackedRestrictedBoltzmann
Machines(SRBMs)[SalakhutdinovandHinton,2009]andGenerativeMomentMatchingNetworks(GenMMNs)[Li
8
Figure2: Learningthestructurefromthetoydatasetusingunivariateleafnodes. Left: after200datapoints. Right:
after500datapoints.
Figure3: Bluedotsarethedatapointsfromthetoydataset,andtheredellipsesshowthediagonalGaussiancompo-
nentslearned. Left: after200datapoints. Right: after500datapoints.
9
Algorithm3createMixture(root,child ,child )
1 2
Input: SPNandtwochildrentobemerged
Output: newmixturemodel
removechild andchild fromroot
1 2
component ←createproductnode
1
addchild andchild aschildrenofcomponent
1 2 1
n ←n
component1 root
jointScope←scope(child )∪scope(child )
1 2
Σ(component1) ←Σ(root)
jointScope,jointScope
component ←createFactoredModel(jointScope)
2
n ←0
component2
mixture←createsumnode
addcomponent andcomponent aschildrenofmixture
1 2
n ←n
mixture root
wmixture,component1 ← ncnommipxotnuernet+12+1
wmixture,component2 ← ncnommipxotnuernet+22+1
addmixtureaschildofroot
returnroot
Algorithm4createMultiVarGaussian(root,child ,child )
1 2
Input: SPN,twochildrentobemergedanddata
Output: newmultivariateGaussian
createmultiVarGaussian
jointScope←{scope(child )∪scope(child )}
1 2
µ(multiVarGaussian) ←µ(root)
jointScope
Σ(multiVarGaussian) ←Σ(root)
jointScope,jointScope
n ←n
multiVarGaussian root
returnmultiVarGaussian
Algorithm5createFactoredModel(scope)
Input: scope(setofvariables)
Output: fullyfactoredSPN
factoredModel←createproductnode
foreachi∈scopedo
addN (µ=0,σ=Σ(root))aschildoffactoredModel
i i,i
endfor
Σ(factoredModel) ←0
n ←0
factoredModel
returnfactoredModel
et al., 2015] are other types of deep generative models. Since it is not possible to compute the likelihood of data
pointswithGenMMNs,themodelisaugmentedwithParzenwindows. Morespecifically,10,000samplesaregener-
atedusingtheresultingGenMMNsandaGaussiankernelisestimatedforeachsamplebyadjustingitsparametersto
maximizethelikelihoodofavalidationset. However,aspointedoutbyTheisetal.[2015]thismethodonlyprovides
anapproximateestimateofthelog-likelihoodandthereforethelog-likelihoodreportedforGenMMNsinTable1may
notbedirectlycomparabletothelog-likelihoodofothermodels.
The network structures for GenMMNs and SRBMs are fully connected while ensuring that the number of pa-
rameters is comparable to those of the SPNs. oSLRAU outperforms these models on 5 datasets while SRBMs and
GenMMNs each outperform oSLRAU on one dataset. Although SRBMs and GenMMNs are more expressive than
SPNssincetheyallowothertypesofnodesbeyondsumsandproducts,trainingGenMMNsandSRBMsisnotoriously
10