On the Relationship between Sum-Product Networks and Bayesian Networks

0.53 MB·English

by  Han Zhao

On the Relationship between Sum-Product Networks and Bayesian Networks
HanZhao [email protected]
MazenMelibari [email protected]
PascalPoupart [email protected]
DavidR.CheritonSchoolofComputerScience,UniversityofWaterloo,Waterloo,ONN2L3G1,Canada
5 Abstract TherelationshipbetweenSPNsandBNs,andmorebroadly
1
withPGMs,isnotclear. SincetheintroductionofSPNsin
0 Inthispaper,weestablishsometheoreticalcon-
2 nectionsbetweenSum-ProductNetworks(SPNs) the seminal paper of Poon & Domingos (2011), it is well
understood that SPNs and BNs are equally expressive in
r andBayesianNetworks(BNs).Weprovethatev-
p thesensethattheycanrepresentanyjointdistributionover
erySPNcanbeconvertedintoaBNinlineartime
A discretevariables1,butitisnotclearhowtoconvertSPNs
andspaceintermsofthenetworksize. Thekey
into BNs, nor whether a blow up may occur in the con-
0 insight is to use Algebraic Decision Diagrams
version process. The common belief is that there exists
3 (ADDs) to compactly represent the local condi-
tionalprobabilitydistributionsateachnodeinthe a distribution such that the smallest BN that encodes this
] distribution is exponentially larger than the smallest SPN
I resultingBNbyexploitingcontext-specificinde-
A that encodes this same distribution. The key behind this
pendence(CSI).ThegeneratedBNhasasimple
beliefliesinSPNs’abilitytoexploitcontext-specificinde-
. directed bipartite graphical structure. We show
s
pendence(CSI)(Boutilieretal.,1996).
c that by applying the Variable Elimination algo-
[
rithm (VE) to the generated BN with ADD rep- WhiletheabovebeliefiscorrectforclassicBNswithtabu-

2 resentations, we can recover the original SPN larconditionalprobabilitydistributions(CPDs)thatignore
v wheretheSPNcanbeviewedasahistoryrecord CSI, and for BNs with tree-based CPDs due to the repli-
9 orcachingoftheVEinferenceprocess. Tohelp cationproblem(Pagallo,1989),itisnotclearwhetheritis
3
statetheproofclearly,weintroducethenotionof correct for BNs with more compact representations of the
2
1 normalSPNandpresentatheoreticalanalysisof CPDs.TheotherdirectionisclearforclassicBNswithtab-
0 the consistency and decomposability properties. ularrepresentation: givenaBNwithtabularrepresentation
. We conclude the paper with some discussion of ofitsCPDs,wecanbuildanSPNthatrepresentsthesame
1
0 theimplicationsoftheproofandestablishacon- jointprobabilitydistributionintimeandspacecomplexity
5 nectionbetweenthedepthofanSPNandalower thatmaybeexponentialinthetree-widthoftheBN.Briefly,
1 boundofthetree-widthofitscorrespondingBN. thisisdonebyfirstconstructingajunctiontreeandtrans-
v: lateitintoanSPN2.However,tothebestofourknowledge,
i it is still unknown how to convert an SPN into a BN and
X
1.Introduction whether the conversion will lead to a blow up when more
r
a compact representations than tables and trees are used for
Sum-Product Networks (SPNs) have recently been pro-
theCPDs.
posedastractabledeepmodels(Poon&Domingos,2011)
for probabilistic inference. They distinguish themselves We prove in this paper that by adopting Algebraic Deci-
fromothertypesofprobabilisticgraphicalmodels(PGMs), sionDiagrams(ADDs)(Baharetal.,1997)torepresentthe
includingBayesianNetworks(BNs)andMarkovNetworks CPDs at each node in a BN, every SPN can be converted
(MNs),bythefactthatinferencecanbedoneexactlyinlin- intoaBNinlineartimeandspacecomplexityinthesizeof
ear time with respect to the size of the network. This has theSPN.ThegeneratedBNhasasimplebipartitestructure,
generated a lot of interest since inference is often a core whichfacilitatestheanalysisofthestructureofanSPNin
task for parameter estimation and structure learning, and terms of the structure of the generated BN. Furthermore,
it typically needs to be approximated to ensure tractabil-
1Joint distributions over continuous variables are also possi-
ity since probabilistic inference in BNs and MNs is #P-
ble,butwewillrestrictourselvestodiscretevariablesinthispa-
complete(Roth,1996).
per.
2http://spn.cs.washington.edu/faq.shtml
OntheRelationshipbetweenSum-ProductNetworksandBayesianNetworks
we show that by applying the Variable Elimination (VE) knowledge compilation. In knowledge compilation, the
algorithm(Zhang&Poole,1996)tothegeneratedBNwith reasoning process is divided into two phases: an offline
ADDrepresentationofitsCPDs,wecanrecovertheorigi- compilationphaseandanonlinequery-answeringphase.In
nalSPNinlineartimeandspacewithrespecttothesizeof theofflinephase,theknowledgebase,eitherpropositional
theSPN. theory or belief network, is compiled into some tractable
target language. In the online phase, the compiled target
Ourcontributionscanbesummarizedasfollows. First,we
model is used to answer a large number of queries effi-
present a constructive algorithm and a proof for the con-
ciently. Thekeymotivationofknowledgecompilationisto
versionofSPNsintoBNsusingADDstorepresentthelo-
shiftthecomputationthatiscommontomanyqueriesfrom
cal CPDs. The conversion process is bounded by a lin-
the online phase into the offline phase. As an example,
earfunctionofthesizeoftheSPNinbothtimeandspace.
ACshavebeenstudiedandusedextensivelyinbothknowl-
Thisgivesanewperspectivetounderstandtheprobabilis-
edgerepresentationandprobabilisticinference(Darwiche,
tic semantics implied by the structure of an SPN through
2000;Huangetal.,2006;Chaviraetal.,2006). Rooshenas
thegeneratedBN.Second,weshowthatbyexecutingVE
&Lowd(2014)recentlyshowedthatACsandSPNscanbe
on the generated BN, we can recover the original SPN in
convertedmutuallywithoutanexponentialblow-upinboth
linear time and space complexity in the size of the SPN.
timeandspace. Asadirectresult,ACsandSPNssharethe
Combined with the first point, this establishes a clear re-
sameexpressivenessforprobabilisticreasoning.
lationship between SPNs and BNs. Third, we introduce
the subclass of normal SPNs and show that every SPN Another representation closely related to SPNs in propo-
can be transformed into a normal SPN in quadratic time sitional logic and knowledge representation is the
and space. Compared with general SPNs, the structure of deterministic-Decomposable Negation Normal Form (d-
normalSPNsexhibitmoreintuitiveprobabilisticsemantics DNNF) (Darwiche & Marquis, 2001). Propositional for-
andhencenormalSPNsareusedasabridgeintheconver- mulas in d-DNNF are represented by a directed acyclic
sionofgeneralSPNstoBNs. Fourth,ourconstructionand graph (DAG) structure to enable the re-usability of sub-
analysisprovidesanewdirectionforlearningtheparame- formulas. The terminal nodes of the DAG are literals and
ter/structureofBNssincetheSPNsproducedbythealgo- the internal nodes are AND or OR operators. Like SPNs,
rithms that learn SPNs (Dennis & Ventura, 2012; Gens & d-DNNF formulas can be queried to answer satisfiability
Domingos,2013;Peharzetal.,2013;Rooshenas&Lowd, andmodelcountingproblems. Wereferinterestedreaders
2014)canbeconvertedintoBNs. to Darwiche & Marquis (2001) and Darwiche (2001) for
moredetaileddiscussions.
2.RelatedWork Since their introduction by Poon & Domingos (2011),
SPNs have generated a lot of interest as a tractable class
Exact probabilistic reasoning has a close connection with
of models for probabilistic inference in machine learn-
propositional logic and weighted model counting (Roth,
ing. Discriminative learning techniques for SPNs have
1996;Gomesetal.,2008;Bacchusetal.,2003;Sangetal.,
been proposed and applied to image classification (Gens
2005). The model counting problem, #SAT, is the prob-
& Domingos, 2012). Later, automatic structure learn-
lemofcomputingthenumberofmodelsforagivenpropo-
ing algorithms were developed to build tree-structured
sitional formula, i.e., the number of distinct truth assign-
SPNsdirectlyfromdata(Dennis&Ventura,2012;Peharz
ments of the variables for which the formula evaluates to
etal.,2013;Gens&Domingos,2013;Rooshenas&Lowd,
TRUE. In its weighted version, each boolean variable X
2014). SPNs have also been applied to various fields and
has a weight Pr(x) [0,1] when set to TRUE and a
∈ have generated promising results, including activity mod-
weight 1 Pr(x) when set to FALSE. The weight of a
− eling(Amer&Todorovic,2012),speechmodeling(Peharz
truthassignmentistheproductoftheweightsofitsliterals.
et al., 2014) and language modeling (Cheng et al., 2014).
The weighted model counting problem then asks the sum
Theoreticalworkinvestigatingtheinfluenceofthedepthof
of the weights of all satisfying truth assignments. There
SPNsonexpressivenessexists(Delalleau&Bengio,2011),
are two important streams of research for exact weighted
but is quite limited. As discussed later, our results rein-
model counting and exact probabilistic reasoning that re-
force previous theoretical results about the depth of SPNs
latetoSPNs: DPLL-styleexhaustivesearch(Birnbaum&
andprovidefurtherinsightsaboutthestructureofSPNsby
Lozinskii, 2011) and those based on knowledge compila-
examiningthestructureofequivalentBNs.
tion, e.g., BinaryDecisionDiagrams(BDDs), Decompos-
ableNegationNormalForms(DNNFs)andArithmeticCir-
3.Preliminaries
cuits(ACs)(Bryant,1986;Darwiche,2001;2000).
The SPN, as an inference machine, has a close connec- Westartbyintroducingthenotationusedinthispaper. We
tionwiththebroaderfieldofknowledgerepresentationand use 1 : N to abbreviate the notation 1,2,...,N . We
{ }
OntheRelationshipbetweenSum-ProductNetworksandBayesianNetworks
use a capital letter X to denote a random variable and a gorithms to do probabilistic reasoning in BNs. See Wain-
boldcapitalletterX todenoteasetofrandomvariables wright&Jordan(2008)foracomprehensivesurvey.
1:N
X = X ,...,X . Similarly,alowercaseletterxis
1:N 1 N
{ }
used to denote a value taken by X and a bold lowercase 3.2.AlgebraicDecisionDiagram
letter x denotes a joint value taken by the correspond-
1:N
WefirstgiveaformaldefinitionofAlgebraicDecisionDi-
ing vector X of random variables. We may omit the
1:N
agrams (ADDs) for variables with Boolean domains and
subscript1 : N fromX andx ifitisclearfromthe
1:N 1:N
context. For a random variable X , we use xj,j 1 : J thenextendthedefinitiontodomainscorrespondingtoar-
i i ∈ bitraryfinitesets.
to enumerate all the values taken by X . For simplicity,
i
we use Pr(x) to mean Pr(X = x) and Pr(x) to mean Definition 1 (Algebraic Decision Diagram (Bahar et al.,
Pr(X = x). We use calligraphic letters to denote graphs 1997)). AnAlgebraicDecisionDiagram(ADD)isagraph-
(e.g., ). Inparticular,BNs,SPNsandADDsaredenoted ical representation of a real function with Boolean input
respecGtively by , and . For a DAG and a node v variables: f : 0,1 N R, where the graph is a rooted
B S A G { } (cid:55)→
in ,weuse todenotethesubgraphof inducedbyv DAG. There are two kinds of nodes in an ADD. Terminal
v
G G G
andallitsdescendants. LetV beasubsetofthenodesof nodes,whoseout-degreeis0,areassociatedwithrealval-
,then isasubgraphof inducedbythenodesetV. ues. Internal nodes, whose out-degree is 2, are associated
V
G G| G
Similarly,weuseX orx todenotetherestrictionofa with Boolean variables X ,n 1 : N. For each internal
|A |A n ∈
vectortoasubsetA. Weusenodeandvertex,arcandedge node Xn, the left out-edge is labeled with Xn = FALSE
interchangeably when werefer to a graph. Other notation andtherightout-edgeislabeledwithX =TRUE.
n
willbeintroducedwhenneeded.
We extend the original definition of an ADD by allowing
To ensure that the paper is self contained, we briefly re- ittorepresentnotonlyfunctionsofBooleanvariables,but
viewsomebackgroundmaterialaboutBayesianNetworks, also any function of discrete variables with a finite set as
AlgebraicDecisionDiagramsandSum-ProductNetworks. domain. This canbe doneby allowing eachinternal node
Readers who are already familiar with those models can X tohave out-edgesandlabeleachedgewithxj,j
skipthefollowingsubsections. 1 :n ,wh|Xerne| isthedomainofvariableX andn ∈
n n n n
|X | X |X |
isthenumberofvaluesX takes. SuchanADDrepresents
n
3.1.BayesianNetwork a function f : 1 N R, where means the
X ×···×X (cid:55)→ ×
Cartesian product between two sets. Henceforth, we will
Consideraproblemwhosedomainischaracterizedbyaset
useourextendeddefinitionofADDsthroughoutthepaper.
of random variables X with finite support. The joint
1:N
probabilitydistributionoverX canbecharacterizedby Forourpurpose,wewilluseanADDasacompactgraphi-
1:N
a Bayesian Network, which is a DAG where nodes repre- calrepresentationoflocalCPDsassociatedwitheachnode
senttherandomvariablesandedgesrepresentprobabilistic inaBN.Thisisakeyinsightofourconstructiveproofpre-
dependencies among the variables. In a BN, we also use sented later. Compared with a tabular representation or
theterms“node”and“variable”interchangeably. Foreach a decision tree representation of local CPDs, CPDs rep-
variable in a BN, there is a local conditional probability resented by ADDs can fully exploit CSI (Boutilier et al.,
distribution(CPD)overthevariablegivenitsparentsinthe 1996) and effectively avoid the replication problem (Pa-
BN. gallo,1989)ofthedecisiontreerepresentation.
ThestructureofaBNencodesconditionalindependencies WegiveanexampleinFig.1wherethetabularrepresenta-
amongthevariablesinit. LetX ,X ,...,X beatopo- tion, decision-tree representation and ADD representation
1 2 N
logical ordering of all the nodes in a BN3, and let π be ofafunctionof4Booleanvariablesispresented. Another
Xi
thesetofparentsofnodeX intheBN.Eachvariableina advantage of ADDs to represent local CPDs is that arith-
i
BNisconditionallyindependentofallitsnon-descendants meticoperationssuchasmultiplyingADDsandsumming-
given its parents. Hence, the joint probability distribution outavariablefromanADDcanbeimplementedefficiently
overX admitsthefactorizationinEq.1. inpolynomialtime. ThiswillallowustouseADDsinthe
1:N
VariableElimination(VE)algorithmtorecovertheoriginal
N N SPNafteritsconversiontoaBNwithCPDsrepresentedby
(cid:89) (cid:89)
Pr(X1:N)= Pr(Xi|X1:i−1)= Pr(Xi|πXi) ADDs. ReadersarereferredtoBaharetal.(1997)formore
i=1 i=1 detailedandthoroughdiscussionsaboutADDs.
(1)
Given the factorization, one can use various inference al-
3.3.Sum-ProductNetwork
3AtopologicalorderingofnodesinaDAGisalinearordering
ofitsnodessuchthateachnodeappearsafterallitsparentsinthis BeforeintroducingSPNs,wefirstdefinethenotionofnet-
ordering. work polynomial, which plays an important role in our
OntheRelationshipbetweenSum-ProductNetworksandBayesianNetworks
X1
X3 X2 X1
X
2
X4 0.3 X3 0.1
X01 X02 X03 X04 f0(.4·) X11 X02 X03 X04 f0(.4·) X3
0 0 0 1 0.6 1 0 0 1 0.6
0 0 1 0 0.3 1 0 1 0 0.3 0.4 0.6 0.1
0 0 1 1 0.3 1 0 1 1 0.3 X4 0.3 X4 0.3
0 1 0 0 0.4 1 1 0 0 0.1
0 1 0 1 0.6 1 1 0 1 0.1
0 1 1 0 0.3 1 1 1 0 0.1
0 1 1 1 0.3 1 1 1 1 0.1 0.4 0.6 0.4 0.6
(a) Tabularrepresentation. (b) Decision-Treerepresentation. (c) ADDrepresentation.
Figure1.DifferentrepresentationsofthesameBooleanfunction. ThetabularrepresentationcannotexploitCSIandtheDecision-Tree
representationcannotreuseisomorphicsubgraphs. TheADDrepresentationcanfullyexploitCSIbysharingisomorphicsubgraphs,
whichmakesitthemostcompactrepresentationamongthethreerepresentations. InFig.1(b)andFig.1(c),theleftandrightbranches
ofeachinternalnodecorrespondrespectivelytoFALSEandTRUE.
proof. WeuseI[X =x]todenoteanindicatorthatreturns nodehaschildrenwiththesamescope.
1whenX = xand0otherwise. Tosimplifythenotation, Definition5(Consistent). AnSPNisconsistentiffnovari-
wewilluseIxtorepresentI[X =x]. able appears negated in one child of a product node and
Definition 2 (Network Polynomial (Poon & Domingos, non-negatedinanother.
2011)). Let f() 0 be an unnormalized probability Definition 6 (Decomposable). An SPN is decomposable
· ≥
distribution over a Boolean random vector X1:N. The iff for every product node v, scope(vi) (cid:84) scope(vj) = ∅
network polynomial of f() is a multilinear function wherev ,v Ch(v),i=j.
(cid:80)xf(x)(cid:81)Nn=1Ixn ofindica·torvariables,wherethesum- i j ∈ (cid:54)
mation is over all possible instantiations of the Boolean Clearly,decomposabilityimpliesconsistencyinSPNs. An
randomvectorX . SPNissaidtobevalidiffitdefinesa(unnormalized)prob-
1:N
abilitydistribution. Poon&Domingos(2011)provedthat
Intuitively, the network polynomial is a Boolean expan- ifanSPNiscompleteandconsistent,thenitisvalid. Note
sion (Boole, 1847) of the unnormalized probability dis- thatthisisasufficient,butnotnecessarycondition. Inthis
tribution f(). For example, the network polynomial of a paper, we focus only on complete and consistent SPNs as
·
BN X1 → X2 is Pr(x1,x2)Ix1Ix2 +Pr(x1,x¯2)Ix1Ix¯2 + we are interested in their associated probabilistic seman-
Pr(x¯1,x2)Ix¯1Ix2 +Pr(x¯1,x¯2)Ix¯1Ix¯2. tics. ForacompleteandconsistentSPNS,eachnodev in
Definition 3 (Sum-Product Network (Poon & Domingos, definesanetworkpolynomialfv()whichcorrespondsto
S ·
2011)). ASum-ProductNetwork(SPN)overBooleanvari- thesub-SPNrootedatv. Thenetworkpolynomialdefined
ablesX isarootedDAGwhoseleavesaretheindicators bytherootoftheSPNcanthenbecomputedrecursivelyby
1:N
Ix1,...,IxN andIx¯1,...,Ix¯N andwhoseinternalnodesare takingaweightedsumofthenetworkpolynomialsdefined
sums and products. Each edge (v ,v ) emanating from a by the sub-SPNs rooted at the children of each sum node
i j
sum node v has a non-negative weight w . The value of and a product of the network polynomials defined by the
i ij
aproductnodeistheproductofthevaluesofitschildren. sub-SPNsrootedatthechildrenofeachproductnode. The
Thevalueofasumnodeis(cid:80) w val(v )where probabilitydistributioninducedbyanSPN isdefinedas
Ch(vi) are the children of vi vajn∈dCvha(vli()vj)ijis thejvalue of PrS(x) (cid:44) (cid:80)fxSf(Sx()x), where fS(·) is the neStwork polyno-
nodevj. ThevalueofanSPNS[Ix1,Ix¯1,...,IxN,Ix¯N]is mial defined by the root of the SPN S. An example of a
thevalueofitsroot. completeandconsistentSPNisgiveninFig.2.
ThescopeofanodeinanSPNisdefinedasthesetofvari-
4.MainResults
ables that have indicators among the node’s descendants:
For any node v in an SPN, if v is a terminal node, say, In this section, we first state the main results obtained in
an indicator variable over X, then scope(v) = X , else thispaperandthenprovidedetailedproofswithsomedis-
(cid:83) { }
scope(v)= v˜∈Ch(v)scope(v˜).Poon&Domingos(2011) cussionoftheresults. Tokeepthepresentationsimple,we
furtherdefinethefollowingpropertiesofanSPN: assumewithoutlossofgeneralitythatalltherandomvari-
Definition4(Complete). AnSPNiscompleteiffeachsum ablesareBooleanunlessexplicitlystated. Itisstraightfor-
OntheRelationshipbetweenSum-ProductNetworksandBayesianNetworks
Remark3. ThecombinationofTheorems1and3shows
+
that distributions for which SPNs allow a compact repre-
10 9
6 sentation and efficient inference, BNs with ADDs also al-
low a compact representation and efficient inference (i.e.,
× × × noexponentialblowup).
To make the upcoming proofs concise, we first define a
+ + + +
4 9 14 2 normal form for SPNs and show that every complete and
6 1 6 8 consistent SPN can be transformed into a normal SPN in
Ix1 Ix¯1 Ix2 Ix¯2 quadratic time and space without changing the network
polynomial. WethenderivetheproofswithnormalSPNs.
Figure2.AcompleteandconsistentSPNoverBooleanvariables NotethatweonlyfocusonSPNsthatarecompleteandcon-
X1,X2.ThisSPNisalsodecomposablesinceeveryproductnode sistent. Hence,whenwerefertoanSPN,weassumethatit
haschildrenwhosescopesdonotintersect. Thenetworkpolyno- iscompleteandconsistentwithoutexplicitlystatingthis.
mialdefinedby(therootof)thisSPNis:f(X1,X2)=10(6Ix1+
4Ix¯1)(6Ix2+14Ix¯2)+6(6Ix1+4Ix¯1)(2Ix2+8Ix¯2)+9(9Ix1+ 4.1.NormalForm
Ix¯1)(2Ix2 +8Ix¯2) = 594Ix1Ix2 +1776Ix1Ix¯2 +306Ix¯1Ix2 +
824Ix¯1Ix¯2andtheprobabilitydistributioninducedbySisPrS = ForanSPN ,letfS()bethenetworkpolynomialdefined
3559040Ix1Ix2 + 13757060Ix1Ix¯2 + 3350060Ix¯1Ix2 + 3852040Ix¯1Ix¯2. at the root oSf . Defi·ne the height of an SPN to be the
S
lengthofthelongestpathfromtheroottoaterminalnode.
ward to extend our analysis to discrete random variables Definition7. AnSPNissaidtobenormalif
withfinitesupport. ForanSPN ,let bethesizeofthe
1. Itiscompleteanddecomposable.
S |S|
SPN,i.e.,thenumberofnodesplusthenumberofedgesin
2. For each sum node in the SPN, the weights of the
thegraph. ForaBN ,thesizeof , ,isdefinedbythe
edges emanating from the sum node are nonnegative
B B |B|
sizeofthegraphplusthesizeofalltheCPDsin (thesize
andsumto1.
B
ofaCPDdependsonitsrepresentation,whichwillbeclear
3. Every terminal node in the SPN is a univariate dis-
fromthecontext). Themaintheoremsare:
tribution over a Boolean variable and the size of the
Theorem 1. There exists an algorithm that converts any scope of a sum node is at least 2 (sum nodes whose
completeanddecomposableSPN overBooleanvariables scopeisofsize1arereducedintoterminalnodes).
S
X1:N intoaBN withCPDsrepresentedbyADDsintime Theorem4. ForanycompleteandconsistentSPN ,there
bOu(tNion|Sa|n).dFurth=eBrOm(oNre,S)a.ndBrepresentthesamedistri- exi(cid:48)st=s aOn(orm2a)l.SPN S(cid:48) such that PrS(·) = PrS(cid:48)S(·) and
|B| |S| |S | |S|
As it will be clear later, Thm. 1 immediately leads to the Toshowthis,wefirstprovethefollowinglemmas.
followingcorollary:
Lemma 5. For any complete and consistent SPN over
S
Corollary 2. There exists an algorithm that converts any X , there exists a complete and decomposable SPN (cid:48)
1:N
S
complete and consistent SPN over Boolean variables over X1:N such that fS(x) = fS(cid:48)(x), x and (cid:48) =
X intoaBN withCPDsrepSresentedbyADDsintime O( 2). ∀ |S |
1:N B |S|
O(N 2). Furthermore, and representthesamedis-
|S| S B
tributionand =O(N 2). Proof. Let be a complete and consistent SPN. If it is
|B| |S| S
also decomposable, then simply set (cid:48) = and we are
Remark1. TheBN generatedfrom inTheorem1and
S S
B S done. Otherwise, let v ,...,v be an inverse topologi-
Corollary 2 has a simple bipartite DAG structure, where 1 M
calorderingofallthenodesin , includingbothterminal
all the source nodes are hidden variables and the terminal
S
nodesandinternalnodes,suchthatforanyv ,m 1:M,
nodesaretheBooleanvariablesX . m
1:N ∈
alltheancestorsofv inthegraphappearafterv inthe
m m
Remark 2. Assuming sum nodes alternate with product
ordering. Let v be the first product node in the order-
m
nodesinSPN ,thedepthof isproportionaltothemax-
ing that violates decomposability. Let v ,v ,...,v
imum in-degreSe of the nodesSin , which, as a result, is m1 m2 ml
be the children of v where m < m < <
B m 1 2
proportionaltoalowerboundofthetree-widthof . ···
m < m (due to the inverse topological ordering). Let
B l
Theorem3. GiventheBN withADDrepresentationof (v ,v ),i < j,i,j 1 : l be the first ordered pair of
B mi mj ∈ (cid:84)
CPDs generated from a complete and decomposable SPN nodes such that scope(cid:84)(vmi) scope(vmj) (cid:54)= ∅. Hence,
overBooleanvariablesX ,theoriginalSPN canbe let X scope(v ) scope(v ). Consider f and
S 1:N S ∈ mi mj vmi
recovered by applying the Variable Elimination algorithm f which are the network polynomials defined by the
vmj
to inO(N ). sub-SPNsrootedatv andv .
B |S| mi mj
OntheRelationshipbetweenSum-ProductNetworksandBayesianNetworks
Expand network polynomials f and f into a sum- Algorithm1DecompositionTransformation
vmi vmj
of-product form by applying the distributive law between Input: CompleteandconsistentSPN .
S
products and sums. For example, if f(X1,X2) = (Ix1 + Output: CompleteanddecomposableSPNS(cid:48).
9Ix¯1)(4Ix2+6Ix¯2),thentheexpansionoff isf(X1,X2)= 1: Let v1,v2,...,vM be an inverse topological ordering
4Ix1Ix2+6Ix1Ix¯2+36Ix¯1Ix2+54Ix¯1Ix¯2. SinceS iscom- ofnodesinS.
plete,thensub-SPNsrootedatvmi andvmj arealsocom- 2: form=1toM do
plete, which means that each monomial in the expansion 3: ifv isanon-decomposableproductnodethen
m
fovfmfjv.miSminucestXsha∈resthcoepsea(mvmeis)c(cid:84)opsec.opTeh(evmsajm),ethapenplieevsertoy 45:: ΩV(v←m){v←∈(cid:83)Siv(cid:54)=mjs|csocpoep(ev(mv)i)(cid:84)(cid:84)Ωs(cvompe)((cid:54)=vm∅j)}
monomial in the expansion of fvmi and fvmj must con- 6: SV ←Svm|V
tain an indicator variable over X, either Ix or Ix¯. Fur- 7: D(vm) descendantsofvm
←
thermore, since is consistent, then the sub-SPN rooted 8: fornodev V vm do
at vm is a(cid:81)lso conSsistent. Consider fvm = (cid:81)lk=1fvmk = 9: ifPa(v)\∈DS(vm\{)(cid:54)=(cid:81)}∅then
fvmifvmj k(cid:54)=i,jfvmk. Because vm is consistent, we 10: Createp←v⊗ X∈Ω(vm)∩scope(v)Ix∗
know that each monomial in the expansions of f and 11: Connectpto f Pa(v) D(v )
vmi ∀ ∈ \ m
f mustcontainthesameindicatorvariableofX,either 12: Disconnectvfrom f Pa(v) D(v )
vmj ∀ ∈ \ m
Ix orIx¯,otherwisetherewillbeatermIxIx¯ infvm which 13: endif
violates the consistency assumption. Without loss of gen- 14: endfor
erality, assume each monomial in the expansions of fvmi 15: fornodev ∈SV inbottom-uporderdo
andfvmj containsIx. Thenwecanre-factorizefvm inthe 16: Disconnectv˜∈Ch(v)∀scope(v˜)⊆Ω(vm)
followingway: 17: endfor
(cid:81)
18: Connect X∈Ω(vm)Ix∗ tovmdirectly
19: endif
20: endfor
fvm = (cid:89)l fvmk =I2xfvImxi fvImxj (cid:89) fvmk 2212:: DDeelleetteeaallllnporoddeuscutnnroedacehsawbiltehforoumt-dtehgereroeo0tofS
k=1 k(cid:54)=i,j 23: Contractallproductnodeswithout-degree1
=IxfvImxi fvImxj (cid:89) fvmk =Ixf˜vmif˜vmj (cid:89) fvmk
k(cid:54)=i,j k(cid:54)=i,j
(2) followingidentity:
 
wpfuohnteecrnteito,nwi.eeb.yu,sIfea2xcttho=erifzaIinxctgatnhIxdatofi˜unvmtdiifc(raof˜tvmomrjfv)vamrisiia(bdflevefimsnjae)rd.eaiEdsqet.mh2e- whfevrmeΩ=(vmX)∈(cid:44)(cid:89)Ω((cid:83)vmi,)j∈Ix1∗:l,i(cid:54)=k(cid:89)j=ls1co(cid:81)peX(∈vmΩ(iv)m∩f)sv∩cmsockoppee((vvmmkj))I,∗xi.(e3.),
meansthatinordertomakev decomposable,wecansim- Ω(v )istheunionofallthesharedvariablesbetweenpairs
m m
ply remove all the indicator variables Ix from sub-SPNs ofchildrenofvm andIx∗ istheindicatorvariableofX
∈
rooted at vmi and vmj and later link Ix to vm directly. Ω(vm)appearinginSvm. Basedontheanalysisabove,we
Such a transformation will not change the network poly- knowthatforeachX Ω(v )therewillbeonlyonekind
m
∈
nomialfvm(cid:84)asshownbyEq.2,butitwillremoveX from ofindicatorvariableIx∗ thatappearsinsideSvm,otherwise
scope(v ) scope(v ). Inprinciple, wecanapplythis v isnotconsistent.InLine6, isdefinedasthesub-
mi mj m Svm|V
transformationtoallorderedpairs(v ,v ),i<j,i,j SPNof inducedbythenodesetV,i.e.,asubgraphof
mi mj ∈ Svm
1:lwithnonemptyintersectionsofscope.However,thisis where the node set is restricted to V. In Lines 5-6,
Svm
notalgorithmicallyefficientandmoreimportantly,forlocal wefirstextracttheinducedsub-SPN from rooted
SV Svm
componentscontainingIxinfvm whicharereusedbyother at vm using the node set in which nodes have nonempty
nodesvn outsideofSvm,wecannotremoveIx fromthem intersectionswithΩ(vm). WedisconnectthenodesinSV
otherwisethenetworkpolynomialsforeachsuchv willbe fromtheirchildreniftheirchildrenareindicatorvariables
n
changedduetotheremoval.Insuchcase,weneedtodupli- ofasubsetofΩ(v )(Lines15-17). AtLine18,webuilda
m
catethelocalcomponentstoensurethatlocaltransforma- newproductnodebymultiplyingalltheindicatorvariables
tionswithrespecttof donotaffectnetworkpolynomials inΩ(v )andlinkittov directly. Tokeepunchangedthe
vm m m
f . WepresentthetransformationinAlg.1. Alg.1trans- networkpolynomialsofnodesoutside thatusenodes
vn Svm
forms a complete and consistent SPN into a complete in , we create a duplicate node p for each such node v
V
S S
anddecomposableSPN (cid:48). Informally,itworksusingthe andlinkptoalltheparentsofv outsideof andatthe
S Svm
OntheRelationshipbetweenSum-ProductNetworksandBayesianNetworks
sametimedeletetheoriginallink(Lines9-13). LetRbetherootof .
S
In summary, Lines 15-17 ensure that v is decomposable Base case. SPNs of height 0 are indicator variables
m
•
by removing all the shared indicator variables in Ω(v ). over some Boolean variable whose network polyno-
m
Line 18 together with Eq. 3 guarantee that f is un- mialsimmediatelysatisfyLemma6.
vm
changed after the transformation. Lines 9-13 create nec- Inductionstep. AssumeLemma6holdsforanySPN
•
essaryduplicatestoensurethatothernetworkpolynomials withheight k.ConsideranSPN withheightk+1.
≤ S
arenotaffected.Lines21-23simplifythetransformedSPN Weconsiderthefollowingtwocases:
tomakeitmorecompact. AnexampleisdepictedinFig.3 – TherootR of isaproductnode. Theninthis
S
toillustratethetransformationprocess. case the network polynomial f () for is de-
S
(cid:81) · S
finedasf = f . Wehave
S v∈Ch(R) v
+ + (cid:88) (cid:88) (cid:89)
f (x)= f (x ) (4)
S v scope(v)
vm vn vm vn x x v∈Ch(R) |
× × × p ×
(cid:89) (cid:88)
vm+1 +vm2 Ix3 v+m1 +vm2 × = fv(x|scope(v)) (5)
v∈Ch(R)x|scope(v)
(cid:89)
× × × × = 1=1 (6)
Ix1 Ix3 Ix2 Ix¯1 Ix1 Ix2 Ix3 Ix¯1 v∈Ch(R)
where x means that x is restricted to the
Figure3.TransformationprocessdescribedinAlg.1toconstruct |scope(v)
setscope(v). Eq.5followsfromthedecompos-
acompleteanddecomposableSPNfromacompleteandconsis-
abilityofRandEq.6followsfromtheinduction
tentSPN.Theproductnodev intheleftSPNisnotdecompos-
m
able.Inducedsub-SPNS ishighlightedinblueandS ishigh- hypothesis.
vm V
lightedingreen. v highlightedinredisreusedbyv whichis – The root R of is a sum node. The network
m2 n S (cid:80)
outsideSvm. Tocompensateforvm2, wecreateanewproduct polynomialisfS = v∈Ch(R)wR,vfv.Wehave
nodepintherightSPNandconnectittoindicatorvariableIx3 (cid:88) (cid:88) (cid:88)
andvm2.DashedgraylinesintherightSPNdenotedeletededges fS(x)= wR,vfv(x) (7)
andnodeswhilerededgesandnodesareaddedduringAlg.1.
x x v∈Ch(R)
(cid:88) (cid:88)
= w f (x) (8)
R,v v
WenowanalyzethesizeoftheSPNconstructedbyAlg.1.
v∈Ch(R) x
Foragraph , letV( )bethenumberofnodesin and
(cid:88)
let E( ) be Sthe numbSer of edges in . Note that inSLines = wR,v =1 (9)
S S
8-17 we only focus on nodes that appear in the induced v∈Ch(R)
SPN , which clearly has . Furthermore,
SV |SV| ≤ |Svm| Eq.8followsfromthecommutativeandassocia-
we create a new product node p at Line 10 iff v is reused
tivelawofadditionandEq.9followsbythein-
by other nodes which do not appear in . This means
Svm ductionhypothesis.
thatthenumberofnodescreatedduringeachiterationbe-
tween Lines 2 and 20 is bounded by V( ) V( ).
SV ≤ Svm
Line10alsocreates2newedgestoconnectptov andthe
Corollary7. ForanycompleteanddecomposableSPN
indicatorvariables.Lines11and12firstconnectedgestop S
over X that satisfies condition 2 of Def. 7, Pr () =
andthendeleteedgesfromv,hencethesetwostepsdonot 1:N S ·
f ().
yieldincreasesinthenumberofedges. Sotheincreasein S ·
thenumberofedgesisboundedby2V( ) 2V( ). Lemma 8. For any complete and decomposable SPN ,
CombiningincreasesinbothnodesandeSdVges≤,duringSevamch thereexistsanSPN (cid:48) wheretheweightsoftheedgesemS-
S
outeriterationtheincreaseinsizeisboundedby3 anating from every sum node are nonnegative and sum to
V
3 = O( ). TherewillbeatmostM = V( |S)ou|t≤er 1,andPrS()=PrS(cid:48)(), (cid:48) = .
|Svm| |S| S · · |S | |S|
iterations hence the total increase in size will be bounded
byO(M )=O( 2). Proof. Alg. 2 runs in one pass of to construct the re-
|S| |S| quired SPN (cid:48). We proceed to provSe that the SPN (cid:48) re-
oLveemrmXa1:N6.thFaotrsaatnisyfiecsocmopnldeitteioann2dodfeDcoemf.p7o,s(cid:80)abxlefSS(PxN)=S taunrdnethdabty (cid:48)AslSagt.is2fiessatcisofinedsitPiornS(cid:48)2(·o)f=DePf.r7S.(·I)t,i|sSc(cid:48)l|ea=Sr t|hSat|
1. (cid:48) = S because we only modify the weights of to
|S | |S| S
construct (cid:48) at Line 7. Based on Lines 6 and 7, it is also
S
Proof. We give a proof by induction on the height of . straightforward to verify that for each sum node v in (cid:48),
S S
OntheRelationshipbetweenSum-ProductNetworksandBayesianNetworks
Algorithm2WeightNormalization – R is a sum node with weights w ,...,w 0.
1 l
≥
Input: SPN Wehave
S
Output: SPN (cid:48) l
1: S(cid:48) ←S S fS(cid:48)(x)=(cid:88)wi(cid:48)fSi(cid:48)(x) (15)
2: val(Ix) 1, Ix i=1
← ∀ ∈S
3: Ltheetnvo1d,e.s..in,vM be an inverse topological ordering of =(cid:88)l wival(Ri) Pr(x) (16)
4: form=1toSM do i=1 (cid:80)lj=1wjval(Rj) Si
567::: ifvvwma(cid:48)li(svmas)u←mwvn(cid:80)mo,dvve∈vaCtlh(hve()vn,m)wvvm,vvCahl((vv) ) =(cid:88)i=l1 (cid:80)ljw=1ivwajlv(Rali()Rj)(cid:80)fxSfi(Sxi()x)
vm,v ← val(vm) ∀ ∈ m (17)
8: elseifv isaproductnodethen
m
9: val(vm)←(cid:81)v∈Ch(vm)val(v) =(cid:88)l wival(Ri) fSi(x) (18)
10: endif i=1 (cid:80)lj=1wjval(Rj)val(Ri)
11: endfor
(cid:80)l w f (x) f (x)
= (cid:80)lj=i=11wjivaSli(Rj) = (cid:80)xSfS(x)
the weights of the edges emanating from v are nonnega-
(19)
tive and sum to 1. We now show that PrS(cid:48)() = PrS().
· · =Pr(x) (20)
Using Corollary 7, PrS(cid:48)() = fS(cid:48)(). Hence it is suf- S
· ·
ficient to show that fS(cid:48)() = PrS(). Before deriving where Eqn. 16 follows from the induction hy-
· ·
a proof, it is helpful to note that for each node v ,
pothesis,Eq.18and19followfromthefactthat
(cid:80) ∈ S
val(v) = f (x ). Wegiveaproofbyin- (cid:80)
ductiononthxe|shcoepei(gvh)tovf |.scope(v) val(v)= x|scope(v)fv(x|scope(v)),∀v ∈S.
S
ThiscompletestheproofsincePrS(cid:48)()=fS(cid:48)()=PrS().
Basecase. SPNswithheight0areindicatorvariables · · ·
•
whichautomaticallysatisfyLemma8.
Inductionstep. AssumeLemma8holdsforanySPN
• GivenacompleteanddecomposableSPN ,wenowcon-
ofheight k.ConsideranSPN ofheightk+1.Let S
≤ S structandshowthatthelastconditioninDef.7canbesat-
Rbetherootnodeof without-degreel. Wediscuss
S isfiedintimeandspaceO( ).
thefollowingtwocases. |S|
Lemma 9. Given a complete and decomposable SPN ,
– R is a product node. Let R1,...,Rl be the thereexistsanSPN (cid:48)satisfyingcondition3inDef.7sucSh
icnhgildsruebn-SofPNRs.andBSy1,i.n.d.u,cStilobne, tAhelgc.or2rerseptounrnds- thatPrS(cid:48)()=PrS(S)and (cid:48) =O( ).
· · |S | |S|
(cid:48),..., (cid:48) that satisfy Lemma 8. Since R is a
S1 Sl Proof. We give a proof by construction. First, if is not
productnode,wehave
S
weight normalized, apply Alg. 2 to normalize the weights
l (i.e., the weights of the edges emanating from each sum
(cid:89)
fS(cid:48)(x)= fSi(cid:48)(x|scope(Ri)) (10) nodesumto1).
i=1
Now check each sum node v in in a bottom-up order.
l S
(cid:89) If scope(v) = 1, by Corollary 7 we know the network
= PSir(x|scope(Ri)) (11) pol|ynomial f|v is a probability distribution over its scope,
i=1
say, X . Reduce v into a terminal node which is a dis-
=(cid:89)l (cid:80) fSi(x|scope(Ri)) (12) tribut{ion}over X induced by its network polynomial and
i=1 x|scope(Ri)fSi(x|scope(Ri)) dmisocvoenanlelctthevufnroremacahlalbiltesncohdiledsrefnro.mThetolaostbtsatienp is(cid:48).toNorete-
(cid:81)l f (x ) S S
= i=1 Si |scope(Ri) (13) thatinthisstepwewillonlydecreasethesizeof ,hence
(cid:80)x(cid:81)li=1fSi(x|scope(Ri)) |S(cid:48)|=O(|S|). S
f (x)
= (cid:80)xSfS(x) =PSr(x) (14) ProofofThm.4. The combination of Lemma 5, 8 and 9
completestheproofofThm.4.
Eq.11followsfromtheinductionhypothesisand
Eq. 13 follows from the distributive law due to AnexampleofanormalSPNconstructedfromtheSPNin
thedecomposabilityof . Fig.2isdepictedinFig.4.
S
OntheRelationshipbetweenSum-ProductNetworksandBayesianNetworks
Since p is a decomposable product node, admits the
+ Sp
10 9 + abovefactorizationbythedefinitionofaproductnodeand
6 4 9
7 6 35 Corollary7.
35
× × × NormalForm
× × ×
Note that there may exist multiple paths from the root to
+ + + +
4 9 14 2 X1 X1 X2 X2 p in . Each such path admits the factorization stated in
6 1 6 8 (0.6,0.4) (0.9,0.1) (0.3,0.7) (0.2,0.8) S
Ix1 Ix¯1 Ix2 Ix¯2 Eq. 21. Eq. 21 explains two key insights implied by the
structureofanSPNthatwillallowustoconstructanequiv-
Figure4.TransformanSPNintoanormalform. Terminalnodes alent BN with ADDs. First, CSI is efficiently encoded by
whichareprobabilitydistributionsoverasinglevariablearerep- thestructureofanSPNusingProposition21. Second,the
resentedbyadouble-circle. DAG structure of an SPN allows multiple assignments of
hiddenvariablestosharethesamefactorization,whichef-
fectively avoids the replication problem presents in deci-
4.2.SPNtoBN
siontrees.
In order to construct a BN from an SPN, we require the
Based on the observations above and with the help of the
SPN to be in a normal form, otherwise we can first trans-
normal form for SPNs, we now proceed to prove the first
formitintoanormalformusingAlg.1and2.
main result in this paper: Thm. 1. First, we present the
Let be a normal SPN over X . Before showing how algorithm to construct the structure of a BN from in
to coSnstruct a corresponding BN1,:Nwe first give some intu- Alg.3. Inanutshell,Alg.3createsanobservaBblevariSable
itions. One useful view is to associate each sum node in
an SPN with a hidden variable. For example, consider a Algorithm3BuildBNStructure
sumnodev without-degreel. Since isnormal,we Input: normalSPN
hraalvlyes(cid:80)uglig=e1swt∈sitSh=at1weancdanwaiss≥oci0a,t∀eia∈hid1dSe:nl.disTchreistenraatun-- O1u:tpRut: BroNotBof=(BSV,BE)
← S
domvariableH withmultinomialdistributionPr (H = 2: ifRisaterminalnodeovervariableX then
v v v
i) = w ,i 1 : l for each sum node v . Therefore, 3: CreateanobservablevariableX
i
can be th∈ought as defining a joint prob∈abSility distribu- 4: V V X
Stion over X and H = H v ,visasumnode 5: elsBe ←B ∪{ }
1:N v
where X1:N are the obser{vable|var∈iabSles and H are th}e 6: foreachchildRiofRdo
hiddenvariables. Whendoing inferencewithan SPN,we 7: ifBNhasnotbeenbuiltforSRi then
implicitlysumoutallthehiddenvariablesHandcompute 8: RecursivelybuildBNStructureforSRi
Pr (x) = (cid:80) Pr (x,h). Associating each sum node in 9: endif
S h S
anSPNwithahiddenvariablenotonlygivesusaconcep- 10: endfor
tual understanding of the probability distribution defined 11: ifRisasumnodethen
byanSPN,butalsohelpstoelucidateoneofthekeyprop- 12: CreateahiddenvariableHR associatedwithR
erties implied by the structure of an SPN as summarized 13: V V HR
B ←B ∪{ }
below: 14: foreachobservablevariableX R do
∈S
15: (H ,X)
Proposition10. GivenanormalSPN ,letpbeaproduct BE ←BE ∪{ R }
S 16: endfor
node in with l children. Let v ,...,v be sum nodes
S 1 k 17: endif
whichlieonapathfromtherootof top. Then
S 18: endif
(cid:12)
Pr(x (cid:12)H =v∗,...,H =v∗)=
S |scope(p)(cid:12) v1 1 vk k Xin foreachterminalnodeoverXin (Lines2-4).For
B S
(cid:89)l PSr(x|scope(pi)(cid:12)(cid:12)(cid:12)Hv1 =v1∗,...,Hvk =vk∗) (21) aebaclehHinvteransasolcsuiamtednowdeithvvinaSn,dAblugi.ld3scdreiraetecsteadheiddgdeesnfvraormi-
i=1 H toallobservablevariablesX appearinginthesub-SPN
v
rootedatv (Lines11-17). TheBN createdbyAlg.3has
where Hv = v∗ means the sum node v selects its v∗th a directed bipartite structure withBa layer of hidden vari-
branchandx denotesrestrictingxbysetA,p istheith
|A i ablespointingtoalayerofobservablevariables. Ahidden
childofproductnodep.
variable H points to an observable variable X in iff X
B
appearsinthesub-SPNrootedatH in .
Proof. Considerthesub-SPN prootedatp. pcanbeob- S
S S
tainedbyrestrictingH = v∗,...,H = v∗,i.e.,going We now present Alg. 4 and 5 to build ADDs for each ob-
v1 1 vk k
fromtherootof alongthepathH =v∗,...,H =v∗. servablevariableX andhiddenvariableH in . Foreach
S v1 1 vk k B
OntheRelationshipbetweenSum-ProductNetworksandBayesianNetworks
Algorithm4BuildCPDusingADD,observablevariable the contraction of product nodes will not add more edges
Input: normalSPN ,variableX into X since the out-degree of each product node in the
S A
Output: ADD inducedsub-SPNmustbe1duetothedecomposabilityof
X
A
1: ifADDhasalreadybeencreatedfor andX then theproductnode. WeillustratetheapplicationofAlg.3,4
S
2: retrieveADDfromcache and5onthenormalSPNinFig.4,whichresultsintheBN
X
A ←
3: else withCPDsrepresentedbyADDsshowninFig.5.
B
4: R rootof
← S WenowshowthatPrS(x)=PrB(x) x.
5: ifRisaterminalnodethen ∀
6: decisionstumprootedatR Lemma11. GivenanormalSPN ,theADDsconstructed
7: elsAeXifR←isasumnodethen byAlg.4and5encodelocalCPDSsateachnodein .
B
8: CreateanodeH into
R X
A
9: foreachR Ch(R)do Proof. ItiseasytoverifythatforeachhiddenvariableH
i
∈
10: LinkBuildADD(SRi,X)asithchildofHR in B, AH represents a local CPD since AH is a decision
11: endfor stumpwithnormalizedweights.
12: elseifRisaproductnodethen
ForanyobservablevariableX in , letPa(X)betheset
13: Findchild suchthatX scope(R ) B
SRi ∈ i of parents of X. By Alg. 3, every node in Pa(X) is a
14: BuildADD( ,X)
AX ← SRi hidden variable. Furthermore, H, H Pa(X) iff there
15: endif ∀ ∈
exists one terminal node over X in that appears in the
16: store incache S
AX sub-SPN rooted at H. Hence given any joint assignment
17: endif
h of Pa(X), there will be a path in from the root to
X
A
aterminalnodethatisconsistentwiththejointassignment
Algorithm5BuildCPDusingADD,hiddenvariable oftheparents. Also,theleavesin containnormalized
X
A
Input: normalSPN ,variableH weightscorrespondingtotheprobabilitiesofX(seeDef.7)
S
Output: ADD H inducedbythecreationofdecisionstumpsoverXinLines
A
1: FindthesumnodeH in 5-6ofAlg.4.
S
2: decisionstumprootedatH in
H
A ← S Theorem12. ForanynormalSPN overX ,theBN
1:N
S B
constructedbyAlg.3,4and5encodesthesameprobability
hiddenvariableH,Alg.5builds H asadecisionstump4 distribution,i.e.,PrS(x)=PrB(x),∀x.
A
obtained by finding H and its associated weights in .
Consider ADDs built by Alg. 4 for observable variablSes Proof. Again, we give a proof by induction on the height
Xs. LetX bethecurrentobservablevariablewearecon- of .
S
sidering. Basically,Alg.4isarecursivealgorithmapplied
Base case. The height of SPN is 0. In this case,
toeachnodein whosescopeintersectswith X . There • S
will be a single terminal node over X and will
S { }
are three cases. If current node is a terminal node, then it S B
beasingleobservablenodewithdecisionstump
X
mustbeaprobabilitydistributionoverX. Inthiscasewe A
constructed from the terminal node by Lines 5-6 in
simplyreturnthedecisionstumpatthecurrentnode. Ifthe
Alg.4. ItisclearthatPr (x)=Pr (x), x.
S B
currentnodeisasumnode,thenduetothecompletenessof ∀
Induction step. Assume Pr (x) = Pr (x), x for
B S
,weknowthatallthechildrenofRsharethesamescope • ∀
any withheight k,where isthecorresponding
S
with R. We first create a node H corresponding to the S ≤ B
R BNconstructedbyAlg.3,4and5from . Consider
hidden variable associated with R into (Line 8) and S
X anSPN withheightk+1.LetRbetherootof and
A
recursively apply Alg. 4 to all the children of R and link S S
R ,i 1 : l be the children of R in . We consider
i
them to H respectively. If the current node is a product ∈ S
R thefollowingtwocases:
node,thenduetothedecomposabilityof ,weknowthat
– R is a product node. Let scope(R ) = X ,t
S t t
there will be a unique child of R whose scope intersects ∈
1 : l. Claim: there is no edge between
i
with X . We recursively apply Alg. 4 to this child and S
and ,i = j, where ( ) is the sub-SPN
{ } j i j
returntheresultingADD(Lines12-15). S (cid:54) S S
rooted at R (R ). If there is an edge, say, from
i j
Equivalently, Alg. 4 can be understood in the following vj to vi where vj j and vi i, then
∈ S ∈ S
way: we extract the sub-SPN induced by X and con- scope(vi) scope(vj) scope(Rj). On the
tract5 all the product nodes in it to obtain A{X}. Note that ohtahveer∅ha=nds⊆,coscpoep(vei()vi) s⊆c⊆opsec(oRpie)((cid:84)Ris)c.opSe(oRwj)e,
4Adecisionstumpisadecisiontreewithonevariable. which c(cid:54)ontradicts th⊆e decomposability of the
5Ingraphtheory,thecontractionofanodevinaDAGisthe
operation that connects each parent of v to each child of v and thendeletevfromthegraph.