Multicuts and Perturb & MAP for Probabilistic Graph Clustering

3.4 MB·

by  Jörg Hendrik Kappes

Multicuts and Perturb & MAP for Probabilistic Graph
Clustering
J¨org Hendrik Kappes Paul Swoboda Bogdan Savchynskyy Tamir
· · ·
Hazan Christoph Schn¨orr
·
6
1
0
2

n
a
Abstract We present a probabilistic graphical model From the viewpoint of statistics and Bayesian in-
J
formulation for the graph clustering problem. This en- ference, such Maximum-A-Posteriori (MAP) point es-
9
ables to locally represent uncertainty of image parti- timates have been always criticised as falling short of

tionsbyapproximatemarginaldistributionsinamath- thescopeofprobabilisticinference,thatistoprovide–
]
V ematically substantiated way, and to rectify local data alongwiththeMAPestimate–“errorbars”thatenable
C term cues so as to close contours and to obtain valid toassesssensitivitiesanduncertaintiesforfurtherdata
. partitions. analysis.Approachestothismoregeneralobjectiveare
s
WeexploitrecentprogressongloballyoptimalMAP less uniquely classified than for the MAP problem. For
c
[ inferencebyintegerprogrammingandonperturbation- example, a variety of approaches have been suggested

based approximations of the log-partition function, in from the viewpoint of clustering (see more comments
1
order to sample clusterings and to estimate marginal and references below) which, on the other hand, differ
v
8 distributions of node-pairs both more accurately and from the variational marginalisation problem in con-
8 moreefficientlythanstate-of-the-artmethods.Ourap- nection with discrete graphical models [42]. From the
0 proach works for any graphically represented problem computational viewpoint, these more general problems
2 instance. This is demonstrated for image segmentation are even more involved than the corresponding MAP(-
0
and social network cluster analysis. Our mathematical like) combinatorial inference problems.
.
1 ansatz should be relevant also for other combinatorial Inthispaper,weconsidergraphpartitioninginterms
0
problems. of the minimal cost multicut problem [12], also known
6
as correlation clustering in other fields [6], which in-
1 Keywords Correlation Clustering Multicut
: Graphical Models Perturb and MA·P · cludes modularity clustering [10], the image partition-
v · ingproblem[4]andothergraphpartitionproblems[32]
i
X as special case. Our work is based on
r 1 Introduction (i) recent progress [34,16] on the probabilistic anal-
a
ysis of perturbed MAP problems applied to our
Clustering,imagepartitioningandrelatedNP-hardde-
setting in order to establish mathematically the
cision problems abound in the fields image analysis,
connection to basic variational approximations of
computer vision, machine learning and data mining,
inference problems [42],
and much research has been done on alleviating the
(ii) recent progress on exact [22,23] and approxima-
combinatorial difficulty of such inference problems us-
tive[8]solversoftheminimumcostmulticutprob-
ingvariousformsofrelaxations.Arecentassessmentof
lem, which is required in connection with (i).
thestate-of-the-artusingdiscretegraphicalmodelshas
beenprovidedby[20].Asubsetofthespecificproblem Figure 1 provides a first illustration of our approach
instances considered there (Potts-like functional min- when applied for image partitioning. Instead of only
imisation) are closely related to continuous formula- calculating the most likely partition, our approach ad-
tions investigated, e.g., by [11,27]. ditionally provides alternative probable partitions and
returnsquantitativemeasuresofcertaintyofthebound-
ary parts.
2 J¨orgHendrikKappesetal.
ity,andtoexplicitlycomputealternativestothesingle
MAPsolution.Thelatterhasbeenachievedin[33]us-
ing a non-parametric Bayesian framework. Due to the
complexityofmodelevaluation,however,authorshave
to resort to MCMC sampling.
Concerning continuous problem formulations, a re-
markableapproachtoassess“errorbars”ofvariational
segmentations has been suggested by [36]. Here, the
startingpointisthe“smoothed”versionoftheMumford-
ShahfunctionalintermsoftherelaxationofAmbrosio
and Tortorelli [2] that is known to Γ-converge to the
Fig.1 Twoexamplesdemonstratingourapproach.Leftcol-
Mumford-Shah functional in the limit of correspond-
umn: images subject to unsupervised partitioning. Center
column:globallyoptimalpartitions.Rightcolumn:proba- ing parameter values. Authors of [36] apply a particu-
bilisticinferenceprovidedalongwiththepartition.Thecolor lar perturbation (“polynomial chaos”) that enables to
order:white yellow red black,togetherwithdecreas- locally infer confidence of the segmentation result. Al-
→ → →
ingbrightness,indicateuncertainty,cf.Fig.3.Wepointout
thoughbeingsimilarinscopetoourapproach,thisap-
thatalllocal informationprovidedbyourapproachisintrin-
sically non-locally inferred and relates to partitions, that is proach is quite different. An obvious drawback results
toclosed contours. from the fact that minima of the Ambrosio-Tortorelli
functional do not enforce partitions, i.e. may involve
contours that are not closed.
Although probabilistic image partitioning has been
motivating our work, the resulting approach is more Finally,we mention recent work [37] that addresses
widely applicable. This is demonstrated in the exper- the same problem using – again – a quite different ap-
imental section by analyzing a problem instance from proach:“stochastic”in[37]justreferstotherelaxation
the field of machine learning in terms of network data of binary indicator vectors to the probability simplex,
defined on a general graph. and this relaxation is solved by a local minimisation
method.
All the previous approaches require to select or op-
1.1 Related Work timize over an unknown number of clusters. This in-
troduces a bias into the model, as we will show later.
The susceptibility of clustering to noise is well known. Ourapproach,ontheotherhand,worksonanexponen-
This concerns, in particular, clustering approaches to tial family over edge-variables. It models clusterings in
image partitioning that typically employ spectral re- terms of multicuts, which inherently includes selection
laxation [39,19,28]. Measures proposed in the litera- ofthenumberofclusters.Samplesfromanapproxima-
ture[29,32]toquantitativelyassessconfidenceinterms tionofthisdistributionareobtainedasMAP-solutions
ofstability,employdataperturbationsandvariousforms of randomly perturbed partition problems.
of cluster averaging. While this is intuitively plausible,
atheoreticallymoreconvincingsubstantiationseemsto
be lacking, however.
In [18], a deterministic annealing approach to the
unsupervised graph partitioning problem (called pair- 1.2 Basic Notation
wise clustering) was proposed by adding an entropy
term weighted by an artificial temperature parameter. For the set of natural numbers from 1 to k we use the
Unlike the simpler continuation method of Blake and shorthand [k] and denote the cardinality of a set A by
Zisserman[9],thiswayofsmoothingthecombinatorial A.ForasetAwedenoteby[V]k thesetofk-element
| |
partitioning problem resembles the variational transi- subsets of A and for the sets A and A by [A ,A ]
1 2 1 2
tion from marginalisation to MAP estimation, by ap- the set a ,a a A ,a A . By 2A we denote
1 2 1 1 2 2
{{ }| ∈ ∈ }
plying the log-exponential function to the latter ob- the power set of A, which is defined as the set of all
jective [42]. As in [9], however, the primary objective subsetsofA.Theinnerproductofvectorsisdenotedby
of[18]istocomputeasingle“good”localoptimumby , ,andtheindicatorfunctionI(expression)is1ifthe
h· ·i
solvingasequenceofincreasinglynon-convexproblems expressionistrueand0otherwise.Weusetheclassical
parametrised by an artificial temperature parameter, notation for a undirected graph G=(V,E) where V is
rather than sampling various “ground states” (close to asetofnodesandE [V]2isasetofedges.Thedegree
⊂
zero-temperature solutions) in order to assess stabil- ofanodevisgivenbydeg(v):= u V : u,v E .
|{ ∈ { }∈ }|
MulticutsandPerturb&MAPforProbabilisticGraphClustering 3
1.3 Organization variablesx , i V.Fromanoptimizerx of (3)wecan
i ∗
∈
getanoptimizerS of (2)bycalculatingtheconnected
∗
The remaining paper is organized as follows. We will components on G =(V, ij E: x =x ).
0 i j
{ ∈ }
give the formal definition of partitioning, the minimal A major drawback of this formulation is the huge
costmulticutproblemanditspolyhedralrepresentation space needed to represent the assignments. Further-
inSec.2.Thisisfollowedbythedefinitionofprobabilis- more,duetothelackofanexternalfield(unaryterms),
tic distributions over partitions and methods that esti- anypermutationofanoptimalassignmentresultsinan-
mate marginals and generate samples in Sec. 3. A de- other optimal labeling. As a consequence of this sym-
tailednumericalevaluationonsyntheticandrealworld metry,thestandardrelaxationintermsoftheso-called
data is given in Sec. 4. localpolytope[42]becomestooweakandcannothan-
dle the necessary non-local constraints, cf. Sec.3.1.
In order to overcome these problems, we adopt an
2 Graph Partitioning and Multicuts
alternativerepresentationofpartitionsbasedontheset
of inter cluster edges as suggested in [12]. We call the
Theminimalcostmulticutproblem,alsoknownascor-
edge set
relation clustering, is defined in terms of partitions of
an undirected weighted graph δ(S ,...,S ):= [S ,S ] E (4)
1 k i j
∩
G=(V,E,w), V =[n], E [V]2, (1a) i6=j,[i,j∈[k]
⊆ amulticutassociatedwiththepartitionS = S ,...,S .
w: E R, e we :=w(e) (1b) { 1 k}
→ 7→ To obtain a polyhedral representation of multicuts, we
withasignededge-weightfunctionw.Apositiveweight defineforeachsubsetE0 Eanindicatorvectorsχ(E0)
⊆ ∈
we > 0, e ∈ E indicates that it is beneficial to put {0,1}|E| by
the two nodes into the same cluster, whereas a neg-
1, if e E ,
ative weight indicates that it is beneficial to separate χe(E0):= ∈ 0
them. We formally define below valid partitions and (0, if e E E0.
∈ \
interchangeably call them clusterings.
Themulticutpolytope (G)thenisgivenbythecon-
MC
Definition 1 (partition, clustering) A set of sub- vex hull
sets S , ...,S , called shores, components or clus-
{ 1 k} (G):=conv χ δ(S) : S (G) . (5)
ters, is a (valid) partition of a graph G = (V,E,w) iff MC ∈S
(a) S V, i [k], (b) S = , i [k], (c) the in- Theverticesofth(cid:8)is(cid:0)polyt(cid:1)opearethein(cid:9)dicatorfunctions
i i
⊆ ∈ 6 ∅ ∈
duced subgraphs G := S ,[S ]2 E are connected, of valid partitions and denoted by
i i i
∩
(d) S =V,(e)S S = , i,j [k], i=j.The
set ofia∈l[lk]valiid partitionsi(cid:0)∩of Gj is∅denot(cid:1)∈ed by 6(G). Y(G):= χ δ(S) : S ∈S(G) . (6)
S S
Based(cid:8)on(cid:0)this(cid:1)representatio(cid:9)n,theminimalcostmul-
The number (G) of all possible partitions is upper-
|S | ticut problem amounts to find a partition S (G)
boundedbytheBellnumber[1]thatgrowsveryquickly ∈ S
that minimizes the sum of the weights of edges cut by
with V .
| | the partition
Definition 2 (minimalcostmulticutproblem)The
argmin w χ (δ(S))
correlationclustering orminimalcostmulticutproblem S∈S(G) e· e
e E
istofindapartitionS thatminimizesthecostofintra X∈
∗
argmin w y . (7)
cluster edges as defined by the weight function w. ≡ y∈MC(G) e· e
e E
X∈
S
| | ThisproblemisknowntobeNP-hard[6]andmore
S∗ ∈argmin wij I(i∈Sk∧j 6∈Sk) (2) over APX-hard [13]. Although problem (7) is a linear
S∈S(G) iXj∈E kX=1 program, the representation of the multicut polytope
The minimal cost multicut problem can be formu- (G) by half-spaces is of exponential size and more-
MC
lated as a node labeling problem given by the problem over,unlessP =NP,noefficientseparationprocedure
of minimizing a Potts model for the complete multicut polytope exist [14].
However, one can develop efficient separation pro-
x∗ ∈argmin wijI(xi 6=xj). (3) cedures for an outer relaxation of the multicut poly-
x∈V|V| iXj∈E tope which involves all facet-defining cycle inequali-
Since any node can form its own cluster, V labels are ties. Together with integrality constraints, this guar-
| |
neededtorepresentallpossibleassignmentsintermsof antees globally optimal solutions of problem (7) and is
4 J¨orgHendrikKappesetal.
still applicable for real world problems [21,20]. For the by the right-hand side of (12),
tractability of huge models and to provide better any-
time performance, several greedy move-making meth- −H(pθ(µ)) if µ∈MC◦(G)
ods [24,5,8,7] has been suggested. These methods are A∗(µ)=nlim −H(pθ(µn))if µ∈MC(G)\MC◦(G)
able to find nearly optimal solutions for larger models  →∞ if µ (G)
∞ 6∈MC
much faster.
Here ◦(G) is the relative interior of (G), and
MC MC
µn asequencein ◦(G)thatconvergestoµ.Thelog
MC
3 Probabilistic Graph Partitioning partition function has the variational representation
Additionally to the most likely partitioning, one might A(θ)= sup {hθ,µi−A∗(µ)} (13)
µ (G)
also be interested in the probability of partitions and ∈MC
theprobabilitythatacertainedgeispartofacut.For and the supremum in Eq. (13) is attained uniquely
this we define for a given graph G a probability dis- at the vector µ (G)1 specified by the moment-
∈ MC
tribution over all partitions in terms of an exponential matching condition µ=Eθ(y).
family Thus, calculating the marginals amounts to solving
theconvexproblem(13).Thisisatleastashardassolv-
w
p(yθ):=exp( θ,y A(θ)) θ:= − (8) ingtheMAP-inferenceproblem,whichrequirestosolve
| h i− T
alinearprogram,sinceboththeconvexset (G)and
MC
the entropy A (µ) are not tractable in general.
A(θ):=ln exp( θ,y ) , (9) − ∗
 h i  Wewilldiscussnexthowtoapproximatethisprob-
y∈XY(G) lem by variational approaches and perturb & MAP.
 
where w R|E| is the vector due to the edge weights
∈
(1b) and (G) is given by (6). T 0 is called temper-
Y ≥ 3.1 Variational Approach
ature parameter. This exponential family differs from
usual discrete exponential families in the log partition
Sincesolvingproblem(13)isNP-hard,weneedtractable
functionA(θ).Insteadofcalculatingthesumover 0,1 E ,
{ }| | approximations of (G) and of the negative entropy
asitisusualfordiscretegraphicalmodels,weimplicitly MC
A (µ).Commonvariationalapproximationsforuncon-
take into account the topological constraints encoded ∗
strainedgraphicalmodelsconsiderouterrelaxationson
by the multicut polytope by restricting the feasible set
the feasible set and tractable approximations on the
to (G).
Y entropy in terms of pseudo-marginals, which are then
WhiletheMAP-inferenceproblemconsiderstofind
solved by message passing algorithms.
y that maximizes p(yθ), it is also useful to do proba-
∗ | Although formulation (3) has been used before in
bilistic inference and calculate the probability that an
the literature (e.g., [38]), we have to cope with several
edgee Eiscutandthuscontributestoseparatingdif-
∈ particularissues.Firstly,contraryto[38]ourmodelin-
ferentclusters.Thisprobabilityisgivenbythemarginal
cludesalsonegativecouplings,whichrenderstheprob-
distributions
lemmorechallenging.Secondly,thelabel-spacecanbe-
p(ye θ):=Eθ(ye)= p(yθ). (10) come very large when using a formulation in the node-
| y0∈Y(XG),ye0=ye | domain,asdonein(3)2.Finally,themostcrucialprob-
lem is that there is not a one-to-one correspondence
Sincep(yθ)isanexponentialfamilyweknow[42]that
| between a clustering and a node-labeling of the nodes.
A(θ) is a convex function of θ and
Forexample,ifwehavek labels,thenknode-labelings
∂A(θ) existstorepresenttheclusteringintooneclass.Onthe
Eθ(ye)= ∂θ , (11) other hand, any clustering into 2 clusters can be mod-
e
eled by k (k 1) node-labelings. As a consequence
The (Legendre-Fenchel) conjugate function of A(θ) is · −
wewouldtakesomepartitionsmoreoftenintoaccount
given by
thanothersduringmarginalisation.Thisambiguitycan
A∗(µ)=θ∈suRp|E| <θ,µ>−A(θ) (12) of1 For(Gθ)∈areRNonlwyerehaacvheedµw∈heMn aCt◦l(eGa)s.tBonoeunednatrryy opfoiθntiss
MC
and takes values in R . we call µ R|E| dual vari- infinity.
∪∞ ∈ 2 Toovercomethisproblemwewillrestrictthenumberof
ables.Theorem3.4in[42]givesanalternativeinterpre-
possible labels and exclude thereby some partitions. If the
tationofA (µ)asthenegativeentropyofthedistribu-
∗ numberofusedlabelsisgreaterorequalthanthechromatic
tion pθ(µ) with Eθ(µ)[y]=µ, where θ =θ(µ) is defined numberofthegraphallpartitionsarerepresentable.
MulticutsandPerturb&MAPforProbabilisticGraphClustering 5
bepartiallyreducedbyassigningthelabel1tothefirst
3 1 3 1
node w.l.o.g. as suggested in [38], however ambiguities 0
betweenremaininglabelsarestillpresentifk>2.Even
0 0 0 0
exact probabilistic inference by the junction tree algo-
2 2
rithm(JTA-n)thendoesnotprovidecorrectmarginals,
cf. Figs. 4 and 5. When using approximative inference, p(y12)=p(y23)=0.5 p(y12)=p(y23)=0.6
in this case loopy belief propagation (LBP), the error
(a)
increases.
To avoid these ambiguities, we switch to graphical
4 4
modelsintheedgedomain,asdefinedbyEq.(8).Asan
0 0
outer relaxation of the multicut polytope we consider
0 0
the cycle-multicut polytope (G). It is defined as
C
MC 3 1 3 1
theintersectionofhalf-spacesgivenbythecordlesscy- 0
cles of G denoted by C(G) 2E:
⊂ 0 0 0 0
MCC(G)={[0,1]|E| |∀e∈C ∈C(G): ye0 ≥ye}. 2 2
e0∈XC\{e} p(y12) = p(y23) = p(y12) = p(y23) =
As shown by Chorpa [12] we have (G)= (G) p(y14)=p(y34)= 172 p(y14)=p(y34)= 183
C
0,1 E .So (G)isaefficientlyYseparabMleCouterre∩- (b)
| | C
{ } MC
laxation of (G) and consequently an outer relaxation
Y Fig. 2 Adding zero weighted edges to a graph can change
for (G), too. Accordingly, we define an alternative thelikelihoodofpartitionsandmarginaldistributionforcut
MC
exponential family edges.Inexample(a),theadditionaledgedestroystheinde-
pendenceofrandomvariablesforedge-cutsbyenforcingtopo-
p(yθ¯):=exp θ¯,φ(y) A(θ¯) (14) logicalconsistencyandforbidstocutonlyoneofthe3edges.
| h i− This causes a bias towards a cut. Similarly, in example (b),
(cid:0) (cid:1) the triangulation of the graph causes a bias towards cutting
A(θ¯):=ln exp θ¯,φ(y) (15) edges.Thecut(1,1,1,1)intheleftgraphhastwovalidcorre-
 h i  spondingcuts((1,1,1,1,0),(1,1,1,1,1))intherightgraph.
y∈{X0,1}|E| (cid:0) (cid:1)
 
with the exponential parameter θ¯ and the sufficient
not tractable. To overcome this problem, we add zero-
statistic φ(y) both in R|E|+ C∈C(G)2|C| and defined by weighted edges in order to make the graph chordal.
P
w While the corresponding factor graph has then order
θ¯ = − e (16)
e T three, an additional bias is introduced into the model,
0 , if e C : y y cf. Fig. 2.
θ¯C,yC = , oth∀er∈wise e0∈C\{e} e0 ≥ e Theadvantageofthisansatzisthateq.(14)-(19)de-
(cid:26)−∞ P fineadiscretegraphicalmodel,forwhichseveralmeth-
(17)
ods exists to estimate the marginals. This includes the
φ(y) =y (18)
e e junction tree algorithm (JTA) [26] for exact inference
φ(y)C,yc0 =I(yC0 ≡yC) (19) ongraphswithsmalltree-width,aswellasloopybelief
propagation (LBP) [25], tree re-weighted belief prop-
Contrary to the formulation in Eq. (8), the constraints
agation (TRBP) [41], and generalized belief propaga-
areencodedintheexponentialparameter,whichisthat
tion (GBP) [43] for approximative inference by mes-
why marked by an additional bar.
sage passing. In addition to the above-mentioned bias
Itisworthtomentionthatthedimensionoftheex-
fornon-chordalgraphs,theapproximativemethodssuf-
ponential parameter vector grows exponentially with
fer from the combinatorial nature of the higher-order
the length of the longest non-chordal cycle. If G is
terms (17) that can not be dealt with by local mes-
chordal3,thenC(G)includesonlyapolynomialnumber
sages, cf. Sec. 4.1.3.
of cycles of length three, which can be easily enumer-
ated. If G is not chordal, we have to deal factors of
high order, i.e. the objective includes terms that de- 3.2 Perturbation & MAP
pend on more than two variables, which is practically
Recently, Hazan and Jaakkola [16] showed the connec-
3 Agraphiscalledchordal,ifeachcycleoflengthlarger3
tionbetweenextremevaluestatisticsandthepartition
have a chord, which is an edge that is not part of the cycle
butconnectstwoverticesofthecycle. function,basedonthepioneeringworkofGumbel[15].
6 J¨orgHendrikKappesetal.
In particular they provided a framework for approx- It has been reported [35,16,17] that this distribution p˜
imating and bounding the partition function as well isagoodapproximationforp,butcontrarytothefully
as creating samples by using MAP-inference with ran- perturbedmodelitisnotknownsofarhowtoexplicitly
domly perturbed models. representp˜asmemberoftheexponentialfamilyofdis-
Analytic expressions for the statistics of a random tributions. This precludes a analytic evaluation of this
MAP perturbation can be derived for general discrete approximation.
sets, whenever independent and identically distributed
randomperturbationsareappliedtoeveryassignment.
3.3 Perturb & MAP for Graph Partitioning
Theorem 1 ([15])Givenadiscretedistributionp(z)=
In the following we will review the possible sampling
exp(θ(z) A(θ))withz andθ: R ,let schemes with Perturb & MAP, based on Sec. 3.2, for
− ∈Z Z → ∪{−∞}
Γ be a vector of i.i.d. random variables Γ indexed by
z the graph partition problem described in Sec. 2. Ac-
z ,eachfollowingtheGumbeldistributionwhosecu-
cordingly, we denote again the random variables by y
∈Z
mulativedistributionfunctionisF(t)=exp exp( (t+
instead of z.
− −
c)) (here c is the Euler-Mascheroni constant). Then
(cid:0)
Pr(cid:1)zˆ=argmax θ(z)+Γ =exp θ(zˆ) A(θ) , 3.3.1 Unbiased Sampling
z
z { } −
∈Z
(cid:2) E max θ(z)+Γz (cid:3)=A(θ)(cid:0). (cid:1) Based on Theorem 1 we can define an exact sampler
z { }
∈Z forthedistributionsofpartitionsforsmall graphs.For
(cid:2) (cid:3)
Theorem1offersaprincipledwaybasedonsolvingthe each possible partition we have to add a random vari-
MAP problem for randomly perturbed model parame- able. The number of possible partitions of N elements
ters,tocomputethelogpartitionfunction(13)inview is given by the Bell number. In our setting it can be
of computing the marginals (11) as our objective. For smaller,sincethepartitionhasalsotorespectthegraph
larger problems, however, the number of states is too topology and single clusters have to be connected. In
large and thus Thm. 1 not directly applicable. Hazan prior work [23] we showed how to include such higher-
and Jaakkola [16] developed computationally feasible order terms into the MAP-inference. Sampling a parti-
approximations and bounds of the partition function tion from p(yθ) reduces to solving
|
basedonlow-dimensionalrandomMAPperturbations.
argmax θ,y + γ γ Gumbel(c,1), (22)
y y
Theorem 2 ([16])Givenadiscretedistributionp(z)= y Y(G)h i ∼
exp(θ(z) A(θ)) with z = [L]n, n = V and ∈ y∈XY(G)
− ∈ Z | |
θ: R . Let Γ0 be a collection of i.i.d. ran- where γy is a sample from a Gumbel distribution with
Z → ∪{−∞}
dom variables Γ indexed by i V = [n] and z location c and scale 1 denoted by Gumbel(c,1). Note
{ i0;zi} ∈ i ∈
Z = [L], i V, each following the Gumbel distribu- that contrary to any MCMC-algorithm, which gener-
i
∈
tion whose cumulative distribution function is F(t) = ates true samples only in the limit (after convergence
exp exp( (t+c)) (here c is the Euler-Mascheroni ofaMarkovchain),samplesthatfollowtheoriginaldis-
− −
constant). Then tributioncanbegeneratedbysolvingthecombinatorial
(cid:0) (cid:1)
problem (22) in finite time.
A(θ)=EΓ10;z1hzm1∈aZx1···EΓN0;zn(cid:2)zmn∈aZxnθ(z)+iX∈[n]Γi0;zi(cid:3)i. 3.3.2 Fast Biased Sampling
Note that the random vector Γ includes only nL ran-
0 When the low-dimensional perturbation as in Thm. 2
dom variables. Appying Jensen’s inequality, we arrive
is used, samples are generated by
at a computationally feasible upper bound of the log
partition function [16], argmax θ,y + (γ y +γ (1 y )) (23)
e;1 e e;0 e
h i · · −
A(θ)≤EΓ0hmz∈aZx θ(z)+iX∈[n]Γi0;zii=:A˜(θ). (20) =ayyr∈∈gYYm((GGa))xhθ,yi+eeXX∈∈EE(γe;1−γe;0)·ye (24)
Thesamplesthataregeneratedbythemodesofthe γ Gumbel(c,1)
e;ye ∼
perturbed models follow the distribution
Note that the difference of two independent random
p˜(zˆ|θ)=Pr[zˆ∈argmzaZx θ(z)+ Γi0;zi]. (21) variables that follow a Gumbel distribution follows a
∈ iX∈[n] logistic distribution [40].
MulticutsandPerturb&MAPforProbabilisticGraphClustering 7
(JTA-e),cf.Sec.3.1.Thelattermethodcomputesthe
optimalmarginalsiftheoriginalgraphischordal.Since
0 1 1 3 1 the complexity of JTA is exponential in the tree-width
4 2 4
it does not scale well. We used the implementation
Fig. 3 The colormap used in this paper for visualizing the
within libDAI [31].
probabilitythatanedge(E)iscutPr[ye=1]
Alternatively to JTA we tested loopy belief propa-
gation (LBP) in the node- (LBP-n) and edge-domain
Inordertogenerateofasample,aminimalcostmul- (LBP-e), again with the publicly available implemen-
ticutproblemwithperturbedweightshastobesolved. tation of libDAI [31]. While LBP has no convergence
This can be done either exactly by integer linear pro- guaranteesitshowsgoodconvergencebehaviorifdamp-
gramming[23]orapproximatively[24,8].Thelatterop- ingisused.Inthenodedomaintheprobabilityofacut
tion introduces an additional bias and does no longer is calculated as the sum of terms with different labels
guarantee that eq. (21) holds. Empirically we observe in the pairwise marginals. We also try generalized be-
howeverthattheadditionalerrorissmallforourprob- lief propagation [43] and tree reweighted belief propa-
lems, as the approximate solutions are near optimal gation[41],whicharemoretheoreticallysubstantiated.
and negligible compared to the gap introduced by the They often failed however due to numerical problems
Jensen’s inequality in eq. (21). caused by the constraint-terms.
Thesamplesfollowthedistributionp˜(yˆθ)forwhich FortheproposedPerturb&MAPapproachwecon-
thelog-partitionfunctionisgivenbyA˜(θ),c|f.Eqns.(21), sider global Perturb & MAP (G-P&M) for tiny mod-
(21). Consequently, in view of (11) the marginals µ˜ of els,asdescribedinEq.(22),aswellaslow-dimensional
p˜(yθ) approximate the marginals µ of the original dis- Perturb&MAP,asdescribedinEq.(23),togetherwith
|
tribution (8) by exactinferenceby[23](L-P&M)orapproximativein-
ference by CGC [8] (L-P&M-CGC) and Kernighan
µ µ˜= A˜(θ) (25a)
≈ ∇θ Lin[24](L-P&M-KL),forsolvingtheperturbedprob-
=EΓ0 argmax hθ,yi+ Γe0;ye (25b) lems. For all P&M methods we used OpenGM [3] to
hy∈Y(G)n eX∈E oi solvethesubproblems.Forestimatingthemarginals,we
M used1000samplesforsyntheticmodelsand100samples
1
≈ M argmax hθ,yi+ γe0(;nye) , (25c) for real-world models.
kX=1 y∈Y(G)n eX∈E o All experiments were done on machines equipped
with γe0(;nye) ∼Γe0;ye. with an Intel Core i7-2600K (3.40GHz) CPU, 8 MB
cache and 16 GB RAM, with a single thread.
4 Experiments
4.1 Synthetic Models
Wecomparethefollowingmethodsindicatedbyacronyms
in bold font.
For evaluations in a controlled setting we generated
Bruteforce(BF)enumeratesallvalidpartitions.This
fullyconnectedgraphsandgridgraphswithedge-weights
isonlypracticableforsmallgraphsuptotennodes,but
that are independently normal distributed with mean
provides globally optimal unbiased marginals.
zero and variance 1. As long as the graphs are small,
The local estimates (LOCAL) ignore topological
exact marginals can be calculated by enumerating all
constraints and compute marginals by edge-labelings in 0,1 E or all multicuts (G) which
| |
{ } Y
can be precomputed for a given graph G. The corre-
exp( w )
Pr[ye =1]= exp( w )−+eexp(0). spondingresultsforthefullyconnectedandgridgraphs
e
− are shown in Fig 4 and 5, respectively.
This method can be used as a baseline to measure the For the fully connected graphs JTA-e is exact and
improvements by the use of topology. JTA-n has small error in the marginals. Since both
The junction tree algorithm (JTA) can be applied methods would not scale to larger graphs we resorted
inthenodedomain(JTA-n),cf.Sec.3.1.Weuse4la- toLBPinthenodeandedgedomain.Inbothcasesthe
bels which is sufficient to represent each partition of a estimated marginals became worse as the graph size
planargraphandareasonabletrade-ofbetweenexpres- grows. L-P&M gave slightly worse results on the aver-
sivenessandcomputationalcostforgeneralgraphs.Al- age, but produced no outliers. Using the KL-heuristic
ternatively, in the edge-domain JTA can be applied on (L-P&M-KL) instead of exact inference did not effect
thetriangulatedgraphwithadditionaltriple-constraints thequalitymuch.Forgridmodelsevenexactinference
8 J¨orgHendrikKappesetal.
JTA-e and JTA-n returned results worse than L-P&M y 1
t
for graphs larger than 3×3, caused by the inherent abili 0.5
bias of the underlying approximations. Again, the use b
o
of approximative sub-solvers in L-P&M-CGC did not pr 0
deteriorateresultscomparedtoexactinferenceforthese 0.1 1 10 100
problem sizes. By and large the error increases less for temperature
L-P&M than for JTA and LBP. Fig. 6 Discrete distributions p(i) ∝ exp(−θi/T) for θ =
(2,7,4,1,10,0) and different temperatures. For T the
→ ∞
distributions becomes uniform, for T 0 all its mass con-
→
4.1.1 Effect of Graph-topology centratesonasinglemode.
As illustrated in Fig. 2 the topology of a graph has an
Fig. 9 while those are not visible for low temperatures
impact on the probability that an edge is cut and a
when using the same number of samples.
graphwithzeroweightshavenotnecessaryaprobabil-
As a rule of thumb we found that the temperature
ity 0.5 that an edge is cut. Adding edges to a graph
should be selected such that on the average to 10% of
introduce a bias, which we can observe in Fig. 5. The
the variables the optimal label is not assigned. This
junction tree algorithm (JTA-e) calculates the optimal
generatessamplesthatarediverseenough,butnottoo
marginals on the triangulated graph. However, these
diverse.
marginals can be far from the exact marginals of the
non-triangulatedgraph.Forlargergraphsthiserrorcan
4.1.3 Evaluation with Ground-truth
belargerthantheerrorcausedbylow-dimensionalper-
turbation, as it is the case for 3 3 and 4 4 grids.
× × An exact evaluation of the marginal distributions is
The experiments furthermore show, that the choice
only possible for very small graphs for which we can
of the topology already has an influence on the likeli-
compute the log-partition function by enumerating all
hoodthatanedgeiscut.Inrealworldapplicationsthat
partitions. As fully connected graphs are chordal, the
shouldbeconsideredwhene.g.selectingsuper-pixelsfor
junction tree algorithm can then be used. However,
image segmentation.
its complexity is not less than enumerating all states.
Message passing methods are exact for graphs with 3
4.1.2 Effect of Temperature nodes. For larger graphs the underlying outer relax-
ation and entropy approximations are no longer tight
The temperature parameter T controls the standard andgetworsewithlargergraphs.Whilemarginalspro-
deviationofthedistributionandtheimpactoftheper- duced with the Perturbed & MAP technique have a
turbation. When the temperature drops to 0 all the slightly bigger mean error for full connected graphs,
mass of the distribution concentrates on the optimal Perturb&MAPdoesnotcreateoutliersasthemessage
solutions and samples are drawn uniformly from these passingmethods.Forgridgraphslargerthan3 3Per-
×
”ground” states. When the temperature grows to turbed&MAPproducesbetterresultsintermsofmean
∞
the mass covers the entire feasible set and samples are and maximal error. The use of approximative multicut
drawn uniformly from all valid segmentations. Fig. 6 solvers,namelyKLandCGC,forPerturb&MAPleads
illustrates this for a discrete distribution. For low tem- to similar results as when using exact solvers, but for
peratures it becomes extremely unlikely that the per- the synthetic datasets the difference is negligible.
turbationisstrongenoughto”lift”anon-optimalstate
abovetheoptimalone.Forlargetemperatureshowever 4.1.4 Number of Samples
the difference in the original model are negligible com-
pared to the perturbation variance. Perturb&MAPgeneratessamplesfromadistribution.
From a theoretical point of view the temperature An natural question is how many samples are required
parameterizes a non-linear transformation of the mar- forarobustestimationofthemarginals.Theanswerto
ginals.Fromapracticalviewpoint,itcontrolsthevari- this question obviously depends on the distribution we
ance of the samples. In other words, if we could sam- are sampling from.
pleinfinitelymanyclusteringswecouldrecalculatethe Fig.7showsthebehaviorforafullyconnectedgraph
marginalsforanydensity.Withafinitenumberofsam- withfivenodes(K5).WhiletheerrorforexactPerturb
pleswecanadjusttheresolutionformoreorlesslikely & MAP (G-P&M) vanishes, the low-dimensional Per-
edge-cuts. For example less likely cuts become visible turb & MAP (L-P&M) has a systematic error since it
for high temperatures (at the cost of more noise) in doesnotsamplefromtheoriginaldistribution.Forthis
MulticutsandPerturb&MAPforProbabilisticGraphClustering 9
0.8 0.5 0.35
3nodes slanigram ni rorre tulosba0000000.......1234567 slanigram ni rorre tulosba0000....1234 slanigram ni rorre tulosba000...000012...555123
0 0 0
JTA-e JTA-n LBP-e LBP-n L-P&ML-P&M-KL LOCAL JTA-e JTA-n LBP-e LBP-n L-P&ML-P&M-KL LOCAL JTA-e JTA-n LBP-e LBP-n L-P&ML-P&M-KL LOCAL
0.8
0.4
0.7
5nodes slanigram ni rorre tulosba0000....2468 slanigram ni rorre tulosba000000......123456 slanigram ni rorre tulosba0000....0000123...5555123
0 0 0
JTA-e JTA-n LBP-e LBP-n L-P&ML-P&M-KL LOCAL JTA-e JTA-n LBP-e LBP-n L-P&ML-P&M-KL LOCAL JTA-e JTA-n LBP-e LBP-n L-P&ML-P&M-KL LOCAL
1 0.8
7nodes slanigram ni rorre tulosba0000....2468 slanigram ni rorre tulosba0000000.......1234567 slanigram ni rorre tulosba0000....1234
0 0 0
JTA-e JTA-n LBP-e LBP-n L-P&ML-P&M-KL LOCAL JTA-e JTA-n LBP-e LBP-n L-P&ML-P&M-KL LOCAL JTA-e JTA-n LBP-e LBP-n L-P&ML-P&M-KL LOCAL
1 0.6
0.8
9nodes slanigram ni rorre tulosba0000....2468 slanigram ni rorre tulosba000...246 slanigram ni rorre tulosba00000.....12345
0 0 0
JTA-e JTA-n LBP-e LBP-n L-P&ML-P&M-KL LOCAL JTA-e JTA-n LBP-e LBP-n L-P&ML-P&M-KL LOCAL JTA-e JTA-n LBP-e LBP-n L-P&ML-P&M-KL LOCAL
T =0.25 T =0.5 T =1
Fig. 4 Evaluationoftheabsoluteerrorofcomputedmarginalsforfullyconnectedgraphswith3,5,7,and9nodes.Forlarger
models we are no longer able to calculate the exact marginals. The edge weights were sampled from the normal distribution
with mean 0 and variance 1. Boxplots show the mean (diamond) and median (redline). The edges of the boxes are the 25th
and 75th percentiles. Outliers, with respect to an 1.5 IQR, are marked by red crosses. The black lines mark the largest and
lowestnon-outliers.
Whilefor3nodesLBP-eisexact,L-P&Malreadyhaveasystematicbiascausedbylow-orderperturbation.Forgraphswith
morethan3nodes,LBP-estartssufferingfromitsunderlyingrelaxation.LBP-nproducesmanybadestimates.L-P&Mand
L-P&M-KLprovidemarginalswithbetterorequalaccuracyasLBPandnomarginalswithlargeerrorslikeitscompetitors,
exceptJTA-nandJTA-e,whichdonotscale,however.
10 J¨orgHendrikKappesetal.
0.8 0.4
0.7 0.5 0.35
2x2grid slanigram ni rorre tulosba000000......123456 slanigram ni rorre tulosba0000....1234 slanigram ni rorre tulosba000...000012...555123
0 0 0
BF JTA-e JTA-n LBP-e LBP-n LPALMPAM-CGC LOCAL BF JTA-e JTA-n LBP-e LBP-n LPALMPAM-CGC LOCAL BF JTA-e JTA-n LBP-e LBP-n LPALMPAM-CGC LOCAL
0.5
0.6 0.35
2x3grid slanigram ni rorre tulosba00000.....12345 slanigram ni rorre tulosba0000....1234 slanigram ni rorre tulosba000...000012...555123
0 0 0
BF JTA-e JTA-n LBP-e LBP-n LPALMPAM-CGC LOCAL BF JTA-e JTA-n LBP-e LBP-n LPALMPAM-CGC LOCAL BF JTA-e JTA-n LBP-e LBP-n LPALMPAM-CGC LOCAL
0.35
0.7
0.5 0.3
3x3grid slanigram ni rorre tulosba000000......123456 slanigram ni rorre tulosba0000....1234 slanigram ni rorre tulosba000...00012..55512
0 0 0
BF JTA-e JTA-n LBP-e LBP-n LPALMPAM-CGC LOCAL BF JTA-e JTA-n LBP-e LBP-n LPALMPAM-CGC LOCAL BF JTA-e JTA-n LBP-e LBP-n LPALMPAM-CGC LOCAL
0.5
0.8 0.7
4x4grid slanigram ni rorre tulosba000...246 slanigram ni rorre tulosba000000......123456 slanigram ni rorre tulosba0000....1234
0 0 0
BF JTA-e JTA-n LBP-e LBP-n LPALMPAM-CGC LOCAL BF JTA-e JTA-n LBP-e LBP-n LPALMPAM-CGC LOCAL BF JTA-e JTA-n LBP-e LBP-n LPALMPAM-CGC LOCAL
T =0.25 T =0.5 T =1
Fig.5 Evaluationoftheabsoluteerrorofcomputedmarginalsforgridgraphswith4,6,9and16nodes.Forlargermodelswe
arenolongerabletocalculatetheexactmarginals.Theedgeweightsaresampledfromanormaldistributionwithmean0and
variance1.Boxplotsshowthemean(diamond)andmedian(redline).Theedgesoftheboxesarethe25thand75thpercentiles.
Outliers,withrespecttoan1.5IQR,aremarkedbyredcrosses.Theblacklinesmarkthelargestandlowestnon-outliers.
JTA-e and LBP-e are not exact and include some bias caused by the additional edges included for triangulation. While for
smallmodelsthiserrorismoderate,itincreasesforlargergrids.L-P&MandL-P&M-CGChaveasystematicbiascausedby
low-orderperturbation,whichdoesnotgrowtoomuchwhenthegridsizeincreases.