NegCut: Automatic Image Segmentation based on MRF-MAP

0.31 MB·English

by  Zhao Qiyang

2012 2012
#2223 #2223
000 054
001 055
002 056
003 NegCut: Automatic Image Segmentation based on MRF-MAP 057
004 058
005 059
006 QiyangZhao 060
007 061
NLSDE, Beihang University
008 062
[email protected]
009 063
010 064
0211 065
1
012 066
0
0213 Abstract tion. The targetfunctionis then to be optimizedto obtain 067
0n 14 a good segmentation result. Clearly, the user interactions 068
015 069
a Solving the Maximum a Posteriori on Markov Random areimportantinthewholetask,inotherword,it’sthefirst
0J16 Field, MRF-MAP, is a prevailing method in recent inter- powertopushtheoptimizationtorun. Especiallywhenthe 070
06 17 activeimagesegmentationtools. Althoughmathematically smoothnessterm only reflects the noisy tolerance capabil- 071
0118 072
explicit in its computational targets, and impressive for ity,asinGrabCutandPaint,theuser-definedcolordistribu-
0 19 073
thesegmentationquality,MRF-MAPishardtoaccomplish tionsaremuchcriticalindeterminingwhichpartthepixels
0]20 074
V without the interactive information from users. So it is belongto.
021 075
C rarely adopted in the automatic style up to today. In this Compared with the generally dissatisfactory segmenta-
022 076
. paper, we present an automatic image segmentation algo- tionqualityincurrentautomaticsegmentationmethods,it’s
0s23 077
rithm, NegCut, based on the approximationto MRF-MAP.
c tempting of considering the good segmentation results in
024 078
[ FirstweproveMRF-MAPisNP-hardwhentheprobabilis- MRF-MAP-basedinteractivetools. Soaninterestingprob-
0 25 079
02 26 ticmodelsareunknown,andthenpresentanapproximation lem arises naturally here: whether or not we can design a 080
v functionintheformofminimumcutsongraphswithnega- segmentation algorithm in the MRF-MAP framework, but
027 081
5 tiveweights. Finally,thebinarysegmentationistakenfrom withoutanyuserinteractions? Howevertherearefewcon-
028 082
0 the largest eigenvector of the target matrix, with a tuned
creteattemptsexceptSE-Cut,tothebestofourknowledge.
0929 versionoftheLanczoseigensolver. Itisshowncompetitive In order to produce the required color models, the user- 083
0230 084
0.31 atthesegmentationqualityinourexperiments. designatedzonesinGrabCutarereplacedbyautomatically 085
1
chosen seeds in SE-Cut([4]). SE-Cut choosesits seeds by
0032 086
randomwalksandspectralembeddingtechniquesinsteadin
0233 1Introduction 087
0134 theMRF-MAPframeworkitself. Clearly,it’snotaunified 088
: processinginonlyonedominatingthreadinthetheoretical
0v35 Image segmentation is an important research field in 089
sense. In practice, there’s an additional but necessary re-
0i36 computer vision and computational graphics. In recent 090
X quirementforallseedsto belargeenoughlestbeingelim-
037 years, many interactive segmentation methods, such as 091
r inated in the advanced processing. However, it’s hard to
0a38 GrabCut([1]), Paint([2]), are designed in the Bayesian 092
derive from the MRF-MAP directly, and it makes SE-Cut
039 framework, to say, Maximum a Posteriori (MAP) on 093
a little more inconvenient both in practice and theoretical
040 MarkovRandomFields(MAP)([3]).Duetothehighquality 094
views.
041 oftheirsegmentationresults,especiallyofaccuratebound- 095
042 aries of extracted objects/foregrounds, these methods are In this paper, we presentan automaticsegmentational- 096
043 capableofprocessingmanytediousandtime-resumingin- gorithm,NegCut,thoroughlyundertheMRF-MAPframe- 097
044 teractivetasksinimageediting,medicaldiagnosing,etc. work. It is designed to produce binary segmentation re- 098
045 InMRF-MAP-basedsegmentationmethods,thesample sults, and likely to figure the dominating objects or out- 099
046 foregroundand/orbackgroundzonesaredesignatedbyuser standing scenes out of the images, the same as the inter- 100
047 interactions first, either in the scribble-based or painting- activesegmentationtools. Ourmaincontributionsare: (1) 101
048 based style. The sample data is then adopted to typically a mathematically strict analysis on the hardness of MRF- 102
049 producetwo appropriateprobabilisticmodelsforthe color MAP without any predefined information (or uninformed 103
050 distributionsintheforegroundandbackgroundzones. Ac- MRF-MAPasabbr.) (2)afeasiblewaytoapproximatethe 104
051 companiedwithsomewell-designedsmoothnessterms,the MRF-MAPfunctions,andaconcisemappingfromtheop- 105
052 datatermsappearintheformofthelog-hoodsofallprob- timizationof approximatingenergyontominimumcutsof 106
053 abilitiesofpixelcolors,thencomposethefinaltargetfunc- graphswithnegativeweights. (3)asimpleimplementation 107
1
2012 2012
#2223 #2223
108 162
with the help of the Lanczos eigensolver([5]), along with already adopted in GrabCut. Also we could add a posis-
109 163
someexperimentsandcorrespondinganalysis. stiveconstanttoallsmoothnesscoefficientstoenhancethe
110 164
local continuity, or reduce the factor λ down to eliminate
111 2Background thedifferencesbetweendifferentcolors. 165
112 166
Afterthecolorprobabilisticmodelsandsmoothnessco-
113 In a MRF-MAP-based interactive segmentation algo- 167
efficientsaredetermined,thereareseveralmethodstomin-
114 rithm, the task is mapped into the minimization of a par- 168
imize the target energy function, such as graph cut, loopy
115 ticulartargetenergyfunctionE,whichisthesumofadata 169
belief propagation (LBP) and iterated conditional mode
116 termED andasmoothnesstermES multipliedbyafactor (ICM)([6]). In the case of binary labels, the graph cut 170
117 λ: 171
methodisshowntobepracticallyefficient:thetargetenergy
118 E =ED+λES (1) functioncouldbemappedintocutsonanundirectedgraph 172
119 where the data term ED is the sum of all data penalty on wheretheglobalminimumsolutioncouldbeobtainedwith 173
120 all pixels according to their labels, the smoothness term the max-flowalgorithmsin the polynomialtime. Here the 174
121 ES isthesumofallpotentialofadjacencyinteractionsbe- datapenaltiesareregardedastheweightsofedgeslinkedto 175
122 tweenallneighboringpixelsofdifferentlabels. Inoursit- thesource/sinknodeinnetworks.Astothecaseofmultiple 176
123 uation,theselabelsarejustforegroundorbackground(fore labels, however,the hardnesschanges, to say, it’s a k-way 177
124 andbackasabbr.) Typicallyincurrentalgorithms,thedata cutproblemwhichisprovenNP-hard. 178
125 penaltyofeachpixelisthenegativelog-hoodofitsproba- The determined coefficients of the data terms, derived 179
126 bility accordingto the color distribution in the foreground from the probabilistic color models, are important to con- 180
127 orbackgroundzone,andthesmoothnesstermsareusually struct the graph to be cut. Undoubtedly the user interac- 181
128 definedonalladjacentpixelson4/8-connectiongrids: tions are necessary for the current MRF-MAP based in- 182
129 183
teractive image segmentations, whereas other tools such
130 E = lnPr (p),E = S(p,q) 184
D L(p) S as random walks could present some candidate fore-
−
131 Xp (p,q)∈NX,L(p)6=L(q) ground/backgroundzonesinstead. 185
132 (2) 186
133 whereL(p)isthelabelofpixelpinthelabelconfiguration 3 Hardness ofuninformed MRF-MAP 187
134 L, and(p,q) N meansp is adjacentto q. Althoughthe 188
∈
135 smoothnesstermsarecriticalandimportantalso,it’snotthe Remind that in the interactive image segmentations 189
136 point in this paper. In general the smoothness term could based on MRF-MAP, the probabilistic color models are 190
137 beregardedasa priorin formof a MarkovRandomField, computed from sample pixel colors, to be more detailed, 191
138 therefore the problem is actually to pursue an appropriate foreground/backgroundzonesdesignatedbyusers.Itmeans 192
139 labelconfigurationto reach the lowest energyE, which is thatthewholeenergyfunctionisstillcomputableevenifwe 193
140 theMaximumaposteriorundertheMRF-MAPframework. onlyknowtheexactlabelsofallpixels. Fromthispointof 194
141 Althoughit’smoreexacttoregardthepriorasaconditional view,nowtheenergyfunctionE canbewrittenas 195
142 random field or discriminating random field, we here still 196
143 takeMRF as the notationfollowingthe pastliteraturesfor 197
144 convenience. E(L)= lnGen(L,p)+λ S(p,q) 198
−
145 There are many choices when determining the coeffi- Xp (p,q)∈NX,L(p)6=L(q) 199
146 cientsin the energyfunctionE. Forthe data term, the or- (3) 200
147 dinarystatisticalhistogramsaresufficientforthesituations where we aim to obtain an appropriate probabilistic color 201
148 withonlyafewpossiblecolors,suchas16graylevels,256 model from all pixel p’s with the same label Lp in fore, 202
{
149 colorsandsoon. However,itisnotsuitableforlargecolor back , with the help of a histogram- or GMM-generating 203
}
150 spacessuchas24-bittruecolors,asthesampleswererela- functionGen. GivenGen,allweneedtocalculateE isjust 204
151 tivelytoosparsetoraiseameaningfulprobabilisticmodel. thelabelsofallpixelsintheimage.Inotherword,thereisa 205
152 Thenothermorecomplicatedmodels,suchastheGaussian correspondingenergyvalueforeachlabelconfigurationof 206
153 mixture model(GMM),fit this situationbetter considering pixels.ClearlythereexistsaminimumenergyE∗according 207
154 thecompromisebothofefficiencyandaccuracy. Itshould toacertainlabelconfigurationL∗ amongthem: 208
155 bepointedoutthat, theGMMmethodiscompatibletothe 209
L∗ =argmin(E(L)) (4)
156 histogrammethod, if we take each color valueas an exact 210
L
157 Gaussiancomponentwithvariance0. 211
158 For the smoothnessterm, the coefficientsshouldbe de- Recall that we usually obtain a good segmentation result 212
159 signedtoencouragecontinuitybetweenadjacentpixels,es- according to the minimum energy value in the interactive 213
160 pecially those of similar colors. A good choice is the ex- segmentations, then whether or not L∗ is also meaningful 214
161 ponentialfunctionof the Euclideandistancesof colors, as intheimagesegmentationproblem?Furthermore,couldwe 215
2
2012 2012
#2223 #2223
216 obtainsuchaglobaloptimalsolutionL∗ inthepolynomial 270
217 271
time? However,itisnotaddressedbeforetothebestofour 1 t(cid:13)o (cid:13)x(cid:13) 1 t(cid:13)o (cid:13)x(cid:13) ......(cid:13) 1 t(cid:13)o (cid:13)x(cid:13) ......(cid:13) 1 t(cid:13)o (cid:13)x(cid:13)
1(cid:13) 2(cid:13) i(cid:13) m(cid:13)
218 272
knowledge.
219 273
Since the histograms are easy to deal with, it’s reason-
220 274
able to investigatethe hardnessofuninformedMRF-MAP color(cid:13) color(cid:13) color(cid:13) color(cid:13)
1(cid:13) 2(cid:13) i(cid:13) m(cid:13)
221 275
onsmallnumberofcolors,whiletheconclusionstillholds
222 276
under the general cases of GMMs. To be more concen- Figure1.An1×2kimagewithmcolors.
223 277
trated, we neglect all smoothness term, i.e., λ = 0. Sup-
224 278
posewehavenpixelsofmcolorsintheinputimage,and
225 the pixels of color i counts to ni. Then for each label thatf2 reachesitsminimumvalueatx = d2. Sofinallywe 279
226 configuration L in fore, back n, we set n as the pixel knowthattheenergyfunctionEreachesitsminimumvalue 280
0,i
227 amount of color i a{nd label fo}re, while it’s the same for when k≤n∗nk = k>n∗nk ifpossible. Inthefollowing 281
228 wewillreducethesetpartitionproblemtotheminimization 282
n to color i and back. Furthermore, F and B are the
1,i P P
229 ofourenergyfunctionE. Sincethesetpartitionproblemis 283
two sets of fore pixels and back pixels respectively, while
230 an NPC problem([7]), uninformed MRF-MAP is NP-hard 284
theirpixelamountsares =n +n +...+n and
0 0,1 0,2 0,m
231 then.Rememberthat,thesetpartitionproblemisabout 285
s =n +n +...+n . Thesenotationswillalsobe
1 1,1 1,2 1,m
232 takeninthefollowingsections. Now,theenergyfunctionE GivenapositiveintegersetX = x1,x2,...,xm sum- 286
{ }
233 mingupto2k,whetherornotthereisasubsetX′satisfying 287
onLbecomes
234 thatthesumofallentriesinX′equalsk? 288
n n
235 E(L)= ln 0,p + ln 1,p The reduction is rather straightforward. First we con- 289
236 − s0 − s1 struct an image img with the size 1 2k (only one pixel 290
pX∈F pX∈F ×
237 m high)inthecolorspace color1,color2,...,colorm as 291
{ }
238 =(s lns +s lns ) (n lnn +n lnn ) Then in the smoothness terms, we make it sufficiently 292
0 0 1 1 0,i 0,i 1,i 1,i
−
239 i=1 large for pixels of the same colors, but close to zero for 293
X
240 (5) pixelsofdifferentcolors. Thereforetheblocksofidentical 294
241 colorswould neverbe cut, andthe extremelysmallcoeffi- 295
242 Thenconsiderafunctionf1definedoninterval[0,c]: cientsforadjacentpixelsofdifferentcolorswouldhaveno 296
243 effecton the energyfunctionE. When minimizingE, we 297
f (0)=alna+(b+c)ln(b+c) clnc,
244 1 − wouldget 298
245 f1(c)=(a+c)ln(a+c)+blnb clnc, 299
−
246 f (x)=(a+x)ln(a+x)+(b+c x)ln(b+c x) 300
224478 1 −xlnx−(c−x)ln(c−x−),(0<x<c)−. min iX∈X′xi!ln iX∈X′xi!+iX∈X¯′xilniX∈X¯′xi 330012
(6)     (8)
249 303
onacertainsubsetX′satisfying x = x =
250 Obviously f1 is continuous on the entire interval, and we k, according to the above analysisi.∈XS′oithe seti∈pXa¯′rtitiion 304
251 haveitsderivativef1′(x)=0onx= aa+cb,f1′(x)>0when problem could be determined bPy minimizingPthe energy 305
252 0 < x < aa+cb, and f1′(x) < 0 for aa+cb < x < c. Hence functionE andverifyingwhethertheminimumisequalto 306
225534 wthheemreaaxsiimtsumminviamluuemovfaflu1eoinso[0n,xc]=ap0poerarxs=oncx. T=henaa+fcobr, (2kHlnerke−wehami=ve1xesitlanbxliis)h.eSdoththeelirnekdubcettiwoneeinsfithneissheetdp.arti- 330078
255 eachk=1,...,m,ifs0,s1,nk andalln0,i,n1,i(i =k)as tionproblemPandtheuninformedMRF-MAPinimageseg- 309
256 fixed,wewouldgettheminimumvalueofE(L)at6n0,k=0 mentations. Recall that, the hardnessof normalized cut is 310
257 orn1,k=0. Withoutthelossofgenerality,weassumethere alsolinkedtothesameNPCproblem([8]),andthereduction 311
258 exists an m∗ satisfying that n1,k= 0 for all k m∗, and thereimplicitlyensuresthatitsbinarysegmentationresults 312
259 n0,k=0forallk >m∗. Sowehave ≤ are somewhatfair in sizes. Clearlythis favorableproperty 313
260 314
alsoholdsforthepotentialimagesegmentationalgorithms
261 315
minE(L)=( nk)ln( nk)+( nk)ln( nk) basedonuninformedMRF-MAP,to say,nomuchisolated
262 316
k≤m∗ k≤m∗ k>m∗ k>m∗ small-sizedpieceswereproduced.
263 X X X X 317
m
264 n lnn 4 Approximatinguninformed MRF-MAP 318
k k
265 − 319
k=1
266 X (7) Considering the NP-hardness when minimizing the en- 320
267 ergyfunctionE,itisreasonabletopursueitsapproximating 321
268 Now consider another function f2(x) = xlnx + (d solutioninstead. However,E is nota polynomialwith re- 322
269 x)ln(d x). Sincef′(x) = ln x , it’s easy to conclud−e spectofthelabelconfigurationL,evennotaclosedformat 323
− 2 d−x
3
2012 2012
#2223 #2223
324 all.Soitisfeasibletofinditsreplacementwhichissolvable 4.2Forlargecolorspaces 378
325 379
andsufficientlyclosetoE.
326 Itisalittlemorecomplicatedforthecasesoflargecolor 380
First let’s consider a functionf (x) = xlnx + (1
327 x)ln(1 x). We have f (x) = 3−5xy ∆(x) where−y amounts, such as 224 colors in the 3-bytes RGB space. 381
328 − 3 2 − Sincetheamountofpotentialcolorsisusuallymuchlarger 382
=(1–x)and
329 than that of the image pixels, there is hardly remarkable 383
330 ∆(x)=xy(1(x2+y2)+ 1(x3+y3)+ ), (9) number of samples in each bin of the color histogram. 384
331 3 4 ··· GrabCuttakesGMMasagoodreplacementtohistograms 385
332 by performingtwo Taylor expansions. The mean of ∆(x) inthecolorimagesegmentation,butitisnotfluentandeasy 386
333 on[0,1]is 1∆(x)dx= 1 ,andwecouldbasicallytaken to find the approximation of E from GMMs. Instead we 387
334 0 12 adopt a trivial but also accurate scheme in the continuous 388
it as the value of ∆(x) on the entire interval, given that
335 the mean sqRuare error 1(∆(x) 1 )2dx 3 10−4 is RGBcolorspace: first,chooseafidelityparameterσ2 em- 389
336 0 − 12 ≈ × pirically or by calculations on the entire image as in [1], 390
considerable small. Hence it is totally acceptable to ap-
337 proximatef with −5xR(1 x) 1 ,asshowninFig. 2. thensetthestatisticalcontributionofsamplecoloroneach 391
338 3 2 − − 12 pixelptobeanormaldistributionwithmeanpandvariance 392
339 Nowreturntooure(cid:0)nergyfunctionE.(cid:1)Sincetheuninformed σ2 393
340 Prp(k)= 1 e−Dis2tσ(2p,k) (12) 394
341 √2πσ2 395
original
342 396
approximation whereDist(p,k)meansthesquaredEuclideandistancebe-
343 397
tweenthecolorsofpixelpandk. Thenconstructtheprob-
344 398
abilistic color models on all foreground pixels and back-
345 399
groundpixelsas
346 400
347 1 1 401
Pr (p)= Pr (p),Pr (p)= Pr (p)
348 F n q B n q 402
0 1
q∈F q∈B
349 X X 403
(13)
350 404
And the energy function E becomes into the smoothness
351 405
termplusanintegralontheentirecontinuousRGBspace
352 406
353 Figure2.Curvesoffunctionf3anditsapproximationon[0,1]. n 407
354 E(L)= 0 PrF(p)lnPrF(p) dp 408
MRF-MAPonsmallnumberofcolorsisrelativelysimple, − n ·
355 Z (cid:16) (cid:17) 409
356 wesetthecaseasourstart. + n1 PrB(p)lnPrB(p) dp (14) 410
− n ·
357 4.1Forsmallcolorspaces Z (cid:16) (cid:17) 411
358 +λ S(p,q) 412
Histograms are generally adopted in the cases of small
359 coloramounts. Basedtheapproximationoff ,wecanap- (p,q)∈NX,L(p)6=L(q) 413
360 3 414
361 proximatetheenergyfunctionE with Since qPrq(p) is a fixed value in the input image, it is 415
362 m actuallytominimize 416
5 1 5 1 P
363 (−2ns0(n−s0)− 12)− 2−n n0,i(ni−n0,i)− 12 417
364 Xi=1(cid:18) i (cid:19) 5 + 5 Prp(k)·Prq(k)dk 418
336656 +λ S(p,q) Xp,q −2n2 2n ·Z jPrj(k) ! (15) 441290
367 (p,q)∈NX,L(p)6=L(q) (10) +λ S(p,Pq) 421
368 (p,q)∈N,L(p)6=L(q) 422
X
369 Sincem,nandallni’sareallfixedintheinputimage,itis 423
370 actuallytominimize when pursuingthe minimum of E. Again we can make it 424
simplerbyenlargingthefactorλ ntimes
371 m 425
5 5
372 − s (n s ) (− n (n n )) 426
373 2n 0 − 0 −Xi=1 2ni 0,i i− 0,i (11) 5 + 5 Prp(k)·Prq(k)dk 427
337745 +λ S(p,q) Xp,q −2n 2 ·Z jPrj(k) ! (16) 442289
376 (p,q)∈NX,L(p)6=L(q) +λ SP(p,q) 430
377 whenpursuingtheminimumofE. (p,q)∈NX,L(p)6=L(q) 431
4
2012 2012
#2223 #2223
432 486
433 487
434 488
435 489
436 490
437 491
438 492
439 493
440 494
441 495
442 496
443 497
444 498
445 499
446 Figure3.Leftmostcolumn:originalgrayimages,otherthreecolumns:segmentationresultswithλ=1,5,10inturn.(16grayscales) 500
447 501
448 502
449 5NegCut basedonMin-Cut AccordingtotheLagrangefactormethod([6]),thesolution 503
450 to (18) is the eigenvector D corresponding to the largest 504
5.1NegCutonsmallcolorspaces
451 eigenvalue. Then we might straightforwardly get the re- 505
452 Take thesimple case ofhistogramson smallnumberof quired binary indicator from D by setting the ith entry as 506
453 colorsasourstart. Thenourtargetistoobtainasegmenta- 1ifdi 0,or+1ifdi > 0. HeretheLanczosalgorithm 507
− ≤
454 tionresultbyminimizing(11).Nowconstructanundirected is adopted to finish the time-consuming operations of the 508
455 graphGofnnodescorrespondingtothepixels,andsetthe largesteigenvectorsforlargematrices. Infact,theLanczos 509
456 edgeweightw(p,q)asthesumofthefollowingthreeterms algorithmis wellknownasthe fastest methodto solvethe 510
457 largesteigenvectorsinO(N)timemostlyforsparsematri- 511
5
458 w1(p,q)= −2n ces, whereasourgraphGiscompletelyconnectedandthe 512
459 correspondingweightmatrix W is full. Fortunately, W is 513
5 ,pandqhavethesamecolori,
460 w2(p,q)= 02,noitherwise. (17) onlycalledtoperformamatrix-vectormultiplicationthere, 514
461 (cid:26) andthemajorcalculationisidenticalforpixelsofthesame 515
462 S(p,q),pisadjacenttoq, color. So we might improve it to sharply reduce the time 516
w (p,q)=
463 3 0,otherwise. consumptiondowntolinear,asdescribedinalgorithm1. 517
(cid:26)
464 Now we have established the NegCut algorithm on the 518
It is easy to prove that the approximatingenergy value on
465 smallcolorspacesasbelow: 519
a label configurationL is just equal to the capacity of the
466 520
correspondingcutC= F,B onG. Thereforeitistofind
467 { } 521
the minimum cut on G minimizing (11). Although there Algorithm1NegCutonsmallcolorspaces
468 522
exists several O(N3) algorithm solving the minimum cuts 1: On each adjacent pair of pixels (p, q), compute the
469 523
ongraphswithnon-negativeweights,itisnotsuitablehere.
smoothnesstermS(p,q);
470 524
471 Infact,thenegativeweightededgesmaketheMin-Cuttobe 2: SolvethelargesteigenvectorD =[d1,d2, ,dn]T of 525
aMax-Cutonaauxiliarygraphwithnon-negativeweights. ···
472 thematrixW = [w(p,q)]withtheLanczosalgorithm, 526
SinceMax-CuthasalreadybeproventobeaNP-complete
butperformthematrix-vectormultiplicationofW and
473 527
474 problem, it can be concluded that there is no polynomial somevectorR=[r1,r2, ,rn]T as 528
algorithmforustominimize(11). ···
475 529
However,wemightgeneralizethisdiscreteoptimization setϕ=0. Thenfork=1ton,ϕ ϕ 5 r ;
476 taskinto thecontinuousrealspace Rn. Firstwe set anin- • ← − 2n · k 530
477 dicatorvectorD = [d ,d , ,d ]T +1, 1 n denot- set θi = 0 for all i’s. Then for k = 1 to n, set 531
478 1 2 ··· n ∈ { − } • θ θ + 5 r accordingtoitscolori; 532
479 ing the ith pixelof fore by +1 or back by −1, then estab- i ← i 2ni · k 533
lisha matrixW = [w(p,q)] whoseentriessumuptoS .
480 Clearly the cut value is just equal to 1 S DTWDW. • fork =1ton,calculateµk = j:(j,k)∈N S(j,k). 534
481 2 W − Thenaccordingtoitscolori,andoutputϕ+θi+ 535
482 Sinutebrsveaqlu[ent1l,y1,]wiensgteeanderoaflize+t1h,e la1be,lasnidn(cid:0)ttohethoericgoinntailnoupotu(cid:1)is- (25n − 25ni)·rk asthekthentrPyofvectorW ·R. 536
483 − { − } 537
mizationtaskon(11)becomes 3: let the label of pixel k be back if dk 0, or fore if
484 ≤ 538
d >0.
485 maxDTWD,s.t. D =n. (18) k 539
k k2
5
2012 2012
#2223 #2223
540 Apparently we could finish step 1 and 3 in O(n) time. Algorithm2NegCutonlargecolorspaces 594
541 595
Fromstep2.ato2.d,itrequiresO(n)timetocalculatethe 1: Cluster all pixels into m color classes with the mean
542 596
productvectorinvolvingW, soweneedtotallyO(n)time color c , ..., c . On each pixelk, calculate its prob-
1 m
543 597
fortheslightlychangedLanczosalgorithminstep2.Finally abilistic densitiesinallm Gaussian Distributions, and
544 theentiretimecomplexityofNegCutforsmallcolorspaces additscontribution Prp(k)·Prq(k) tow (p,q)forevery 598
545 remainsO(n). PjPrj(k) 2 599
twocolorclassesofpixelpandq;
546 600
547 5.2NegCutonlargecolorspaces 2: SolvethelargesteigenvectorD =[d1,d2,··· ,dn]T of 601
the matrixW = [w(p,q)] with the Lanczos algorithm,
548 602
The case of large color spaces is a little more compli- butperformthematrix-vectormultiplicationofW and
549 603
cated. Recallthatin (16)we needto calculatethe integral somevectorR=[r ,r , ,r ]T as
550 1 2 ··· n 604
onthewholecontinuouscolorspace,anditisunrealisticin
551 setϕ=0. Thenfork=1ton,ϕ ϕ 5 r ; 605
552 mostapplications. Afeasiblesubstitutionistoperformthe • ← − 2n · k 606
calculation on the typical samples according to the prob-
553 set θi = 0 for all i’s. Then for k = 1 to n, i = 1 607
abilistic distribution of (13). Fortunately the pixels of the •
554 tom,setθi θi+w2(j,k) rk accordingtoits 608
inputimagejustmeetthisrequirementwell. Hencethein- ← ·
555 colorclassj; 609
tegralbecomesinto the sum as Prp(k)·Prq(k), however
556 k PjPrj(k) fork =1ton,calculateµ = S(j,k). 610
557 thecomputationalloadisstillheaPvyaswehavetoscaneach • Thenaccordingtoitscolokrclassij,:(oju,kt)p∈uNtϕ+θ + 611
555589 ptiirxeecloamndplreexciotyrdisitsupcotnotrOib(unti2o)n,swthoicnhPirsq’tso.taWllyhiulenathceceepnt-- µk− 25n ·rk asthekthentryoPfvectorW ·R. i 661123
560 ableevenonordinary-sizedimages. Inordertoimproveit, 3: let the label of pixel k be back if dk 0, or fore if 614
561 we cluster all colorsinto a limited numberof classes first, d >0. ≤ 615
k
562 sothatthesummationswouldbeidenticalforthepixelsin 616
563 thesamecolorclass. Now,allweneedisjusttoscaneach 617
564 pixelandrecorditscontributionstothesecolorclasses,and 618
andallgrayscale imagesarederivedfromthem. Herethe
565 thetimeconsumptionisreducedtoO(n)giventheamount 619
amountsofcolorclassesinallcolorimagesaresettobe16,
566 ofcolorclassesislimited. 620
andallgray-scaledimagesalso have16graylevelswithin
567 Asinthesmallcolorspaces,weconstructanundirected 621
them. Based onthe block-basedLanczoseigensolver([9]),
568 graphGofnnodescorrespondingtothepixels,andsetthe 622
we established two versions for NegCut in Matlab codes.
569 edgeweightw(p,q)asthesumoffollowingthreeterms, 623
Atlast,alltestimagesareresizedtobeabout256 256to
570 × 624
5 reducethetime-consumingcalculationsintheexperiments.
571 w (p,q)= − 625
572 1 2n TherearethreegroupsofresultsinFig.3,4and5:gray- 626
573 5 Prp(k) Prq(k) scaled on different λ’s, segmentation results of color im- 627
574 w2(p,q)= 2 · k jP·rj(k) (19) ages on different λ’s, and different color images on fixed 628
575 X λ=1. Ourtargetimagesareintendlychosentobeofclose 629
576 w3(p,q)= S0,(opth,qe)rw,pPisies.adjacenttoq, grayscales,splendidlycolorsanddelicatelocalchanges,so 630
577 (cid:26) thattoverifytheperformanceofNegCutondifferentchal- 631
lenges. ThesmallcolorspaceversionofNegCutworkson
578 NowwehaveestablishedtheNegCutalgorithmforthelarge 632
gray-scaledimages,whilethosecolorimagesareprocessed
579 colorspaces: 633
withthemorecomplicatedalgorithm.
580 There are some lightdifferencesbetween the two algo- 634
581 rithmsinstep1and2. Howevergiventheamountofcolor In general, we obtained basically acceptable segmenta- 635
582 classesislimitedandfixed,itrequiresalsoO(n)timetofin- tion results in all experiments, especially that most domi- 636
583 ishthemultiplicationinvolvingW. Finallytheentiretime nating objects are figured out of these images. There are 637
584 complexityofNegCutforlargecolorspacesisalsoO(n). more isolated, but vivid pieces in the segmentationresults 638
585 whenλ=1,bothongray-scaledimagesandcolorimages. 639
586 6Experiments Andit’smuchbetterforcolorimagesbecausetheconnec- 640
587 tionsbetweendifferentcolorsareinvolvedinthelargecolor 641
588 SinceNegCutisdifferentonthesmallcolorspacesand space version of NegCut. When λ = 1, the segmentation 642
589 large color spaces, our experiments are finished on both boundariesaremorelikelylocatedonthedesirededgesof 643
590 gray scale images and color images. We trivially set all the objects in these images. However, it also brings too 644
591 smoothnesstermstobe1overentireimages,andletλvary much emphasison these discontinuousline segments, and 645
592 from1to10.Allcolortestimagesarechosenfromtheseg- resultsin muchmoreisolated pieces in the segmentations. 646
593 mentationdatasetsfromBerkeleyandMSRatCambridge, Whenλvariesfrom1to5,thento10,itisshowntheseg- 647
6
2012 2012
#2223 #2223
648 702
649 703
650 704
651 705
652 706
653 707
654 708
655 709
656 710
657 711
658 712
659 713
660 714
661 715
662 716
663 717
664 718
665 719
666 720
667 721
668 722
669 723
670 724
671 725
672 726
673 727
674 728
675 729
676 730
677 731
678 732
679 733
680 Figure4.Leftmostcolumn:originalimages,otherthreecolumns:segmentationresultswithλ=1,5,10inturn.(m=16) 734
681 735
682 736
683 mentationboundariesbecomesmootherand smoother,but oneappears. 737
684 onthecostoftheloseofelaboratedetails. Whenλ = 10, Recall that in the interactive segmentation tools, their 738
685 thetwosegmentationzonesaremorefairinsizes. Therea- targetis to obtain the minimum of particular energyfunc- 739
686 son for that is, the large portion of the smoothness terms tions with user interactions. Since NegCut approximates 740
intheenergyfunctionweakenstheeffectofthedatapenal- the global minimum energy value, it is meaningful in the
687 741
688 ties. NegCuttendstocuttwopiecessizedofonehalfofthe analysisontheunderlyingmathematicaltrendswhenthese 742
689 entireimagetoreachtheminimumenergies. interactivetoolsperformingcalculations. 743
Different from the ordinary minimum-cut-based seg-
690 744
691 7Conclusions mentation or clustering algorithms, the negative weights 745
692 ensure that NegCut likely obtain fair segmentation re- 746
693 Despite the NP-hardness of uninformed MRF-MAP, sults instead of isolated extremelysmall pieces. However, 747
694 NegCutisaninitialattemptindevelopingautomaticimage thedouble-foldNP-hardnessencounteredinsolvingMRF- 748
695 segmentationtoolsundertheMRF-MAPframework.Given MAPremindsus,itisnecessarytomakeaattemptonmore 749
696 that the Lanczos algorithm is specially designed for solv- efficientcalculationmethods. 750
697 ing the extreme eigenvectors, NegCut are much easier to 751
References
698 developin principle than those require inner eigenvectors, 752
699 suchas the normalizedcut. ThoughNegCutonlyproduce [1] C.Rother, V. Kolmogorov and A.Blake. GrabCut: Interac- 753
700 binary segmenation results, it can be recurrently called to tiveForegroundExtractionusingIteratedGraphCuts. ACM 754
701 refinethepastsegmentationzonesresultsuntiltheexpected transactionsongraphics,23(3):307-312,2004. 755
7
2012 2012
#2223 #2223
756 810
757 811
758 812
759 813
760 814
761 815
762 816
763 817
764 818
765 819
766 820
767 821
768 822
769 823
770 824
771 825
772 826
773 827
774 828
775 829
776 830
777 831
778 832
779 833
780 834
781 835
782 836
783 837
784 838
785 839
786 840
787 841
788 842
789 843
790 844
791 845
792 846
Figure5.Leftcolumns:originalimages,rightcolumns:segmentationresults.(λ=1,m=16)
793 847
794 848
795 849
[2] J.Liu,J.SunandH.Shum.PaintSelection.ACMtransactions A GuidetotheTheoryof NP-Completeness, W.H.Freeman
796 ongraphics,28(3):1-7,2009. Press,1979. 850
797 [3] A.Blake,C.Rother,M.Brown,P.Perez,andP.Torr. Interac- [8] J.B. Shi and J. Malik. Normalized cuts and image segmen- 851
798 tiveImageSegmentationUsinganAdaptiveGMMRFModel. tation. IEEEtrans.onPatternAnalysisandMachineIntelli- 852
799 EuropeanConferenceonComputerVision,428C441,2004. gence,22(8):888-905,2000. 853
800 [4] F.Estrada,A.Jepson,andC.Chennubhotla. SpectralEmbed- [9] J.Baglama,D.Calvetti,andL.Reichel. IRBL:AnImplicitly 854
801 dingandMin-CutforImageSegmentation. BritishMachine Restarted Block Lanczos Method for large-scale Hermitian 855
802 VisionConference,317-326,2004. eigenproblems. SIAMJournal ofScienceComputing, 27(1), 856
803 [5] G.Golub,C.VanLoan. MatrixComputations,JohnHopkins 2003. 857
804 Press,1996. 858
805 [6] R.Szeliski,R.Zabih,D.Scharstein,andetc. Acomparative 859
806 study of energy minimization methods for markov random 860
807 fieldswithsmoothness-based priors. IEEEtrans.onPattern 861
808 AnalysisandMachineIntelligence,30(6):1068–1980,2008. 862
809 [7] M.R.GareyandD.S.Johnson.ComputersandIntractability: 863
8