LayerBuilder: Layer Decomposition for Interactive Image and Video Color Editing

2.9 MB·

by  Sharon Lin

LayerBuilder: Layer Decomposition for
Interactive Image and Video Color Editing
SharonLin12MatthewFisher13AngelaDai1PatHanrahan1
7 1StanfordUniversity
1 2Zazzle
0 3AdobeResearch
2

n
a
J

6
1

]
R
G
.
s
c
[
InputImage LayerDecomposition Recolorings

2
Figure1:LayerBuildertakesaninputimageandautomaticallycomputesasetofcoloredlayerswhoselinearcombination
v
4 reconstructstheoriginalimage.Expressedinthisform,itisstraightforwardtointeractivelyrecolortheimagebychoosinga
5 newsetofcolorsforthelayers.Ontheright,weshowtworecoloringsofeachinputimage.Thelayercolorsareshowninthe
7 upper-rightofeachimage.Layerweightsrangefrom0(black)to1(white).
3
0 Abstract
1. Exploringandeditingcolorsinimagesisacommontaskingraphicdesignandphotography.However,allowing
0 forinteractiverecoloringwhilepreservingsmoothcolorblendsintheimageremainsachallengingproblem.We
7 presentLayerBuilder,analgorithmthatdecomposesanimageorvideointoalinearcombinationofcoloredlayers
1 tofacilitatecolor-editingapplications.Theselayersprovideaninteractiveandintuitivemeansformanipulating
: individualcolors.Ourapproachreducescolorlayerextractiontoafastiterativelinearsystem.LayerBuilderuses
v
locallylinearembedding,whichrepresentspixelsaslinearcombinationsoftheirneighbors,toreducethenumber
i
X ofvariablesinthelinearsolveandextractlayersthatcanbetterpreservecolorblendingeffects.Wedemonstrate
r ouralgorithmonrecoloringavarietyofimagesandvideos,andshowitsoveralleffectivenessinrecoloringquality
a andtimecomplexitycomparedtopreviousapproaches.Wealsoshowhowthisrepresentationcanbenefitother
applications,suchasautomaticrecoloringsuggestion,texturesynthesis,andcolor-basedfiltering.
Categories and Subject Descriptors (according to ACM CCS): Image Processing and Computer Vision [I.4.9]:
Applications—;
1. Introduction nentsandlocalizechangestothedesiredportionofthefinal
image.Whencreatingadigitalimage,artistscarefullymain-
Representinganimagebyseparatingdifferentelementsinto
tain the set of layers to make it easy to perform edits that
layersisanintegralfeatureofdigitalimageeditingsoftware
willenablerapiditerationthroughthedesignspace.Inmost
suchasAdobePhotoshoporGIMP.Layersofferanintuitive
cases, combining the layers is computationally straightfor-
waytoeditthecolorandgeometryofindependentcompo-
S.Linetal./LayerBuilder:LayerDecompositionforInteractiveImageandVideoColorEditing
ward,allowinganartisttointeractivelyevaluatehowlayer parameterchoice.Asshowninthesupplementalvideo,once
editsaffectthefinalimage. thelayerdecompositionhasbeenperformed,allvideoand
imagecoloreditsareinteractive,allowingartiststoquickly
A layer representation is a powerful editing tool, but it
explorethepotentialcoloringspace.
isnotalwaysavailable.Onceanimageisrasterized,itsde-
compositionintolayersistypicallylost.Furthermore,pho- We demonstrate the effectiveness of our system for re-
tographsorvideoscapturedbycameraorproducedbyren- coloring a variety of artistic and natural images. We fur-
dering systems may not have an accessible decomposition ther show that our layers can be used to augment existing
intomeaningfullayers.Theproblemofrecoveringagoodset image processing or synthesis applications. We use Layer-
oflayersforanimageorvideoisnoteasytospecify,asdif- Builder to extend work on automatic pattern coloring sug-
ferentusersandapplicationswillpreferdifferentlayerings. gestion, which previously required pre-layered images and
Nevertheless,forflattenedimagesandvideos,wewouldstill operatedonanunblendedsetofcolors,tosupportanyinput
liketorecoversomeofthepowerfulinteractiveeditingop- imageandtakeadvantageofourcontinuouslayerdecompo-
erationsenabledbyameaningfullayerdecomposition. sition[LRFH13].Texturesynthesisalgorithmswhichoper-
ateonageneralfeaturespacecaneasilyusethelayerdecom-
Inthispaper,wefocusontheproblemofextractinglayers
positionasalocalfeaturevectorforeachpixel[LH06],al-
foruseinimageandvideocolorediting.Oursystemtakes
lowingsynthesizedtexturestomorefaithfullyrespectlong-
aninputimageorvideoandcomputesadecompositioninto
rangeeffectsintheexemplar.Finally,weshowthatlikewith
coloredlayers,whosecombinationreconstructstheoriginal
digitalimageeditingtools,itiseasytoproducesignificant
input.Userscanoptionallyspecifyadditionalconstraintsto
butlocalizedmodificationstotheimagebydirectlyfiltering
furtherrefinethelayers.Giventhisrepresentation,itiseasy
thelayersandrecombiningthem.
toperformeditsandinteractivelyrecomposethefinalrecol-
oredresult.Figure1visualizestheprocessforbothphoto-
graphicandartisticimages. 2. RelatedWorkandBackground
Our layers enable easy and interactive editing of im- Relatedworkforourmethodspanstwomainareas:image
ages and videos, even in the presence of challenging color recoloringviaeditpropagationandimagelayerization.We
blending effects such as semi-transparency, motion, and alsoprovidesomebackgroundonlocallylinearembedding.
unfocused pixel blur. LayerBuilder enables finer control
over each color range compared to color transfer tech-
2.1. ImageRecoloringviaEditPropagation
niques [RAGS01,TJT05], which require a reference im-
age, and material or object matting techniques [CLT13], Affinity-based recoloring methods use pixel similarity to
which tend to extract larger patterned regions. Recolor- propagateusercoloredits,intheformofstrokesorpalette
ing methods using affinity-based edit propagation [AP08, changes,totherestoftheimage.Levinandcolleaguescon-
LJH10,CFL∗15] have similar control over colors but pro- sider the similarity between neighboring pixels based on
duce more halo artifacts in color blending regions because luminance [LLW04]. To handle textured regions, An and
theydonotnecessarilypreservecolorinterpolationrelation- Pellacini consider the similarity between all pairs of pix-
ships. Manifold-preserving edit propagation reduces these els [AP08], a computation that can be sped up by cluster-
artifactsbutrequiressolvingalinearsystemforeachcolor ing pixels beforehand [XLJ∗09]. Some methods compute
change[CZZT12].Ourmethodhandlestheseregionsaswell analphainfluencematteforeachstroke [LAA08,FFL10],
asmanifold-preservingeditpropagation,whilemakingcolor allowingquicksubsequentcolortweaks.Liandcolleagues
editinginteractiveafterlayersetup. achieveinteractiverecoloringtimebyformulatingeditprop-
agationasaradialbasisfunction(RBF)interpolationprob-
We show how to use the concept of locally linear em-
lemwherefunctioncoefficientsdependonuser-editedpix-
beddingtoreducetheproblemofextractinglayersforcolor
els[LJH10].Changandcolleaguesfocusonrecoloringvia
editingtoasimpleiterativelinearsystem.Weadaptandac-
extracting and editing the image’s color palette [CFL∗15].
celeratetheeditpropagationtechniquefromChenandcol-
They introduce a faster recoloring method by using RBF
leages [CZZT12] to work for computing layers. Instead of
functions where the coefficients depend only on the input
performing our solve directly over pixels, we demonstrate
palette colors, and spatial information is not considered.
thatitissufficienttosolveforthelayerdecompositionona
However,recoloringbyshiftingpixelcolorsbasedontheir
much smaller set of superpixels, then extrapolate the pixel
similaritytootherpixels(orpalettecolors)doesnotneces-
decompositionfromthesuperpixelsolution.Fortheimages
sarilypreservethecolorinterpolationrelationshipsbetween
shown,thisresultsinasystemwiththreeordersofmagni-
pixelsalongcolortransitions(e.g.edgesandgradients).As
tudefewerfreevariables,permittingsolvesonlargeimages
aresult,thesemethodstendtoproducehaloartifactsinthese
withmanylayerswhileremainingwithinreasonablemem-
colorblendingregions[CZZT12].
ory and computational budgets. Finally, we compare our
method against previous recoloring approaches to demon- To reduce halo effects, Chen and colleagues propose a
strateitsinteractivityandrobustnesstocolorblendingand smoothnesstermthatpreservesthepixelmanifoldstructure
S.Linetal./LayerBuilder:LayerDecompositionforInteractiveImageandVideoColorEditing
of the image [CZZT12]. Each pixel’s color is represented occlusion-relatededits,involveminimizingasimplerenergy
by a weighted combination of its K-nearest pixels in ap- functionandstillenableavarietyofcoloredits.
pearanceandspace.Recoloringanimageshouldthenboth
respect user input as well as maintain these color relation-
2.3. LocallyLinearEmbedding
shipsintheresult.Thismethodcanbeacceleratedbysolv-
ingeditpropagationforasmallersetofrepresentativesam- Locallylinearembedding(LLE)isanon-lineardimension-
plesandtheninterpolatingsamplestogetthefinalpixelcol- alityreductiontechniquefordatathatlieonamanifoldem-
ors [CZL∗14]. Our work takes this idea of preserving the beddedinahigher-dimensionalspace.LLEisbasedonthe
pixelmanifolduponelevelofindirectionwhendecompos- intuitionthateachpointanditsneighborsformalocallylin-
inganimageintolayers.Thelayerdecompositionofapixel ear patch of the manifold [RS00]. Then for a set of data
shouldbeaweightedcombinationofitsKneighbors’layer pointsx1,...,xn,wecanapproximateeachxiasalinearcom-
decomposition, and the weights should be determined by binationofitsKnearestneighbors.Specifically,wecompute
thecolorrelationshipsofthepixels.Oncewecomputethe the weights wij that minimize the residual sum of squares
layers, image recolorization is very fast, and only requires fromreconstructingeachxi,
addinglayerstogether.
n
∑(cid:13)(cid:13)xi−∑wijxj(cid:13)(cid:13)2
i=1 j(cid:54)=i
2.2. ImageLayers
whereforeachi,∑jwij=1.Theseweightscharacterizethe
Asimilarnotionoflayersisusedinimagematting[CCSS01,
manifoldstructureandcanbecomputedefficiently[RS00].
LLW08,SV11], which separates objects from the back-
Thisideahasbeenusedtopropagatescribbleeditsthrough
ground, and material matting [LL11,CLT13], which sepa-
images and video, noting that in natural images, the color
rates different textures. A stroke-based or trimap input is
atapixelisoftenablendofthecolorsofneighboringpix-
usedtoestimatealphamattesforeachlayeraswellaslayer
els[CZZT12].
colors.Thesetechniquesfocusonfacilitatingobjectormate-
rialmanipulation,andtheirlayerstendtocaptureobjectsor
texturesthatmayhavemultiplecolors.Ourlayersrepresent 3. LayerDecomposition
a single color, allowing for cleaner and more fine-grained
Our goal is to take an input image or video representing a
colorchangesaswedemonstrateinSection4.Bothtypesof
setofPpixels,atargetnumberoflayersN,andcomputethe
layershaveorthogonalgoalsandmaybecombinedtobetter
following:
localizecoloreditstosimilarly-coloredobjectsormaterials.
• AsetofNlayercolors
Carroll and colleagues aim to recolor objects in pho-
• ForeachoftheNlayers,aP×1columnvectorXjwhose
tographswhileupdatingtheirinterreflectionswithotherob-
entriesdenotethecontributionoflayer jtoeachpixel.X
jects consistently [CRA11]. Their approach involves first
isthePN×1columnvectorformedbyconcatenatingall
breaking the photo down into its material reflectance and
Xj.
illumination intrinsic images [BPD09]. Assuming the ma-
terialreflectanceimageispiecewiseconstantandthuseasily OuralgorithmissummarizedinFigure2andconsistsof
recolorable, the authors focus on decomposing the illumi- fourmainsteps.First,computethelayercolorsfromthein-
nationimageintocolor-basedlayersgivenauser-specified putpixelsorobtainthemfromtheuser(Figure2a).Second,
set of main colors. This method works best for recoloring computeasuperpixelsegmentationoftheimage(Figure2b).
objectsthatareinfocuswherecolorblendingisexplained Third,solveforthelayerweightsforeachsuperpixel.This
mainly by illumination. We use a similar, but simpler, en- problemreducestosolvingalinearsystem(Figure2c).Fi-
ergyfunctionforcomputingcolor-basedlayers.Ratherthan nally,foreachlayer,computeaper-pixelweightbylinearly
specializing on interreflections, our color-based layers tar- combiningthesetofper-superpixelweights(Figure2d).
get a more general class of color blending effects, such as
onesduetodepth-of-field,motionblur,andincludingmore
3.1. LayerColors
pronouncedinterreflections.Ourcolor-basedlayerscancre-
ateplausibleresultsforawiderangeofcoloreditswithout Becausepeoplemayhavedifferentgoalswhenrecoloringan
requiringadecompositionintoreflectanceandillumination image,weallowtheusertoeitherspecifyapaletteorusean
images.InSection4,wecompareourmethodtotheoneby automaticone.Automaticpalettesprovideausefulstarting
Carrollandcolleagesforthecaseofinterreflections. pointandalsoenableautomaticrecolorings(Section5.2).In
allresults,wespecifywhenanautomaticpaletteisused.
Concurrently, Tan and colleagues developed a method
for extracting ordered single-color layers that use the To provide intuitive controls over the space of possible
more traditional over operator to reconstruct the original colorings, our algorithm should choose the N colors that a
image [TLG16]. In contrast, our method extracts order- person would likely choose as well as choose colors that
agnostic additive layers that, although do not support canlinearlyreconstructtheoriginalimageinRGB(orLAB)
S.Linetal./LayerBuilder:LayerDecompositionforInteractiveImageandVideoColorEditing
From each superpixel we compute a 6-dimensional fea-
turevector(r,g,b,x,y,t)formedbyconcatenatingtheaverage
pixel color with the superpixel’s spatiotemporal centroid.
Thecolor,spatial,andtemporalcoordinatesarenormalized
between0and1andre-weightedaccordingtoimportance.
Inourexperiments,weweightthespatialcoordinates(x,y)
(a)Inputimage (b)Superpixelsegmentation
by0.5,tobiasdistancecalculationstowardscolorandtem-
poralsimilarity.Figure2bshowsanexamplesuperpixelseg-
mentationwithsuperpixelboundariesoutlinedinblack.
3.3. EnergyFunction
Inthissectionwewillcomputeasetofper-superpixellayer
(c)Per-superpixellayerweights (d)Per-pixellayerweights
contributionsLj∈L,whereLj isaS×1columnvectorde-
notingthecontributionoflayer jtoeachsuperpixel,andLis
Figure2:Algorithmoverview.(a)theinputimageandthe
extractedpalette.(b)thesuperpixelsegmentationusing250 the(SN)×1columnvectorformedbyconcatenatingallLj.
Toquantitativelydefinewhatconstitutesagoodlayering,we
superpixels. (c) the red layer weights for each superpixel,
minimizeanenergyfunctionΘ(L)overthesuperpixel-layer
computed using Equation 8. (d) the result shown in (c) is
contributionsL.Θ(L)isaweightedsumoffourterms:man-
used to compute per-pixel weights for the red layer using
ifold consistency M(L), image reconstruction R(L), unity
Equation10.Figure1visualizesthecompletelayerdecom-
U(L),andexplicitconstraintsE(L).
positionforthisimageusing2000superpixels.
Θ(L)=λmM(L)+λrR(L)+λuU(L)+λeE(L) (1)
space. To this end, we adapt the color palette model intro-
ducedbyLinandHanrahan,whichhasbeentrainedtomatch
Manifold consistency. Layer contributions should be
palettespeoplewouldextractfromimages[LH13].Weadda
locally consistent, in that nearby and similar superpixels
penaltytermtothismodelbasedontheaveragecolorrecon-
should have similar layer decompositions. We implement
structionerror,whereerrorperpixelismeasuredbythedis-
this term using the superpixel manifold determined by lo-
tancefromthepixelcolortotheconvexhullofthepalette.In
cally linear embedding, similar to the per-pixel constraints
ourexperiments,weweightthetermby−0.01λwhereλis
usedforeditpropogation[CZZT12].
thesumofabsoluteweightsintheLinandHanrahanmodel.
Following the approach used in Section 2.3, we express
thecolorofeachsuperpixelsiasalinearcombinationofthe
3.2. SuperpixelSegmentation
coloroftheKs-nearestsuperpixelssij:
Csuolmtspinutainsgysthteempewr-iltahyaerfrweeeivgahrtisabdlierefcotrlyeaocnhppixixeellvaanludeesarceh- (cid:13)(cid:13)(cid:13)Color(si)−∑Ks wijColor(sij)(cid:13)(cid:13)(cid:13)2
layer weight. This rapidly becomes intractable to solve di- (cid:13) j=1 (cid:13)
rectly: for large images this may result in millions of free
Wesolvefortheweightswij andrepresenttheminanS×
variables,growingtobillionsoffreevariablesforvideose-
Ssuperpixel-manifoldmatrixW.Themanifoldconsistency
quences.Tocombatthisproblem,wetakeadvantageofthe
termisthen:
high correlation between nearby pixels. We solve for the
N
layerweightsonamuchsmallernumberofsuperpixels(or M(L) = ∑ ||(IS−W)Lj||2
supervoxels),andthencomputethelayersolutionforeach j=1 (2)
pixelusingalinearcombinationofnearbysuperpixels.The
imageandvideoresultsinthispaperuseatmost4000su- = ||(I −M)L||2
SN
perpixels,resultinginsystemswithmultipleordersofmag-
where In is the n×n identity matrix and M is a block di-
nitudefewerfreevariables.
agonalmatrixwithoveralldimensionSN×SNoftheform:
Tocomputesuperpixels,weuseavariationoftheseeded
region growing algorithm (SRG) [AB94] with iterative k-  W 0 ... 0 
meansre-centering.Wemanuallyspecifythenumberofsu-
 0 W ... 0 
pSeurppeirxpeilxseSlsaarnedthineintiacloizmepeuatecdhbtoySaRrGan,daonmdtsheeeidrcleonctartoioidns. M= ... ... ... ...  (3)
andcolorsareusedasseedsforthenextiterationofSRG,re- 0 0 ... W
peatingforatotalof5k-meansiterations.Forsupervoxels,
wedefinepixelneighborsthrough6-connectivity:4spatial Imagereconstruction.Thelayersshouldreconstructthe
and2temporalneighbors. original image when multiplied by their colors and added
S.Linetal./LayerBuilder:LayerDecompositionforInteractiveImageandVideoColorEditing
together. We enforce this by measuring the squared image 3.4. ComputingPer-pixelWeights
reconstructionerror:
Weusetheconceptoflocallylinearembeddingdescribedin
N
R(L) = ∑ || ∑(cdjLj)−Bd||2 Section2.3toextrapolatetheper-pixellayercontributionsX
d∈(r,g,b) j=1 (4) fromtheper-superpixelcontributionsL.Westartbycomput-
ingforeachpixelasetofKp nearestsuperpixelneighbors.
= ||RL−B||2 Thedistancebetweenapixelandasuperpixeliscomputed
usingtheEuclideandistancebetweenthe6-dimensionalsu-
whereB istheS×1vectorofchanneldsuperpixelcolors
d perpixelfeaturevectorandthecorrespondingvaluesforeach
and c is channel d of the color of layer j. R is a block
dj pixel. Next, we express the color of each pixel as a linear
matrixwithoveralldimension3S×(SN):
combination of the colors of its Kp superpixel neighbors.
 
cr1IS cr2IS ... crNIS Specifically,forpixelpiwecomputetheweightsqijbymin-
R= cg1IS cg2IS ... cgNIS  (5) imizing
c I c I ... c I
b1 S b2 S bN S (cid:13)(cid:13) Kp (cid:13)(cid:13)2
andBistheconcatenationofallBd. (cid:13)(cid:13)Color(pi)−∑qijColor(sij)(cid:13)(cid:13) (9)
j=1
Unity.Weencouragelayercontributionsforeachsuper-
pixeltosumuptoone,tohelpnormalizethetotalcontribu- subjecttotheconstraint∑Kj=p1qij =1.Werepresenttheset
tionthatcanaffectasinglesuperpixel.Formally: ofallqijwithaP×SmatrixQ.Finally,foreachlayer jwe
U(L)=||UL−1ˆ||2 (6) independently compute the per-pixel layer values Xj from
theper-superpixellayervaluesLjbysimplematrixmultipli-
where1ˆisaS×1columnvectorofonesandUisaS×(SN) cation:
indicatormatrixwhoserowscorrespondtosuperpixels,with
onesineachcolumnindexcorrespondingtothatsuperpixel’s Xj=QLj (10)
layercontributions:
(cid:40) Figure 2d shows the result of multiplying the Lj visual-
1 if j−i≡0 (mod S) izedinFigure2cbyQ.Observethathigh-frequencydetails
u =
ij
0 if j−i(cid:54)≡0 (mod S) suchasthesmallsliversofredinthelower-leftoftheim-
age are faithfully preserved, even though these slivers are
Explicitconstraints.Insomecases,hintsfromtheuser notcontainedinisolatedsuperpixels.
indicatingthatagivenlayershouldcontributetoagivenim-
age region can help create a better layering. For example,
3.5. BoundingLayerContributions
these hints can separate similarly-colored but semantically
differentregions.GivenCuserconstraints,wepenalizelayer Equation 8 does not constrain the range of layer contribu-
contributionswhentheydeviatefromuserconstraints: tions. Negative or large positive values can be unintuitive,
E(L)=||EL−T||2 (7) causingsmalleditstoalayer’scolortohaveadisproportion-
ateimpactonapixelcolor.Oneapproachtosolvethisprob-
whereEisaC×(SN)indicatormatrixthatselectstheuser- lemistoformulatetheoptimizationofLasaleast-squares
specified superpixel-layer contributions from L, and T is a problemwithhardboxconstraints[0,1].Althoughproblems
lengthCvectorcontainingtheuser-specifiedtargetvalues. ofthisformhaveknownsolutions,forlargesystemstheycan
When the user does not specify layer contribution con- stillbeprohibitivelyslowtosolvewhencomparedagainstan
straints,ouralgorithmautomaticallyaddsconstraintstopix- unconstrainedleast-squaressolve[Adl88].
elswithverysimilarcolorsforeachlayer.Thiscanencour-
Another approach is to first solve the unconstrained lin-
age better separation between layers in many images. For
earproblem,thenaddinboundaryconditionsthathavebeen
trickierimageswithindependentregionssharingsimilarcol-
violatedassoftconstraintsinfutureiterationsofthesolve.
ors,user-specifiedconstraintsareneededtobetterseparate
Althoughitrequiresmultipleiterations,thisapproachcanef-
theseregions.Figure3showsacasewherelettingtheuser
ficientlysolvesometypesofconvexproblems[BV04].We
assignsuperpixelstoaparticularlayerresultsinbettersepa-
firstperformtheunconstrainedsolveandcheckwhichvalues
rationbetweenthecat’swhitefurandthebackground.
arenegative.Inthenextiteration,weaddasoftconstraintfor
Θ(L)isaquadraticfunctioninLwhichcanbeminimized these values to be 0 with weight λn, adding to the explicit
bysolvingalinearsystem: constraintsinEquation7.Combinedwiththeunitytermin
Equation6,thisencouragescontributionstofallwithinthe
(cid:16) (cid:17)
λm(I−M)T(I−M)+λrRTR+λuUTU+λeETE L= range [0,1]. We refer to this approach as iterative negative
suppression.
λrRTB+λuUT1ˆ+λeETT (8)
We find that strictly enforcing the box constraints often
Figure2cvisualizesLjforonelayeroftheinputimage. makes image reconstruction from the layers very sensitive
S.Linetal./LayerBuilder:LayerDecompositionforInteractiveImageandVideoColorEditing
Auto
User
Figure3:Automaticallydeterminedconstraintsvs.user-specifiedconstraints.Usinguser-specifiedconstraintsallowsthelayers
tobetterseparatethecat’swhitefurfromthebackground.
Hardbounds
Neg.Suppr.
Figure4:Layerandreconstructionresultsusing2000superpixels.Palettecolorswerechosenautomatically.Greenindicates
layervaluesgreaterthan1whileredindicatesvalueslessthan0.Usinghardboundingconstraintsfadestheorangeinthis
image,andismoresensitivetothegamutofthepalettechosen.Usingsoftconstraintswithnegativesuppressiongivesmore
flexibilityandcanbetterreconstructtheimage.
to the initial color palette. This is because the image can- 4. Evaluation
notreconstructcolorsoutsidetheconvexhullofthepalette.
Weevaluateouralgorithmbycomparingrecoloringresults
Figure4showsacomparisonbetweenhardboxconstraints
withpreviouswork,analyzingperformancetime,andshow-
(using CVX and the SDPT3 solver) and iterative negative
ing robustness under different number of superpixels. For
suppression.Inthisexample,usinghardboxconstraintsre-
morecomparisons,pleaseseethesupplementarymaterials.
sultsinlayersthatcannotreconstructtheorangeintheorig-
inalimage,whileiterativenegativesuppressionreconstructs
theimagecorrectlyandresultsinanotherwisesimilarlay- 4.1. ComparisonsWithEditPropagation
ering.For4000superpixels,theconvexsolvertookoveran
hourtocomputealayering,whiletheunconstrainedlinear We first compare our image recoloring results with those
solvetookonlyafewseconds.Alltheresultsshowninthis of three previous algorithms: RBF interpolation [LJH10],
paperusefouriterationsofiterativenegativesuppressionto palette-based RBF interpolation [CFL∗15], and manifold-
boundthemagnitudeoflayercontributions. preserving edit propagation [CZZT12]. RBF interpolation
uses the common scribble-based paradigm for edit propa-
gation,whereasparsesetofcoloredstrokesarepropagated
throughout the input. Palette-based RBF interpolation uses
3.6. Parameters thesamepaletteeditinginputasourmethod,andmanifold-
preservingeditpropagationhasbeendemonstratedonboth
Inourexperiments,wefindthattheweightsλm=1,λr = typesofinputs.Becauseofthedifferenceininputs,itisdif-
0.5,λu=0.1,andλe=0.1,andthenumberofsuperpixel- ficulttomakedirectcomparisons.Weinsteadattempttoper-
to-superpixelandpixel-to-superpixelneighborsKs=30and formequivalenteditsacrossthethreemethods;scribblein-
Kp=10workwell.Wehavenotfoundouralgorithmtobe putisprovidedtoRBFinterpolationwhilecolorpaletteed-
very sensitive to the choice of these parameters, although
its are provided to LayerBuilder, manifold-preserving edit
it is important that λm (which preserves color blending re- propagation,andpalette-basedRBFinterpolation.
lationships)andλr(whichensuresthelayersreconstructthe
originalimage)besufficientlylargerthanλuandλe.Solving We ran each method on a variety of paintings and pho-
forlayersinRGBandLABspacealsoyieldssimilarresults. tographs. Figure 5 shows recoloring results from different
Fortheresultsshown,weuseRGBspace. methods on five images from this set. For more examples,
S.Linetal./LayerBuilder:LayerDecompositionforInteractiveImageandVideoColorEditing
Input [LJH10] [CZZT12] [CFL∗15] LayerBuilder
Figure 5: Image recoloring comparison. The first column shows the input strokes to [LJH10] and the palette change for
LayerBuilder, [CZZT12],and[CFL∗15].Coloredstrokesindicateachangeincolorontheselectedpixels,whileblackstrokes
constrain the pixels to stay the same color. Our method produces natural results even in the presence of challenging color
blendingscenarios.RBFinterpolation[LJH10]andpalette-basedRBFinterpolation[CFL∗15]producessignificantartifacts
atcolorboundariesandblendedregions.
please see the supplementary materials. We used the same hue changes in blended regions. However, they are more
parametersforallLayerBuilderresults,exceptfortheinput pronetohaloartifactsatcolorblendingboundariesasseen
palette. For the previous methods, we show the best result inFigure5.Palette-basedRBFinterpolationdoesnotcon-
thatdoesnotovershootthetargetregionforrecoloring,after sider spatial distance when shifting pixel colors, which re-
varying parameters (such as the number of nearest neigh- sultsinafastersetuptimethanLayerBuilder,butcanalso
borsK formanifold-preservingeditpropagation[CZZT12] leadtomorecross-contaminationwheneditingsimilarlycol-
andcolorweightσcforRBFinterpolation[LJH10]).Weuse oredobjects(e.g.theredcarandthemaroonsoccerjerseys
thedefaultgriddimensionof12forpalette-basedRBFin- inthesoccerimage).
terpolation[CFL∗15].
4.2. ComparisonswithMattingandIllumination-Based
Ourmethodpreservescolortransitionsandblendsaswell
Layers
asmanifold-preservingeditpropagationandismuchfaster.
After layer computation, any color change only requires Asimilarideaoflayeringisusedinimagemattingtosep-
computing a weighted sum of layer colors for each pixel, arateobjectsormaterialsinanimage.Althoughtheselay-
whilemanifold-preservingeditpropagationrequiressolving erscanbeusedforrecoloring,theycontainmanydifferent
alargelinearsystem.Inaddition,becauseLayerBuilderpe- colors.Webelievecolor-basedlayersaremoreoftensuited
nalizesnegativelayerweights,layerscanoftenberecolored to recoloring tasks as they offer finer control over individ-
toadramaticdegree,asseeninthesoccerimage.Manifold- ualcolors.InFigure6,wecomparerecoloringusinglayers
preservingeditpropagationdoesnotpreventpropagationof from LayerBuilder and KNN matting [CLT13]. The KNN
negative changes, so changing one color is more likely to pinklayermixescolorswiththewhiteflower,resultingina
leadtounintuitivechangeselsewhere,likethegreenergrass moreoverblownrecoloring.Ourcolor-basedlayerscanbet-
in the soccer image. RBF interpolation and palette-based terseparatethepinkpigmentfromtherestoftheflowerand
RBFinterpolationworkwellonsharpregionsandwithsmall fromthedefocusedareas.
S.Linetal./LayerBuilder:LayerDecompositionforInteractiveImageandVideoColorEditing
Input KNNMatting LayerBuilder
Stage Time
Superpixelcomputation 14.9s
Per-pixelLLEweighting 7.2s
Linearsolve(alliterations) 8.4s
Table1:Timingsfordifferentstagesofourlayerextraction
algorithm.
PinkLayer
4.3. TimingandComplexity
WeevaluateourlayerextractionalgorithmonaPCwithan
OtherLayers
IntelCorei73.7GHzprocessorand8GBofRAM.Wedid
notoptimizeorparallelizeourlayerextractionimplementa-
tion. Timings for computing a five-layer decomposition of
Figure6:ImagerecoloringcomparisonbetweenKNNmat- a 720x405, 70 frame video (20M pixels) are given in Ta-
ting [CLT13] and LayerBuilder. The seed points used for ble1.Weuseasingle-threadedversionofEigenforourlin-
KNN matting are shown along with our layer colors and earsolve†.Becauseoursolveisperformedonsuperpixels,
coloredits.Ourlayersbetterseparatethepinkpigmentfrom the size of the linear system remains manageable (20000
thewhite,resultinginamorenaturalcoloring. free variables). Although the time required to compute the
Input [CRA11] LayerBuilder per-pixellinear-embeddingweightsissignificant,it’sworth
notingthatthesearecomputedindependentlyforeachpixel
and could easily be parallelized. Our peak memory usage
waslessthan700MB,muchofwhichisneededtostorethe
layersforeachvideoframewithoutcompression.Oncethe
layersarecomputed,coloreditscanbecomputedatinterac-
tiverates(lessthan5msperframe).
4.4. VaryingSuperpixelCount
InFigure8,weinvestigatetheeffectofsuperpixelcounton
theextractedlayersandthequalityoftherecoloredimage.
With50superpixels,thewhiteandbluelayersarenotcor-
rectly separated and overlap significantly. Recoloring with
these layers results in undesirable color blending and does
not produce the expected saturated shade of red requested
Figure7:Imagerecoloringcomparisonbetween [CRA11]
in the recolored palette. As the number of superpixels in-
and LayerBuilder. We show the original image and the
creases,thelayersseparatemorecleanlyandthisimprove-
palette edits used for our method in the first column. Our
mentisreflectedintherecoloredimage.Forimagesshown
methodcanachieveplausiblerecoloringsforpronouncedin-
inthepaper,theextractedlayersdonotchangesignificantly
terreflectioneffectswhilehavingasimplereditingpipeline.
whenmorethan2000superpixelsareused.
Finally, we compare our method to the method by Car- 5. Applications
roll and colleagues [CRA11], which focuses on the prob-
Wedemonstrateourlayerdecompositionalgorithmonmore
lemofrecoloringindirectreflectioncolorstobeconsistent
complexrecoloringofimagesandvideo,andgiveexamples
witheditedobjectcolorsandinvolvesfirstdecomposingthe
forthreeotherapplications-automatedpatterncoloring,tex-
image into reflectance and illumination components. Fig-
turesynthesis,andlayerfiltering.
ure7showsrecoloringcomparisonsontwoortheirimages,
whichfeaturecolorblendingduetointerreflections.Carroll
andcolleagues’methodcanbetterhandlemoresubtleinter- 5.1. InteractiveRecoloringwithLayers
reflectionsinthebabyphotoduetotheirseparatehandling
Itisstraightforwardtoapplyourlayerdecompositionalgo-
of reflectance and illumination. However, we believe our
rithm to interactively recolor images and videos. Once the
methodstillachievesplausibleresultsformorepronounced
layeringisknown,recoloringanimageorvideoframewith
interreflectioneffects,suchasintheclothesphoto,aswell
asforothertypesofcolorblendingwithoutrequiringade-
compositionintoreflectanceandilluminationimages. † eigen.tuxfamily.org/
S.Linetal./LayerBuilder:LayerDecompositionforInteractiveImageandVideoColorEditing
OriginalImage 50Superpixels 250Superpixels 2000Superpixels
Figure8:Changingthenumberofsuperpixels.Top:recoloringresultwitheachsuperpixelcount.Bottom:twooftheextracted
layers,correspondingtotheblueandwhitecolorsintheoriginalimage.Usingtoofewsuperpixelscanresultinlayersthatare
notwellseparated.
3
5
2 e
e m
m a
Fra Fr
2 0
1 1
e 1
m e
m
a
Fr Fra
2
2 7
e 9
m 1
a e
Fr m
a
Fr
Figure 9: Video recoloring. Left Columns: frames from the original video. Middle and Right Columns: frames from edited
videos.Changesinthecolorpalettesofthevideos,visualizedbelow,propagatethroughtheentirevideoinreal-timeusingthe
layerdecomposition.
anewpaletteisassimpleasperformingalinearcombination state space. Lin and colleagues propose a graphical model
ofNimages.Figure1showstwoimagerecoloringexamples forscoringpatterncolorings,whereimagesarerepresented
fornaturalanddigitalimages.Thepaletteandlayerswere as non-overlapping regions of solid color [LRFH13]. The
automaticallycomputedwithnoadditionaluserconstraints. modelscorescoloringsbylearningcolordistributionsover
Thealgorithmproducespleasingresults,evenwhenunder- individual and adjacent regions from a training dataset of
goingsignificantcolorvariationsuchaschangingtheseason images,whereeachregionisrepresentedasasetofshape
inaphotographfromsummertofall. features. However, it can only recolor images that are ac-
companiedbyalayerdecompositionwhichisnotavailable
InFigure9,weshowtwoexamplesofvideorecoloring.
for most images. We adapt this model to support arbitrary
Ourmethodsupportssignificanteditsonlargepalettes,such
imagesbyaddingfeaturesthattakeintoaccountoverlapping
asmodifyingthegrassfromgreentoblueandadaptingthe
imagelayersfromLayerBuilder.Seethesupplementalmate-
othercolorsinapleasingfashion.Asdemonstratedintheac-
rialsforthelistoffeaturesconsidered.Figure10showscol-
companyingvideo,thereal-timefeedbacktochangesmade
oringsuggestionsforfourimagessampledfromtheadapted
tothepalettehelpsusersgeneratevisuallyappealingresults
modelwhichwastrainedon907COLOURLoverspatterns.
when editing both images and video. Even though we use
By using our layer decomposition algorithm, we can auto-
only4000superpixelsinalltheseimagesandvideos,there-
maticallyrecoloramuchwidervarietyofimages.
sultingimagesdonotexhibitsignificantspatialortemporal
artifactsfromthissuperpixelrepresentation.
5.3. TextureSynthesis
Texturesynthesisalgorithmstraditionallysynthesizeanim-
5.2. PatternRecoloring
age by comparing incomplete pixel neighborhoods in the
Inadditiontoallowinguserstointeractivelyrecoloranim- result with existing neighborhoods in an exemplar. These
age, we can also automatically generate coloring sugges- neighborhoods are often computed in RGB space, but can
tionsforthatimagetohelpusersbetterexplorethecoloring easily be extended to incorporate image layer features to
S.Linetal./LayerBuilder:LayerDecompositionforInteractiveImageandVideoColorEditing
Figure10:Recoloringsuggestionsfrommodeltrainedon907COLOURLoverspatterns,usinglayerpropertiesasfeatures.
Exceptforthefirstexample,luminancewasconstrainedtobeclosetotheoriginal.Weuseanautomaticpalette.
provide a more informative representation of texture fea- Exemplar RGB RGB&Layers
tures. We adapt the approach proposed by Lefebvre and
Hoppe [LH06]tocomputefeaturevectorsforeachneigh-
borhood, including both the RGB values at each pixel and
thelayerweightscorrespondingtoeachpalettecolor.Since
smallspatialneighborhoodscannotencodelargetexturefea-
turesorsemanticstructure,appearance-spacetexturesynthe-
sis and other approaches use the notion of a user-provided
feature mask to guide synthesis. Our layer decompositions
automatically generate layers that encode the same types
of detail represented by these feature masks. Figure 11
showsappearance-spacetexturesynthesisresultsusingonly
RGB features and using both RGB and layer features. Al-
thoughsynthesisremainschallengingforimageswithnon-
stationary features, for images such as the ground pattern
(Figure11secondrow)incorporatingthelayersintothelo-
calfeaturevectorbettercapturesthedistributionimpliedby
theexemplar.
5.4. LayerFiltering
Variousimageeditingoperationscanbeperformedinalo- Figure 11: Texture synthesis augmented with layers. Left:
calizedmannerbyeditingindividuallayers,andthenrecom- exemplartexturesandtheirlayerdecompositions.Anauto-
posingthem.InFigure12,weshowtheeffectsofseveralfil- maticpalettewasused.Middle:resultsoftexturesynthesis
teringoperationsonindividuallayers:aGaussianfilterem- usingpixelRGBvalues.Right:synthesisusingRGBandthe
phasizes the light in the sky, the star is embossed, and the layercontributionvectorfromeachpixel.
bluefireismotionblurred,respectively.Althoughtheedited
layersaretightlyblendedintotheimage,ourlayerscorrectly
separate the different color sources in the image and these
high-leveloperationsproducefewunintendedartifacts.