Plausible Shading Decomposition For Layered Photo Retouching

8.3 MB·

by  Carlo Innamorati

Plausible Shading Decomposition For Layered Photo Retouching
CarloInnamorati TobiasRitschel TimWeyrich NiloyMitra
UniversityCollegeLondon
7
1
0
2
b Figure1: Ourapproachautomaticallysplitsinputimagesintolayersmotivatedbylighttransport,suchas(a):occlusion,albedo,irradiance
e andspecular,or(b):thesixmajorspatiallightdirections,whichcanthenbemanipulatedindependentlyusingoff-the-shelfphotomanipulation
F softwareandcomposedbacktoanimprovedimage.For(a)shadowsweremadedeeper,albedohuechanged,saturationofirradianceincreased
andthespecularwasblurredforamoreglossymaterial.For(b)Thefrontlightingwasmadeweakerandlightfromthelefthadbeentintedred.
2

] Abstract Inthiswork,wesetouttodeviseasystemthatcombinesthestrength
R
ofbothapproaches: theabilitytoworkonrealphotos,combined
G withaseparationintolighttransportlayers.Startingfromasingle
Photographersroutinelycomposemultiplemanipulatedphotosof
s. thesamescene(layers)intoasingleimage, whichisbetterthan photograph,oursystemproducesadecompositionintolayers,which
canthenbeindividuallymanipulatedandrecombinedintothede-
c anyindividualphotocouldbealone. Similarly,3Dartistssetup
siredimageusingoff-the-shelfimagemanipulationsoftware.Fig.1
[ renderingsystemstoproducelayeredimagestocontainonlyindi-
showsanexample.
vidualaspectsofthelighttransport,whicharecomposedintothe
2
finalresultinpost-production.Regrettably,bothapproacheseither Whilemanydecompositionsarepossible,wesuggestaspecificlay-
v
takeconsiderabletimetocapture, orremainlimitedtosynthetic eringmodelthatisinspiredbyhowmanyartistsaswellaspractical
7
scenes. Inthispaper,wesuggestasystemtoallowdecomposing contemporaryrenderingsystems(e.g.,ininteractiveapplications
0
asingleimageintoaplausibleshadingdecomposition(PSD)that suchascomputergames)work:adecompositionintoshadow,dif-
5
approximateseffectssuchasshadow,diffuseillumination,albedo, fuseillumination,albedo,andspecularshading.Thismodelisnot
6
andspecularshading.Thisdecompositioncanthenbemanipulated completelyphysical,butsimple,intuitiveforartistsanditsinverse
0
inanyoff-the-shelfimagemanipulationsoftwareandrecomposited modeliseffectivelylearnable. Weformulateshadowasasingle
.
1 back.Wedosobylearningaconvolutionalneuralnetworktrained scalarfactortobrightenordarkentheappearance,resultingfrom
0 usingsyntheticdata.Wedemonstratetheeffectivenessofourdecom- addingthediffuseandspecularshading.Thediffuseshadingisfur-
7 positiononsynthetic(i.e.,rendered)andrealdata(i.e.,photographs), therdecomposedintoillumination(colorofthelight)andreflectance
1 andusethemforcommonphotomanipulation, whicharenearly (coloroftheobject),whilethespecularshadingismodeleddirectly.
: impossibletoperformotherwisefromsingleimages. Toinvertthismodel,weemployadeepconvolutionalarchitecture
v
(CNN)thatistrainedusingsyntheticdata, forwhichtheground
i
X truth-decompositionofaphotointolighttransportlayersisknown.
1 Introduction
r Insummary,wemakethefollowingcontributions:
a
Professionalphotographersregularlycomposemultiplephotosof
• splittingandre-combinationofimagesbasedonlighttransport
the same scene into one image, giving themselves more flexibil-
layers(shadow,diffuselight,albedoandspecularshading);
ity and artistic freedom than achievable by capturing in a single
photo.Theydoso,by‘decomposing’thesceneintoindividuallay- • aCNNtrainedonsyntheticdatatoperformsuchasplit;and
ers,e.g.,bychangingthelight,manipulatingtheindividuallayers
• evaluatingourapproachonarangeofrealphotographsand
(e.g.,typicallyusingasoftwaresuchasAdobePhotoshop), and
demonstratingutilityforphoto-manipulations.
thencomposingthemintoasingleimage.Onotheroccasionsthis
processiscalledstacking.Unfortunately,thisprocessrequiresthe
effortofsettingupandtakingmultipleimages.Analternativethat 2 Previous Work
overcomesthislimitationisrenderingsyntheticimages.Inthiscase,
theimagecanbeclearlydecomposedintotheindividualaspects Combiningmultiplephotos(alsoreferredtoasa“stack”)ofascene
of light transports (e.g., specular highlights vs. diffuse shading). where one aspect has changed in each layer is routinely used in
Thelightpathnotation[Heckbert1990]providesastrictcriterion computergraphics[Cohenetal.2003].Forexample,NVIDIAIRay
forthisdecomposition. Regrettably,thisrequirestheimagetobe activelysupportsrenderedLPElayers(lightpathexpressions[Heck-
“synthesizable”,i.e.,material,geometry,andilluminationshouldbe bert 1990]) to be individually edited to simplify post-processing
knownaswellasasuitablesimulationalgorithm.Thisisoftennot towardsartisticeffectswithoutresortingtosolvingtheinverseren-
thecaseforscenesobtainedassingleimages. deringproblem.Oneaspecttochangeisillumination,suchasflash-
no-flash[EisemannandDurand2004]orexposurelevels[Mertens choice,alsobeyondwhatisphysicallycorrect. Forexample,the
etal.2009].Moreadvancedeffectinvolvedirectionoflight[Akers StyLitsystemproposedbyFisˇseretal.[2016]correlatesartisticstyle
etal.2003;Rusinkiewiczetal.2006;Fattaletal.2007],eventually withlighttransportexpressions.Specifically,itrequirespixelsinthe
resultinginamoresophisticateduserinterface[Boyadzhievetal. imagetobelabeledwithlightpathinformation,e.g.,byrendering
2013]. Alltheseapproachesrequirespecializedcapturetogather andaligning. Hence,itcantaketheoutputofourfactorizationto
multipleimagescapturedbymakinginvasivechangestothescene, enable stylization single photographs without being restricted to
limitingtheiruseinpracticetochangeanimagepost-capture.Infact, renderedcontent.
severalwebsitesanddedicatedYouTubechannelshaveemergedto
provideDIYinstructionstosetupsuchstudioconfigurations. 3 Our Approach
For single images, a more classic approach is to perform intrin-
sicdecompositionintoshading(irradiance)anddiffusereflectance Overview. Oursystemhasthreemaincomponents:(i)producing
(albedo)[BarrowandTenenbaum1978;Garcesetal.2012;Belletal. trainingdata(Sec.3.2);(ii)aconvolutionalneuralnetworktode-
2014],possiblysupportedbyadedicatedUIforimages[Bousseau composesingleimagesintolighttransportlayers(Sec.3.3); and
et al. 2009; Boyadzhiev et al. 2012], using annotated data [Bell aninteractivesystemtomanipulatethelighttransportlayersbefore
etal.2014],orvideos[Yeetal.2014; Bonneeletal.2014]. Re- recomposingthemintoanimage(Sec.3.5).
cently,CNNshavebeensuccessfullyappliedtothistaskproducing
Thetrainingdata(Sec.3.2)isproducedbyrenderingalargenumber
state-of-the-artresults[Narihiraetal.2015].
of 3D scenes into image tuples, where the first is the composed
We also using a data-driven CNN-based approach to go beyond image,whiletheotherimagesarethelighttransportlayers. This
classic intrinsic image decomposition light transport layers with stepneedsonlytobeperformedonceandthetrainingdatawillbe
furtherseparationintoocclusionandspecularcomponents,thatare madeavailableuponpublication.
routinelyusedwhenpost-compositinglayeredrenderings(seeSec.4
Thelayerdecomposition(Sec.3.3)isdoneusingaCNNthatcon-
andsupplementarymaterials).
sumesaphotoandoutputsallitslighttransportlayers.ThisCNNis
trainedusingthetrainingdatafromthepreviousstep.Weselected
Inotherrelatedefforts,researchershavelookedintofactorizingcom-
aconvolution-deconvolutionarchitecturethatisonlytobetrained
ponents,suchasspecular[Tanetal.2004;Mallicketal.2006]from
once,canbeexecutedefficientlyonnewinputimages,anditsdefi-
singleimages,orambiantocclusion(AO)fromsingle[Yangetal.
nitionwillbemadepubliclyavailableuponpublication(pleaserefer
2015]ormultiplecaptures[Hauaggeetal.2013].Weshowthatour
tothesupplementaryforthearchitecture).
approachcansolvethisproblematacomparablequality,butrequires
onlyasinglephotoandincombinationyieldsfurtherseparationof Optionally,weemployanupsamplingstepSec.3.4,thatre-samples
diffuseshadingandalbedowithoutrequiringaspecializedmethod. the fixed-resolution light transport layer CNN output, such that
composingtheminthearbitrarilyhighresolutionoftheoriginal
Despitetheadvancesinrecoveringreflectance(e.g.,withtwocap-
resolutionproducespreciselytheoriginalimagewithoutanybias,
tures and a stationarity assumption [Aittala et al. 2015], or with
blur,ordrift.
dedicatedUIs[Dongetal.2011]),illumination(e.g.,Lalondeetal.
[2009]estimateskyenvironmentmapsandRematasetal.[2016] Finally,wesuggestasystem(Sec.3.5)thatexecutestheCNNona
reflectancemaps)anddepth(e.g.,Eigenetal.[2014]useaCNN photoatdeploymenttimetoproducelighttransportlayers,which
to estimate depth) from photographs, no system doing a practi- canthenbeindividuallyandinteractivelymanipulated,inanyoff-the-
caljointdecompositionisknown. Mostrelevanttooureffort,is shelfimagemanipulationsoftware,allowingoperationsonphotos
SIRFS[BarronandMalik2015]thatbuilddata-drivenpriorsfor thatpreviouslywereonlypossibleonlayeredrenderings,orusing
shape,reflectance,illumination,andusetheminanoptimization multiplecaptures.
setuptorecoverthemostlikelyshape,reflectance,andillumination
underthesepriors(seeSec.4forexplicitcomparisontoSIRFS). Beforedetailingallthethreesteps,wewillnextintroducethespe-
cificimageformationmodelunderlyingourframeworkinSec.3.1.
Inthecontextofimagemanipulations,specializedsolutionsexist:
Ohetal.[2001]representasceneasalayeredcollectionofcolor
3.1 Model
anddepthtoenabledistortion-freecopyingofpartsofaphotograph,
andallowdiscountingeffectofilluminationonuniformlytextured
Weproposetwodifferentimageformationmodels: onethatisin-
areasusingbilateralfiltering;Khanetal.[2006]enableautomati-
variantunderlightdirectionandonethatcapturesthedirectional
callyreplacingonematerialwithanother(e.g.,increase/decrease
dependency.
specularity,transparency,etc.)startingfromasinglehighdynamic
range image by exploiting our ‘blindness’ to certain physical in-
accuracies;Carolletal.[2011]achieveconsistentmanipulationof
inter-reflections;orthesystemofKarschetal.[2011]thatcombines
manyoftheaboveintostate-of-theartandcompellingaugmented
imagesynthesis.
Splittingintolightpathlayersistypicalinrenderinginspiredbythe
classiclightpathnotation[Heckbert1990].Inthiswork,different
fromHeckbbert’sphysicalE(S|D)∗Lformalism,weuseamoreedit-
friendlyfactorizationintoshadow, diffuselight, diffusematerial,
andspecular,insteadofseparatingdirectandindirecteffects.While
alltheaboveworksonphotos,itwasacknowledgedthatrendering Figure2:Thecomponentsofourtwoimagingmodels.
beyondthelawsofphysicscanbeusefultoachievedifferentartistic
goals[Todoetal.2007; Vergneetal.2009; Ritscheletal.2010;
Richardtetal.2014;Dongetal.2015;Schmidtetal.2015]. Our Non-directionalModel. WemodelthecolorCofapixelas
approach naturally supports this option, allowing users to freely
changelighttransportlayers,usinganyimage-levelsoftwareoftheir C=O(ρI+S), (1)
whereO∈(0,1)∈Rdenotestheocclusion,whichisthefractionof Shape. Shapegeometrycomprisesof300randomcarsfromfrom
directionsintheupperhemispherethatisblockedfromthelight;the ShapeNet[Changetal.2015].Notethatthemodelswereassumed
variableI∈(0,1)3∈R3denotesthediffuseillumination(irradadi- tobeupright. Thisclasswaschosen, asitpresentsbothsmooth
ance),i.e.,colorofthelight,withoutanydirectionaldependence; surfacesaswellashardedgestypicalformechanicalobjects.Note
ρ∈(0,1)3∈R3describesthealbedo(diffusereflectance),i.e.,the thatourresultsshowmanyclassesverydifferentfromcars,such
colorofthesurfaceitself;andfinally,S∈(0,1)3∈R3isthespecu- asfruits,statues,mechanicalappliances,etc. Pleasenotethatwe
larshading,wherewedonotseparatebetweenthereflectanceand specificallyrestrictedtrainingtoonlycarstoevaluatehowtheCNN
theillumination,anddonotcaptureanydirectionaldependence. generalizes to other object classes. Other problems like optical
flowhavebeensolvedusingCNNsongeneralscenesdespitebeing
trainedonverylimitedgeometry,suchastrainingexclusivelyon
DirectionalModel. Thedirecitonalmodelis chairs[Dosovitskiyetal.2015].
N (cid:90)
C=∑R( Li(ω)bi(ω)dω), (2) Reflectance. Reflectanceusingthephysically-correctedPhong
i=1 Ω model [Lafortune and Willems 1994], sampled as follows: The
diffusecolorscomedirectlyfromShapeNetmodels.Thespecular
whereL istheincominglight,andRthereflectionoperator,map-
pinginciomingtooutgoinglight.Inotherwords,weexpresspixel componentksisassumedtobeasinglecolor. Arandomdecision
ismadeifthematerialisassumedtobeelectricordielectric. Ifit
colorasasumofreflectionsofnbasisilluminations.
iselectric,wechoosethespecularcolortobetheaveragecolorof
thediffusetexture.Otherwise,wechooseittobeauniformrandom
Htoe1rea,t(ebv1e,r.y..d,ibrne)ctcioann,bi.ee.a,n∑ynisbeit(ωof)s=ph1e(rpicaarltiftiuonnctoiofnusnitthya)t.sOunme vgareluyevianluUe[.0,G1l]o.ssinessissetasn=3.010ξ,whereξ isarandom
suchdecompositionisthesphericalharmonicsbasisofanyorderor
thecubebasis(thatisoneforasinglecubeface).Inourapproach,
wesuggesttouseanovelsoftcubedecompositionthatcombines
Illumination. Illumination is sampled from a set of 122 HDR
strengthsofboth:Itifveryselectiveinthedirectionaldomain,has
environmentmapsinresolution512×256thathaveanuncalibrated
finitelymanycomponentsbutalsodoesnotintroduceasharpcut
absoluterangeofvaluesbutarerepresentativefortypicallighting
inthedirectionaldomain.Itisdefinedastheclampeddotproduct
settings:indoor,outdoor,aswellasstudiolights.
betweenthei-thcubesidedirectionciraisedtoasharpeningpower
σ=20:
Rendering. After fixing shape, material, and illumination, we
6
bi(ω)=max((cid:104)ω,ci(cid:105),0)σ/∑max((cid:10)ω,cj(cid:11),0)σ. synthesize a single image from a random view (rotation is only
aboutvertical).TocomputeC,wecomputeallcomponentsindivid-
j=1
ually,andcomposethemaccordingtoEq.1. Theocclusionterm
Oiscomputedusingscreen-spaceocclusion[Ritscheletal.2009].
An additional benefit of the (soft) cube decomposition is, that it
ThediffuseshadingIiscomputedusingpre-computedirradiance
is,otherthanSH,isstrictlypositive,facilitatingloadingoflayers
environmentmaps[RamamoorthiandHanrahan2001].Similarly,
intoapplicationsthatdonot(well)supportnegativevalues,suchas
Photoshop.Otherbasesarepossibleinthisframework,allowingto specular shading is the product of the specular color ks selected
accordingtotheaboveprotocol,andapre-convolvedillumination
tailorittospecificdomain,whereaprioronlightdirectionsmight
exist(e.g.,portraitphotos). map for gloss level n. Diffuse albedo ks is directly available in
ShapeNet.Whilewecouldalsotrytoinfertheglossiness,itwould
There are many values of O, I, ρ and S to explain an observed notbeclearhowtouseitsnon-lineareffectwithclassiclayering.
colorC,sothedecompositionisnotunique.Invertingthismapping Noindirectilluminationorlocalinteractionsarerendered. When
fromasingleobservationislikelytobeimpossible. Atthesame learningthedirectional-dependentvariant,werendersiximages,
time,humansareclearlyabletosolvethistask. Oneexplanation wheretheilluminationwaspre-convolvedwiththei-thdecomposed
canbethattheyrelyoncontext,onthespatialstatisticsofmultiple illumination.
observations c(x), such that a decomposition into light transport
layersbecomespossible.Inotherwords,simplynotallarrangements Whilethisimagesynthesisisfarfrombeingphysicallyaccurate,it
ofdecompositionsareequallylikely.Asdescribednext,weemploy canbeproducedeasily,systematicallyandforaverylargenumber
aCNNtolearnthisdecompositioninasimilarfashion. ofimages,makingitsuitableforlearningthelayerstatistics.Overall
weproduce100kimagesinaresolutionof256×256(ca.10GB)
in5hoursonacurrentPCwithadecentGPU.
Units. Carehastobetakeninwhatcolorspacelearnedandtrain-
ingdataistobeprocessed.AstheilluminationisHDR,theresulting
imageisanHDRrendering.However,asourinputimageswillbe
LDRatdeploymenttime, weneedtomatchtheirrange. Tothis
end,automaticexposurecontrolisusedtomapthosevaluesintothe
Figure3:Samplesfromoursetofsynthetictrainingdata.
LDRrange,byselectingthe0.95luminancepercentileofarandom
subsetofthepixelsandscaleallvaluessuchthatthisvaluemaps
to1. TherenderedresultCisstoredaftergamma-correctionwith
3.2 Trainingdata γ=2.0.Allothercomponentsarestoredinphysicallylinearunits
(γ =1.0) and are processed in physical linear units by the CNN
Trainingdatacomprisesofsyntheticimagesthatshowarandom andtheend-applicationusingthelayers. Doingthefinalgamma-
shape,withpartiallyrandomreflectanceshadedbyrandomenviron- correctionwillconsequentiallybeuptotheapplicationusingthe
mentmapillumination. layerslateron(asshowninoureditexamples).
Figure4:Decompositionofinputimages(left)intothesixdirectionallayers(left)fordifferentobjects.
3.3 Decomposition havebeentrainedfor12hours).Amoredetaileddescriptionofthe
network’sarchitecturecanbefoundinthesupplementalmaterials.
WeperformdecompositionusingaCNN[Krizhevskyetal.2012]
trainedusingthedataproducedasdescribedabove. Inputtothe 3.4 Upsampling
network is a single image such as a photograph. Output for the
non-directionalvariantarethefourimages(occlusion,diffuseillu-
Upsampling (Fig. 5) is an optional step applied to input images
mination,albedo,specularshading),thelighttransportlayers,where
thathavearesolutionarbitrarilyhigherthantheonetheCNNis
occlusionisscalarandtheothersarethree-vector-valued. Output
trained on (256×256). Input to this process are all the layers in
forthedirectionalvariant,theoutputare6directionaldiffuseand
the low resolution. Output are the layers in that arbitrarily high
specularlayers,tobecomposedwithoneAO,andonealbedolayer.
resolution,suchthatapplyingthecompositionequationresultsin
Thisdesignfollowstheconvolution-deconvolutionwithcrosslinks thehigh-resolutionimage.
idea,resultinginanhourglassscheme[Ronnebergeretal.2015].
Thenetworkisfully-convolutional.Westartataresolutionof256×
256thatisreduceddownto2×2throughstride-twoconvolutions.
Wethenperformtwostride-oneconvolutionstoincreasethenumber
offeaturelayersinaccordancetotherequirednumberofoutput
layers (i.e. quadruple for the light transport layers, sextuple for
thedirectionallightlayers).Thedeconvolutionpartofthenetwork
comprisesofblocksperformingaresize-convolution(upsampling
followedbyastride-oneconvolution),crosslinkingandastride-one
convolution. Every convolution in the network is followed by a
Figure5:OurCNNcomputesadecompositioninafixedresolution
ReLU[NairandHinton2010]non-linearityexceptforthelastlayer,
of256×256withtheresults(topinsets).GiventheHDoriginal,we
forwhichaSigmoidnon-linearityisusedinstead. Thisisdoneto
performupsampling(bottominsets)thatassurestheycombineto
normalizetheoutputtotherange[0,1]. Imageslargerorsmaller
theHDinputwhenblended.
thantherequiredinputsizeof256×256willbeappropriatelyscaled
and/orpaddedtobesquarewithwhitepixels. Allreceptivefields
are3×3pixelsinsizeexceptforthefirstandlasttwolayersthat Thisisachievedasfollows,independentlyforeveryhigh-resolution
are5×5. pixelandin100iterationsperpixel: Initiallyallvaluearesetto
theCNNoutput. Ateachiteration,weholdalllayercomponents
Asthelossfunction,wecombineaper-lighttransportlayerL2loss fixed,andinturnsolveforthemissingone,giventhecolor. We
withanovelthree-foldrecombinationloss,thatencouragesthenet- thenblendthisresultgradually(weight0.001)withtheprevious
worktoproducecombinationsthatresultintheinputimageand result.Thisisrepeatedforalllayers.Additionally,thelightlayeris
fulfilsthefollowingrequirements: (i)thelayershavetoproduce forcedtonotchangethechroma.WhenvaluesleavetheunitRGB
theinput,soC=AO(Iρ+S);(ii)thecomponentsshouldexplain cube,theyareback-projected.Theresultisalayeringinanarbitrary
the image without AO, i.e.,C/AO=Iρ+S; and (iii) diffuse re- resolutionthatfollowstheCNNdecomposition,yetproducesthe
flectedlightshouldexplaintheimagewithoutAOandspecular,so highresolutionimageprecisely(energy-conserving).Animmediate
C/AO−S=Iρ.Ifthenetworkwasabletoalwaysperformaperfect practical consequence of this is, that any image loaded into our
decomposition,theL2lossalonewouldbesufficient.Asitmakes systemafterthedecompositionintolayerslookspreciselylikethe
errorsinpractice,thesecondlossbiasesthoseerrorstoatleasthap- inputwithoutanyinitialbias(blurrorcolorshift)introducedbythe
peninsuchawaythatthecombinedresultdoesnotdeviatefrom CNNprocessing.Pleasenotethatweexplicitlymentionupsampling
theinput.AlllossesareinthesameRGB-differencerangeandwere fortheresultswhereweusethismode(onlyforedits).
weightedequallyforsimplicity.
Overall,thenetworkisaratherstraight-forwardmoderndesign,but 3.5 Composition
trainedtosolveanoveltask(lighttransportlayers)onnovelkind
oftrainingdata(synthesized,directionally-dependantinformation). For composition any arbitrary software that can handle layering,
WeusedTensorFlow[Changetal.2015]forourimplementation suchasAdobePhotoshopandAdobeAfterEffects, canbeused.
platformandeachmodelrequiresonlyseveralhourstotrain(both We do not limit the manipulation to produce a composition that
Figure7:Directionalediting(Seetext“Edit”Sec.4).
Figure6:O·(ρ·I+S)editing(Seetext“Edit”Sec.4).
is physically valid, because this is typically limiting the artistic changingtheweightofallvaluesinalayer,andlocalmanipulations,
freedomatthispartofthepipeline[Todoetal.2007;Ritscheletal. suchasblurringthehighlightsoralbedoindividually.
2009;Schmidtetal.2015].Ourdecompositionissosimplethatit
canbeimplementedusingaPhotoshopmacrothatmerelysetsthe
InFig.6,thefirstcar(a)changethealbedohuewithoutaffecting
appropriateadditiveandmultiplicativeblendmodes,followedbya
thehighlightcolor.Thesecondcar(b)removedthediffusepartre-
finalgammamapping.Thecontentisthenreadytobemanipulated
sultinginsveryspecularcar.Thebanana(c)imageshowsincreased
withexistingtoolswithWYSIWYGfeedback.
highlights and deepened shadows. The first shoe image (d) was
mademorespecularandthesecond(e)less,whilealsochanging
4 Results albedohueandmakingshadowsdarker.Finally,thestatuematerial
(f)waschangedtoplasterbyremovingspecularandsettingalbedo
Wereportresultsinformoftypicaldecompositionsonimages,edits toidentity, tobronzebyremovingdiffuseshadingandtoyellow
enabledbythisdecomposition,andnumericevaluation. Thefull plasticbyadjustingallcomponents.
supplementalmaterialwithmanymoredecompositionsisfoundat
geometry.cs.ucl.ac.uk/projects/2017/PSD/results.php. InFig.7thefirstedit(a)changedthehueoftherightcolortoblue.
Thebacklightonthemouse(b)wasturnedvioletandthefrontlight
green.Thebee(c)islitmoreformthesidewithcoloredlight.The
Decompositions. Howwellanetworkperformsisbestseenwhen
bronzestatue(d)isleftblueishfromtheleft.Thestatue(e)isedit
applyingittorealimages.Regrettably,wedonotknowthereference
tobelitfromtheback. Thestrawberyillumination(f)wasmade
lighttransportlayer-decompositionordirectionaldecomposition,
coloredfromthetop.
sothequalitycanonlybejudgedqualitatively. Therefore,results
ofdecomposingimagesintolighttransportlayersisseeninFig.8
whiledecompositionintodirectionsisshowinFig.4.
Numericalevaluation. IntrinsicimagesassumeStobezero(no
Edits.TypicaleditsareshowninFig.6andthedirectionalvariant specular)andcombineourtermsoandD, theocclusionandthe
inFig.7.Notethatwesupportbothglobalmanipulations,suchas diffuseillumination,intoasingle“shading”termthatisseparated
Figure8:Decompositionofinputimagesintolighttransportlayers.Pleasesee“Decomposition”inSec.4.
fromthereflectanceρ. againstthesametechniquesbutonourtestdataset.
C=O(ρ·I+S)≈O(ρ·I+0)= O·I ·ρ (3)
(cid:124)(cid:123)(cid:122)(cid:125) Limitations. LikeinmanyCNNbasedlearningapproaches,the
ShadingI(cid:48)
shortcomingofourtwonetworksarehardtopindown.Notsurpris-
inglytheyperformwellontrainingdataandgeneralizesreasonably
Acomparisononofourdecompositionandtypicalapproachesto acrossotherobjectclasses,stilltheyfailwhentheyseecompletely
generateintrinsicimagesisshowninFig.9.Intable1wecompare newtypeofdata. Whileoneobviouswaytotrytoimproveper-
Figure9:Comparisonofourapproachtothreedifferentreflectanceandshadingestimationtechniques[BarronandMalik2015],[Belletal.
2014],[Narihiraetal.2015].Weruntheirmethodonrealimagesandcomparetheirresultstoours.
Method DSSIM NRMSE withimage-basedrelighting. InProc.IEEEVIS.
Ours .0661±.0146 .3323±.1310 BARRON, J.T., ANDMALIK, J. 2015. Shape,illumination,and
DI .0862±.0165 .7698±.4818 reflectancefromshading. IEEEPAMI.
IIW .0775±.0158 .7698±.4594
SIRFS .0846±.0187 1.315±1.074 BARROW,H., ANDTENENBAUM,J. 1978. Recoveringintrinsic
scenecharacteristics. Comput.Vis.Syst..
Table1:Evaluationofourtestdatasetandotherintrinsicimageal-
BELL,S.,BALA,K.,ANDSNAVELY,N. 2014. Intrinsicimagesin
gorithms.Wereportthemeanandstandarddeviationresultsoftwo
thewild. ACMTrans.Graph.(Proc.SIGGRAPH)33,4,159.
well-knownerrormetrics:DSSIMandNRMSE.Weruntheexperi-
mentonabatchof100examplesfromourtestdatasetcomparing
BONNEEL,N.,SUNKAVALLI,K.,TOMPKIN,J.,SUN,D.,PARIS,
thegrounthtruthalbedotoourresultsandourcompetitors’.
S.,ANDPFISTER,H. 2014. Interactiveintrinsicvideoediting.
ACMTrans.Graph.(Proc.SIGGRAPHAsia)33,6.
formancewouldbetoaddmoretrainingdata(e.g.,differenttypes BOUSSEAU,A.,PARIS,S.,ANDDURAND,F. 2009. User-assisted
intrinsicimages.
of shape families, different illumination and materials, etc.) we
wouldliketounderstandbetterwhatdatasetstoaddtomaximize
BOYADZHIEV,I.,BALA,K.,PARIS,S.,ANDDURAND,F. 2012.
improvement.ThisremainsanelusivegoalinCNN-basedsystems
User-guidedwhitebalanceformixedlightingconditions. ACM
asofnow.
Trans.Graph.31,6.
5 Conclusion BOYADZHIEV,I.,PARIS,S.,ANDBALA,K. 2013. User-assisted
imagecompositingforphotographiclighting.ACMTrans.Graph.
32,4.
Wehavesuggestedthefirstdecompositionofgeneralimagesinto
lighttransportlayers,thatwerepreviouslyonlypossibleeitheron
CARROLL,R.,RAMAMOORTHI,R.,ANDAGRAWALA,M. 2011.
syntheticimages,orwhencapturingmultipleimagesandmanipu-
Illuminationdecompositionformaterialrecoloringwithconsis-
latingthescene.Wehaveshownthatovercomingtheselimitations tentinterreflections. ACMTrans.Graph.30,4.
allowsproducinghigh-qualityimages,butitalsosavescapturetime
andremovesthelimitationtorenderings.Futureworkcouldinves- CHANG,A.X.,FUNKHOUSER,T.A.,GUIBAS,L.J.,HANRAHAN,
tigateotherdecompositionssuchasglobalanddirectillumination, P.,HUANG,Q.,LI,Z.,SAVARESE,S.,SAVVA,M.,SONG,S.,
sub-surface-scatteringordirectionalilluminationorotherinputs, SU, H., XIAO, J., YI, L., AND YU, F. 2015. Shapenet: An
suchasvideos. information-rich3dmodelrepository. CoRRabs/1512.03012.
COHEN,M.F.,COLBURN,A.,ANDDRUCKER,S. 2003. Image
References
stacks.
AITTALA,M.,WEYRICH,T.,ANDLEHTINEN,J. 2015. Two-shot DONG,Y.,TONG,X.,PELLACINI,F.,ANDGUO,B. 2011. App-
svBRDFcaptureforstationarymaterials. ACMTrans.Graph Gen:interactivematerialmodelingfromasingleimage.
(Proc.SIGRAPH)34,4.
DONG, B., DONG, Y., TONG, X., AND PEERS, P. 2015.
AKERS,D.,LOSASSO,F.,KLINGNER,J.,AGRAWALA,M.,RICK, Measurement-based editing of diffuse albedo with consistent
J., ANDHANRAHAN, P. 2003. Conveyingshapeandfeatures interreflections. ACMTrans.Graph.34,4.
DOSOVITSKIY, A., FISCHERY, P., ILG, E., HAZIRBAS, C., RICHARDT, C., LOPEZ-MORENO, J., BOUSSEAU, A.,
GOLKOV,V.,VANDERSMAGT,P.,CREMERS,D.,ANDBROX, AGRAWALA, M., AND DRETTAKIS, G. 2014. Vectoris-
T. 2015. Flownet: Learning optical flow with convolutional ingbitmapsintosemi-transparentgradientlayers. Comp.Graph.
networks. InProc.ICCV,2758–66. Forum(Proc.EGSR)33,4,11–19.
EIGEN, D., PUHRSCH, C., AND FERGUS, R. 2014. Depthmap RITSCHEL,T.,GROSCH,T.,ANDSEIDEL,H.-P. 2009. Approx-
predictionfromasingleimageusingamulti-scaledeepnetwork. imatingdynamicglobalilluminationinimagespace. InProc.
InProc.NIPS. I3D.
EISEMANN, E., ANDDURAND, F. 2004. Flashphotographyen- RITSCHEL,T.,THORMA¨HLEN,T.,DACHSBACHER,C.,KAUTZ,
hancementviaintrinsicrelighting. ACMTrans.Graph.(Proc. J., AND SEIDEL, H.-P. 2010. Interactive on-surface signal
SIGGRAPH)23,3. deformation. InACMTrans.Graph.(Proc.SIGGRAPH),vol.29.
FATTAL, R., AGRAWALA, M., AND RUSINKIEWICZ, S. 2007. RONNEBERGER, O., FISCHER, P., AND BROX, T. 2015. U-net:
Multiscaleshapeanddetailenhancementfrommulti-lightimage Convolutionalnetworksforbiomedicalimagesegmentation. In
collections. ACMTrans.Graph.26,3. Proc.Med.ImageComp.andComp.-AssistedInt.
FISˇER, J., JAMRISˇKA, O., LUKA´Cˇ, M., SHECHTMAN, E., RUSINKIEWICZ,S.,BURNS,M.,ANDDECARLO,D. 2006. Ex-
ASENTE, P., LU, J., AND SY´KORA, D. 2016. StyLit: aggeratedshadingfordepictingshapeanddetail. ACMTrans.
Graph.(Proc.SIGGRAPH)25,3.
Illumination-guidedexample-basedstylizationof3Drenderings.
ACMTrans.Graph.(Proc.SIGGRAPH)35,4. SCHMIDT, T.-W., PELLACINI, F., NOWROUZEZAHRAI, D.,
GARCES,E.,MUNOZ,A.,LOPEZ-MORENO,J.,ANDGUTIERREZ, JAROSZ, W., AND DACHSBACHER, C. 2015. Stateoftheart
inartisticeditingofappearance,lightingandmaterial. InComp.
D. 2012. Intrinsicimagesbyclustering.
Graph.Forum.
HAUAGGE, D., WEHRWEIN, S., BALA, K., AND SNAVELY, N.
2013. Photometricambientocclusion. InIEEECVPR. TAN, R. T., NISHINO, K., AND IKEUCHI, K. 2004. Separating
reflectioncomponentsbasedonchromaticityandnoiseanalysis.
HECKBERT,P.S.1990.Adaptiveradiositytexturesforbidirectional IEEEPAMI26,10.
raytracing. ACMSIGGRAPHComputerGraphics24,4.
TODO,H.,ANJYO,K.-I.,BAXTER,W.,ANDIGARASHI,T. 2007.
KARSCH,K.,HEDAU,V.,FORSYTH,D.,ANDHOIEM,D. 2011. Locallycontrollablestylizedshading. ACMTrans.Graph.(Proc.
Renderingsyntheticobjectsintolegacyphotographs. InACM SIGGRAPH)26,3,17.
Trans.Graph.(Proc.SIGGRAPHAsia),vol.30.
VERGNE,R.,PACANOWSKI,R.,BARLA,P.,GRANIER,X.,AND
KHAN,E.A.,REINHARD,E.,FLEMING,R.W.,ANDBU¨LTHOFF, SCHLICK,C.2009.Lightwarpingforenhancedsurfacedepiction.
H.H. 2006. Image-basedmaterialediting. ACMTrans.Graph. InACMTrans.Graph.(Proc.SIGGRAPH),vol.28,ACM.
(Proc.SIGGRAPH)25,3.
YANG,W.,JI,Y.,LIN,H.,YANG,Y.,BINGKANG,S.,ANDYU,
J. 2015. Ambientocclusionviacompressivevisibilityestimation.
KRIZHEVSKY, A., SUTSKEVER, I., AND HINTON, G. E. 2012.
InProc.CVPR.
Imagenetclassificationwithdeepconvolutionalneuralnetworks.
InNIPS. YE, G., GARCES, E., LIU, Y., DAI, Q., AND GUTIERREZ, D.
2014.Intrinsicvideoandapplications.ACMTrans.Graph.(Proc.
LAFORTUNE, E. P., AND WILLEMS, Y. D. 1994. Using the
SIGGRAPH)33,4,80.
modifiedphongreflectancemodelforphysicallybasedrendering.
LALONDE,J.-F.,EFROS,A.A.,ANDNARASIMHAN,S.G. 2009.
Estimatingnaturalilluminationfromasingleoutdoorimage. In
Proc.ICCV.
MALLICK,S.P.,ZICKLER,T.,BELHUMEUR,P.N.,ANDKRIEG-
MAN,D.J. 2006. Specularityremovalinimagesandvideos:A
PDEapproach. InECCV.
MERTENS,T.,KAUTZ,J.,ANDVANREETH,F. 2009. Exposure
fusion:Asimpleandpracticalalternativetohighdynamicrange
photography. Comp.Graph.Forum(Proc.PG)28,1.
NAIR,V.,ANDHINTON,G.E.2010.Rectifiedlinearunitsimprove
restrictedboltzmannmachines. InProc.ICML,807–14.
NARIHIRA,T.,MAIRE,M.,ANDYU,S.X. 2015. Directintrin-
sics:Learningalbedo-shadingdecompositionbyconvolutional
regression. InProc.ICCV.
OH, B. M., CHEN, M., DORSEY, J., AND DURAND, F. 2001.
Image-basedmodelingandphotoediting. InProc.SIGGRAPH.
RAMAMOORTHI,R.,ANDHANRAHAN,P.2001.Anefficientrepre-
sentationforirradianceenvironmentmaps. InProc.SIGGRAPH.
REMATAS, K., RITSCHEL, T., FRITZ, M., GAVVES, E., AND
TUYTELAARS,T. 2016. Deepreflectancemaps. InCVPR.