PixelCNN++: Improving the PixelCNN with Discretized Logistic Mixture Likelihood and Other Modifications

2.5 MB·

by  Tim Salimans

UnderreviewasaconferencepaperatICLR2017
PIXELCNN++: IMPROVING THE PIXELCNN WITH
DISCRETIZED LOGISTIC MIXTURE LIKELIHOOD AND
OTHER MODIFICATIONS
TimSalimans,AndrejKarpathy,XiChen,DiederikP.Kingma
tim,karpathy,peter,dpkingma @openai.com
{ }
7
1 ABSTRACT
0
2 PixelCNNs are a recently proposed class of powerful generative models with

n tractable likelihood. Here we discuss our implementation of PixelCNNs which
a we make available at https://github.com/openai/pixel-cnn. Our
J implementationcontainsanumberofmodificationstotheoriginalmodelthatboth

9 simplifyitsstructureandimproveitsperformance.1)Weuseadiscretizedlogistic
1 mixturelikelihoodonthepixels,ratherthana256-waysoftmax,whichwefindto
speeduptraining.2)Weconditiononwholepixels,ratherthanR/G/Bsub-pixels,

] simplifying the model structure. 3) We use downsampling to efficiently capture
G
structure at multiple resolutions. 4) We introduce additional short-cut connec-
L tionstofurtherspeedupoptimization. 5)Weregularizethemodelusingdropout.
. Finally,wepresentstate-of-the-artloglikelihoodresultsonCIFAR-10todemon-
s
c stratetheusefulnessofthesemodifications.
[

1 1 INTRODUCTION
v
7
1 ThePixelCNN,introducedbyvandenOordetal.(2016b),isagenerativemodelofimageswitha
5 tractablelikelihood. Themodelfullyfactorizestheprobabilitydensityfunctiononanimagexover
5 allitssub-pixels(colorchannelsinapixel)asp(x) = ip(xi|x<i). Theconditionaldistributions
0 p(xi x<i)areparameterizedbyconvolutionalneuralnetworksandallshareparameters. ThePixel-
. CNN| isapowerfulmodelasthefunctionalformoftheQseconditionalsisveryflexible. Inaddition
1
it is computationally efficient as all conditionals can be evaluated in parallel on a GPU for an ob-
0
servedimagex. Thankstotheseproperties,thePixelCNNrepresentsthecurrentstate-of-the-artin
7
generative modeling when evaluated in terms of log-likelihood. Besides being used for modeling
1
: images, the PixelCNN model was recently extended to model audio (van den Oord et al., 2016a),
v video(Kalchbrenneretal.,2016b)andtext(Kalchbrenneretal.,2016a).
i
X
For use in our research, we developed our own internal implementation of PixelCNN and made a
r number of modifications to the base model to simplify its structure and improve its performance.
a
Wenowreleaseourimplementationathttps://github.com/openai/pixel-cnn,hoping
thatitwillbeusefultothebroadercommunity. OurmodificationsarediscussedinSection2, and
evaluated experimentally in Section 3. State-of-the-art log-likelihood results confirm their useful-
ness.
2 MODIFICATIONS TO PIXELCNN
WenowdescribethemostimportantmodificationswehavemadetothePixelCNNmodelarchite-
cure as described by van den Oord et al. (2016c). For complete details see our code release at
https://github.com/openai/pixel-cnn.
2.1 DISCRETIZEDLOGISTICMIXTURELIKELIHOOD
ThestandardPixelCNNmodelspecifiestheconditionaldistributionofasub-pixel,orcolorchannel
ofapixel,asafull256-waysoftmax.Thisgivesthemodelalotofflexibility,butitisalsoverycostly
in terms of memory. Moreover, it can make the gradients with respect to the network parameters
1
UnderreviewasaconferencepaperatICLR2017
very sparse, especially early in training. With the standard parameterization, the model does not
knowthatavalueof128isclosetoavalueof127or129,andthisrelationshipfirsthastobelearned
before the model can move on to higher level structures. In the extreme case where a particular
sub-pixelvalueisneverobserved,themodelwilllearntoassignitzeroprobability. Thiswouldbe
especiallyproblematicfordatawithhigheraccuracyontheobservedpixelsthantheusual8bits:In
theextremecasewhereveryhighprecisionvaluesareobserved,thePixelCNN,initscurrentform,
would require a prohibitive amount of memory and computation, while learning very slowly. We
thereforeproposeadifferentmechanismforcomputingtheconditionalprobabilityoftheobserved
discretizedpixelvalues. Inourmodel,likeintheVAEofKingmaetal.(2016),weassumethereis
a latent color intensity ν with a continuous distribution, which is then rounded to its nearest 8-bit
representationtogivetheobservedsub-pixelvaluex.Bychoosingasimplecontinuousdistribution
formodelingν(likethelogisticdistributionasdonebyKingmaetal.(2016))weobtainasmoothand
memoryefficientpredictivedistributionforx. Here,wetakethiscontinuousunivariatedistribution
to be a mixture of logistic distributions which allows us to easily calculate the probability on the
observeddiscretizedvaluex,asshowninequation(2).Forallsub-pixelvaluesxexceptingtheedge
cases0and255wehave:
K
ν π logistic(µ ,s ) (1)
i i i
∼
i=1
X
K
P(xπ,µ,s) = π [σ((x+0.5 µ )/s ) σ((x 0.5 µ )/s )], (2)
i i i i i
| − − − −
i=1
X
whereσ()isthelogisticsigmoidfunction. Fortheedgecaseof0,replacex 0.5by ,andfor
− −∞
255replacex+0.5by+ . Ourprovidedcodecontainsanumericallystableimplementationfor
∞
calculatingthelogoftheprobabilityinequation2.
Our approach follows earlier work using continuous mixture models (Domke et al., 2008; Theis
et al., 2012; Uria et al., 2013; Theis & Bethge, 2015), but avoids allocating probability mass to
values outside the valid range of [0,255] by explicitly modeling the rounding of ν to x. In addi-
tion,wenaturallyassignhigherprobabilitytotheedgevalues0and255thantotheirneighboring
values, which corresponds well with the observed data distribution as shown in Figure 1. Experi-
mentally, we find that only a relatively small number of mixture components, say 5, is needed to
accurately model the conditional distributions of the pixels. The output of our network is thus of
muchlowerdimension,yieldingmuchdensergradientsofthelosswithrespecttoourparameters.In
ourexperimentsthisgreatlyspedupconvergenceduringoptimization,especiallyearlyonintrain-
ing. However,duetotheotherchangesinourarchitecturecomparedtothatofvandenOordetal.
(2016c)wecannotsaywithcertaintythatthiswouldalsoapplytotheoriginalPixelCNNmodel.
Figure 1: Marginal distribution of all sub-pixel values in CIFAR-10. The edge value of 255 is
muchmorefrequentthanitsneighbouringvalues: Thisiseasytomodelusingourroundingbased
approach,butharderusingcontinuousortruncateddistributions.
2
UnderreviewasaconferencepaperatICLR2017
2.2 CONDITIONINGONWHOLEPIXELS
Thepixelsinacolorimageconsistofthreerealnumbers,givingtheintensitiesofthered,blueand
greencolors. TheoriginalPixelCNNfactorizesthegenerativemodeloverthese3sub-pixels. This
allows for very general dependency structure, but it also complicates the model: besides keeping
track of the spatial location of feature maps, we now have to separate out all feature maps in 3
groupsdependingonwhetherornottheycanseetheR/G/Bsub-pixelofthecurrentlocation. This
addedcomplexityseemstobeunnecessaryasthedependenciesbetweenthecolorchannelsofapixel
arelikelytoberelativelysimpleanddonotrequireadeepnetworktomodel.Therefore,weinstead
conditiononlyonwholepixelsupandtotheleftinanimage,andoutputjointpredictivedistributions
overall3channelsofapredictedpixel.Thepredictivedistributiononapixelitselfcanbeinterpreted
asasimplefactorizedmodel:Wefirstpredicttheredchannelusingadiscretizedmixtureoflogistics
asdescribedinsection2.1. Next,wepredictthegreenchannelusingapredictivedistributionofthe
sameform. Hereweallowthemeansofthemixturecomponentstolinearlydependonthevalueof
theredsub-pixel. Finally,wemodelthebluechannelinthesameway,whereweagainonlyallow
lineardependencyontheredandgreenchannels. Forthepixel(r ,g ,b )atlocation(i,j)in
i,j i,j i,j
ourimage,thedistributionconditionalonthecontextC ,consistingofthemixtureindicatorand
i,j
thepreviouspixels,isthus
p(r ,g ,b C ) = P(r µ (C ),s (C )) P(g µ (C ,r ),s (C ))
i,j i,j i,j i,j i,j r i,j r i,j i,j g i,j i,j g i,j
| | × |
P(b µ (C ,r ,g ),s (C ))
i,j b i,j i,j i,j b i,j
× |
µ (C ,r ) = µ (C )+α(C )r
g i,j i,j g i,j i,j i,j
µ (C ,r ,g ) = µ (C )+β(C )r +γ(C )b , (3)
b i,j i,j i,j b i,j i,j i,j i,j i,j
withα,β,γscalarcoefficientsdependingonthemixturecomponentandpreviouspixels.
Themixtureindicatorissharedacrossall3channels;i.e.ourgenerativemodelfirstsamplesamix-
tureindicatorforapixel, andthensamplesthecolorchannelsone-by-onefromthecorresponding
mixturecomponent. HadweusedadiscretizedmixtureofunivariateGaussiansforthesub-pixels,
insteadoflogistics,thiswouldhavebeenexactlyequivalenttopredictingthecompletepixelusing
a (discretized) mixture of 3-dimensional Gaussians with full covariance. The logistic and Gaus-
sian distributions are very similar, so this is indeed very close to what we end up doing. For full
implementationdetailswerefertoourcodeathttps://github.com/openai/pixel-cnn.
2.3 DOWNSAMPLINGVERSUSDILATEDCONVOLUTION
The original PixelCNN only uses convolutions with small receptive field. Such convolutions are
good at capturing local dependencies, but not necessarily at modeling long range structure. Al-
though we find that capturing these short range dependencies is often enough for obtaining very
good log-likelihood scores (see Table 2), explicitly encouraging the model to capture long range
dependenciescanimprovetheperceptualqualityofgeneratedimages(compareFigure3andFig-
ure5). Onewayofallowingthenetworktomodelstructureatmultipleresolutionsistointroduce
dilated convolutions into the model, as proposed by van den Oord et al. (2016a) and Kalchbren-
neretal.(2016b). Here, weinsteadproposetousedownsamplingbyusingconvolutionsofstride
2. Downsampling accomplishes the same multi-resolution processing afforded by dilated convo-
lutions, but at a reduced computational cost: where dilated convolutions operate on input of ever
increasing size (due to zero padding), downsampling reduces the input size by a factor of 4 (for
strideof2in2dimensions)ateverydownsampling. Thedownsideofusingdownsamplingisthat
itlosesinformation,butwecancompensateforthisbyintroducingadditionalshort-cutconnections
intothenetworkasexplainedinthenextsection. Withtheseadditionalshort-cutconnections, we
foundtheperformanceofdownsamplingtobethesameasfordilatedconvolution.
2.4 ADDINGSHORT-CUTCONNECTIONS
Forinputofsize32 32oursuggestedmodelconsistsof6blocksof5ResNetlayers. Inbetween
×
thefirstandsecondblock,aswellasthesecondandthirdblock,weperformsubsamplingbystrided
convolution. Inbetweenthefourthandfifthblock,aswellasthefifthandsixthblock,weperform
upsampling by transposed strided convolution. This subsampling and upsampling process loses
information,andwethereforeintroduceadditionalshort-cutconnectionsintothemodeltorecover
3
UnderreviewasaconferencepaperatICLR2017
this information from lower layers in the model. The short-cut connections run from the ResNet
layersinthefirstblocktothecorrespondinglayersinthesixthblock,andsimilarlybetweenblocks
two and five, and blocks three and four. This structure resembles the VAE model with top down
inferenceusedbyKingmaetal.(2016),aswellastheU-netusedbyRonnebergeretal.(2015)for
imagesegmentation. Figure2showsourmodelstructuregraphically.
32x32 16x16 8x8 8x8 16x16 32x32 = Sequence of 6

layers
= Downward stream
x
= Downward and 

rightward stream
x
= Identity (skip)

connection
= Convolutional

connection
Figure 2: Like van den Oord et al. (2016c), our model follows a two-stream (downward, and
downward+rightward)convolutionalarchitecturewithresidualconnections;however,therearetwo
significant differences in connectivity. First, our architecture incorporates downsampling and up-
sampling,suchthattheinnerpartsofthenetworkoperateoverlargerspatialscale,increasingcom-
putational efficiency. Second, we employ long-range skip-connections, such that each k-th layer
provides a direct input to the (K k)-th layer, where K is the total number of layers in the net-
−
work. Thenetworkisgroupedintosequencesofsixlayers,wheremostsequencesareseparatedby
downsamplingorupsampling.
2.5 REGULARIZATIONUSINGDROPOUT
The PixelCNN model is powerful enough to overfit on training data. Moreover, rather than just
reproducingthetrainingimages, wefindthatoverfittedmodelsgenerateimagesoflowperceptual
quality,asshowninFigure8.Oneeffectivewayofregularizingneuralnetworksisdropout(Srivas-
tavaetal.,2014).Forourmodel,weapplystandardbinarydropoutontheresidualpathafterthefirst
convolution. ThisissimilartohowdropoutisappliedinthewideresidualnetworksofZagoruyko
& Komodakis (2016). Using dropout allows us to successfully train high capacity models while
avoidingoverfittingandproducinghighqualitygenerations(comparefigure8andfigure3).
3 EXPERIMENTS
WeapplyourmodeltomodelingnaturalimagesintheCIFAR-10dataset.Weachievestate-of-the-
artresultsintermsoflog-likelihood,andgenerateimageswithcoherentglobalstructure.
3.1 UNCONDITIONALGENERATIONONCIFAR-10
WeapplyourPixelCNNmodel,withthemodificationsasdescribedabove,togenerativemodelingof
theimagesintheCIFAR-10dataset.FortheencodingpartofthePixelCNN,themodeluses3Resnet
blocksconsistingof5residuallayers,with2 2downsamplinginbetween. Thesamearchitecture
×
isusedforthedecodingpartofthemodel,butwithupsamplinginsteadofdownsamplinginbetween
blocks. Allresiduallayersuse192featuremapsandadropoutrateof0.5. Table1showsthestate-
of-the-arttestlog-likelihoodobtainedbyourmodel.Figure3showssomesamplesgeneratedbythe
model.
4
UnderreviewasaconferencepaperatICLR2017
Figure3: SamplesfromourPixelCNNmodeltrainedonCIFAR-10.
Model Bitspersub-pixel
DeepDiffusion(Sohl-Dicksteinetal.,2015) 5.40
NICE(Dinhetal.,2014) 4.48
DRAW(Gregoretal.,2015) 4.13
DeepGMMs(vandenOord&Dambre,2015) 4.00
ConvDRAW(Gregoretal.,2016) 3.58
RealNVP(Dinhetal.,2016) 3.49
PixelCNN(vandenOordetal.,2016b) 3.14
VAEwithIAF(Kingmaetal.,2016) 3.11
GatedPixelCNN(vandenOordetal.,2016c) 3.03
PixelRNN(vandenOordetal.,2016b) 3.00
PixelCNN++ 2.92
Table1:Negativelog-likelihoodforgenerativemodelsonCIFAR-10expressedasbitspersub-pixel.
3.2 CLASS-CONDITIONALGENERATION
Next, we follow van den Oord et al. (2016c) in making our generative model conditional on the
class-labeloftheCIFAR-10images. Thisisdonebylinearlyprojectingaone-hotencodingofthe
class-labelintoaseparateclass-dependentbiasvectorforeachconvolutionalunitinournetwork.We
findthatmakingthemodelclass-conditionalmakesithardertoavoidoverfittingonthetrainingdata:
ourbesttestlog-likelihoodis2.94inthiscase. Figure4showssamplesfromtheclass-conditional
model, with columns 1-10 corresponding the 10 classes in CIFAR-10. The images clearly look
qualitatively different across the columns and for a number of them we can clearly identify their
classlabel.
5
UnderreviewasaconferencepaperatICLR2017
Figure 4: Class-conditional samples from our PixelCNN for CIFAR-10 (left) and real CIFAR-10
imagesforcomparison(right).
3.3 EXAMININGNETWORKDEPTHANDFIELDOFVIEWSIZE
Itishypothesizedthatthesizeofthereceptivefieldandadditionallytheremovalofblindspotsin
thereceptivefieldareimportantforPixelCNN’sperformance(vandenOordetal.,2016b). Indeed
van den Oord et al. (2016c) specifically introduced an improvement over the previous PixelCNN
modeltoremovetheblindspotinthereceptivefieldthatwaspresentintheirearliermodel.
HerewepresentthesurprisingfindingthatinfactaPixelCNNwithrathersmallreceptivefieldcan
attaincompetitivegenerativemodellingperformanceonCIFAR-10aslongasithasenoughcapacity.
Specifically,weexperimentedwithourproposedPixelCNN++modelwithoutdownsamplingblocks
andreducethenumberoflayerstolimitthereceptivefieldsize. Weinvestigatetworeceptivefield
sizes: 11x5 and 15x8, and a receptive field size of 11x5, for example, means that the conditional
distributionofapixelcandependsonarectangleabovethepixelofsize11x5aswellas 112−1 =5x1
blocktotheleftofthepixel.
Aswelimitthesizeofthereceptivefield,thecapacityofthenetworkalsodropssignificantlysince
itcontainsmanyfewerlayersthananormalPixelCNN.WecallthetypeofPixelCNNthat’ssimply
limitedindepth“Plain”SmallPixelCNN.Interestingly,thismodelalreadyhasbetterperformance
than the original PixelCNN in van den Oord et al. (2016b) which had a blind spot. To increase
capacity, we introduced two simple variants that make Small PixelCNN more expressive without
growingthereceptivefield:
NIN(NetworkinNetwork):insertadditionalgatedResNetblockswith1x1convolutionbe-
•
tweenregularconvolutionblocksthatgrowreceptivefield. Inthisexperiment,weinserted
3NINblocksbetweeneveryotherlayer.
Autoregressive Channel: skip connections between sets of channels via 1x1 convolution
•
gatedResNetblock.
Both modifications increase the capacity of the network, resulting in improved log-likelihood as
shown in Table 2. Although the model with small receptive field already achieves an impressive
likelihoodscore,itssamplesdolackglobalstructure,asseeninFigure5.
6
UnderreviewasaconferencepaperatICLR2017
Table2: CIFAR-10bitspersub-pixelforSmallPixelCNN
Model Bitspersub-pixel
Field=11x5,Plain 3.11
Field=11x5,NIN 3.09
Field=11x5,AutoregressiveChannel 3.07
Field=15x8,Plain 3.07
Field=15x8,NIN 3.04
Field=15x8,AutoregressiveChannel 3.03
Figure5: Samplesfrom3.03bits/dimSmallPixelCNN
3.4 ABLATIONEXPERIMENTS
InordertotesttheeffectofourmodificationstoPixelCNN,werunanumberofablationexperiments
whereforeachexperimentweremoveaspecificmodification.
3.4.1 SOFTMAXLIKELIHOODINSTEADOFDISCRETIZEDLOGISTICMIXTURE
Inordertotestthecontributionofourlogisticmixturelikelihood,were-runourCIFAR-10experi-
mentwiththe256-waysoftmaxastheoutputdistributioninstead. Weallowthe256logitsforeach
sub-pixeltolinearlydependontheobservedvalueofprevioussub-pixels,withcoefficientsthatare
givenasoutputbythemodel. Ourmodelwithsoftmaxlikelihoodisthusstrictlymoreflexiblethan
ourmodelwithlogisticmixturelikelihood,althoughtheparameterizationisquitedifferentfromthat
usedbyvandenOordetal.(2016c).Themodelnowoutputs1536numbersperpixel,describingthe
logitsonthe256potentialvaluesforeachsub-pixel,aswellasthecoefficientsforthedependencies
betweenthesub-pixels. Figure6showsthatthismodeltrainsmoreslowlythanouroriginalmodel.
In addition, the running time per epoch is significantly longer for our tensorflow implementation.
Forourarchitecture,thelogisticmixturemodelthusclearlyperformsbetter. Sinceourarchitecture
differsfromthatofvandenOordetal.(2016c)inotherwaysaswell,wecannotsaywhetherthis
wouldalsoapplytotheirmodel.
3.4.2 CONTINUOUSMIXTURELIKELIHOODINSTEADOFDISCRETIZATION
Insteadofdirectlymodelingthediscretepixelvaluesinanimage,itisalsopossibletode-quantize
thembyaddingnoisefromthestandarduniformdistribution,asusedbyUriaetal.(2013)andothers,
andmodelingthedataasbeingcontinuous. Theresultingmodelcanbeinterpretedasavariational
autoencoder(Kingma&Welling,2013;Rezendeetal.,2014),wherethedequantizedpixelszform
alatentcodewhosepriordistributioniscapturedbyourmodel. Sincetheoriginaldiscretepixelsx
canbeperfectlyreconstructedfromzunderthismodel,theusualreconstructiontermvanishesfrom
7
UnderreviewasaconferencepaperatICLR2017
Figure 6: Training curves for our model with logistic mixture likelihood versus our model with
softmaxlikelihood.
thevariational lowerbound. Theentropy ofthestandard uniformdistributionis zero, sothe term
thatremainsistheloglikelihoodofthedequantizedpixels,whichthusgivesusavariationallower
boundontheloglikelihoodofouroriginaldata.
We re-run our model for CIFAR-10 using the same model settings as those used for the 2.92 bits
perdimensionresultinTable1,butnowweremovethediscretizationinourlikelihoodmodeland
insteadaddstandarduniformnoisetotheimagedata. Theresultingmodelisacontinuousmixture
modelinthesameclassasthatusedbyTheisetal.(2012);Uriaetal.(2013);Theis&Bethge(2015)
andothers.Afteroptimization,thismodelgivesavariationallowerboundonthedataloglikelihood
of 3.11 bits per dimension. The difference with the reported 2.92 bits per dimension shows the
benefitofusingdiscretizationinthelikelihoodmodel.
3.4.3 NOSHORT-CUTCONNECTIONS
Next,wetesttheimportanceoftheadditionalparallelshort-cutconnectionsinourmodel,indicated
bythedottedlinesinFigure2. Were-runourunconditionalCIFAR-10experiment,butremovethe
short-cut connections from the model. As seen in Figure 7, the model fails to train without these
connections. Thereasonforneedingtheseextrashort-cutsislikelytobeouruseofsub-sampling,
whichdiscardsinformationthatotherwisecannoteasilyberecovered,
Figure7: Trainingcurvesforourmodelwithandwithoutshort-cutconnections.
3.4.4 NODROPOUT
Were-runourCIFAR-10modelwithoutdropoutregularization. Thelog-likelihoodweachieveon
thetrainingsetisbelow2.0bitspersub-pixel,butthefinaltestlog-likelihoodisabove6.0bitsper
8
UnderreviewasaconferencepaperatICLR2017
sub-pixel. At no point during training does the unregularized model get a test-set log-likelihood
below 3.0 bits per sub-pixel. Contrary to what we might naively expect, the perceptual quality of
thegeneratedimagesbytheoverfittedmodelisnotgreat,asshowninFigure8.
Figure8: SamplesfromintentionallyoverfittedPixelCNNmodeltrainedonCIFAR-10,withtrain
log-likelihoodof2.0bitsperdimension: Overfittingdoesnotresultingreatperceptualquality.
4 CONCLUSION
We presented PixelCNN++, a modification of PixelCNN using a discretized logistic mixture like-
lihood on the pixels among other modifications. We demonstrated the usefulness of these mod-
ifications with state-of-the-art results on CIFAR-10. Our code is made available at https:
//github.com/openai/pixel-cnnandcaneasilybeadaptedforuseonotherdatasets.
REFERENCES
LaurentDinh,DavidKrueger,andYoshuaBengio. Nice: Non-linearindependentcomponentsesti-
mation. arXivpreprintarXiv:1410.8516,2014.
LaurentDinh,JaschaSohl-Dickstein,andSamyBengio. Densityestimationusingrealnvp. arXiv
preprintarXiv:1605.08803,2016.
Justin Domke, Alap Karapurkar, and Yiannis Aloimonos. Who killed the directed model? In
Computer Vision and Pattern Recognition, 2008. CVPR 2008. IEEE Conference on, pp. 1–8.
IEEE,2008.
KarolGregor,IvoDanihelka,AlexGraves,andDaanWierstra. Draw: Arecurrentneuralnetwork
forimagegeneration.InProceedingsofthe32ndInternationalConferenceonMachineLearning,
2015.
KarolGregor,FredericBesse,DaniloJimenezRezende,IvoDanihelka,andDaanWierstra.Towards
conceptualcompression. arXivpreprintarXiv:1604.08772,2016.
NalKalchbrenner,LasseEspeholt,KarenSimonyan,AaronvandenOord,AlexGraves,andKoray
Kavukcuoglu.Neuralmachinetranslationinlineartime.arXivpreprintarXiv:1610.10099,2016a.
Nal Kalchbrenner, Aaron van den Oord, Karen Simonyan, Ivo Danihelka, Oriol Vinyals, Alex
Graves,andKorayKavukcuoglu.Videopixelnetworks.arXivpreprintarXiv:1610.00527,2016b.
DiederikPKingmaandMaxWelling. Auto-EncodingVariationalBayes. Proceedingsofthe2nd
InternationalConferenceonLearningRepresentations,2013.
9
UnderreviewasaconferencepaperatICLR2017
DiederikP.Kingma, TimSalimans, RafalJozefowicz, XiChen, IlyaSutskever, andMaxWelling.
Improvingvariationalinferencewithinverseautoregressiveflow.InAdvancesinNeuralInforma-
tionProcessingSystems,2016.
DaniloJRezende,ShakirMohamed,andDaanWierstra. Stochasticbackpropagationandapproxi-
mateinferenceindeepgenerativemodels. InICML,pp.1278–1286,2014.
OlafRonneberger,PhilippFischer,andThomasBrox. U-net: Convolutionalnetworksforbiomed-
ical image segmentation. In International Conference on Medical Image Computing and
Computer-AssistedIntervention,pp.234–241.Springer,2015.
JaschaSohl-Dickstein,EricA.Weiss,NiruMaheswaranathan,andSuryaGanguli. Deepunsuper-
visedlearningusingnonequilibriumthermodynamics. InProceedingsofthe32ndInternational
ConferenceonMachineLearning,2015.
NitishSrivastava,GeoffreyEHinton,AlexKrizhevsky,IlyaSutskever,andRuslanSalakhutdinov.
Dropout:asimplewaytopreventneuralnetworksfromoverfitting. JournalofMachineLearning
Research,15(1):1929–1958,2014.
LucasTheisandMatthiasBethge. Generativeimagemodelingusingspatiallstms. InAdvancesin
NeuralInformationProcessingSystems,pp.1927–1935,2015.
LucasTheis, ReshadHosseini, andMatthiasBethge. Mixturesofconditionalgaussianscalemix-
turesappliedtomultiscaleimagerepresentations. PloSone,7(7):e39857,2012.
Benigno Uria, Iain Murray, and Hugo Larochelle. Rnade: The real-valued neural autoregressive
density-estimator. InAdvancesinNeuralInformationProcessingSystems,pp.2175–2183,2013.
Aaron van den Oord and Joni Dambre. Locally-connected transformations for deep gmms. In
InternationalConferenceonMachineLearning(ICML):DeeplearningWorkshop,2015.
AaronvandenOord,SanderDieleman,HeigaZen,KarenSimonyan,OriolVinyals,AlexGraves,
Nal Kalchbrenner, Andrew Senior, and Koray Kavukcuoglu. Wavenet: A generative model for
rawaudio. arXivpreprintarXiv:1609.03499,2016a.
AaronvandenOord,NalKalchbrenner,andKorayKavukcuoglu. Pixelrecurrentneuralnetworks.
InInternationalConferenceonMachineLearning(ICML),2016b.
Aaron van den Oord, Nal Kalchbrenner, Oriol Vinyals, Lasse Espeholt, Alex Graves, and Ko-
ray Kavukcuoglu. Conditional image generation with pixelcnn decoders. arXiv preprint
arXiv:1606.05328,2016c.
Sergey Zagoruyko and Nikos Komodakis. Wide residual networks. arXiv preprint
arXiv:1605.07146,2016.
10