deconstructing the ladder network archi

15 Pages·2016·0.99 MB·English

Workshoptrack-ICLR2016
DECONSTRUCTING THE LADDER NETWORK ARCHI-
TECTURE
MohammadPezeshki LinxiFan
Departementd’informatiqueetderechercheope´rationnelle DepartmentofComputerScience
Universite´ deMontre´al ColumbiaUniversity
Montre´al,QCH3C3J7 NewYork,NewYork10027
[email protected] [email protected]
Phile´monBrakel,AaronCourville,YoshuaBengio∗
Departementd’informatiqueetderechercheope´rationnelle
Universite´ deMontre´al
Montre´al,QCH3C3J7
[email protected], {courvila, <findme>}@iro.umontreal.ca
ABSTRACT
The Ladder Network is a recent new approach to semi-supervised learning that
turned out to be very successful. While showing impressive performance, the
Ladder Network has many components intertwined, whose contributions are not
obvious insuch a complexarchitecture. This paper presentsan extensive exper-
imentalinvestigationofvariantsoftheLadderNetworkinwhichwereplacedor
removedindividualcomponentstolearnabouttheirrelativeimportance.Forsemi-
supervisedtasks,weconcludethatthemostimportantcontributionismadebythe
lateralconnections, followedbytheapplicationofnoise, andthechoiceofwhat
we refer to as the ‘combinator function’. As the number of labeled training ex-
amplesincreases,thelateralconnectionsandthereconstructioncriterionbecome
lessimportant,withmostofthegeneralizationimprovementcomingfromthein-
jection of noise in each layer. Finally, we introduce a combinator function that
reducestesterrorratesonPermutation-InvariantMNISTto0.57%forthesuper-
visedsetting,andto0.97%and1.0%forsemi-supervisedsettingswith1000and
100labeledexamples,respectively.
1 INTRODUCTION
Labelingdatasetsistypicallyacostlytaskandinmanysettingstherearefarmoreunlabeledexam-
plesthanlabeledones. Semi-supervisedlearningaimstoimprovetheperformanceonsomesuper-
vised learning problem by using information obtained from both labeled and unlabeled examples.
Since the recent success of deep learning methods has mainly relied on supervised learning based
onverylargelabeleddatasets,itisinterestingtoexploresemi-superviseddeeplearningapproaches
toextendthereachofdeeplearningtothesesettings.
Since unsupervised methods for pre-training layers or neural networks were an essential part of
the first wave of deep learning methods (Hinton et al., 2006; Vincent et al., 2008; Bengio, 2009),
a natural next step is to investigate how ideas inspired by Restricted Boltzmann Machine training
and regularized autoencoders can be used for semi-supervised learning. Examples of approaches
basedonsuchideasarethediscriminativeRBM(Larochelle&Bengio,2008)andadeeparchitec-
turebasedonsemi-supervisedautoencodersthatwasusedfordocumentclassification(Ranzato&
Szummer, 2008). More recent examples of approaches for semi-supervised deep learning are the
semi-supervised Variational Autoencoder (Kingma et al., 2014) and the Ladder Network (Rasmus
et al., 2015) which obtained very impressive, state of the art results (1.13% error) on the MNIST
handwrittendigitsclassificationbenchmarkusingjust100labeledtrainingexamples.
∗YoshuaBengioisaCIFARSeniorFellow
1
Workshoptrack-ICLR2016
The Ladder Network adds an unsupervised component to the supervised learning objective of a
deepfeedforwardnetworkbytreatingthisnetworkaspartofadeepstackofdenoisingautoencoders
or DAEs (Vincent et al., 2010) that learns to reconstruct each layer (including the input) based on
a corrupted version of it, using feedback from upper levels. The term ’ladder’ refers to how this
architectureextendsthestackedDAEinthewaythefeedbackpathsareformed.
ThispaperisfocusingonthedesignchoicesthatleadtotheLadderNetwork’ssuperiorperformance
and tries to disentangle them empirically. We identify some general properties of the model that
makeitdifferentfromstandardfeedforwardnetworksandcomparevariousarchitecturestoidentify
thosepropertiesanddesignchoicesthatarethemostessentialforobtaininggoodperformancewith
theLadderNetwork. WhiletheoriginalauthorsoftheLadderNetworkpaperexploredsomevari-
ants of their model already, we provide a thorough comparison of a large number of architectures
controlling for both hyperparameter settings and data set selection. Finally, we also introduce a
variantoftheLadderNetworkthatyieldsnewstate-of-the-artresultsforthePermutation-Invariant
MNISTclassificationtaskinbothsemi-andfully-supervisedsettings.
2 THE LADDER NETWORK ARCHITECTURE
In this section, we describe the Ladder Network Architecture1. Consider a dataset with
N labeled examples (x(1),y∗(1)),(x(2),y∗(2)),...,(x(N),y∗(N)) and M unlabeled examples
x(N +1),x(N +2),...,x(N +M) where M (cid:29) N. The objective is to learn a function that
modelsP(y|x)byusingboththelabeledexamplesandthelargequantityofunlabeledexamples. In
the case of the Ladder Network, this function is a deep Denoising Auto Encoder (DAE) in which
noiseisinjectedintoallhiddenlayersandtheobjectivefunctionisaweightedsumofthesupervised
CrossEntropycostonthetopoftheencoderandtheunsuperviseddenoisingSquareErrorcostsat
eachlayerofthedecoder. Sincealllayersarecorruptedbynoise,anotherencoderpathwithshared
parameters is responsible for providing the clean reconstruction targets, i.e. the noiseless hidden
activations(SeeFigure1).
Throughlateralskipconnections,eachlayerofthenoisyencoderisconnectedtoitscorresponding
layer in the decoder. This enables the higher layer features to focus on more abstract and task-
specificfeatures. Hence,ateachlayerofthedecoder,twosignals,onefromthelayeraboveandthe
otherfromthecorrespondinglayerintheencoderarecombined.
Formally,theLadderNetworkisdefinedasfollows:
x˜,z˜(1),...,z˜(L),y˜=Encoder (x), (1)
noisy
x,z(1),...,z(L),y =Encoder (x), (2)
clean
xˆ,zˆ(1),...,zˆ(L) =Decoder(z˜(1),...,z˜(L)), (3)
where Encoder and Decoder can be replaced by any multi-layer architecture such as a multi-layer
perceptroninthiscase. Thevariablesx,y,andy˜aretheinput,thenoiselessoutput,andthenoisy
outputrespectively. Thevariablesz(l),z˜(l),andzˆ(l)arethehiddenrepresentation,itsnoisyversion,
and its reconstructed version at layer l. The objective function is a weighted sum of supervised
(CrossEntropy)andunsupervisedcosts(Reconstructioncosts).
Cost=−ΣN logP(y˜(n)=y∗(n)|x(n))+ΣM ΣL λ ReconstructionCost(z(l)(n),zˆ(l)(n)).
n=1 n=N+1 l=1 l
(4)
Note that while the noisy output y˜ is used in the Cross Entropy term, the classification task is
performedbythenoiselessoutputyattesttime.
Intheforwardpath, individuallayersoftheencoderareformalizedasalineartransformationfol-
lowedbyBatchNormalization(Ioffe&Szegedy,2015)andthenapplicationofanonlinearactivation
function:
1Pleasereferto(Rasmusetal.,2015;Valpola,2014)formoredetailedexplanationoftheLadderNetwork
architecture.
2
Workshoptrack-ICLR2016
z˜(l) =W(l)·h˜(l−1), (5)
pre
µ˜(l) =mean(z˜(l) ), (6)
pre
σ˜(l) =stdv(z˜(l) ), (7)
pre
z˜(l) −µ˜(l)
z˜(l) = pre +N(0,σ2), (8)
σ˜(l)
h˜(l) =φ(γ(l)(z˜(l)+β(l))) (9)
whereh˜(l−1) isthepost-activationatlayerl−1andW(l) istheweightmatrixfromlayerl−1to
layerl. BatchNormalizationisappliedtothepre-normalizationz˜(l)usingthemini-batchmeanµ(l)
pre
and standard deviation σ(l). The next step is to add Gaussian noise with mean 0 and variance σ2
tocomputepre-activationz˜(l). Theparametersβ(l) andγ(l) areresponsibleforshiftingandscaling
before applying the nonlinearity φ(·). Note that the above equations describe the noisy encoder.
If we remove noise (N(0,σ2)) and replace h˜ and z˜with h and z respectively, we will obtain the
noiselessversionoftheencoder.
At each layer of the decoder in the backward path, the signal from the layer zˆ(l+1) and the noisy
signalz˜(l)arecombinedintothereconstructionzˆ(l)bythefollowingequations:
u(l+1) =V(l)·zˆ(l+1), (10)
pre
µ(l+1) =mean(u(l+1)), (11)
pre
σ(l+1) =stdv(u(l+1)), (12)
pre
u(l+1)−µ(l+1)
u(l+1) = pre , (13)
σ(l+1)
zˆ(l) =g(z˜(l),u(l+1)) (14)
whereV(l)isaweightmatrixfromlayerl+1tolayerl. Wecallthefunctiong(·,·)thecombinator
functionasitcombinestheverticalu(l+1)andthelateralz˜(l)connectionsinanelement-wisefashion.
The original Ladder Network proposes the following design for g(·,·), which we call the vanilla
combinator:
g(z˜(l),u(l+1))=b +w (cid:12)z˜(l)+w (cid:12)u(l+1)+w (cid:12)z˜(l)(cid:12)u(l+1)
0 0z 0u 0zu
(15)
+w (cid:12)Sigmoid(b +w (cid:12)z˜(l)+w (cid:12)u(l+1)+w (cid:12)z˜(l)(cid:12)u(l+1)),
σ 1 1z 1u 1zu
where(cid:12)isanelement-wisemultiplicationoperatorandeachper-elementweightisinitializedas:
w ←1
w{0,1}z ←0
{0,1}u (16)
w ,b ←0
 {0,1}zu {0,1}
w ←1
σ
Inlatersections,wewillexplorealternativeinitializationschemesonthevanillacombinator.
Finally,theReconstructionCost(z(l),zˆ(l))inequation(4)isdefinedasthefollowing:
zˆ(l)−µ(l)
ReconstructionCost(z(l),zˆ(l))=|| −z(l)||2. (17)
σ(l)
where zˆ(l) is normalized using µ(l) and σ(l) which are the encoder’s sample mean and standard
deviationstatisticsofthecurrentminibatch,respectively. Thereasonforthissecondnormalization
istocanceltheeffectofunwantednoiseintroducedbythelimitedbatchsizeofBatchNormalization.
3
Workshoptrack-ICLR2016
Figure1:TheLadderNetworkconsistsoftwoencoders(oneachsideofthefigure)andonedecoder
(in the middle). At each layer of both encoders (equations 5 to 9), z(l) and z˜(l) are computed by
applying a linear transformation and normalization on h(l−1) and h˜(l−1), respectively. The noisy
version of the encoder (left) has an extra Gaussian noise injection term. Batch normalization cor-
rection (γl,βl) and non-linearity are then applied to obtain h(l) and h˜(l). At each layer of the
decoder,twostreamsofinformation,thelateralconnectionz˜(l)(graylines)andtheverticalconnec-
tion u(l+1), are required to reconstruct zˆ(l) (equations 10 to 14). Acronyms CE and RC stand for
CrossEntropyandReconstructionCostrespectively. Thefinalobjectivefunctionisaweightedsum
ofallReconstructioncostsandtheCrossEntropycost.
3 COMPONENTS OF THE LADDER NETWORK
Now that the precise architecture of the Ladder Network has been described in details, we can
identify a couple of important additions to the standard feed-forward neural network architecture
that may have a pronounced impact on the performance. A distinction can also be made between
those design choices that follow naturally from the motivation of the ladder network as a deep
autoencoderandthosethataremoread-hocandtaskspecific.
The most obvious addition is the extra reconstruction cost for every hidden layer and the input
layer. Whileitisclearthatthereconstructioncostprovidesanunsupervisedobjectivetoharnessthe
unlabeledexamples,itisnotclearhowimportantthepenalizationisforeachlayerandwhatroleit
playsforfully-supervisedtasks.
4
Workshoptrack-ICLR2016
AsecondimportantchangeistheadditionofGaussiannoisetotheinputandthehiddenrepresen-
tations. Whileaddingnoisetothefirstlayerisapartofdenoisingautoencodertraining,itisagain
notclearwhetheritisnecessarytoaddthisnoiseateverylayerornot. Wewouldalsoliketoknow
ifthenoisehelpsbymakingthereconstructiontasknontrivialandusefulorjustbyregularizingthe
feed-forwardnetworkinasimilarwaythatnoise-basedregularizerslikedropout(Srivastavaetal.,
2014)andadaptiveweightnoise(Graves,2011)do.
Finally, the lateral skip connections are the most notable deviation from the standard denoising
autoencoder architecture. The way the vanilla Ladder Network combines the lateral stream of in-
formation z˜(l) and the downward stream of information u(l+1) is somewhat unorthodox. For this
reason,wehaveconductedextensiveexperimentsonboththeimportanceofthelateralconnections
andtheprecisechoiceforthefunctionthatcombinesthelateralanddownwardinformationstreams
(whichwerefertoasthecombinatorfunction).
4 EXPERIMENTAL SETUP
Inthissection,weintroducedifferentvariantsoftheLadderArchitectureanddescribeourexperi-
mentmethodology. Eachvariantdiffersfromeitherthevanillamodelorapreviousvariantinonly
onecomponent. Thisenablesustoisolateeachcomponentandobserveitseffectswhileothercom-
ponents remain unchanged. Table 1 depicts the different variants we have investigated and their
abbreviations.
Table1: Schematicorderingofthedifferentarchitecturesandvariantsweinvestigated.
VanillaLadder
Description
Noise Vanilla TheoriginalLadderNetwork.
FirstNoise FirstNoise Noiseonlytothe1stlayer.
FirstRecons Reconstructiononlyatthe1stlayer.
FirstRecons
FirstN&R Noiseonlytothe1stlayerandalso
FirstN&R
reconstructiononlyatthe1stlayer.
NoLateral NoLateral FirstN&Rwithnolateralconnection
(resemblesaDAEwithaddedsupervisedcost).
Combinator
RandInit Randominitializationofthevanilla
RandInit
combinatorweights.
RevInit RevInit Reverseinitialization(w ←0andw ←1)
z u
NoSig NoSig Nonon-linearityinthevanillacombinator.
NoMult Nomultiplicationinthevanillacombinator.
NoMult
Linear Simplelinearcombinationofz˜l andu(l+1)
Linear
(Nosigmoidandnomultiplication).
Gaussian Gaussian Equations19to21.
GatedGauss Gaussiancombinatorwithgating.
GatedGauss
MLP MLPcombinatorfunction.
MLP
AMLP AugmentedMLPwithmultiplicativeunit.
AMLP
4.1 NOISEVARIANTS
Differentconfigurationsofnoiseinjection,penalizingreconstructionerrors,andthelateralconnec-
tionremovalsuggestfourdifferentvariants:
5
Workshoptrack-ICLR2016
• Addnoiseonlytothefirstlayer(FIRSTNOISE).
• Onlypenalizethereconstructionatthefirstlayer(FIRSTRECONS),i.e. λ(l≥1)aresetto0.
• Applybothoftheabovechanges:addnoiseandpenalizethereconstructiononlyatthefirst
layer(FIRSTN&R).
• Removealllateralconnectionsfrom FIRSTN&R.Therefore, equivalenttotoadenoising
autoencoderwithanadditionalsupervisedcostatthetop,theencoderandthedecoderare
connectedonlythroughthetopmostconnection. WecallthisvariantNOLATERAL.
4.2 VANILLACOMBINATORVARIANTS
Wetrydifferentvariantsofthevanillacombinatorfunctionthatcombinesthetwostreamsofinfor-
mationfromthelateralandtheverticalconnectionsinanunusualway. Asdefinedinequation15,
theoutputofthevanillacombinatordependsonu,z˜,andu(cid:12)z˜2,whichareconnectedtotheoutput
viatwopaths,onelinearandtheotherthroughasigmoidnon-linearityunit. (SeeFigure2(a))
Note that the vanilla combinator is initialized in a very specific way (equation 16), which sets the
initialweightsforlateralconnectionz˜to1andverticalconnectionuto0. Thisparticularscheme
encourages the Ladder decoder path to learn more from the lateral information stream z˜than the
verticaluatthebeginningoftraining.
Weexploretwovariantsoftheinitializationscheme:
• Randominitialization(RANDINIT):allper-elementweightsw∗andbiasesb∗arerandomly
initializedtoN(0,0.2).
w ←0
w{0,1}z ←1
• Reverseinitialization(REVINIT) {0,1}u
w ,b ←0
 {0,1}zu {0,1}
w ←1
σ
Thesimplestwayofcombininglateralconnectionswithverticalconnectionsistosimplyaddthem
in an element-wise fashion, similar to the nature of skip-connections in a recently published work
onResidualLearning(Heetal.,2015). WecallthisvariantLINEARcombinatorfunction. Wealso
derivetwomorevariantsNOSIGandNOMULTinthewayofstrippingdownthevanillacombinator
functiontothesimpleLINEARone:
• Removesigmoidnon-linearity(NOSIG). Thecorrespondingper-elementweightsareini-
tializedinthesamewayasthevanillacombinator.
• Removethemultiplicativetermz˜(cid:12)u(NOMULT).
• Simplelinearcombination(LINEAR)
g(z˜,u)=b+w (cid:12)u+w (cid:12)z˜ (18)
u z
(cid:26)
w ←1
wheretheinitializationschemeresemblesthevanillaone: z
w ,b ←0
u
4.3 GAUSSIANCOMBINATORVARIANTS
Another choice for the combinator function with a probabilistic interpretation is the GAUSSIAN
combinator proposed in the original paper about the Ladder Architecture (Rasmus et al., 2015).
BasedonthetheoryintheSection4.1of(Valpola,2014),assumingthatbothadditivenoiseandthe
conditional distribution P(z(l)|u(l+1)) are Gaussian distributions, the denoising function is linear
withrespecttoz˜(l). Hence,thedenoisingfunctioncouldbeaweightedsumoverz˜(l)andaprioron
z(l). Theweightsandtheprioraremodeledasafunctionoftheverticalsignal:
2Forsimplicity,subscriptiandsuperscriptlareimplicitfromnowon.
6
Workshoptrack-ICLR2016
g(z˜,u)=ν(u)(cid:12)z˜+(1−ν(u))(cid:12)µ(u), (19)
µ(u)=w (cid:12)Sigmoid(w (cid:12)u+w )+w (cid:12)u+w , (20)
1 2 3 4 5
ν(u)=w (cid:12)Sigmoid(w (cid:12)u+w )+w (cid:12)u+w . (21)
6 7 8 9 10
Strictlyspeaking,ν(u)isnotaproperweight,becauseitisnotguaranteedtobepositiveallthetime.
To make the Gaussian interpretation rigorous, we explore a variant that we call GATEDGAUSS,
whereequations19and20staythesamebut21isreplacedby:
ν(u)=Sigmoid(w (cid:12)u+w ). (22)
6 7
GATEDGAUSS guarantees that 0 < ν(u) < 1. We expect that ν(u)i will be close to 1 if the
informationfromthelateralconnectionforunitiismorehelpfultoreconstruction,andcloseto0if
theverticalconnectionbecomesmoreuseful. TheGATEDGAUSScombinatorissimilarinnatureto
thegatingmechanismsinothermodelssuchasGatedRecurrentUnit(Choetal.,2014)andhighway
networks(Srivastavaetal.,2015).
4.4 MLP(MULTILAYERPERCEPTRON)COMBINATORVARIANTS
We also propose another type of element-wise combinator functions based on fully-connected
MLPs. Wehaveexploredtwoclassesinthisfamily. Thefirstone, denotedsimplyas MLP,maps
two scalars [u,z˜] to a single output g(z˜,u) (figure 2(b)). We empirically determine the choice of
activation function for the hidden layers. Preliminary experiments show that the Leaky Rectifier
LinearUnit(LReLU)(Maasetal.,2013)performsbetterthaneithertheconventionalReLUorthe
sigmoidunit. OurLReLUfunctionisformulatedas
(cid:26)
x, ifx≥0,
LReLU(x)= . (23)
0.1x, otherwise
WeexperimentwithdifferentnumbersoflayersandhiddenunitsperlayerintheMLP.Wepresent
resultsforthreespecificconfigurations: [4]forasinglehiddenlayerof4units, [2,2]for2hidden
layerseachwith2units,and[2,2,2]for3hiddenlayers. Forexample,inthe[2,2,2]configuration,
theMLPcombinatorfunctionisdefinedas:
(cid:16) (cid:17)
g(z˜,u)=W ·LReLU W ·LReLU(W ·[u,z˜]+b )+b +b (24)
3 2 1 1 2 3
whereW ,W ,andW are2×2weightmatrices;b ,b ,andb are2×1biasvectors.
1 2 3 1 2 3
The second class, which we denote as AMLP (Augmented MLP), has a multiplicative term as an
augmentedinputunit(figure2(c)). Weexpectthatthismultiplicationtermallowstheverticalsignal
(u(l+1))tooverridethelateralsignal(z˜),andalsoallowsthelateralsignaltoselectwherethevertical
signalistobeinstantiated. Sincetheapproximationofmultiplicationisnoteasyforasingle-layer
MLP,weexplicitlyaddthemultiplicationtermasanextrainputtothecombinatorfunction. AMLP
mapsthreescalars[u,z˜,u(cid:12)z˜]toasingleoutput. WeusethesameLReLUunitforAMLP.Wedo
similarexperimentsasinthe MLP caseandincluderesultsfor[4], [2,2]and[2,2,2]hiddenlayer
configurations.
Both MLP and AMLP weight parameters are randomly initialized to N(0,η). η is considered to
beahyperparameterandtunedonthevalidationset. Precisevaluesforthebestη valuesarelisted
inAppendixA.
4.5 METHODOLOGY
Theexperimentalsetupincludestwosemi-supervisedclassificationtaskswith100and1000labeled
examples and a fully-supervised classification task with 60000 labeled examples for Permutation-
Invariant MNIST handwritten digit classification. Labeled examples are chosen randomly but the
7
Workshoptrack-ICLR2016
Figure2: (a)Vanillacombinator (b)MLPcombinator (c)AugmentedMLPcombinator
numberofexamplesfordifferentclassesisbalanced. Thetestsetisnotusedduringallthehyperpa-
rametersearchandtuning. Eachexperimentisrepeated10timeswith10differentbutfixedrandom
seedstomeasurethestandarderroroftheresultsfordifferentparameterinitializationsanddifferent
selectionsoflabeledexamples.
AllvariantsandthevanillaLadderNetworkitselfaretrainedusingtheADAMoptimizationalgo-
rithm(Kingma&Ba,2014)withalearningrateof0.002for100iterationsfollowedby50iterations
with a learning rate decaying linearly to 0. Hyperparameters including the standard deviation of
thenoiseinjectionandthedenoisingweightsateachlayeraretunedseparatelyforeachvariantand
eachexperimentsetting(100-,1000-,andfully-labeled). Hyperparmetersareoptimizedbyeithera
randomsearch(Bergstra&Bengio,2012), oragridsearch, dependingonthenumberofhyperpa-
rametersinvolved(seeAppendixAforpreciseconfigurations).
5 RESULTS & DISCUSSION
Table 2 collects all results for the variants and the baselines. The results are organized into two
main categories for all of the three tasks. Boxplots of four interesting variants are also shown in
Figure 3. The BASELINE model is a simple feed-forward neural network with no reconstruction
penaltyand BASELINE+NOISE isthesamenetworkbutwithadditivenoiseateachlayer. Thebest
resultsintermsofaverageerrorrateonthetestsetareachievedbytheproposedAMLPcombinator
function: in the fully-supervised setting, the best average error rate is 0.569±0.010, while in the
semi-supervisedsettingswith100and1000labeledexamples,theaveragesare1.002±0.037and
0.974±0.021respectively.
5.1 NOISEVARIANTS
Theresultsinthefirstpartofthetableindicatethatinthefully-supervisedsetting,addingnoiseeither
to the first layer only or to all layers leads to a lower error rate with respect to the baselines. Our
intuitionisthattheeffectofadditivenoisetolayersisverysimilartotheweightnoiseregularization
method(Graves,2011)anddropout(Hintonetal.,2012).
Inaddition,theerrorratesinthefirstpartofthetabletellusthatremovingthelateralconnections
hurts much more than the absence of noise injection or reconstruction penalty in the intermediate
layers. Itisalsoworthmentioningthathyperparametertuningyieldszeroweightsforpenalizingthe
reconstructionerrorsinalllayersexcepttheinputlayerinthefully-supervisedtaskforthevanilla
model. Something similar happens for NOLATERAL as well, where hyperparameter tuning yields
zeroreconstructionweightsforalllayersincludingtheinputlayer.Inotherwords,NOLATERALand
BASELINE+NOISE become the same models for the fully-supervised task. Moreover, the weights
forthereconstructionpenaltyofthehiddenlayersarerelativelysmallinthesemi-supervisedtask.
This is in line with similar observations (relatively small weights for the unsupervised part of the
objective)forthehybriddiscriminantRBM(Larochelle&Bengio,2008).
5.2 COMBINATORFUNCTIONVARIANTS
ThesecondpartofTable2showstherelativeperformanceofdifferentcombinatorfunctions. Un-
surprisingly, the performance deteriorates considerably if we remove the sigmoid non-linearity
(NOSIG)orthemultiplicativeterm(NOMULT)orboth(LINEAR)fromthevanillacombinator.Judg-
ingfromthesizeoftheincreaseinaverageerrorrates,themultiplicativetermismoreimportantthan
thesigmoidunit.
8
Workshoptrack-ICLR2016
Table 2: PI MNIST classification results for the vanilla Ladder Network and its variants trained
on 100, 1000, and 60000 (full) labeled examples. AER and SE stand for Average Error Rate and
its Standard Error of each variant over 10 different runs. BASELINE is a multi-layer feed-forward
neuralnetworkwithnoreconstructionpenalty.
100 1000 60000
Variant AER(%) SE AER(%) SE AER(%) SE
Baseline 25.804 ±0.40 8.734 ±0.058 1.182 ±0.010
Baseline+noise 23.034 ±0.48 6.113 ±0.105 0.820 ±0.009
Vanilla 1.086 ±0.023 1.017 ±0.017 0.608 ±0.013
FirstNoise 1.856 ±0.193 1.381 ±0.029 0.732 ±0.015
FirstRecons 1.691 ±0.175 1.053 ±0.021 0.608 ±0.013
FirstN&R 1.856 ±0.193 1.058 ±0.175 0.732 ±0.016
NoLateral 16.390 ±0.583 5.251 ±0.099 0.820 ±0.009
RandInit 1.232 ±0.033 1.011 ±0.025 0.614 ±0.015
RevInit 1.305 ±0.129 1.031 ±0.017 0.631 ±0.018
NoSig 1.608 ±0.124 1.223 ±0.014 0.633 ±0.010
NoMult 3.041 ±0.914 1.735 ±0.030 0.674 ±0.018
Linear 5.027 ±0.923 2.769 ±0.024 0.849 ±0.014
Gaussian 1.064 ±0.021 0.983 ±0.019 0.604 ±0.010
GatedGauss 1.308 ±0.038 1.094 ±0.016 0.632 ±0.011
MLP[4] 1.374 ±0.186 0.996 ±0.028 0.605 ±0.012
MLP[2,2] 1.209 ±0.116 1.059 ±0.023 0.573 ±0.016
MLP[2,2,2] 1.274 ±0.067 1.095 ±0.053 0.602 ±0.010
AMLP[4] 1.072 ±0.015 0.974 ±0.021 0.598 ±0.014
AMLP[2,2] 1.193 ±0.039 1.029 ±0.023 0.569 ±0.010
AMLP[2,2,2] 1.002 ±0.038 0.979 ±0.025 0.578 ±0.013
As described in Section 4.2 and Equation 16, the per-element weights of the lateral connections
are initialized to ones while those of the vertical are initialized to zeros. Interestingly, the results
areslightlyworseforthe RANDINIT variant,inwhichtheseweightsareinitializedrandomly. The
REVINIT variant is even worse than the random initialization scheme. We suspect that the reason
is that the optimization algorithm finds it easier to reconstruct a representation z starting from its
noisy version z˜, rather than starting from an initially arbitrary reconstruction from the untrained
upper layers. Another justification is that the initialization scheme in Equation 16 corresponds to
optimizingtheLadderNetworkasifitbehaveslikeastackofdecoupledDAEsinitially, therefore
duringearlytrainingitislikethattheAuto-Encodersaretrainedmoreindependently.
The GAUSSIAN combinatorperformsbetterthanthevanillacombinator. GATEDGAUSS,theother
variantwithstrict0 < σ(u) < 1,doesnotperformaswellastheonewithunconstrainedσ(u). In
theGAUSSIANformulation,z˜isregulatedbytwofunctionsofu: µ(u)andσ(u). Thiscombinator
interpolates between the noisy activations and the predicted reconstruction (Equation ??), and the
scalingparametercanbeinterpretedasameasureofthecertaintyofthenetwork.
Finally, the AMLP model yields state-of-the-art results in all of 100-, 1000- and 60000-labeled
experiments for PI MNIST. It outperforms both the MLP and the vanilla model. The additional
multiplicativeinputunitu(cid:12)z˜helpsthelearningprocesssignificantly.
5.3 PROBABILISTICINTERPRETATIONSOFTHELADDERNETWORK
Sincemanyofthemotivationsbehindregularizedautoencoderarchitecturesarebasedonobserva-
tionsaboutgenerativemodels, webrieflydiscusshowtheLadderNetworkcanberelatedtosome
othermodelswithvaryingdegreesofprobabilisticinterpretability. Consideringthatthecomponents
thataremostdefiningoftheLadderNetworkseemtobethemostimportantonesforsemi-supervised
learning in particular, comparisons with generative models are at least intuitively appealing to get
moreinsightabouthowthemodellearnsaboutunlabeledexamples.
9
Workshoptrack-ICLR2016
Figure3: Boxplotssummarizingallindividualexperimentsoffourvariantsfor10differentseeds.
Boxboundariesrepresentthe25th and75th percentiles. Thebluelineandtheredsquarerepresent
themedianandthemean,respectively.Thegraycapsalsoshowtheminimumandmaximumvalues.
ThevariantsthatperformmuchworsethanthevanillaLadderNetworkarenotplotted.
By training the individual denoising autoencoders that make up the Ladder Network with a single
objectivefunction,thiscouplinggoesasfarasencouragingthelowerlevelstoproducerepresenta-
tionsthataregoingtobeeasytoreconstructbytheupperlevels. Wefindasimilarterm(-logofthe
top-level prior evaluated at the output of the encoder) in hierarchical extensions of the variational
autoencoder(Rezendeetal.,2014;Bengio,2014).WhiletheLadderNetworkdifferstoomuchfrom
anactualvariationalautoencodertobetreatedassuch, thesimilaritiescanstillgiveoneintuitions
abouttheroleofthenoiseandtheinteractionsbetweenthelayers.Conversely,onealsomaywonder
howavariationalautoencodermightbenefitfromsomeofthecomponentsofLadderNetworkslike
BatchNormalizationandmultiplicativeconnections.
WhenonesimplyviewstheLadderNetworkasapeculiartypeofdenoisingautoencoder,onecould
extendtherecentworkonthegenerativeinterpretationofdenoisingautoencoders(Alain&Bengio,
2013;Bengioetal.,2013)tointerprettheLadderNetworkasagenerativemodelaswell. Itwould
beinterestingtoseeiftheLadderNetworkarchitecturecanbeusedtogeneratesamplesandifthe
architecture’ssuccessat semi-supervisedlearningtranslatesto this profoundlydifferentuseof the
model.
6 CONCLUSION
The paper systematically compares different variants of the recent Ladder Network architecture
(Rasmusetal.,2015;Valpola,2014)withtwofeedforwardneuralnetworksasthebaselinesandthe
standard architecture (proposed in the original paper). Comparisons are done in a deconstructive
way, starting from the standard architecture. Based on the comparisons of different variants we
concludethat:
• Unsurprisingly,thereconstructioncostiscrucialtoobtainthedesiredregularizationfrom
unlabeleddata.
10