Table Of ContentStochastic Pooling for Regularization of
Deep Convolutional Neural Networks
MatthewD.Zeiler RobFergus
DepartmentofComputerScience DepartmentofComputerScience
CourantInstitute,NewYorkUniversity CourantInstitute,NewYorkUniversity
3 [email protected] [email protected]
1
0
2
n Abstract
a
J
6 We introduce a simple and effective method for regularizing large convolutional
1 neural networks. We replace the conventional deterministic pooling operations
with a stochastic procedure, randomly picking the activation within each pool-
] ingregionaccordingtoamultinomialdistribution, givenbytheactivitieswithin
G
the pooling region. The approach is hyper-parameter free and can be combined
L
withotherregularizationapproaches,suchasdropoutanddataaugmentation. We
. achievestate-of-the-artperformanceonfourimagedatasets, relativetootherap-
s
c proachesthatdonotutilizedataaugmentation.
[
1
v
7
1 Introduction
5
5
3
. Neuralnetworkmodelsarepronetoover-fittingduetotheirhighcapacity. Arangeofregularization
1
techniques are used to prevent this, such as weight decay, weight tying and the augmentation of
0
the training set with transformed copies [9]. These allow the training of larger capacity models
3
than would otherwise be possible, which yield superior test performance compared to smaller un-
1
regularizedmodels.
:
v
i Dropout,recentlyproposedbyHintonetal.[2],isanotherregularizationapproachthatstochastically
X
setshalftheactivationswithinalayertozeroforeachtrainingsampleduringtraining. Ithasbeen
r shown to deliver significant gains in performance across a wide range of problems, although the
a
reasonsforitsefficacyarenotyetfullyunderstood.
Adrawbacktodropoutisthatitdoesnotseemtohavethesamebenefitsforconvolutionallayers,
whicharecommoninmanynetworksdesignedforvisiontasks. Inthispaper, weproposeanovel
type of regularization for convolutional layers that enables the training of larger models without
over-fitting,andproducessuperiorperformanceonrecognitiontasks.
The key idea is to make the pooling that occurs in each convolutional layer a stochastic process.
Conventional forms of pooling such as average and max are deterministic, the latter selecting the
largestactivationineachpoolingregion. Inourstochasticpooling,theselectedactivationisdrawn
fromamultinomialdistributionformedbytheactivationswithinthepoolingregion.
Analternateviewofstochasticpoolingisthatitisequivalenttostandardmaxpoolingbutwithmany
copies of an input image, each having small local deformations. This is similar to explicit elastic
deformationsoftheinputimages[13],whichdeliversexcellentMNISTperformance. Othertypes
ofdataaugmentation,suchasflippingandcroppingdifferinthattheyareglobalimagetransforma-
tions. Furthermore,usingstochasticpoolinginamulti-layermodelgivesanexponentialnumberof
deformationssincetheselectionsinhigherlayersareindependentofthosebelow.
1
2 ReviewofConvolutionalNetworks
Ourstochasticpoolingschemeisdesignedforuseinastandardconvolutionalneuralnetworkarchi-
tecture. We first review this model, along with conventional pooling schemes, before introducing
ournovelstochasticpoolingapproach.
A classical convolutional network is composed of alternating layers of convolution and pooling
(i.e.subsampling). Theaimofthefirstconvolutionallayeristoextractpatternsfoundwithinlocal
regionsoftheinputimagesthatarecommonthroughoutthedataset. Thisisdonebyconvolvinga
templateorfilterovertheinputimagepixels,computingtheinnerproductofthetemplateatevery
locationintheimageandoutputtingthisasafeaturemapc,foreachfilterinthelayer. Thisoutput
isameasureofhowwellthetemplatematcheseachportionoftheimage. Anon-linearfunctionf()
isthenappliedelement-wisetoeachfeaturemapc: a = f(c). Theresultingactivationsaarethen
passedtothepoolinglayer. Thisaggregatestheinformationwithinasetofsmalllocalregions,R,
producingapooledfeaturemaps(ofsmallersize)asoutput. Denotingtheaggregationfunctionas
pool(),foreachfeaturemapcwehave:
s =pool(f(c )) ∀i∈R (1)
j i j
whereR ispoolingregionj infeaturemapcandiistheindexofeachelementwithinit.
j
The motivation behind pooling is that the activations in the pooled map s are less sensitive to the
precise locations of structures within the image than the original feature map c. In a multi-layer
model, the convolutional layers, which take the pooled maps as input, can thus extract features
that are increasingly invariant to local transformations of the input image. This is important for
classificationtasks,sincethesetransformationsobfuscatetheobjectidentity.
Arangeoffunctionscanbeusedforf(),withtanh()andlogisticfunctionsbeingpopularchoices.
In this is paper we use a linear rectification function f(c) = max(0,c) as the non-linearity. In
general, this has been shown [10] to have significant benefits over tanh() or logistic functions.
However, it is especially suited to our pooling mechanism since: (i) our formulation involves the
non-negativityofelementsinthepoolingregionsand(ii)theclippingofnegativeresponsesintro-
duceszerosintothepoolingregions, ensuringthatthestochasticsamplingisselectingfromafew
specificlocations(thosewithstrongresponses),ratherthanallpossiblelocationsintheregion.
Thereare twoconventional choicesfor pool(): averageand max. The formertakes thearithmetic
meanoftheelementsineachpoolingregion:
1 (cid:88)
s = a (2)
j |R | i
j
i∈Rj
whilethemaxoperationselectsthelargestelement:
s =maxa (3)
j i
i∈Rj
Bothtypesofpoolinghavedrawbackswhentrainingdeepconvolutionalnetworks. Inaveragepool-
ing,allelementsinapoolingregionareconsidered,evenifmanyhavelowmagnitude. Whencom-
bined with linear rectification non-linearities, this has the effect of down-weighting strong activa-
tionssincemanyzeroelementsareincludedintheaverage.Evenworse,withtanh()non-linearities,
strongpositiveandnegativeactivationscancanceleachotherout,leadingtosmallpooledresponses.
Whilemaxpoolingdoesnotsufferfromthesedrawbacks,wefinditeasilyoverfitsthetrainingset
in practice, making it hard to generalize well to test examples. Our proposed pooling scheme has
theadvantagesofmaxpoolingbutitsstochasticnaturehelpspreventover-fitting.
3 StochasticPooling
In stochastic pooling, we select the pooled map response by sampling from a multinomial distri-
bution formed from the activations of each pooling region. More precisely, we first compute the
probabilitiespforeachregionj bynormalizingtheactivationswithintheregion:
a
p = i (4)
i (cid:80) a
k∈Rj k
2
Wethensamplefromthemultinomialdistributionbasedonptopickalocationlwithintheregion.
Thepooledactivationisthensimplya :
l
s =a where l∼P(p ,...,p ) (5)
j l 1 |Rj|
The procedure is illustrated in Fig. 1. The samples for each pooling region in each layer for each
training example are drawn independently to one another. When back-propagating through the
networkthissameselectedlocationlisusedtodirectthegradientbackthroughthepoolingregion,
analogoustoback-propagationwithmaxpooling.
Max pooling only captures the strongest activation of the filter template with the input for each
region. However, there may be additional activations in the same pooling region that should be
taken into account when passing information up the network and stochastic pooling ensures that
thesenon-maximalactivationswillalsobeutilized.
Sample!a!loca0on!
from!P():!e.g.!!l = 1
1.6! 0! 0! 0.4! 0! 0!
★! 0! 0! 0! 0! 0! 0! 1.6!
0! 0! 2.4! 0! 0! 0.6!
b)!Filter!
f)!Sampled!!
a)!Image! c)!Rec0fied!Linear! d)!Ac0va0ons,!ai e)!Probabili0es,!pi !!!!Ac0va0on,!s!
Figure 1: Toy example illustrating stochastic pooling. a) Input image. b) Convolutional filter. c)
Rectified linear function. d) Resulting activations within a given pooling region. e) Probabilities
basedontheactivations.f)Sampledactivation.Notethattheselectedelementforthepoolingregion
maynotbethelargestelement. Stochasticpoolingcanthusrepresentmulti-modaldistributionsof
activationswithinaregion.
3.1 ProbabilisticWeightingatTestTime
Usingstochasticpoolingattesttimeintroducesnoiseintothenetwork’spredictionswhichwefound
todegradeperformance(seeSection4.7). Instead,weuseaprobabilisticformofaveraging. Inthis,
theactivationsineachregionareweightedbytheprobabilityp (seeEqn.4)andsummed:
i
(cid:88)
s = p a (6)
j i i
i∈Rj
Thisdiffersfromstandardaveragepoolingbecauseeachelementhasapotentiallydifferentweight-
(cid:80)
ing and the denominator is the sum of activations a , rather than the pooling region size
i∈Rj i
|R |. Inpractice, usingconventionalaverage(orsum)poolingresultsinahugeperformancedrop
j
(seeSection4.7).
Our probabilistic weighting can be viewed as a form of model averaging in which each setting of
the locations l in the pooling regions defines a new model. At training time, sampling to get new
locationsproducesanewmodelsincetheconnectionstructurethroughoutthenetworkismodified.
Attesttime,usingtheprobabilitiesinsteadofsampling,weeffectivelygetanestimateofaveraging
overallofthesepossiblemodelswithouthavingtoinstantiatethem. Givenanetworkarchitecture
withddifferentpoolingregions, eachofsizen, thenumberofpossiblemodelsisnd wheredcan
beinthe104-106 rangeandnistypically4,9,or16forexample(correspondingto2×2,3×3or
4×4poolingregions). Thisisasignificantlylargernumberthanthemodelaveragingthatoccurs
in dropout [2], where n = 2 always (since an activation is either present or not). In Section 4.7
weconfirmthatusingthisprobabilityweightingachievessimilarperformancecomparedtousinga
largenumberofmodelinstantiations,whilerequiringonlyonepassthroughthenetwork.
Using the probabilities for sampling at training time and for weighting the activations at test time
leadstostate-of-the-artperformanceonmanycommonbenchmarks,aswenowdemonstrate.
3
a)) SVHN) d)) MNIST) e)) CIFAR&10) g)) CIFAR&100)
!" !" !"
b)) mean) c)) Local)CN) f)) mean) h)) mean)
Figure 2: A selection of images from each of the datasets we evaluated. The top row shows the
rawimageswhilethebottomrowarethepreprocessedversionsoftheimagesweusedfortraining.
The CIFAR datasets (f,h) show slight changes by subtracting the per pixel mean, whereas SVHN
(b) is almost indistinguishable from the original images. This prompted the use of local contrast
normalization(c)tonormalizetheextremebrightnessvariationsandcolorchangesforSVHN.
4 Experiments
4.1 Overview
Wecompareourmethodtoaverageandmaxpoolingonavarietyofimageclassificationtasks. In
allexperimentsweusemini-batchgradientdescentwithmomentumtooptimizethecrossentropy
betweenournetwork’spredictionoftheclassandthegroundtruthlabels. Foragivenparameterx
attimettheweightupdatesaddedtotheparameters,∆x are∆x = 0.9∆x −(cid:15)g whereg is
t t t−1 t t
thegradientofthecostfunctionwithrespecttothatparameterattimetaveragedoverthebatchand
(cid:15)isalearningratesetbyhand.
All experiments were conducted using an extremely efficient C++ GPU convolution library [6]
wrappedinMATLABusingGPUmat[14],whichallowedforrapiddevelopmentandexperimenta-
tion. We begin with the same network layout as in Hinton et al. ’s dropout work [2], which has 3
convolutionallayerswith5x5filtersand64featuremapsperlayerwithrectifiedlinearunitsastheir
outputs. Weusethissamemodelandtrainfor280epochsinallexperimentsasidefromoneaddi-
tionalmodelinSection4.5thathas128featuremapsinlayer3andistrainedfor500epochs.Unless
otherwisespecifiedweuse3×3poolingwithstride2(i.e.neighboringpoolingregionsoverlapby
1elementalongtheborders)foreachofthe3poolinglayers. Additionally,aftereachpoolinglayer
there is a response normalization layer (as in [2]), which normalizes the pooling outputs at each
location over a subset of neighboring feature maps. This typically helps training by suppressing
extremely large outputs allowed by the rectified linear units as well as helps neighboring features
communicate. Finally, weuseasinglefully-connectedlayerwithsoft-maxoutputstoproducethe
network’s class predictions. We applied this model to four different datasets: MNIST, CIFAR-10,
CIFAR-100andStreetViewHouseNumbers(SVHN),seeFig.2forexamplesimages.
4.2 CIFAR-10
We begin our experiments with the CIFAR-10 dataset where convolutional networks and methods
such as dropout are known to work well [2, 5]. This dataset is composed of 10 classes of natural
imageswith50,000trainingexamplesintotal,5,000perclass. EachimageisanRGBimageofsize
32x32 taken from the tiny images dataset and labeled by hand. For this dataset we scale to [0,1]
andfollowHintonetal.’s[2]approachofsubtractingtheper-pixelmeancomputedoverthedataset
fromeachimageasshowninFig.2(f).
4
Avg (train)
Avg (test)
35
Max (train)
Max (test)
30 Stochastic (train)
Stochastic (test)
25
or
Err20
%
15
10
5
0
50 100 150 200 250
Epochs
Figure3: CIFAR-10trainandtesterrorratesthroughouttrainingforaverage, max, andstochastic
pooling. Max and average pooling test errors plateau as those methods overfit. With stochastic
pooling,trainingerrorremainshigherwhiletesterrorscontinuetodecrease.1
Cross-validating with a set of 5,000 CIFAR-10 training images, we found a good value for the
learning rate (cid:15) to be 10−2 for convolutional layers and 1 for the final softmax output layer. These
rates were annealed linearly throughout training to 1/100th of their original values. Additionally,
wefoundasmallweightdecayof0.001tobeoptimalandwasappliedtoalllayers. Thesehyper-
parametersettingsfoundthroughcross-validationwereusedforallotherdatasetsinourexperiments.
Usingthesamenetworkarchitecturedescribedabove,wetrainedthreemodelsusingaverage,max
andstochasticpoolingrespectivelyandcomparetheirperformance. Fig.3showstheprogressionof
trainandtesterrorsover280trainingepochs. Stochasticpoolingavoidsover-fitting,unlikeaverage
andmaxpooling,andproduceslesstesterrors. Table1comparesthetestperformanceofthethree
poolingapproachestothecurrentstate-of-the-artresultonCIFAR-10whichusesnodataaugmenta-
tionbutaddsdropoutonanadditionallocallyconnectedlayer[2]. Stochasticpoolingsurpassesthis
resultby0.47%usingthesamearchitecturebutwithoutrequiringthelocallyconnectedlayer.
TrainError% TestError%
3-layerConv.Net[2] – 16.6
3-layerConv.Net+1LocallyConn.layerwithdropout[2] – 15.6
AvgPooling 1.92 19.24
MaxPooling 0.0 19.40
StochasticPooling 3.40 15.13
Table1: CIFAR-10Classificationperformanceforvariouspoolingmethodsinourmodelcompared
tothestate-of-the-artperformance[2]withandwithoutdropout.
To determine the effect of the pooling region size on the behavior of the system with stochastic
pooling,wecomparetheCIFAR-10trainandtestsetperformancefor5x5,4x4,3x3,and2x2pooling
sizesthroughoutthenetworkinFig.4.Theoptimalsizeappearstobe3x3,withsmallerregionsover-
fittingandlargerregionspossiblybeingtoonoisyduringtraining. Atallsizesthestochasticpooling
outperformsbothmaxandaveragepooling.
4.3 MNIST
TheMNISTdigitclassificationtaskiscomposedof28x28imagesofthe10handwrittendigits[8].
Thereare60,000trainingimageswith10,000testimagesinthisbenchmark. Theimagesarescaled
to[0,1]andwedonotperformanyotherpre-processing.
During training, the error using both stochastic pooling and max pooling dropped quickly, but the
lattercompletelyoverfitthetrainingdata.Weightdecaypreventedaveragepoolingfromover-fitting,
buthadaninferiorperformancetotheothertwomethods. Table2comparesthethreepoolingap-
proachestostate-of-the-artmethodsonMNIST,whichalsoutilizeconvolutionalnetworks.Stochas-
1Weightdecaypreventedtrainingerrorsfromreaching0withaverageandstochasticpoolingmethodsand
requiredthehighnumberofepochsfortraining.Allmethodsperformedslightlybetterwithweightdecay.
5
9.08
19.38
5x5
0
19.25
6.4
15.86
4.88
e
18.83
Siz4x4
0
18.59
Avg
Train
n
4.38
gio 15.71
Avg
Test
e
g
R 1.8
Max
Train
oolin3x3
0
1199..5523
Max
Test
P 3.4
15.13
Stochas>c
Train
Stochas>c
Test
0.25
20.74
2x2
0
21.11
3.18
16.55
0
5
10
15
20
25
%
Error
Figure4: CIFAR-10trainandtesterrorratesforvariouspoolingregionsizeswitheachmethod.
ticpoolingoutperformsallothermethodsthatdonotusedataaugmentationmethodssuchasjittering
orelasticdistortions[7].Thecurrentstate-of-the-artsinglemodelapproachbyCiresanetal.[1]uses
elastic distortions to augment the original training set. As stochastic pooling is a different type of
regularization,itcouldbecombinedwithdataaugmentationtofurtherimproveperformance.
TrainError% TestError%
2-layerConv.Net+2-layerClassifier[3] – 0.53
6-layerConv.Net+2-layerClassifier+elasticdistortions[1] – 0.35
AvgPooling 0.57 0.83
MaxPooling 0.04 0.55
StochasticPooling 0.33 0.47
Table 2: MNIST Classification performance for various pooling methods. Rows 1 & 2 show the
currentstate-of-the-artapproaches.
4.4 CIFAR-100
TheCIFAR-100datasetisanothersubsetofthetinyimagesdataset,butwith100classes[5]. There
are50,000trainingexamplesintotal(500perclass)and10,000testexamples. AswiththeCIFAR-
10, wescaleto[0,1]andsubtracttheper-pixelmeanfromeachimageasshowninFig.2(h). Due
tothelimitednumberoftrainingexamplesperclass,typicalpoolingmethodsusedinconvolutional
networksdonotperformwell,asshowninTable3. Stochasticpoolingoutperformsthesemethods
bypreventingover-fittingandsurpasseswhatwebelievetobethestate-of-the-artmethodby2.66%.
TrainError% TestError%
ReceptiveFieldLearning[4] – 45.17
AvgPooling 11.20 47.77
MaxPooling 0.17 50.90
StochasticPooling 21.22 42.51
Table3: CIFAR-100Classificationperformanceforvariouspoolingmethodscomparedtothestate-
of-the-artmethodbasedonreceptivefieldlearning.
4.5 StreetViewHouseNumbers
The Street View House Numbers (SVHN) dataset is composed of 604,388 images (using both the
difficulttrainingsetandsimplerextraset)and26,032testimages[11]. Thegoalofthistaskisto
classify the digit in the center of each cropped 32x32 color image. This is a difficult real world
problemsincemultipledigitsmaybevisiblewithineachimage. Thepracticalapplicationofthisis
toclassifyhousenumbersthroughoutGoogle’sstreetviewdatabaseofimages.
We found that subtracting the per-pixel mean from each image did not really modify the statistics
oftheimages(seeFig.2(b))andleftlargevariationsofbrightnessandcolorthatcouldmakeclas-
6
sificationmoredifficult. Instead,weutilizedlocalcontrastnormalization(asin[12])oneachofthe
threeRGBchannelstopre-processtheimagesFig.2(c). Thisnormalizedthebrightnessandcolor
variationsandhelpedtrainingproceedquicklyonthisrelativelylargedataset.
Despitehavingsignificantamountsoftrainingdata,alargeconvolutionalnetworkcanstilloverfit.
Forthisdataset, wetrainanadditionalmodelfor500epochswith64, 64and128featuremapsin
layers1,2and3respectively. Ourstochasticpoolinghelpstopreventoverfittingeveninthislarge
model (denoted 64-64-128 in Table 4), despite training for a long time. The existing state-of-the-
art on this dataset is the multi-stage convolutional network of Sermanet et al. [12], but stochastic
poolingbeatsthisby2.10%(relativegainof43%).
TrainError% TestError%
Multi-StageConv.Net+2-layerClassifier[12] – 5.03
Multi-StageConv.Net+2-layerClassifer+padding[12] – 4.90
64-64-64AvgPooling 1.83 3.98
64-64-64MaxPooling 0.38 3.65
64-64-64StochasticPooling 1.72 3.13
64-64-128AvgPooling 1.65 3.72
64-64-128MaxPooling 0.13 3.81
64-64-128StochasticPooling 1.41 2.80
Table 4: SVHN Classification performance for various pooling methods in our model with 64 or
128layer3featuremapscomparedtostate-of-the-artresultswithandwithoutdataaugmentation.
4.6 ReducedTrainingSetSize
Tofurtherillustratetheabilityofstochasticpoolingtopreventover-fitting,wereducedthetraining
set size on MINST and CIFAR-10 datasets. Fig. 5 shows test performance when training on a
randomselectionofonly1000,2000,3000,5000,10000,half,orthefulltrainingset.Inmostcases,
stochasticpoolingoverfitslessthantheotherpoolingapproaches.
9 Avg 65 Avg
8 Max 60 Max
Stochastic Stochastic
7 55
6 50
45
% Error45 % Error40
35
3
30
2
25
1 20
0 15
1000 2000 3000 5000 10000 30000 60000 1000 2000 3000 5000 10000 25000 50000
# of Training Cases # of Training Cases
Figure5:TesterrorwhentrainingwithreduceddatasetsizesonMNIST(left)andCIFAR-10(right).
Stochasticpoolinggenerallyoverfitstheleast.
4.7 ImportanceofModelAveraging
Toanalyzetheimportanceofstochasticsamplingattrainingtimeandprobabilityweightingattest
time, we use different methods of pooling when training and testing on CIFAR-10 (see Table 5).
Choosing the locations stochastically at test time degrades performance slightly as could be ex-
pected,howeveritstilloutperformsmodelswheremaxoraveragepoolingareusedattesttime. To
confirmthatprobabilityweightingisavalidapproximationtoaveragingmanymodels,wedrawN
samplesofthepoolinglocationsthroughoutthenetworkandaveragetheoutputprobabilitiesfrom
thoseN models(denotedStochastic-N inTable5). AsN increases,theresultsapproachtheprob-
abilityweightingmethod,buthavetheobviousdownsideofanN-foldincreaseincomputations.
Using a model trained with max or average pooling and using stochastic pooling at test time per-
formspoorly. Thissuggeststhattrainingwithstochasticpooling,whichincorporatesnon-maximal
elementsandsamplingnoise,makesthemodelmorerobustattesttime. Furthermore,ifthesenon-
maximal elements are not utilized correctly or the scale produced by the pooling function is not
correct,suchasifaveragepoolingisusedattesttime,adrasticperformancehitisseen.
7
When using probability weighting during training, the network easily over-fits and performs sub-
optimally at test time using any of the pooling methods. However, the benefits of probability
weighting at test time are seen when the model has specifically been trained to utilize it through
eitherprobabilityweightingorstochasticpoolingattrainingtime.
TrainMethod TestMethod TrainError% TestError%
StochasticPooling ProbabilityWeighting 3.20 15.20
StochasticPooling StochasticPooling 3.20 17.49
StochasticPooling Stochastic-10Pooling 3.20 15.51
StochasticPooling Stochastic-100Pooling 3.20 15.12
StochasticPooling MaxPooling 3.20 17.66
StochasticPooling AvgPooling 3.20 53.50
ProbabilityWeighting ProbabilityWeighting 0.0 19.40
ProbabilityWeighting StochasticPooling 0.0 24.00
ProbabilityWeighting MaxPooling 0.0 22.45
ProbabilityWeighting AvgPooling 0.0 58.97
MaxPooling MaxPooling 0.0 19.40
MaxPooling StochasticPooling 0.0 32.75
MaxPooling ProbabilityWeighting 0.0 30.00
AvgPooling AvgPooling 1.92 19.24
AvgPooling StochasticPooling 1.92 44.25
AvgPooling ProbabilityWeighting 1.92 40.09
Table 5: CIFAR-10 Classification performance for various train and test combinations of pooling
methods. The best performance is obtained by using stochastic pooling when training (to prevent
over-fitting),whileusingtheprobabilityweightingattesttime.
4.8 Visualizations
Some insight into the mechanism of stochastic pooling can be gained by using a deconvolutional
networkofZeileretal.[15]toprovideanovelvisualizationofourtrainedconvolutionalnetwork.
The deconvolutional network has the same components (pooling, filtering) as a convolutional net-
workbutareinvertedtoactasatop-downdecoderthatmapsthetop-layerfeaturemapsbacktothe
input pixels. The unpooling operation uses the stochastically chosen locations selected during the
forward pass. The deconvolution network filters (now applied to the feature maps, rather than the
input)arethetransposeofthefeed-forwardfilters,asinanauto-encoderwithtiedencoder/decoder
weights. We repeat this top-down process until the input pixel level is reached, producing the vi-
sualizations in Fig. 6. With max pooling, many of the input image edges are present, but average
pooling produces a reconstruction with no discernible structure. Fig. 6(a) shows 16 examples of
pixel-spacereconstructionsfordifferentlocationsamplesthroughoutthenetwork. Thereconstruc-
tions are similar to the max pooling case, but as the pooling locations change they result in small
localdeformationsofthevisualizedimage.
Despitethestochasticnatureofthemodel,themultinomialdistributionseffectivelycapturethereg-
ularities of the data. To demonstrate this, we compare the outputs produced by a deconvolutional
networkwhensamplingusingthefeedforward(FF)proabilitiesversussamplingfromuniform(UN)
distributions. IncontrasttoFig.6(a)whichusesonlyfeedforwardproabilities, Fig.6(b-h)replace
oneormoreofthepoolinglayers’distributionswithuniformdistributions. Thefeedforwardproba-
bilitiesencodesignificantstructuralinformation,especiallyinthelowerlayersofthemodel. Addi-
tionalvisualizationsandvideosofthesamplingprocessareprovidedassupplementarymaterialat
www.matthewzeiler.com/pubs/iclr2013/.
5 Discussion
Weproposeasimpleandeffectivestochasticpoolingstrategythatcanbecombinedwithanyother
forms of regularization such as weight decay, dropout, data augmentation, etc. to prevent over-
fitting when training deep convolutional networks. The method is also intuitive, selecting from
informationthenetworkisalreadyproviding,asopposedtomethodssuchasdropoutwhichthrow
informationaway. Weshowstate-of-the-artperformanceonnumerousdatasets,whencomparingto
other approaches that do not employ data augmentation. Furthermore, our method has negligible
computationaloverheadandnohyper-parameterstotune,thuscanbeswappedintotoanyexisting
convolutionalnetworkarchitecture.
8
a)
FF(3)
–
FF(2)
–
FF(1)
b)
UN(3)
–
FF(2)
–
FF(1)
c)
FF(3)
–
UN(2)
–
FF(1)
d)
FF(3)
–
FF(2)
–
UN(1)
Image
Avg
e)
UN(3)
–
UN(2)
–
FF(1)
f)
FF(3)
–
UN(2)
–
UN(1)
g)
FF(3)
–
UN(2)
–
UN(1)
h)
UN(3)
–
UN(2)
–
UN(1)
Max
Figure6: Topdownvisualizationsfromthethirdlayerfeaturemapactivationsforthehorseimage
(farleft). Maxandaveragepoolingvisualizationsarealsoshownontheleft. (a)–(h): Eachimage
ina4x4blockisoneinstantiationofthepoolinglocationsusingstochasticpooling. Forsampling
the locations, each layer (indicated in parenthesis) can either use: (i) the multinomial distribution
overapoolingregionderivedfromthefeed-forward(FF)activationsasinEqn.4,or(ii)auniform
(UN) distribution. We can see that the feed-forward probabilities encode much of the structure in
theimage,asalmostallofitislostwhenuniformsamplingisused,especiallyinthelowerlayers.
References
[1] D. C. Ciresan, U. Meier, J. Masci, L. M. Gambardella, and J. Schmidhuber. Flexible, high
performanceconvolutionalneuralnetworksforimageclassification. InIJCAI,2011.
[2] G.E.Hinton,N.Srivastave,A.Krizhevsky,I.Sutskever,andR.R.Salakhutdinov. Improving
neuralnetworksbypreventingco-adaptationoffeaturedetectors. arXiv:1207.0580,2012.
[3] K.Jarrett,K.Kavukcuoglu,M.Ranzato,andY.LeCun. Whatisthebestmulti-stagearchitec-
tureforobjectrecognition? InICCV,2009.
[4] Y. Jia and C. Huang. Beyond spatial pyramids: Receptive field learning for pooled image
features. InNIPSWorkshops,2011.
[5] A. Krizhevsky. Learning multiple layers of featurs from tiny images. Technical Report TR-
2009,UniversityofToronto,2009.
[6] A.Krizhevsky.cuda-convnet.http://code.google.com/p/cuda-convnet/,2012.
[7] Y.LeCun. TheMNISTdatabase. http://yann.lecun.com/exdb/mnist/,2012.
[8] Y.LeCun,L.Bottou,Y.Bengio,andP.Haffner. Gradient-basedlearningappliedtodocument
recognition. ProceedingsoftheIEEE,86(11):2278–2324,1998.
[9] G. Montavon, G. Orr, and K.-R. Muller, editors. Neural Networks: Tricks of the Trade.
Springer,SanFrancisco,2012.
[10] V. Nair and G.E. Hinton. Rectified linear units improve restricted boltzmann machines. In
ICML,2010.
[11] Y.Netzer, T.Wang, A.Coates, A.Bissacco, B.Wu, andA.Y.Ng. Readingdigitsinnatural
imageswithunsupervisedfeaturelearning. InNIPSWorkshop,2011.
[12] P. Sermanet, S. Chintala, and Y. LeCun. Convolutional neural networks applied to house
numbersdigitclassification. InICPR,2012.
[13] P.Simard,D.Steinkraus,andJ.Platt. Bestpracticesforconvolutionalneuralnetworksapplied
tovisualdocumentanalysis. InICDAR,2003.
[14] http://gp-you.org/. GPUmat. http://sourceforge.net/projects/
gpumat/,2012.
[15] M.Zeiler,G.Taylor,andR.Fergus. Adaptivedeconvolutionalnetworksformidandhighlevel
featurelearning. InICCV,2011.
9
