Universal representations:The missing link between faces, text, planktons, and cat breeds

1.9 MB·

by  Hakan Bilen

Universal representations:
The missing link between faces, text, planktons, and cat breeds
HakanBilen AndreaVedaldi
UniversityofOxford UniversityofOxford
[email protected] [email protected]
7
1
0 Abstract code
2

n With the advent of large labelled datasets and high-
a capacity models, the performance of machine vision sys- objects street no. animals universal
J temshasbeenimprovingrapidly. However,thetechnology feature

5 has still major limitations, starting from the fact that dif- extractor
2
ferent vision problems are still solved by different models,

] trained from scratch or fine-tuned on the target data. The planktons digits signs
V human visual system, in stark contrast, learns a universal
C representation for vision in the early life of an individual.
. This representation works well for an enormous variety of
s sketches glyphs faces
c vision problems, with little or no change, with the major
[ advantage of requiring little training data to solve any of

them. Figure1.Humanspossesaninternalvisualrepresentationthat,out
1
ofthebox,worksverywellanynumberofvisualdomains,from
v In this paper we investigate whether neural networks
objects and faces to planktons and characters. In this paper we
5 mayworkasuniversalrepresentationsbystudyingtheirca-
investigate such universal representations by constructing neu-
7
pacity in relation to the “size” of a large combination of
2 ralnetworksthatworksimultaneouslyonmanydomains,learning
vision problems. We do so by showing that a single neu-
7 tosharecommonvisualstructurewherenoobviouscommonality
0 ralnetworkcanlearnsimultaneouslyseveralverydifferent exists. Ourgoalistocontrastthecapacityofsuchmodelagainst
. visualdomains(fromsketchestoplanktonsandMNISTdig- thetotalsizeofthecombinedvisionproblems.
1
its)aswellas,orbetterthan,anumberofspecializednet-
0
7 works.However,wealsoshowthatthisrequirestocarefully andsoon.
1 normalizetheinformationinthenetwork,byusingdomain- Differently from machines, humans develop a powerful
: specific scaling factors or, more generically, by using an internalrepresentationofimagesintheearlyyearsoftheir
v
i instancenormalizationlayer. development [1]. While this representation is subject to
X
slight refinements even later in life, it changes little. This
r is possible because the representation has a universal va-
a
lenceandworksequallywellforanynumberofproblems,
1.Introduction
fromreadingtexttorecognizingpeopleandcontemplating
While the performance of machine vision systems is art.
nowadays believed to be comparable or even superior to Theexistenceofnon-trivialgeneral-purposerepresenta-
theoneofhumanvisionincertaintasks[16],theverynar- tionsmeansthatansignificantpartofvisioncanessentially
row scope of these systems remains a major limitation. In be learned once for all. However, the nature and scope of
fact, while vision in a human works well for an enormous such universal representations remains unclear. In this pa-
varietyofdifferentproblems,differentneuralnetworksare per,weshedsomelightonthisquestionbyinvestigatingto
required in order to recognize faces [20, 42, 47, 51], clas- which extent deep neural networks can be shared between
sify[27,17],detect[44,35]orsegment[9]high-levelobject extremelydiversevisualdomains(Fig.1).
categories,readtext[15,23],recognizebird,flower,dog,or Westartourinvestigationbyaskingwhetheritispossible
catspecies[34],interpretaradiography,anechography,or tolearnneuralnetworkssimultaneouslyfromalargenum-
aMRIimageofdifferentpartsofthehumananatomy[24], berofdifferentproblems(Fig.1). Severalauthors[41,43,
1
58] have shown that neural networks can transfer knowl- Long et al. [36] propose the Deep Adaptation Network, a
edge between tasks through a process of adaptation called multi-task and multi-branch network that matches hidden
finetuning. Whilethisisencouraging, herewelookatthe representationsoftask-specificlayersinareproducingker-
much more challenging problem of learning a single net- nel Hilbert space to reduce domain discrepancy. Misra et
workthatworkswellforalltheproblemssimultaneously. al.[37]proposeCross-Stitchunitsthatcombinetheactiva-
Severalauthorshaveconsideredmulti-taskscenariosbe- tionsfrommultiplenetworksandcanbetrainedend-to-end.
foreus,wherethetaskistoextractmultiplelabelsfromthe Ganin and Lempitsky [13] and Tzeng et al. [54] propose
same visual domain (e.g. image classification, object and deep neural networks that are simultaneously optimized to
part detection and segmentation, and boundary extraction, obtaindomaininvarianthiddenrepresentationsbymaximis-
all in PASCAL VOC [9, 3, 25]). Such tasks are expected ingtheconfusionofdomainclassifiers. Yosinskietal.[58]
topartiallyoverlapsincetheylookatthesameobjecttypes. studytransferabilityoffeaturesindeepneuralnetworksbe-
Ourgoal,instead,istocheckwhetherextremevisualdiver- tweendifferenttasksfromasingledomain. Theauthorsin-
sitystillallowsasharingofinformation. Inordertodoso, vestigatewhichlayersofapre-traineddeepnetworkscanbe
wefixthelabellingtasktoimageclassification,andlookat adaptedtonewtasksinasequentialmanner. Theprevious
combiningnumerousanddiversedomains(e.g.text,faces, workexplorevariousmethodstotransferbetweendifferent
animals,objects,sketches,planktons,etc.). networks,herewelookatlearninguniversalrepresentations
Whilethesetupissimple,itallowstoinvestigateanim- fromverydiversedomainswithasingleneuralnetwork.
portantquestion: whatisthecapacityofmodelsinrelation Ourworkisalsorelatedtomethods[19,45,6]thattrans-
to the “size” of the combination of multiple vision prob- fer the information between networks. Hinton et al. [19]
lems.Ifproblemsarecompletelyindependent,thetotalsize propose a knowledge distillation method that transfers the
shouldgrowproportionally,whichshouldbematchedbyan informationfromanensembleofmodels(teacher)toasin-
equallyunboundedincreaseinmodelcapacity.Ontheother gle one (student) by enforcing it to generate similar pre-
hand,ifproblemsoverlap,thenthecomplexitygrowthgrad- dictions to the existing ones. Romero et al. [45] extend
uallyslowsdown, allowingmodelcomplexitytocatchup, thisstrategybyencouragingsimilaritybetweennotonlythe
sothat, inthelimit, universalrepresentationsbecomepos- predictionsbutalsobetweenintermediatehiddenrepresen-
sible. tations of different networks. Chen et al. [6] address the
Ourfirstcontributionistoshow,throughcarefulexperi- slowprocessofsequentialtrainingbothteacherandstudent
mentation(section4), that thecapacityofneural networks networksfromscratch. Theauthorsacceleratethelearning
is large even when contrasted to the complexity generated processbysimultaneouslytrainingteacherandstudentnet-
by combining numerous and very diverse visual domains. works. Thislineofworkfocusesonlearningcompactand
For example, it is possible to share all layers of a CNN, accuratenetworksonthesametaskbytransferringknowl-
including classification ones, between datasets as diverse edge between different networks, while our work aims to
as CIFAR-10, MNIST and SVHN, without loss in perfor- learnasinglenetworkthatcanperformwellinmultipledo-
mance(section4.1).Ingeneral,extensivesharingofparam- mains.
eters works very well for combination of up to ten diverse
Multi-task learning. Multi-task learning [4] has been ex-
domains.
tensively studied over two decades by the machine learn-
Our second contribution is to show that, while sharing
ing community. It is based on the key idea that the tasks
is possible, it notelessly requires to normalize information
shareacommonrepresentationwhichisjointlylearntalong
carefully, in order to compensate for the different dataset
withthetaskspecificparameters. Multi-tasklearningisap-
statistics (section 3). We test various schemes, including
plied to various computer vision problems and reported to
domain-orientedbatchandinstancenormalization,andfind
achieve performance gains in object tracking [60], facial-
(section4)thatthebestmethodusesdomain-specificscal-
landmark detection [61], surface normals and edge la-
ingparameterslearnedtocompensateforthestatisticaldif-
bels[57],objectdetectionandsegmentation[10]objectand
ferencesbetweendatasets. However,wealsoshowthatin-
partdetection[3]. Incontrasttoourwork,multi-tasklearn-
stancenormalizationcanbeusedtoconstructarepresenta-
ingtypicallyfocusesondifferenttasksinthesamedatasets.
tionthatworkswellforalldomainswhileusingasingleset
ofparameters,withoutanydomain-specifictuningatall.
Life-longlearning. NeverEndingLearning[38]andLife-
long Learning [53, 48] aim at learning many tasks se-
2.RelatedWork
quentiallywhileretainingthepreviouslylearntknowledge.
Terekhov et al. [52] propose Deep Block-Modular Neural
Transfer learning and domain adaptation. Our work is Networks that allow a previously trained network learn a
relatedtomethodsthattransferknowledgebetweendiffer- new task by adding new nodes while freezing the original
ent tasks and between same tasks from different domains. networkparameters. LiandHoiem[33]recentlyproposed
theLearningwithoutForgettingmethodthatcanlearnanew acrossdomains:
task while retaining the responses of the original network
on the new task. The main focus in this line of research (φ∗,...,φ∗ )=
1 D
istopreserveinformationaboutoldtasksasnewtasksare
D
learned, whileourworkisaimedatexploringthecapacity argminλR(φ ,...,φ )+ 1 (cid:88)V (φ ). (1)
ofmodelswhenmultipletasksarelearnedjointly. φ1,...,φD 1 D D d=1 d d
ThetermR(φ ,...,φ )encodesbothregularizationterms
3.Method 1 D
as well as hard constraints, defining the structure of the
We call a representation a vectorial function φ : x (cid:55)→ learningproblem.
φ(x) ∈ RC mapping an image x ∈ RH×W×3 to a C-
No sharing. As a baseline, separate neural networks are
dimensionalcodevectorφ(x)(oftenthisvectorisalsoa3D
learned for each domain. This is obtained when the regu-
tensor). Asrepresentationsweconsiderheredeepconvolu-
larizer in Eq. 1 decomposes additively R(φ ,...,φ ) =
1 D
tionalneuralnetworks(DCNNs). ADCNNcanbedecom- (cid:80)D R(φ ). Inthiscase,thereisnosharingbetweendo-
posedasasequenceφ(x)=φ ◦···◦φ ◦φ (x)oflinear d=1 d
N 2 1 mains.
and non-linear functions φ , called layers, or, in more so-
n
phisticatedcases[50,17],asafeed-forwardcomputational Feature sharing. The baseline is set against the case in
graphwheresuchfunctionsareusedasnodes. which part of the neural networks φ are shared. In the
d
Aconceptthatwillbeimportantlateristheoneofdata simplestinstance,thismeansthatonecanwrite
batches. Neural networks are almost invariably learned
φ =φ(cid:48) ◦φ , ..., φ =φ(cid:48) ◦φ ,
by considering batches of example images together. Here 1 1 0 D D 0
we follow the standard practice of representing a batch of
whereφ isacommonsubsetofthenetworks.Forexample,
T images by adding a fourth index t = 1,...,T to the 0
followingthecommonintuitionthatearlylayersofaneural
data tensors x ∈ RH×W×3×T. As the information propa-
networks have are less specialized and hence less domain-
gatesthroughthenetwork,allintermediatetensorsarealso
specific[7],φ maycontainalltheearlylayersuptosome
batchesofT datapoints. 0
depth, after which the different networks branch off.1 We
callthisordinaryfeaturesharing.
3.1.Learningfrommultipledomains
Adapted feature sharing. In this paper, we propose and
Here we consider the problem of learning neural net-
study alternatives to ordinary feature sharing. More ab-
works from multiple domains D ,...,D . For simplic-
1 D stractly, weareinterestedinminimizingthedifferencebe-
ity, we limit ourselves to image classification problems.
tweentheindividualrepresentationsφ ,...,φ andauni-
Hence, a domain D = (X ,Y ,p ,L ) consists of an 1 D
d d d d d versal representation φ used as a common blueprint. For
input space Xd = RHd×Wd×Cd, a discrete label (output) example, when domain0s differ substantially in their statis-
space Y = {1,2,...,K }, an (unknown) joint proba-
d d tics(e.g.textvsnaturalimages),suchdifferencesmayhave
bility distribution p (x,y) over inputs and labels, and a
d a significant impact in the response of neurons, but it may
loss function L : Y × Y → R measuring the quality
d be possible to compensate for such differences by slightly
L (y,yˆ) of a label prediction yˆ against its ground truth
d adjusting the representation parameters. Another intuition
value y. As usual, the quality of a predictor yˆ = φ (x)
d is that not all features in the universal representation φ
is measured in terms of its expected risk E[L (y,φ (x))]. 0
d d maybeusefulinalldomains,sothatsomecouldbedeacti-
For each domain, furthermore, we also have a training set
vateddependingontheproblem. Inordertoexplorethese
Td = {(x(1,d),y(1,d)),...,(x(Nd,d),y(Nd,d))}ofNd train- ideas, we will consider the case in which representations
ingpairs,whichresultsintheempiricalrisk
decomposeasφ (x)=φ(cid:48) ◦φ (x|w ),wherew isasmall
d d 0 d d
number of domain-dependent parameters and φ (·;w ) is
0 d
1 (cid:88)Nd theuniversalrepresentationblueprint. Wecallthisadapted
V (φ )= L (y(t,d),φ (x(t,d)))
d d N d d featuresharing.
d
t=1
For adapted feature sharing, we consider in particular
WealsoassumethatasimilarbutdisjointvalidationsetVd anextremelysimpleformofparametrizationforφ0(·;wd)
isavailableforeachdomain. (Fig. 2). In order to be able to adjust for different mean
responses of neurons for each domain, as well as to po-
Our goal is to learn D predictors φ ,...,φ , one for
1 D
tentially select subset of features, we consider adding a
each task, in order to minimize their overall risk. While
balancingdifferenttasksisaninterestingprobleminitsown 1Such a constraint can be incorporated in Eq. 1 by requiring that
right, here we simply choose to minimize the average risk R(φ1,...,φd)<∞ ⇔ ∃φ0: ∀d=1,...,D∃φ(cid:48)d: φd=φ(cid:48)d◦φ0.
IN Sc BN Sc BN Sc
mux mux
Figure2.Fromlefttoright,threeexamplemodules: instancenormalization,batchnormalization,andbatchnormalizationwithdomain-
specificscalingbuildingmodules.Theshadedblocksindicatelearnableparameters.Othervariantsaretested,notshownforcompactness.
domain-dependentscalingfactors andabiasb aftereach where
d d
convolutionalorlinearlayerinaCNN.Thisisimplemented
byascalinglayerφscale: µct(x)= H1W (cid:88)xvuct,
vu
∀vuc: y =[φ (x;s,b)] =s x +b .
vuc scale uvc c vuc c 1 (cid:88)
σ2(x)= (x −µ (x))2.
ct HW vuct ct
Alltogether,thescaleandbiasparametersformcollections vu
S = (s ,...,s ) and B = (b ,...,b ). Since all do-
1 D 1 D
mainsaretrainedjointly,weintroducealsoamuxer(Fig.2), In practice, both batch normalization and instance nor-
malization layers are always immediately followed by a
namelyalayerthatextractsthecorrespondingparameterset
giventheindexdofthecurrentdomain: scaling layer (Fig. 2). As discussed earlier in section 3.1,
inthispaperweconsidereitherfixingthesamescalingand
φ (d;S,B)=(s ,b ). biasparametersacrossdomains,ormakethemdomainspe-
mux d d
cific.
For networks that include batch or instance normaliza-
tion layers (Section 3.2), a scaling layer already follows Batchpurity. Whenthemodelistrainedortested,batches
each occurrence of such blocks. In this case, we simply are always pure, i.e. composed of data points from a sin-
adaptthecorrespondingparametersratherthanintroducing gledomain. Thissimplifiestheimplementation,and,most
newscalinglayers. importantly,hasanimportanteffectontheBNlayer. Fora
purebatch, BNcaninfactaggressivelynormalizedataset-
3.2.Batchandinstancenormalization specific biases, which would not be possible for mixed
batches. IN,instead,operatesonanimage-by-imagebasis,
Batchnormalization(BN)[21]isasimpleyetpowerful
andisnotaffectedbythechoiceofpureormixedbatches.
techniquethatcansubstantiallyimprovethelearnabilityof
An important detail is how the BN and IN blocks are
deepneuralnetworks. Thebatchnormalizationlayerisde-
used in testing, after the network has been learned. Upon
finedas
“deploying” the architecture for testing, the BN layers are
y =[φ (x)] = xvuct−µc(x). (2) usuallyremovedbyfixingmeansandvariancestofixedav-
vuct BN vuct (cid:112)σ2(x)+(cid:15) eragesaccumulatedoverseveraltrainingbatches[21]. Un-
c
lessthisisdone,BNcannotbeevaluatedonindividualim-
wherethebatchmeansµandvariancesσ2aregivenby agesattesttime;furthermore,removingBNusuallyslightly
improvesthetestperformanceandisalsoslightlyfaster.
1 (cid:88)
µ (x)= x , DroppingBNrequiressomecareinourarchitecturedue
c HWT vuct to the difference between pure batches from different do-
vut
σc2(x)= HW1 T (cid:88)(xvuct−µc)2. mspaeicnisficDmd.eaInnsthaendexvpaeriraimnceensts(,µwd,eσted2s)t,sceolmecptuedtinbgydaommuaxiner-
vut from collections (U,Σ) (Fig. 2), or share a single set of
Recently,[55,2]notedthatitissometimesadvantageous meansandvariances(µ,σ2)betweenalldomains. Wealso
to simplify batch normalization further, and consider in- consideranalternativesetting,BN+,inwhichBNisapplied
stead instance (or layer) normalization. The instance nor- at training and test times unchanged. The disadvantage of
malization (IN) layer, in particular, has almost exactly the BN+isthatitcanonlyoperateonpurebatchesandnotsin-
sameexpressionasEq.2, butmeanandcovariancearein- gleimages;theadvantageisthatmomentsareestimatedon-
stanceratherthanbatchaverages: the-flyforeachtestbatchinsteadofbeingpre-computed.
NotethattheINlayerissimilartotheBN+layerinthat
y =[φ (x)] = xvuct−µct(x), (3) itestimatesmeansandvariancesonthefly,bothattraining
vuct IN vuct (cid:112)σ2(x)+(cid:15) andtestingtime,andappliesunchangedinbothphases.
ct
CIFAR10 MNIST SVHN dataset is introduced for zero-shot learning, it provides
class labels for each image. Caltech-256 [14] is a stan-
Nosharing 9.4 0.34 3.7
dard object classification benchmark that consists of 256
Deepsharing 10.2 0.37 3.7
object categories and an additional background class. CI-
Fullsharing 10.2 0.38 3.7
FAR10[26]consistsof6000032×32colourobjectclasses
Table 2. Top-1 error rate (%) for three datasets. The top row is in 10 classes. Daimler Mono Pedestrian Classification
forindividuallytrainednetworksperdataset. Deepsharingcorre-
Benchmark [39] contains a collection of pedestrian and
spondstosharingalltheconvolutionalbutthelastclassifierlayer.
non-pedestrianimages.Pedestriansarecroppedandresized
Fullsharingcorrespondstosharingallparametersincludingfinal
to 18 × 36 pixels. The German Traffic Sign Recogni-
classifierparameterswithdomain-specificscaleandbiasparame-
tion (GTSR) Benchmark [49] contains cropped images
ters.Notethatallthreedatasetshavetenclassesandthisallowsus
of 43 traffic signs. Sizes of the traffic signs vary between
toshareclassifierparameters.
15×15and222×193pixels. MNIST[30]contains70000
3.3.Trainingregime handwritten digits which are centred in 28 × 28 images.
Omniglot[28]consistsof1623differenthandwrittenchar-
As noted above, training always consider pure batches.
acters from 50 different alphabets. The dataset is origi-
In more detail, all models are learned by means of SGD,
nally designed for one shot learning. Instead we include
alternatingbatchesfromeachofthedomainD ,inaround-
i all the character categories in train and test time. Plank-
robin fashion. This automatically balances the datasets
ton imagery data [8] is a classification benchmark that
whenthesehavedifferentsizes, aslearningvisitsanequal
contains 30336 images of various organisms ranging from
numberoftrainingsamplesforeachdomain, regardlessof
the smallest single-celled protists to copepods, larval fish,
thedifferenttrainingsetsizes. Thiscorrespondstoweigh-
and larger jellies. Human Sketch dataset [12] contains
ingthedomain-specificlossfunctionsequally.
20000humansketchesofeverydayobjectssuchas“book”,
This design also has some practical advantages. In our
“car”, “house”, “sun”. The Street View House Numbers
implementation,differentdomainsareassignedtodifferent
(SVHN)[40]isareal-worlddigitrecognitiondatasetwith
GPUs. Inthiscase,eachGPUcomputesthemodelparam-
around 70,000 images which are centred around a single
eter gradients with respect to a pure batch extracted from
characterandresizedinto32×32pixels.
aparticulardataset. Gradientsarethenaccumulatedbefore
Asthemajorityofdatasetsdifferintermsofimageres-
thedescentstep.
olutionsandcharacteristics, imagesareresizedto64×64
Finally, note that architectures may only partially share
pixels,greyscaleonesareconvertedintoRGBbysettingthe
features, up to some depth. Obviously, domain-specific
threechannelstothesamevalue.Thoughitwouldbepossi-
parameters are updated only from the pure batches corre-
bletomaintaintheimagesintheoriginalresolution, using
spondingtothatdomain.
asinglescalesimplifiesthenetworkdesign. Eachdatasetis
alsowhitened,bysubtractingitsmeananddividingitbyits
4.Experiments
standarddeviationperchannel. Forthedatasetsthatdonot
Experiments focus on image classification problems in haveafixed trainandtestsplits, weuse 80% to 20% ratio
two scenarios. In the first one, different architectures and fortrainandtestdatarespectively.
learning strategies are evaluated on a portfolio of 10 very
Architectures. Wechoosetousethestate-of-the-artResid-
diverse image classification datasets (section 4.1), from
ual Networks [18] due to their remarkable capacity and
planktons to street numbers. For computational reasons,
performance. More specifically, for this experiment we
these experiments consider relatively small 64 × 64 pix-
select the ResNet-38 model. This network has a stack
els images and tens of thousands training images per do-
of 4 residual units with 3 × 3 convolutions for each fea-
main. Inthesecondscenario,wetestsimilarideasonlarger
ture map size ({64,32,16,8}) and with number of filters
datasets,includingImageNet,butinalessextensivemanner
{16,32,128,256} respectively. The network ends with a
duetothecomputationalcost(section4.2).
global average pooling layer and a fully connected layer
4.1.Smalldatasets followed by softmax for classification. As the majority
of the datasets have a different number of classes, we use
Data. We choose 10 image classification tasks from very a dataset-specific fully connected layer in our experiments
diverse domains including objects, hand-written digit and unlessotherwisestated.
characters, pedestrians, sketches, traffic signs, planktons As explained in section 3.3, datasets are balanced by
and house numbers. The dataset statistics are summarized samplingbatchesfromdifferentonesinaround-robinfash-
inTable1andafewexampleimagesaregiveninTable3. ionduringtraining. Wefollowthesamedataaugmentation
In more detail, Animals with Attributes (AwA) [29] strategyin[18],the64×64sizewhitenedimageispadded
contains 30475 images of 50 animal species. While the with8pixelsonallsidesanda64×64patchrandomlysam-
Dataset AwA Caltech CIFAR10 Daimler GTSR MNIST Omniglot Plankton Sketches SVHN
#classes 50 257 10 2 43 10 1623 121 250 10
#images 30k 31k 60k 49k 52k 70k 32k 30k 20k 99k
content animal object object pedestrian trafficsign digit character plankton sketch digit
Table1.Statisticsofvariousdatasets.
AwA Caltech CIFAR10 Daimler GTSR MNIST Omniglot Plankton Sketches SVHN
Figure3.Exampleimagesfromvariousdatasets.
pledfromthepaddedimageoritshorizontalflip. Notethat Fullsharing. Next,weconsidertheoppositeofnosharing
asMNIST,OmniglotandSVHNcontaindigitsandcharac- andsharealltheparametersofthenetwork(i.e.φ = φ ).
d 0
ters,wedonotaugmentflippedimagesfromthesedatasets. A common belief is that only the relatively shallow lay-
The networks are trained using stochastic gradient descent ersofaCNNareshareable,whereasdeeperonesaremore
withmomentum. Thelearningrangeissetto0.1andgrad- domain-specific[32]. Fullsharingchallengesthisnotion.
uallyreducedto0.0001afterashortwarm-uptrainingwith In this experiment, ResNet-38 is configured with BN
alearningrateof0.01asin[18].Theweightdecayandmo- with domain-specific scaling parameters (s ,b ) and mo-
d d
mentumaresetto0.9and0.0001respectively. Intesttime, ments(µ ,σ2). AsingleCNNistrainedonthreedomains,
d d
weuseonlythecentralcropofimagesandreportpercent- CIFAR, MNIST, and SVHN, because such domains hap-
agesoftop-1errorrate. pens to contain exactly 10 classes each. Although CIFAR
Next, we experiment with sharing features up to differ- objectsandMNIST/SVHNdigitshavenothingincommon,
entdepthsinthearchitectures. Tothisend, theResNet-38 werandomlypairdigitswithobjects. Thisallowstoshare
model φd = φ(cid:48)d ◦φ0 is modified to have a branch φ(cid:48)d for allfilterparameters,includingthefinalclassificationlayer,
eachtaskD,stemmingfromacommontrunkφ0. realisingfullsharing.
Baseline: no sharing. As a baseline, a different ResNet- AsshowninTable2,evaluatedonthedifferentdatasets,
38 model (i.e. φ(cid:48) = φ ) is trained from scratch for each theperformanceofthisnetworkisnearlythesameaslearn-
d d
datasetuntilconvergence(25kiterationsaresufficient)us- ingthreeindependentmodels. Thissurprisingresultmeans
ing a batch size of 128 images and BN. Although our fo- thatthemodelhassufficientcapacitytolearnclassifiersthat
cusisnotobtainingstate-of-the-artresultsbutdemonstrat- respondstronglyeithertoadigitinMNISTorSVHN,orto
ingtheeffectivenessofsharingarepresentationamongdif- anobjectinCIFAR,essentiallylearninganoroperator.The
ferentdomains,thechosenCNNprovidesagoodspeedand questiontheniswhethercombiningmoreproblemstogether
performance trade-off and achieves comparable results to caneventuallyexceedthecapacityofthenetwork.
state-of-the-art methods (see Table 3). The much deeper
ResNet-1001 [18] obtains 4.62% error rate in CIFAR-10 Deepsharing. Next,weexperimentwithsharingalllayers
(compared to our 9.4%), DropConnect [56] with a heav- except the last one, which performs classification. In this
ier multi-column network obtains 0.21% in MNIST (com- case, therefore, φ(cid:48)d is a single convolutional layer and φ0
paredtoour0.3%)andLeeetal.[31]report1.69%(com- contains the rest of the network, including all but the last
pared to our 3.7%) in SVHN by using more sophisticated fully connected layer. This setup is similar to full sharing,
pooling mechanisms. While the network yields relatively butallowstocombineclassificationtasksevenwhenthese
lowerrorratesinthemajorityofthedatasets, theabsolute donothaveanequalnumberofclasses.
performance is less good in AwA and Caltech256. How- Theresultsin Table3andTable 2showthatthe shared
ever, this is inline with the results reported in the litera- CNN performs better than training domain-specific net-
ture,wheregoodperformanceonsuchdatasetswasshown works. Remarkably, this is true for all the tasks, and re-
torequirepre-trainingonaverylargedatasetsuchasIma- duces the average error rate by 1%. Remarkably, this im-
geNet [11, 59]. In short, this validates our ResNet-38 ba- provementisobtainedwhilereducingtheoverallnumberof
seineasagoodandrepresentativearchitecture. parametersbyafactorof10.
Sharing AwA Caltech CIFAR10 Daimler GTSR MNIST Omniglot Plankton Sketches SVHN mean
nosharing 77.9 85.1 9.4 10.4 4.2 0.34 13 25.7 31.1 3.7 26.1
deep 73.7 82.6 11.7 5.2 3.5 0.38 13.1 24.9 30.8 4.1 25.0
partial(block1-3) 76.6 84.0 12.4 7.0 4.0 0.29 13.8 26.2 33.1 4.4 26.2
partial(block2-4) 76.6 84.0 9.9 7.7 3.3 0.49 12.1 25.3 30.7 3.5 25.3
deep(×2params) 74.0 81.7 9.1 4.2 4.1 0.43 12.3 25.8 29.1 3.5 24.4
deep(×4params) 73.1 81.5 7.2 4.1 3.5 0.35 11.8 25 27.6 3.5 23.8
Table3.Top-1errorrate(%)forvarioustasks. Thetableshowstheresultsincaseofnofeaturesharingbetweendifferentdomains(first
row),deepfeaturesharingofallconvolutionalweights(deep),partialsharingforselectedconvolutionalweightsinblock1-3andblock
2-4anddeepsharingwithmoreconvolutionalfilters(×2and4×numberoffilters).
Partial sharing. Here, we investigate whether there can normalization (s,b) (µ,σ) meanerror
beabenefitinspecializingatleastpartofthesharedmodel
BN universal universal –
forindividualtasks.Wetesttwosettings.Inthefirstsetting,
BN+ universal – 46.3
thenetworkhasdataset-specificparametersintheshallower
BN universal domain 27.3
block(i.e.thefirststackof4residualunits)—thisshould
BN domain domain 25
bebeneficialtocompensatefordifferentlow-levelstatistics
IN universal – 30.2
inthedomains. Inthesecondsetting,instead,thenetwork
IN domain – 30.4
specializesthelastblock—thisshouldbebeneficialinor-
dertocapturedifferenthigher-levelconceptsinfordifferent Table4.Meantop-1erroroverthe10datasetsfordifferentnormal-
izationstrategiesanddomain-specificordomain-agnostic(univer-
tasks. Interestingly, the results in Table 3 show that deep
sal)choiceofthescalingfactors(s,b)andBNmoments(µ,σ).
sharing is, for this choice of datasets and model, the best
BN+correspondstoapplyingBNattesttimeaswell,whichdoes
configuration.Specializingthelastblockisonlymarginally
notusepre-computedmoments.
worse (−0.3%) and better (+0.9%) than specializing the
firstblock. Thismayindicatethathigh-levelspecialization itsinabilitytocompensateforthelargevarianceamongthe
ispreferable. differentdomains. IfBNisappliedattesttime(BN+),such
thatmomentsarecomputedontheflybutdomain-agnostic
Network capacity. Experiments so far suggested that
scalingisstillused,resultsarebetterbutstillpoor(46.3%).
the model has sufficient capacity to accommodate all the
Domain-agnosticscalingworkswellaslongasatleastthe
tasks, despite their significant diversity. In fact, ten in-
moments are domain-specific (27.3%). However, the best
dividual networks perform worse than a single, shared
combination is to use domain-specific moments and scal-
one. Next, we increase the capacity of the model, but
ings(25%).
we keep sharing all such parameters between tasks. In
IncontrasttoBN,IN,whichnormalizesimagesindivid-
order to do so, we increase the number of convolu-
ually,worksjustaswellwithdomain-specificanddomain-
tional filters twice ({64,128,256,512}) and four times
agnostic scaling. The price to pay is a 5% drop in perfor-
({128,256,512,1024}),whichincreasesthenumberofpa-
mance compared to BN with domain-specific parameters.
rameters 4 and 16 times. Differently from learning 10 in-
However,thisstrategyhasasignificantpracticaladvantage:
dependent networks, this setup allows the model to better
INwithdomain-agnosticscalingeffectivelyusesonlyasin-
usetheaddedcapacitytoaccommodatethedifferenttasks,
glesetofparameters,includingallfilterweightsandscaling
reducingthemeanerrorrateby0.6%and1.2%points,re-
factors,foralldomains. Thissuggestthatsucharepresen-
spectively. Thefactthatjointtrainingcanexploittheadded
tationmaybeapplicabletonoveldomainwithoutrequiring
capacity better suggests that the different domains overlap
anydomain-specifictuningofthenormalizationparameters
synergistically,despitetheirapparentdifferences.
atall.
Normalization strategies. So far, we have shown that
learningasingleCNNforthe10domainsisnotonlypos- 4.2.Largedatasets
sible, but in fact preferable to learining individual models.
However, thisCNN used aspecific normalization strategy, Data. In this part, we consider three large scale com-
BN,aswellasdomain-specificscalingparameters(s ,b ) puter vision tasks: object classification in ImageNet [46],
d d
andmoments(µ ,σ2). face identification in VGG-Face [42], and word classifica-
d d
InTable4weexaminetheimportanceofthesedesignde- tioninSynth90k[22]dataset(Table5). ImageNetcontains
cisions. First, wenotethatBNwithdomain-agnosticscal- 1000objectcategoriesand1.2millionimages. VGG-Face
ing (s,b) and moments (µ ,σ2) performs very poorly on datasetconsistsof2.6millionfaceimagesof2622different
d
the test set, comparable to random chance, clearly due to peoplewhicharecenteredandresizedintoafixedheightof
ImageNet VGG-Face Synth90k
Nosharing 40.5 25.7 -
Deep(conv1-5) 41.8 26.3 -
Nosharing 40.8 25.7 12.1
Nosharing(1/ ) 49.2 57.1 39.0
3
Deep(conv1-5) 43.0 27.8 26.8
Table 5. Top-1 error rate (%). The first and second rows show
the results for the AlexNet without and with sharing parameters
onImageNetandVGG-Facedatasetsrespectively. Thethirdand
fourthdepicttheresultsforaVGG-M-128withoutparametershar-
ImageNet VGG-Face Synth90k ingindifferentcapacities.1/ indicatesthenumberofparameters
3
Figure4.Exampleimagesfromthelarge-scaledatasetsareshown reduced to a thir in each individual network. The last one show
intheirrelativesizes. theresultsforthesamenetworkwithsharingparametersonIma-
geNet,VGG-FaceandMJSynthdatasetsrespectively.
128pixelsandavariablewidth. TheSynth90kdatasetcon-
tains approximately 9 million synthetic images for a 90k ated words for 90k different word classes. For this ex-
wordlexiconwhicharegeneratedwithafixedheightof32 periment, we use the higher-capacity model VGG-M-128
pixels and variable width. We show example images from from [5]. This model has only 128 filters in the second
thesedatasetsinTable4. to last fully connected layer (fc7), instead of 4096. As
the Synth90k dataset contains 90k classes, having a small
Implementation details. ImageNet images are resized to
128-dimensional bottleneck is necessary in order to main-
256pixelsontheirshortestsideandmaintainingtheoriginal
tain the size of the 90k classes classifier matrix (which is
aspectratio. Duringtraining,random224×224,112×56
128×90k) reasonable. Since Synth90k images are much
and 32×128 patches are cropped from ImageNet, VGG-
smallerthantheothertwodatasets, thelastdownsampling
Face and Synth90k respectively. As different input sizes
layer(pool5)isnotusedforthisdomain.
leadtodifferentfeaturemapsizesatthelastconvolutional
Withoutparametersharing,thisnetworkperformssimi-
layer(a6×6mapforImageNetdowntoatiny1×6map
larlytoAlexNet(duetothebottleneckwhichpartiallyoff-
for the smallest Synth90k images – see below), we share
setsthehighercapacity). Asbefore,theconvolutionallayer
the convolutional feature maps among the three tasks but
parameters are shared, and the fully-connected layers pa-
usedomain-specificfullyconnectedlayers. Intraining,we
rameters are not. The results (see the bottom row in Ta-
augmentthedatabyrandomlycropping,flippingandvary-
ble 5) show that the capacity of the model is pushed to its
ing aspect ratio with the exception that we do not flip the
limit.PerformanceonImageNetandVGG-Faceisstillvery
imagesfromSynth90kastheycontainwords. Attesttime,
good,withaminorhitof2-3%,butthereisalargerdropfor
weonlyuseasinglecentercropandreporttop1-errorrates.
Synth90k(26.9%error). Notethatthetotalnumberofpa-
The best normalization strategy (BN with domain specific
rameters in the joint network is a third of the sum of the
scaling)identifiedin4.1isused.
individualnetworkparameters. Inordertohaveafaircom-
Results. We conduct two experiments. In the first one, parison, weevaluatetheperformanceofthreeindependent
a network is trained simultaneously on the ImageNet and modelswithathirdoftheparameterseach(seethe4th row
VGG-Facedatasets,withasignificantdifferenceincontent inTable5).Weshowthatthejointlytrainedmodelperforms
aswellasresolutionbetweendomains. WeuseanAlexNet dramatically better than the individual models, despite the
model[27], adaptthedimensionalityofthefirstfullycon- factthatthetotalnumberofparametersisthesame.
nectedlayer(fc6)fortheVGG-Facedataset,andtrainthe
networksfromscratch. Forthissetting, thebaselineisob- 5.Conclusions
tainedbytrainingtwoindividualnetworkswithoutanypa-
rameter sharing, obtaining 40.5% and 25.7% top-1 error Asmachinevisionconsolidates,thechallengeofdevel-
ratesontheImageNetandVGG-Facerespectively(seeTa- opinguniversalmodelsthat,similarlytohumanvision,can
ble5—thisisthesameaspublishedresults). Sharingthe betrained onceand solvealarge varietyofproblems, will
convolutionalweightsbetweenthesetasksachievecompa- comeintofocus. Acomponentofsuchsystemswillbeuni-
rable performance (there is a marginal drop of 1% accu- versalrepresentations,i.e.featureextractorsthatworkwell
racy),illustratingoncemorethehighdegreeofshareability for all visual domains, despite the significant diversity of
ofsuchrepresentations. thelatter.
In the second setting, we push the envelope by adding In this paper, we have shown that standard deep neu-
the Synth90k dataset which contains synthetically gener- ral networks are already capable of learning very differ-
entvisualdomainstogether,withahighdegreeofinforma- [16] K. He, X. Zhang, S. Ren, and J. Sun. Delving deep into
tionsharing. However,wehavealsoshownthatsuccessful rectifiers:Surpassinghuman-levelperformanceonimagenet
sharingrequirestuningcertainnormalizationparametersin classification. CoRR,(arXiv:1502.01852v1),2015.
the networks, preferably by using domain-specific scaling [17] K.He,X.Zhang,S.Ren,andJ.Sun. Deepresiduallearning
factors, in order to compensate for inter-domain statistical forimagerecognition. InProc.CVPR,2016.
[18] K.He,X.Zhang,S.Ren,andJ.Sun. Identitymappingsin
shifts. Alternatively, techniques such as instance normal-
deep residual networks. arXiv preprint arXiv:1603.05027,
ization can compensate for such difference on the fly, in a
2016.
domain-agnosticmanner.
[19] G.Hinton,O.Vinyals,andJ.Dean.Distillingtheknowledge
Overall, our findings are very encouraging. Universal
inaneuralnetwork.arXivpreprintarXiv:1503.02531,2015.
representationsseemtobewithinthegraspofcurrenttech-
[20] Y.Hong,R.Arandjelovic´,andA.Zisserman.Facesinplaces:
nology, at least for a wide array of real-world problems. Compoundqueryretrieval. InProc.BMVC,2016.
In fact, while our most convincing results have been ob- [21] S.IoffeandC.Szegedy. Batchnormalization: Accelerating
tainedforsmallerdatasets,webelievethatlargerproblems deep network training by reducing internal covariate shift.
canbeaddressedjustassuccessfullybyamoderateincrease CoRR,2015.
ofmodelcapacityandotherrefinementstothetechnology. [22] M.Jaderberg,K.Simonyan,A.Vedaldi,andA.Zisserman.
Syntheticdataandartificialneuralnetworksfornaturalscene
References textrecognition. InNIPSDeepLearningWorkshop,2014.
[23] M.Jaderberg,K.Simonyan,A.Vedaldi,andA.Zisserman.
[1] J.Atkinson.TheDevelopingVisualBrain.OxfordUniversity Readingtextinthewildwithconvolutionalneuralnetworks.
Press,2002. IJCV,116(1):1–20,2016.
[2] J.L.Ba,J.R.Kiros,andG.E.Hinton. Layernormalization. [24] A.Jamaludin,T.Kadir,andA.Zisserman. Spinenet: Auto-
InarXiv,2016. maticallypinpointingclassificationevidenceinspinalmris.
[3] H.BilenandA.Vedaldi.Integratedperceptionwithrecurrent InProc.MICCAI,2016.
multi-taskneuralnetworks. InProc.NIPS,2016. [25] I. Kokkinos. Ubernet: Training auniversal’convolutional
[4] R. Caruana. Multitask learning. Machine Learning, 28, neural network for low-, mid-, and high-level vision us-
1997. ing diverse datasets and limited memory. arXiv preprint
[5] K. Chatfield, K. Simonyan, A. Vedaldi, and A. Zisserman. arXiv:1609.02132,2016.
Returnofthedevilinthedetails: Delvingdeepintoconvo- [26] A.Krizhevsky and G. Hinton. Learning multiple layersof
lutionalnets. InProc.BMVC,2014. featuresfromtinyimages. 2009.
[6] T. Chen, I. Goodfellow, and J. Shlens. Net2net: Accel- [27] A. Krizhevsky, I. Sutskever, and G. E. Hinton. Imagenet
erating learning via knowledge transfer. arXiv preprint classification with deep convolutional neural networks. In
arXiv:1511.05641,2015. Proc.NIPS,2012.
[7] M. Cimpoi, S. Maji, and A. Vedaldi. Deep convolutional [28] B.M.Lake,R.Salakhutdinov,andJ.B.Tenenbaum.Human-
filterbanksfortexturerecognitionandsegmentation. CoRR, levelconceptlearningthroughprobabilisticprograminduc-
abs/1411.6836,2014. tion. Science,350(6266):1332–1338,2015.
[8] R.K.Cowen,S.Sponaugle,K.Robinson,andJ.Luo.Plank- [29] C.H.Lampert,H.Nickisch,andS.Harmeling. Learningto
tonset1.0:Planktonimagerydatacollectedfromf.g.walton detectunseenobjectclassesbybetween-classattributetrans-
smithinstraitsoffloridafrom2014-06-03to2014-06-06and fer. InProc.CVPR,pages951–958,2009.
usedinthe2015nationaldatasciencebowl. NODCAcces- [30] Y.LeCun,C.Cortes,andC.J.Burges. Themnistdatabase
sion127422,2015. ofhandwrittendigits,1998.
[9] J.Dai,K.He,andJ.Sun.Instance-awaresemanticsegmenta- [31] C.-Y.Lee,P.W.Gallagher,andZ.Tu. Generalizingpooling
tionviamulti-tasknetworkcascades. InProc.CVPR,2016. functions in convolutional neural networks: Mixed, gated,
[10] J.Dai,K.He,andJ.Sun.Instance-awaresemanticsegmenta- and tree. In International Conference on Artificial Intelli-
tionviamulti-tasknetworkcascades. InProc.CVPR,2016. genceandStatistics,2016.
[11] J. Donahue, Y. Jia, O. Vinyals, J. Hoffman, N. Zhang, [32] K. Lenc and A. Vedaldi. Understanding image representa-
E. Tzeng, and T. Darrell. Decaf: A deep convolutional tions by measuring their equivariance and equivalence. In
activation feature for generic visual recognition. CoRR, Proc.CVPR,2015.
abs/1310.1531,2013. [33] Z.LiandD.Hoiem. Learningwithoutforgetting. InProc.
[12] M.Eitz,J.Hays,andM.Alexa. Howdohumanssketchob- ECCV,pages614–629.Springer,2016.
jects? ACMTrans.Graph.(Proc.SIGGRAPH),31(4):44:1– [34] T.-Y.Lin,A.RoyChowdhury,andS.Maji.Bilinearcnnmod-
44:10,2012. elsforfine-grainedvisualrecognition. InProc.ICCV,2015.
[13] Y.GaninandV.Lempitsky.Unsuperviseddomainadaptation [35] W.Liu,D.Anguelov,D.Erhan,C.Szegedy,S.Reed,C.-Y.
bybackpropagation. Proc.ICML,2015. Fu,andA.C.Berg. Ssd: Singleshotmultiboxdetector. In
[14] G.Griffin,A.Holub,andP.Perona. Caltech-256objectcat- Proc.ECCV,2016.
egorydataset. 2007. [36] M.LongandJ.Wang. Learningtransferablefeatureswith
[15] A.Gupta,A.Vedaldi,andA.Zisserman. Syntheticdatafor deep adaptation networks. CoRR, abs/1502.02791, 1:2,
textlocalisationinnaturalimages. Proc.CVPR,2016. 2015.
[37] I.Misra, A.Shrivastava, A.Gupta, andM.Hebert. Cross- [56] L.Wan,M.Zeiler,S.Zhang,Y.L.Cun,andR.Fergus. Reg-
stitchnetworksformulti-tasklearning. Proc.CVPR,2016. ularization of neural networks using dropconnect. In Pro-
[38] T.Mitchell. Never-endinglearning. Technicalreport,DTIC ceedings of the 30th International Conference on Machine
Document,2010. Learning(ICML-13),pages1058–1066,2013.
[57] X. Wang, D. Fouhey, and A. Gupta. Designing deep net-
[39] S. Munder and D. M. Gavrila. An experimental study on
pedestrianclassification. PAMI,28(11):1863–1868,2006. worksforsurfacenormalestimation. InProceedingsofthe
IEEEConferenceonComputerVisionandPatternRecogni-
[40] Y.Netzer,T.Wang,A.Coates,A.Bissacco,B.Wu,andA.Y.
tion,pages539–547,2015.
Ng. Readingdigitsinnaturalimageswithunsupervisedfea-
[58] J.Yosinski,J.Clune,Y.Bengio,andH.Lipson. Howtrans-
turelearning. 2011.
ferablearefeaturesindeepneuralnetworks? Proc.NIPS,
[41] M.Oquab,L.Bottou,I.Laptev,andJ.Sivic. Learningand
2014.
Transferring Mid-Level Image Representations using Con-
[59] M.D.ZeilerandR.Fergus. Visualizingandunderstanding
volutionalNeuralNetworks. InProc.CVPR,2014.
convolutionalnetworks. InProc.ECCV,2014.
[42] O. M. Parkhi, A. Vedaldi, and A. Zisserman. Deep face
[60] T. Zhang, B. Ghanem, S. Liu, and N. Ahuja. Robust vi-
recognition. InBritishMachineVisionConference,2015.
sualtrackingviastructuredmulti-tasksparselearning.IJCV,
[43] A. S. Razavian, H. Azizpour, J. Sullivan, and S. Carlsson. 101(2):367–383,2013.
Cnnfeaturesoff-the-shelf:anastoundingbaselineforrecog-
[61] Z.Zhang,P.Luo,C.C.Loy,andX.Tang. Faciallandmark
nition. InProc.CORR,2014.
detectionbydeepmulti-tasklearning.InProc.ECCV,2014.
[44] S.Ren,K.He,R.Girshick,andJ.Sun.Fasterr-cnn:Towards
real-timeobjectdetectionwithregionproposalnetworks. In
arXiv:1506.01497,2015.
[45] A.Romero,N.Ballas,S.E.Kahou,A.Chassang,C.Gatta,
and Y. Bengio. Fitnets: Hints for thin deep nets. arXiv
preprintarXiv:1412.6550,2014.
[46] O. Russakovsky, J. Deng, H. Su, J. Krause, S. Satheesh,
S. Ma, Z. Huang, A. Karpathy, A. Khosla, M. Bernstein,
A.C.Berg,andL.Fei-Fei.Imagenetlargescalevisualrecog-
nitionchallenge,2014.
[47] F.Schroff,D.Kalenichenko,andJ.Philbin. Facenet:Auni-
fiedembeddingforfacerecognitionandclustering.InarXiv,
volumearXiv:1503.03832v1,2015.
[48] D.L.Silver,Q.Yang,andL.Li. Lifelongmachinelearning
systems: Beyondlearningalgorithms. InAAAISpringSym-
posium: LifelongMachineLearning,pages49–55.Citeseer,
2013.
[49] J.Stallkamp,M.Schlipsing,J.Salmen,andC.Igel. Manvs.
computer: Benchmarking machine learning algorithms for
trafficsignrecognition. NeuralNetworks,(0):–,2012.
[50] C.Szegedy,V.Vanhoucke,S.Ioffe,andJ.Shlens. Rethink-
ingtheinceptionarchitectureforcomputervision. InProc.
CVPR,2016.
[51] Y.Taigman, M.Yang, M.A.Ranzato, andL.Wolf. Deep-
face: Closing the gap to human-level performance in face
verification. InProc.CVPR,2014.
[52] A.V.Terekhov,G.Montone,andJ.K.ORegan. Knowledge
transferindeepblock-modularneuralnetworks. InConfer-
enceonBiomimeticandBiohybridSystems,pages268–279.
Springer,2015.
[53] S.Thrun.Lifelonglearningalgorithms.InLearningtolearn,
pages181–209.Springer,1998.
[54] E.Tzeng,J.Hoffman,T.Darrell,andK.Saenko.Simultane-
ousdeeptransferacrossdomainsandtasks. InProc.CVPR,
pages4068–4076,2015.
[55] D. Ulyanov, A. Vedaldi, and V. S. Lempitsky. Instance
normalization: The missing ingredient for fast stylization.
CoRR,abs/1607.08022,2016.