DeepHash: Getting Regularization, Depth and Fine-Tuning Right

0.62 MB·English

by  Jie Lin

DeepHash: Getting Regularization, Depth and Fine-Tuning Right
JieLin∗,1,3,OlivierMore`re∗,1,2,3,VijayChandrasekhar1,3,AntoineVeillard2,3,HanlinGoh1,3
I2R1,UPMC2,IPAL3
Abstract stricted Boltzmann machines (RBM) are primarily known
5 aspowerfuldimensionalityreductiontechniques[8].
1 Thisworkfocusesonrepresentingveryhigh-dimensional We propose DeepHash, a deep binary hashing scheme
0
global image descriptors using very compact 64-1024 bit thatcombinespurpose-specificregularizationwithweakly-
2
binary hashes for instance retrieval. We propose Deep- supervisedfine-tuning. Athoroughempiricalevaluationon

n Hash: a hashing scheme based on deep networks. Key a number of publicly available data sets shows that Deep-
a
to making DeepHash work at extremely low bitrates are Hash consistently and significantly surpasses other state-
J
threeimportantconsiderations–regularization,depthand of-the-art methods at bitrates from 1024 down to 64. This
0
2 fine-tuning – each requiring solutions specific to the hash- is due to the correct mix of regularization, depth and fine-
ing problem. In-depth evaluation shows that our scheme tuning. ThisworkrepresentsastrongsteptowardstheHoly

] consistentlyoutperformsstate-of-the-artmethodsacrossall Grailofaperfect64-bithash.
V
data sets for both Fisher Vectors and Deep Convolutional
C
NeuralNetworkfeatures,byupto20%overotherschemes. 2.RelatedWorkandContributions
s. The retrieval performance with 256-bit hashes is close to
c thatoftheuncompressedfloatingpointfeatures–aremark- Hashing schemes can be broadly categorized into un-
[ able512×compression. supervised and supervised (including semi-supervised)
1 schemes. Examples of unsupervised schemes are Itera-
v tive Quantization [9], Spectral Hashing [10], Restricted
1 1.Introduction Boltzmann Machines [8], while some examples of state-
1
of-the-artsupervisedschemesincludeMinimalLossHash-
7
Acompactbinaryimagerepresentationsuchasa64-bit ing[11], Kernel-basedSupervisedHashing[12], Ranking-
4
0 hashisadefinitemustforfastimageretrieval. 64bitspro- based Supervised Hashing [13] and Column Generation
. videmorethanenoughcapacityforanypracticalpurposes, Hashing[14].Supervisedhashingschemesaretypicallyap-
1
includinginternet-scaleproblems.Inaddition,a64-bithash pliedtothesemanticretrievalproblem. Inthiswork,weare
0
5 is directly addressable in RAM and enables fast matching focused on instance retrieval: semantic retrieval is outside
1 usingHammingdistances. thescopeofthiswork.
:
v State-of-the-art global image descriptors such as Fisher There is plenty of work on binary codes for descriptors
i Vectors (FV) [1] and Deep Convolutional Neural Network likeSIFTorGIST[9,15,16,17,10,12,18,11,19,20,21].
X
(DCNN) features [2, 3] allow for robust image matching. There is comparatively little work on hashing descriptors
r
a However,thedimensionalityofsuchdescriptorsistypically like Fisher Vectors (FV) which are two orders of magni-
veryhigh:8192to65536floatingpointnumbersforFVs[1] tudehigherindimensionality. Perronninetal.[1]propose
and4096forDCNNs[2]. Bringingsuchhigh-dimensional ternary quantization of FV, quantizing each dimension to
floatingpointrepresentationsdowntoa64-bithashisacon- +1,-1or0. Perronninetal. alsoexploreLocalitySensitive
siderablechallenge. Hashing[22]andSpectralHashing[10]. SpectralHashing
Deeplearninghasachievedremarkablesuccessinmany performspoorlyathighrates,whileLSHandsimpleternary
visual tasks such as image classification [2, 4], image re- quantizationneedthousandsofbitstoachievegoodperfor-
trieval [3], face recognition [5, 6] and pose estimation [7]. mance. Gong et al. propose the popular Iterative Quanti-
Furthermore, specific architectures such as stacked re- zation (ITQ) scheme and apply it to GIST [9]. In subse-
quentwork,Gongetal.[23]focusongeneratingverylong
∗ O.More`reandJ.Lincontributedequallytothiswork. codes for global descriptors, and the Bilinear Projection-
1.InstituteforInfocommResearch,A*STAR,Singapore. basedBinaryCodes(BPBC)schemerequirestensofthou-
2.Universite´PierreetMarieCurie,Paris,France.
sandsofbitstomatchtheperformanceoftheuncompressed
3.Image&PervasiveAccessLab,UMICNRS2955,Singapore.
WethankNVIDIACorp.fordonatingtheGPUusedforthiswork. globaldescriptor.Jegouetal.proposeProductQuantization
1
Global Feature Extraction
Training Phase 1: Unsupervised
High-dimensional
Image Descriptor Stacked Regularized RBMs
W W W
1 2 L
Input Image . . .
Training
8K-64K
Fisher Vector dim.
Transfer
or or Training Phase 2: Fine-Tuning model
Deep Convolutional Neural Network d4imK. Deep Siamese Network
W W W
1 2 L
. . .
Testing
Matching &
non-matching Loss Loss Loss
Compact pairs 1 2 L
Trained DeepHash Model Binary Hash
Image
Descriptor W1 W2 WL W1 W2 WL
Hashing . . . . . .
(Testing)
64-1K bits
Figure1.Ourproposedhashingandmodeltrainingpipeline. Ahigh-dimensionalglobalimagedescriptor,suchasfishervectoranddeep
convolutionalneuralnetfeature,isextractedfromanimage. ThetrainedDeepHashmodeltransformsthisimagedescriptortoacompact
binary hash (between 64 to 1K bits), via a succession of L nonlinear feedforward projections. The DeepHash model is trained in two
phases:aunsupervisedpre-trainingphaseandaweaklysupervisedfine-tuningphase. Inphase1,restrictedBoltzmannmachines(RBMs)
aretrainedinalayer-wisemannerandstackedintoadeepnetwork. Inphase2,matchingandnon-matchingpairsareusedtoconstruct
deepSiamesenetworksforparameterfine-tuning.
(PQ)forobtainingcompactrepresentations[24].Whilethis consistentlyoutstandingacrossawiderangeofdatasets
produces compact descriptors, the resulting representation andbothDCNNandFV,showingtherobustnessofour
is not binary and cannot be compared with Hamming dis- scheme.
tances. Asopposedtopreviouswork,ourfocusisongener-
atingextremelycompactbinaryrepresentationsforFVand 3.DeepHash
DCNNfeaturesinthe64bits-1024bitsrange.
DeepHashisahashingschemebasedonadeepnetwork
Inthispaper,weproposeDeepHash(Figure1),ahash-
to generate binary compact hashes for image instance re-
ing scheme based on deep networks for high-dimensional
trieval (Figure 1).1 Given a global image descriptor z0, a
global descriptors. The key to making the DeepHash
deep network performs a series of L layers of nonlinear
scheme work at extremely low bitrates are three impor-
projections to generate a compact hash zL. The model is
tant considerations – regularization, depth and fine-tuning
trained in two phases: 1) greedy layer-wise unsupervised
–eachrequiringsolutionsspecifictothehashingproblem.
pre-training with hashing regularization and 2) weakly-
• We pre-train a deep network using a RBM regulariza- supervisedSiamesefine-tuning.
tion scheme that is specifically adapted to the hash- Intheunsupervisedphase,stackedrestrictedBoltzmann
ing problem. This enhances the efficiency of compact machines(RBMs)[26]areusedtolearntheinitialparame-
hashes, while achieving performance close to the un- tersofthedeepnetwork. Eachnewlayerinthenetworkis
compresseddescriptor. trainedtomodelthedatadistributionofthepreviouslayer
• Using stacked RBMs as a starting point, we fine-tune and is regularized specifically for hashing. A key feature
themodelasadeepSiamesenetwork.Criticalimprove- is that this unsupervised pre-trained model is easily trans-
mentsinthelossfunctionleadtofurtherimprovements ferable. The unsupervised RBM parameters, which can
inretrievalresults. be used to generate good hashes, can be further optimized
withafine-tuningphase. Fine-tuningisdonethroughweak
• DeepHash training is only required to be performed
supervision by treating the deep model as a Siamese net-
onceonasinglelargeindependenttrainingset.Through
work [27]. Fine-tuning is also carried out an independent
a thorough evaluation against state-of-the-art hashing
data set. In the rest of this section, we will describe the
schemesusedforinstanceretrieval,weshowthatDeep-
Hashoutperformsotherschemesbyasignificantmargin 1DeepHashwillbemadepubliclyavailableonCaffeModelZoo[25]
ofup20%,particularlyatlowbitrates. Theresultsare (https://github.com/BVLC/caffe/wiki/Model-Zoo).
detailsofthetrainingprocessforourdeephashingscheme. sparsity[31,32,33]. Forclassification,representationsare
encouraged to be very sparse to improve separability. For
3.1.StackedReguarizedRBMs
hashing,however,itisdesirabletoencouragetherepresen-
The deep network with L layers is initially pre-trained tationtomakeefficientuseofthelimitedlatentsubspace.
layer-by-layer from the bottom up through unsupervised Foragivenlandaminibatchofinputinstanceszlα−1,we
learning,whereeachpairofsuccessivelayers(zl−1andzl) addaregularizationtermtotheRBMoptimizationproblem
istrainedasanRBMbuildingblock. AnRBMisanbipar- toencourage(a)halfthebitstobeactiveforagivenhash,
tite Markov random field with the input layer zl−1 ∈ RI and(b)eachbitvaluetobeequiprobableacrosshashes:
connectedtoalatentlayerzl ∈ RJ viaasetofundirected
(cid:18) (cid:19)
weights Wl ∈ RIJ. The input units zl−1 and latent units argmin −(cid:88)log (cid:88) P(zl−1,zl )+λh(E ) , (4)
i α α α
zjl arealsoparameterisedbytheircorrespondingbiasescli−1 {Wl,bl,cl−1} α zlα∈Eα
andbl,respectively.
j
whereE istheminibatchofsampledlatentunitsforlayer
α
landλistheregularizationconstant.
Binary RBMs. The first layer of the deep network takes
Weadaptthefine-grainedregularizationproposedin[33]
a high-dimensional image descriptor as input. Previous
tosuitourhashingproblem. Foreachinstancezl ,theregu-
works [1, 28] have shown that binarization of FV and α
larizationtermforbinaryunitspenaliseseachunitzl with
DCNN features results in negligible loss in performance. jα
thecrossentropylosswithrespecttoatargetactivationtl
For this work, binarization is done by component-wise jα
basedonapredefineddistribution,
meanthresholdingfortheinputs. Weusebinarylatentunits
withsigmoidactivationfunction,becausebinaryoutputbits (cid:88)(cid:88)
h(E )=− tl logzl +(1−tl )log(1−zl ).(5)
are desired for our hash. Binary RBMs are also faster and α jα jα jα jα
simplertotrainascomparedtocontinuousRBMs[29]. All zlα∈Eαj
layers in the deep network will consist of binary units and
Unlike [33], we choose the tl such that each {tl } for
binaryhashescanbeextractedfromallintermediatelayers. jα jα j
fixedαandeach{tl } forfixedj isdistributedaccording
The units within a layer are conditionally independent jα α
toU(0,1). Theuniformdistributionissuitableforhashing
pairwise.Therefore,theactivationprobabilitiesofonelayer
high-dimensional vectors because the regularizer encour-
can be sampled by fixing the states of the other layer, and
ages the each latent unit to be active with a mean of 0.5,
using distributions given by logistic functions for binary
whileavoidingactivationsaturation. Theresultisaspace-
RBMs:
fillingeffectinthelatentsubspace,wheredataisefficiently
P(zl|zl−1)=1/(1+exp(−w zl−1−b )), (1) represented.
j j j
After RBM training, we further enforce space utiliza-
P(zl−1|zl)=1/(1+exp(−w(cid:62)zl−c )). (2) tion by substituting the learned RBM bias by the data set
i i i mean (cid:104)w zl−1(cid:105) of the linear projection preceding the lo-
j
As a result, alternating Gibbs sampling can be performed gistic. Equation (3) is modified such that the final hash is
between the two layers. The sampled states are used to centeredaround0.5:
update the parameters {Wl,bl,cl−1} through minibatch
(cid:40)
gradient descent using the contrastive divergence algo- 1, if w zl−1−(cid:104)w zl−1(cid:105)>0
zl = j j (6)
rithm [30] to approximate the maximum likelihood of the j 0, otherwise.
inputdistribution.
Given a trained RBM with fixed parameters and an in-
Stacked RBMs. The set of global image descriptors lie
put vector, a hash can be generated through a feedforward
in a complex manifold in a very high-dimensional feature
projectionandthresholdingEquation(1)at0.5.
space. Deepernetworkshavethepotentialtodiscovermore
(cid:40) complex nonlinear hash functions and improve image in-
1, ifP(zl−1|zl)>0.5
zl = i (3) stanceretrievalperformance.Following[26],westackmul-
j 0, otherwise.
tipleRBMsbytrainingonelayeratatimetocreateadeep
networkwithseverallayers.
Hashing Regularization. The unsupervised RBM is Eachlayermodelstheactivationdistributionofthepre-
naively trained without considering the task, which in this viouslayerandcaptureshigherordercorrelationsbetween
case is image hashing. It is, however, important for the those units. For the hashing problem, we are interested in
RBMs to project the data in a latent subspace that is suit- low-ratepointsof64,256and1024bits,whicharetypical
able for hashing. One way to encourage the learning of operatingpointsasdiscussedinSection4.Weprogressively
suitablerepresentationsistoperformregularization,suchas decrease the dimensionality of latent layers by a factor of
0.5
0
1
=
@ R0.4 SNionng−lem Matacrhgiinng Only .
all Double Margin
Rec0.3
0.2
0 4K 8K 12K 16K 20K 24K
Iterations
Figure3.Recall@R=10ontheHolidaysdataset(SeeSection4.1fora
descriptionofthedatasets)overseveraliterationsofSiamesefine-tuning.
(a) (b) Therecallratequicklycollapseswhenusingthesinglemarginlossfunc-
tionsuggestedbyHadselletal. [34]whileperformanceisbetterretained
Figure2.Asamplepoint(blackdot)withcorrespondingmatching(red
whenonlynon-matchingpairsarepassed. Thedouble-marginlosssolves
dots)andnon-matching(bluedots)samples.Thecontrastivelossusedfor
theproblem.ThenetworkisastackedRBM(8192-4096-2048-64)trained
fine-tuningcanbeinterpretedasapplyingattractiveforcesbetweenmatch-
withFisherdescriptorsontheImageNetdataset.Matchingpairsaresam-
ingelements(redarrows)andrepulsiveforcesbetweennon-matchingel-
pledfromtheYandexdataset. Foreverymatchingpair, arandomnon-
ements(bluearrows). (a)Thelossfunction(7)proposedin[34]witha
matchingelementischosenfromthedatasettoformtwonon-matching
singlemarginparameterfornon-matchingpairs(bluecircle). Matching
pairs. Thereare33matchingpairsand66correspondingnon-matching
elementsaresubjecttoattractiveforcesregardlessofwhethertheyareal-
pairwitheveryiteration.ThetestsetistheHolidaysdataset.
readycloseenoughfromeachotherwhichadverselyaffectsfine-tuning.
(b)Ourproposedlossfunction(8)withanadditionalmarginparameter
affectingmatchingpairsreciprocally(redcircle).
matching pairs. As shown in Figure 2(a), the effect is to
apply a contractive force between elements of any match-
2n perlayer,wherenisatuneableparameter. Forourfinal ing pairs and a repulsive force between elements of non-
models,nisempiricallyselectedforeachlayerresultingin matchingpairswhichelement-wisedistanceisshorterthan
√
variablenetworkdepth. m.
However, experiment results in Figure 3 show that the
3.2.DeepSiameseFine-Tuning
loss function (7) causes a quick drop in retrieval results.
Retrieval results are driven by the structure of the local Resultswithnon-matchingpairsalonesuggestthatthehan-
neighborhoodaroundthequery. Theunsupervisedtraining dlingofmatchingpairsisresponsibleforthedrop. Thein-
is followed by a fine-tuning step in order to improve the definite contraction of matching pairs well beyond what is
local structure of the embedding. The fine-tuning is per- necessarytodistinguishthemfromnon-matchingelements
formedwithalearningarchitectureknownasSiamesenet- isadamagingbehaviour, speciallyinafine-tuningcontext
worksfirstintroducedin[27]. Theprinciplewaslatersuc- since the network is first globally optimized with a differ-
cessfullyappliedtodeeparchitecturesforfaceidentification ent objective. Figure 4 shows that any two elements, even
[35]andshowntoproducerepresentationsrobusttovarious matching,arealwaysfarapartinhighdimension. Asaso-
transformationsintheinputspace[34]. TheuseofSiamese lution,weproposeadouble-marginlosswithanadditional
architecturesinthecontextofimageretrievalfromDCNN parameteraffectingmatchingpairs:
featureswasrecentlysuggestedasapossibleimprovement
D (z0,z0)=ymax((cid:107)zl −zl(cid:107)2−m ,0)
tothestate-of-the-artonthesubject[3]. l α β α β 2 1
(8)
A Siamese network is a weakly-supervised scheme for +(1−y)max(m −(cid:107)zl −zl(cid:107)2,0)
2 α β 2
learning a similarity measure from pairs of data instances
As shown in Figure 2(b), the new loss can thus be inter-
labeledasmatchingornon-matching. Inouradaptationof
pretedaslearning“locallarge-marginclassifiers”(ifm ≤
the concept, the weights of the trained RBM network are 1
m )todistinguishbetweenmatchingandnon-matchingele-
fine-tuned by learning a similarity measure at every inter- 2
ments.Inpractice,wefoundthatthetwomarginparameters
mediate layer in addition to the target space. Given a pair
of data (z0,z0), a contrastive loss D is defined for every canbesetequal(m1 = m2 = m)andtunedautomatically
α β l
layerlandtheerrorisbackpropagatedthoughgradientde- fromthestatisticaldistributionofthesampledmatchingand
non-matchingpairs(Figure4).
scent. Backpropagationforthelossesofindividuallayers
(l=1..L)isperformedatthesametime. Applyingtheloss
4.ExperimentalResults
functionproposedbyHandselletal.in[34]yields:
4.1.EvaluationFramework
D (z0,z0)=y(cid:107)zl −zl(cid:107)2+(1−y)max(m−(cid:107)zl −zl(cid:107)2,0)
l α β α β 2 α β 2
(7) Global Descriptors. For the FV, we extract SIFT [36]
where y = 1 if (z0,z0) is a matching pair or y = 0 oth- features obtained from Difference-of-Gaussian (DoG) in-
α β
erwise, and m > 0 is a margin parameter affecting non- terest points. We use PCA to reduce dimensionality of
Effect of Regularization
90 80
b=1024
b=256
80 70 bb==614024 (HR)
b=256 (HR)
@ R70 R=1060 b=64 (HR)
Recall 60 Recall @ 10 Recall @ 50
50 Recall @ 10 (HR) . 40
Recall @ 100
Recall @ 100 (HR)
40 64 256 1024 301 2 3 4 5
Bits Depth
(a) (b)
Figure4.HistogramsofsquaredEuclideandistancesfor20,000matching Figure6.HashingFVforHolidays. (HR)referstoschemestrainedwith
pairsandcorresponding40,000non-matchingpairsforan8192-4096(top)- hashingregularization. (a)Hashingregularizationimprovesperformance
2048(middle)-64(bottom)stackedRBMnetwork.Theredandbluevertical significantlyforsinglelayermodels8192-basbisdecreased. (b)Recall
linesindicatethemedianvaluesforthematchingandnon-matchingpairs improves as depth is increased for lower rate points b = 64 and b =
respectively.TheSiameselosssharedmarginvaluemissystematicallyset 256.Withregularization,wecanachievethesameorbetterrecallatlower
tobethemeanofthetwovalues(blackverticallines). depth.
theSIFTdescriptorfrom128to64dimensions,whichhas ments, we present results on Holidays and UKB data sets,
shown to improve performance [37]. We use a Gaussian combined with the 1 million MIR-FLICKR distractor data
Mixture Model (GMM) with 128 centroids, resulting in set[45].
8192 dimensions each for first and second order statistics.
Only the first-order statistics are retained in the global de- Comparisons. We compare several state-of-the-art
scriptor representation, as second-order statistics only re- schemes. Some have been proposed for lower dimen-
sultsinasmallimprovementinperformance[38]. TheFV sional vectors like SIFT and GIST, but we evaluate their
isL -normalizedtounit-norm,aftersignedpowernormal- performanceonbothFVandDCNNfeatures.
2
ization.WedenotethisconfigurationastheFVfeaturefrom
• ITQ [9]. For the Iterative Quantization (ITQ) scheme,
here-on.
the authors propose signed binarization after applying
DCNNfeaturesareextractedusingtheopen-sourcesoft-
twotransforms: firstthePCAmatrix,followedbyaro-
ware Caffe [25] for the 7-layer AlexNet proposed for Im-
tation matrix, which minimizes the quantization error
ageNet classification in their seminal contribution [2]. We
of mapping PCA-transformed data to the vertices of a
findthatlayerfc6(beforesoftmax)performsthebestforim-
zero-centeredbinaryhypercube.
ageretrieval, similartotherecentlyreportedresultsin[3].
• BPBC[23]. InsteadofthelargePCAprojectionmatri-
WerefertothisfeatureastheDCNNfeaturefromhere-on.
ces used in [9], the authors apply bilinear projections,
whichrequirefarlessmemory.
Training Data. Most schemes, including our proposed
• LSH [22]. LSH is based on random unit-norm projec-
scheme, requireatrainingstep. WeusetheImageNetdata
tionsfollowedbysignedbinarization.
set for training, which consists of 1 million images from
1000differentimagecategories[39]. Werandomlysample • PQ [24]. For FV and Product Quantization, we con-
a subset of images from ImageNet. For the proposed deep siderblocksofdimensionsD = 64,256and1024,and
Siamese fine-tuning scheme proposed, we use the 200K train K = 256 centroids for each block, resulting in
matching image pairs data set provided by Yandex in their b = 64,256 and 1024 bit descriptors respectively. For
recent work [3], consisting primarily of landmark images. DCNN,weconsiderblocksofdimensionsD =32,128
Foreverymatchingpair,arandomsampleispickedtogen- and 1024, with K = 256 centroids, resulting in the
erate 2 corresponding non-matching pairs. This training same bitrates. Here, we do not apply Random Rota-
setisindependentofthequeryanddatabasedatadescribed tions, or PCA before applying PQ [24]. Such prepro-
next. cessingcanbeappliedtootherschemestoo. Thisisnot
a binary hashing scheme and only included for refer-
ence.
Testing Data. We use 4 popular data sets for small
scale experiments: Oxford (55 queries, 5062 database im- WeignoreSpectralHashing[10]duetoitsinferiorper-
ages)[40],INRIAHolidays(500queries,991databaseim- formanceonFVin[1].
ages)[41],StanfordMobileVisualSearchGraphics(1500
4.2.DeepHashExperiments
queries, 1000 database images) [42, 43] and University
of Kentucky Benchmark (UKB) (10200 queries, 10200 HashingRegularization. InFigure6(a),weshowtheef-
database images) [44]. For large-scale retrieval experi- fectofapplyingregularizationproposedinSection3.1ona
1 70 70
C)
U0.95 60 60
A
Under ROC Curve (0.008..589 Recall @ R=1023450000 MAP23450000 DDPILBTSQPeeQHeeBppCHHaasshh+
Area 0.75 10 10 Uncompressed
0.7 64 256 1024 0 64 256 1024 0 64 256 1024
Bits Bits Bits
(a) (b) (c)
Figure5.ComparingAUC,RecallandMAPperformanceofdifferentschemesatvaryingbin(a),(b)and(c)respectively. HolidaysandFVareusedfor
retrievalexperiments,andSMVSforAUC.DeepHashoutperformsallschemes. Also,theperformanceorderingofschemesislargelyconsistentbetween
AUCresultsandretrievalresults,bothMAPandRecall.AUCcanbeusedforfastoptimizationofparameters.
singlelayerRBM8192-b,forb=64,256,1024. TheHoli- Table1.RetrievalresultsbeforeandafterSiamesefine-tuning,withcor-
respondingdifferences. ThestackedRBMnetwork(8192-4096-2048-64)
daysdatasetandFVfeaturesarechosen. Hashingregular-
istrainedwithFisherdescriptorsfromtheImageNetdataset.Fine-tuning
izationimprovesperformancesignificantly,∼10%absolute consistentlyimprovesretrievalresultsatanybit-rate.
recall@R=10atlow-ratepointb=64.Theperformance Recall@R=10 Recall@R=100
Layer
gapincreasesasratedecreases.Thisisintuitiveastheregu- bef. aft. diff. bef. aft. diff.
larizationpushesthenetworktowardskeepinghalfthebits days 42009468 7607..8734 7731..6172 23..8348 8898..9727 9912..4004 13..4287
aliveandequiprobable(acrosshashes),withitseffectbeing oli 64 52.06 53.04 0.98 80.38 83.91 3.53
H
morepronouncedatlowerrates. d 4096 19.38 21.73 2.35 41.09 45.19 4.10
or 2048 14.32 17.23 2.91 36.03 41.03 5.00
xf
O 64 10.69 12.01 1.32 23.75 29.99 6.24
Depth. In Figure 6(b), we plot recall @ R = 10 for the 4096 79.22 82.22 3.00 92.04 93.73 1.69
B
HolidaysdatasetandFVfeatures,asdepthisincreasedfor K 2048 75.62 79.37 3.75 90.79 92.82 2.03
U 64 47.94 49.25 1.31 73.02 73.94 0.92
a given rate point b. For b = 1024, we consider configu-
rations8192-1024,8192-4096-1024,and8192-4096-2048-
1024 corresponding to depth 1,2,3 respectively. For rate
ered: 7.46%@R=10and7.24%@R=100relatively.
points b = 64 and 256, similar configurations of varying
Wenoticedifferencesacrosstestsetswithimprovements
depth are chosen. We observe that, with no regulariza-
ontheOxfordsetbeingmorepronounced. TheYandexdata
tion,recallimprovesasdepthisincreasedforb = 256and
setusedforfine-tuningismadewithmatchingpairsofland-
b = 64,withoptimaldepthof3and4respectively,beyond
markstructureswhichcanexplainthebetterperformanceof
whichperformancedrops. Athigherratesofb = 1024and
theOxforddatasetmadeofbuildingsonly. Thesystematic
beyond,increasingdepthdoesnotimproveasperformance
improvements on all sets are nevertheless evidence of the
saturates. Forhashing,asweetspotinperformanceforthe
hightransferabilityofbothunsupervisedtrainingandsemi-
depth parameter is observed for each rate point, as deeper
supervisedfine-tuning.
networks can cause performance to drop due to loss of in-
formation over the layers. Similar trends are obtained for
recall @ R = 100. Importantly, we observe that with the
Fast Optimization with ROC Experiments. The Stan-
proposed regularization, we can achieve the same perfor-
ford Mobile Visual Search (SMVS) data set [42] contains
mancewithlowerdepthateachratepoint. Thisiscritical,
a list of 16,319 matching image pairs, comprising a wide
aslowerthedepth,thefasterthehashgeneration,andlower
rangeofobjectcategories. WeextractFVformatchingand
thememoryrequirements.
non-matching pairs from the SMVS data set, hash the data
to different rate points, and compute the Receiver Operat-
Fine-Tuning. Table 1 provides detailed retrieval results ing Characteristic (ROC) curve. In Figure 5(a), we plot
for a 3-layer model before and after Siamese fine-tuning. ROC Area Under Curve (AUC) for different schemes for
Theresultsshowconsistentimprovementswitheverytrain- the SMVS data set. In Figure 5(b) and 5(c), we plot re-
ing data set and at any bit-rate with a global average dif- call@10andMAPontheHolidaysdataset. Theretrieval
ference of 2.78% (up to 6.24%). The difference is more performance of a scheme at a given database size depends
significant at higher recall rates with an average of 2.43% on the ROC curve at different TPR/FPR operating points,
@R=10comparedto3.13%@R=100. Theyarehowever as shown in [46]. Low FPR points are important [46, 47].
quitecomparablewhenrelativeimprovementrateisconsid- We observe that the Area Under Curve (AUC) results pre-
Bits FV DCNN
DeepHash+ DeepHash PQ ITQ
1024 8192-1024 4096-1024 ITQ LSH BPBC Uncompressed
256 8192-4096-256 4096-2048-256 100 100
64 8192-4096-2048-64 4096-2048-1024-64 90 90
Table2.DeepHashArchitecture 00 80 00 80
1 1
R= 70 R= 70
@ @
all 60 all 60
dict well the performance ordering (MAP and Recall) of ec 50 ec 50
R R
different schemes for retrieval experiments. The retrieval 40 40
andAUCexperimentsareperformedonverydifferentdata 30 30
64 256 1024 64 256 1024
sets, but the AUC results generalize well, and are used for Bits Bits
fastoptimizationofparameters. (a)FV,Holidays (b)DCNN,Holidays
4 4
DeepHash Parameters. For RBM learning, we set the
43 43
= =
learning rate to 0.001 for the weight and bias parameters, @ R @ R
momentum to 0.9, and ran the training on 150,000 images all 2 all 2
fromtheImageNetdatasetforamaximum30epochs. Bi- Rec Rec
nary descriptors for the first layer are generated by sub- 4 x 1 4 x 1
tracting the mean for each data set. For each rate point,
0 0
we consider a set of models with dimensionality progres- 64 256 1024 64 256 1024
Bits Bits
sivelyreducedbyafactorof2fromthestartingrepresenta- (c)FV,UKB (d)DCNN,UKB
tionforFVandDCNNrespectively. Thebestmodelischo- 60 60
senbasedongreedyoptimizationofAUContheSMVSdata
50 50
set, which works well as seen in detailed experimental re- 0 0
0 0
1 1
sultsofSection4.3. Thechosenarchitecturesaredescribed R=40 R=40
@ @
inTable2. Thedepthofthenetworkincreasesashashsize all 30 all 30
decreases.Eachtargetsettingrequiresseveralhourstotrain Rec Rec
20 20
onamodernCPU.
10 10
64 256 1024 64 256 1024
4.3.RetrievalExperiments Bits Bits
(e)FV,Oxford (f)DCNN,Oxford
We present retrieval results using FV and DCNN fea-
100 100
tures in Figure 7 and Figure 8. For instance retrieval, it is
important for the relevant image to be present in the first 0 80 0 80
0 0
step of the pipeline, matching global descriptors, so that a R=1 60 R=1 60
Geometric Consistency Check (GCC) [48] step can find it @ @
all 40 all 40
subsequently. Wepresentrecall@typicaloperatingpoints, ec ec
R R
R=100andR=1000forsmallandlargedatasetsrespec- 20 20
tively. For UKB small experiments, we plot 4× recall @
64 256 1024 64 256 1024
R = 4 to be consistent with the literature. We refer to Bits Bits
(g)FV,Graphics (h)DCNN,Graphics
hashesbeforeandafterfine-tuningasDeepHashandDeep-
Hash+ respectively in all figures. We refer to deep hashes Figure 7. Small-scale retrieval results. DeepHash outperforms other
schemesbyasignificantmargin.
basedonDCNNandFVfeaturesasDCNN-DeepHashand
FV-DeepHashrespectively.
pared to the second performing scheme. Up to 15% im-
Performance of DeepHash. For DCNN and FV fea- provement is seen at b = 256, which can be a practical
tures, the proposed DeepHash outperforms state-of-the-art ratepointformanyapplications,asthereisonlyamarginal
schemesbyasignificantmarginonalldatasets. Thestatis- drop in performance for DCNN features compared to un-
tics of FV and DCNN features are very different. FV are compressedfeatures. Similartrendsareobtainedforrecall
dense descriptors with zero blocks corresponding to cen- @ R = 10 and MAP, as seen by comparing Holidays re-
troids not visited, while deep DCNN features tend to be sults in Figure 5(b),(c) and Figure 7(a), with a higher gap
sparse. Ourmethodworkswellforbothtypesoffeatures. for larger R. Consistent trends are also obtained for the
For the retrieval experiments in Figure 7, there is up to large-scaleretrievalresultsinFigure8.
20% improvement in absolute recall at b = 64 bits com- Theperformanceorderingofotherschemesdependson
thebitrateandtypeoffeature,whileDeepHashisconsistent
DeepHash+ DeepHash PQ
acrossdatasets. ComparedtoITQschemewhichappliesa
DeepHash PQ ITQ LSH BPBC
singlePCAtransform,eachoutputbitforDeepHashisgen-
eratedbyaseriesofprojections. ThePQschemeperforms 80 80
poorly at the low rates in consideration, as large blocks of 00 00
1060 1060
theglobaldescriptorarequantizedwithasmallnumberof = =
R R
centroids, as previously observed in [23]. LSH performs all @ 40 all @ 40
poorlyatlowrates,butcatchesupgivenenoughbits. ec ec
R R
20 20
We observe a consistent improvement using Siamese
fine-tuning, which learns more discriminative projections. 64 256 1024 64 256 1024
Bits Bits
Thelearntprojectionsgeneralizewell,whichiskeyfordi-
FV,Holidays+1M DCNN,Holidays+1M
verseretrievaltasks,thusshowingtherobustnessofourpro-
100 100
posedmethod.
90 90
00 80 00 80
0 0
Comparing FV-DeepHash and DCNN-DeepHash. At R=1 70 R=1 70
a given rate point, DCNN-DeepHash outperforms FV- @ @
DeepHashhashesforalldatasetsexceptGraphicsasseen ecall 5600 ecall 5600
R R
bycomparingacrossrowsofFigure7.Atlowrates,DCNN- 40 40
DeepHash improves performance by more than 10% on 30 30
64 256 1024 64 256 1024
some of the small data sets, while the gap increases up to Bits Bits
20%forthelMexperiments. FV,UKB+1M DCNN,UKB+1M
The results are data-set dependent. DCNN features are Figure 8. Large-scale retrieval results (with 1M distractor images) for
differentcompressionschemes. DeepHashoutperformsotherschemesat
able to capture more complex low level features and have
mostratepointsanddatasets.
a lower starting dimensionality compared to FV. However,
DCNN features have limited rotation and scale invariance,
based on the level of data invariance seen at training time. bitsforDCNNfeatures.
FV,ontheotherhand,aggregatehand-craftedSIFTdescrip-
torsfromscaleandrotationinvariantinterestpoints,which
Future Work. Improving performance further at very
resultsinascaleandrotationinvariantrepresentation. The
low-rate points like 64 bits for even larger databases is an
Graphicsdatasethasmoreobjectswithlargevariationsin
interestingdirectionforfuturework. Studyingmathemati-
scale and rotation compared to the other data sets: this is
calmodelswhichrelatehashsizetoperformanceforvary-
oneofthereasons, whypeakperformanceofFVishigher
ingdatabasesizeisalsoanexcitingdirectiontopursue. Fi-
thanDCNNforGraphics.
nally,inthiswork,welearntcompacthashesstartingfroma
pre-trainedDCNNmodel. Learningthehashdirectlyfrom
Comparison to Uncompressed Descriptors. We com- pixelsinaDCNNframeworkmightleadtofurtherimprove-
pare the performance of DeepHash to the uncompressed ments.
descriptorinFigure7. Weobtainremarkablecompression
performance - at 256 bits for DCNN hashes, we only ob- 5.Conclusion
serve a marginal drop (a few%) compared to the uncom-
pressedrepresentationforretrievalonawiderangeofdata A perfect image hashing scheme would convert a high-
sets:a512×compressioncomparedtoafloatingpointrep- dimensionaldescriptorintoalow-dimensionalbitrepresen-
resentation,and16×comparedtoabinaryrepresentation. tationwithoutlosingretrievalperformance. Webelievethat
ForFV,wecanmatchtheperformanceoftheuncompressed DeepHash,whichfocusesonachievingcomplexhashfunc-
descriptorwith1024bitsforHolidaysandUKB,withadrop tions with deep learning, is a significant step in this direc-
for Graphics and Oxford. At some rate points, DeepHash tion. Our method is focused on a deep network which ef-
performsbetterthantheuncompresseddescriptor,whichis ficiently utilizes the binary subspace through hashing reg-
duetoquantizationofnoiseintheuncompresseddescriptor. ularization and further fine-tuning using a Siamese train-
Theinstanceretrievalhashingproblembecomesincreas- ing algorithm. Through a rigorous evaluation process, we
inglydifficultaswemovetowardsa64-bithash. At64bits, showthatourmodelperformswellacrossvariousdatasets,
thereisa5-10%dropinperformancecomparedto256bits regardless of the type of image descriptors used, sparse
for DCNN features, while a drop is also observed for FV. or dense. The marked improvement over existing hashing
For the million scale experiments, however, we observe a schemes attests to the importance of regularization, depth
10-20% drop in performance at 64 bits compared to 1024 andfine-tuningforhashingimagedescriptors.
References
rama,andT.Darrell,“Caffe:Convolutionalarchitectureforfastfeatureembed-
ding,”arXivpreprintarXiv:1408.5093,2014.2,5
[1] F.Perronnin,Y.Liu,J.Sanchez,andH.Poirier,“Large-scaleImageRetrieval [26] G.E.Hinton,S.Osindero,andY.-W.Teh,“Afastlearningalgorithmfordeep
withCompressedFisherVectors,”inProceedingsofIEEEConferenceonCom- beliefnetworks,”NeuralComputation,vol.18,no.7,pp.1527–1554,2006.2,
puterVisionandPatternRecognition(CVPR),SanFrancisco,CA,USA,June 3
2010,pp.3384–3391.1,3,5 [27] J.Bromley,I.Guyon,Y.Lecun,E.Sckinger,andR.Shah,“SignatureVerifi-
[2] A.Krizhevsky,I.Sutskever,andG.E.Hinton,“Imagenetclassificationwith cationusinga”Siamese”TimeDelayNeuralNetwork,”inAdvancesinNeural
deepconvolutionalneuralnetworks,”inAdvancesinNeuralInformationPro- InformationProcessingSystems(NIPS),1994.2,4
cessingSystems(NIPS),2012.1,5 [28] P.Agrawal,R.Girshick,andJ.Malik,“Analyzingtheperformanceofmulti-
[3] A.Babenko,A.Slesarev,A.Chigorin,andV.Lempitsky,“NeuralCodesfor layerneuralnetworksforobjectrecognition,”inProceedingsoftheEuropean
ImageRetrieval,”inProceedingsofEuropeanConferenceonComputerVision ConferenceonComputerVision(ECCV),2014.3
(ECCV),2014.1,4,5 [29] G.Hinton,“APracticalGuidetoTrainingRestrictedBoltzmannMachines,”in
[4] K. Simonyan and A. Zisserman, “Very deep convolutional networks for NeuralNetworks:TricksoftheTrade,ser.LNCS,vol.7700. SpringerBerlin
large-scale image recognition,” CoRR, vol. abs/1409.1556, 2014. [Online]. Heidelberg,2012,pp.599–619.3
Available:http://arxiv.org/abs/1409.15561 [30] G.E.Hinton,“Trainingproductsofexpertsbyminimizingcontrastivediver-
[5] “DeepFace: ClosingtheGaptoHuman-LevelPerformanceinFaceVerifica- gence,”NeuralComputation,vol.14,no.8,p.1771–1800,2002.3
tion,”inConferenceonComputerVisionandPatternRecognition(CVPR).1 [31] H.Lee,C.Ekanadham,andA.Ng,“Sparsedeepbeliefnetmodelforvisual
[6] Y.Sun,X.Wang,andX.Tang,“Deeplearningfacerepresentationfrompre- areaV2,”inAdvancesinNeuralInformationProcessingSystems(NIPS),2008,
dicting10,000classes,”June2014.1 pp.873–880.3
[7] A.ToshevandC.Szegedy,“Deeppose:Humanposeestimationviadeepneural [32] V.NairandG.Hinton,“3DObjectRecognitionwithDeepBeliefNets,”in
networks,”June2014.1 AdvancesinNeuralInformationProcessingSystems(NIPS),2009,pp.1339–
[8] R.Salakhutdinov,A.Mnih,andG.Hinton,“RestrictedBoltzmannMachines 1347.3
forCollaborativeFiltering,”inProceedingsofthe24thInternationalConfer- [33] H.Goh,N.Thome,M.Cord,andJ.-H.Lim,“Unsupervisedandsupervised
enceonMachineLearning,ser.ICML’07. NewYork,NY,USA:ACM, visualcodeswithrestrictedBoltzmannmachines,”inEuropeanConferenceon
2007,pp.791–798.1 ComputerVision(ECCV),2012.3
[9] Y.GongandS.Lazebnik,“IterativeQuantization: AProcrusteanApproach [34] R.Hadsell,S.Chopra,andY.LeCun,“DimensionalityReductionbyLearning
toLearningBinaryCodes,”inProceedingsofthe2011IEEEConferenceon anInvariantMapping,”inProceedingsofIEEEConferenceonComputerVision
ComputerVisionandPatternRecognition,ser.CVPR’11,Washington,DC, andPatternRecognition,vol.2,2006,pp.1735–1742.4
USA,2011,pp.817–824.1,5 [35] S.Chopra,R.Hadsell,andY.Lecun,“Learningasimilaritymetricdiscrimina-
[10] Y.Weiss,A.Torralba,andR.Fergus,“SpectralHashing,”inProceedingsof tively,withapplicationtofaceverification,”inProceedingsofIEEEConference
NeuralInformationProcessingSystems(NIPS),Vancouver,BC,Canada,De- onComputerVisionandPatternRecognition(CVPR). IEEEPress,2005,pp.
cember2008,pp.1753–1760.1,5 539–546.4
[11] M.NorouziandD.Fleet,“MinimalLossHashingforCompactBinaryCodes,” [36] D.Lowe,“DistinctiveImageFeaturesfromScale-invariantKeypoints,”Inter-
inProceedingsofICML,2011,pp.353–360.1 nationalJournalofComputerVision,vol.60,no.2,pp.91–110,November
[12] B.KulisandK.Grauman,“KernelizedLocality-SensitiveHashingforScalable 2004.4
ImageSearch,”inProceedingsofIEEEInternationalConferenceonComputer [37] H.Je´gou,M.Douze,C.Schmid,andP.Perez,“AggregatingLocalDescriptors
Vision(ICCV,2009.1 intoaCompactImageRepresentation,”inProceedingsofIEEEConferenceon
[13] J.Wang,W.Liu,A.X.Sun,andY.Jiang,“LearningHashCodeswithListwise ComputerVisionandPatternRecognition(CVPR),SanFrancisco,CA,USA,
Supervision,”inProceedingsofIEEEInternationalConferenceonComputer June2010,pp.3304–3311.5
Vision,ICCV2013,Sydney,Australia,December1-8,2013,2013,pp.3032– [38] J.Lin,L.-Y.Duan,T.Huang,andW.Gao,“RobustFisherCodesforLarge
3039.1 ScaleImageRetrieval,”inProceedingsofInternationalConferenceonAcous-
[14] X.Li,G.Lin,C.Shen,A.vandenHengel,andA.R.Dick,“LearningHash ticsandSignalProcessing(ICASSP),2013.5
FunctionsUsingColumnGeneration.”inProceedingsofICML,vol.28,2013, [39] J.Deng,W.Dong,R.Socher,L.-J.Li,K.Li,andL.Fei-Fei,“Imagenet: A
pp.142–150.1 large-scalehierarchicalimagedatabase,”inIEEEConferenceonComputerVi-
[15] J.Wang,S.Kumar,andS.-F.Chang,“Semi-SupervisedHashingforScalable sionandPatternRecognition(CVPR),2009.5
ImageRetrieval,” inProceedingsofIEEEComputerSocietyConferenceon [40] J.Philbin,O.Chum,M.Isard,J.Sivic,andA.Zisserman,“ObjectRetrieval
ComputerVisionandPatternRecognition(CVPR),SanFrancisco,USA,June withLargeVocabulariesandFastSpatialMatching,”inProceedingsofCVPR,
2010.1 Minneapolis,Minnesota,June2007,pp.1–8.5
[16] Y.Lee,“SphericalHashing,”inProceedingsofthe2012IEEEConferenceon [41] H.Je´gou,M.Douze,andC.Schmid,“HammingEmbeddingandWeakGeo-
ComputerVisionandPatternRecognition(CVPR). Washington,DC,USA: metricConsistencyforLargeScaleImageSearch,”inProceedingsofEuropean
IEEEComputerSociety,2012,pp.2957–2964.1 ConferenceonComputerVision(ECCV),Berlin,Heidelberg,October2008,pp.
[17] K.GraumanandR.Fergus,“LearningBinaryHashCodesforLarge-ScaleIm- 304–317.5
ageSearch,”inMachineLearningforComputerVision,2013,vol.411,pp. [42] V. Chandrasekhar, D.M.Chen, S.S.Tsai, N.M.Cheung, H.Chen, G.Takacs,
49–87.1 Y.Reznik,R.Vedantham,R.Grzeszczuk,J.Back,andB.Girod,“StanfordMo-
[18] W.Liu,J.Wang,R.Ji,Y.-G.Jiang,andS.-F.Chang,“SupervisedHashingwith bileVisualSearchDataSet,”inProceedingsofACMMultimediaSystemsCon-
Kernels,”inComputerVisionandPatternRecognition(CVPR),2012.1 ference(MMSys),SanJose,California,February2011.5,6
[19] A.Torralba,R.Fergus,andY.Weiss,“SmallCodesandLargeImageDatabases [43] MPEGCDVS(CompactDescriptorsforVisualSearch)Benchmark.Stanford
forRecognition,”inProceedingsofIEEEConferenceonComputerVisionand DigitalRepository,http://purl.stanford.edu/qy869qz5226.5
PatternRecognition(CVPR),Anchorage,Alaska,June2008,pp.1–8.1 [44] D.Niste´randH.Stewe´nius,“ScalableRecognitionwithaVocabularyTree,”in
[20] V.Chandrasekhar, M.Makar, G.Takacs, D.Chen, S.S.Tsai, N.M.Che- ProceedingsofIEEEConferenceonComputerVisionandPatternRecognition
ung, R. Grzeszczuk, Y. Reznik, and B. Girod, “Survey of SIFT Compres- (CVPR),NewYork,USA,June2006,pp.2161–2168.5
sionSchemes,”inProceedingsofInternationalMobileMultimediaWorkshop [45] B.T.MarkJ.HuiskesandM.S.Lew,“NewTrendsandIdeasinVisualConcept
(IMMW),IEEEInternationalConferenceonPatternRecognition(ICPR),Is- Detection:TheMIRFlickrRetrievalEvaluationInitiative,”inProceedingsof
tanbul,Turkey,August2010.1 the2010ACMInternationalConferenceonMultimediaInformationRetrieval,
[21] V.Chandrasekhar, G.Takacs, D.M.Chen, S.S.Tsai, R.Grzeszczuk, and NewYork,NY,USA,2010,pp.527–536.5
B.Girod,“CHoG:CompressedHistogramofGradients-ALowBitRateFea- [46] D.M.Chen,“Memory-EfficientImageDatabasesforMobileVisualSearch,”
tureDescriptor,”inProceedingsofIEEEConferenceonComputerVisionand Ph.D.thesis,DepartmentofElectricalEngineering,StanfordUniversity,April
PatternRecognition(CVPR),Miami,Florida,June2009,pp.2504–2511.1 2014.6
[22] C.Yeo,P.Ahammad,andK.Ramchandran,“Rate-efficientVisualCorrespon- [47] D. M. Chen, S. S. Tsai, V. Chandrasekhar, G. Takacs, R. Vedantham,
dencesusingRandomProjections,”inProceedingsofIEEEInternationalCon- R.Grzeszczuk,andB.Girod,“ResidualEnhancedVisualVectorasaCompact
ferenceonImageProcessing(ICIP),SanDiego,California,October2008,pp. SignatureforMobileVisualSearch,”inSignalProcessing,Elsevier,InPress,
217–220.1,5 June2012.6
[23] Y.Gong,S.Kumar,H.Rowley,andS.Lazebnik,“LearningBinaryCodesfor [48] M.A.FischlerandR.C.Bolles,“RandomSampleConsensus:AParadigmfor
High-DimensionalDataUsingBilinearProjections.”inProceedingsofCVPR, ModelFittingwithApplicationstoImageAnalysisandAutomatedCartogra-
2013,pp.484–491.1,5,8 phy,”CommunicationsofACM,vol.24,no.6,pp.381–395,June1981.7
[24] H.Je´gou,F.Perronnin,M.Douze,J.Sa´nchez,P.Pe´rez,andC.Schmid,“Ag-
gregatinglocalimagedescriptorsintocompactcodes,”IEEETransactionson
PatternAnalysisandMachineIntelligence,vol.34,no.9,pp.1704–1716,Sep.
2012.2,5
[25] Y.Jia,E.Shelhamer,J.Donahue,S.Karayev,J.Long,R.Girshick,S.Guadar-