Unsupervised Joint Mining of Deep Features and Image Labels for Large-scale Radiology Image Categorization and Scene Recognition

0.85 MB·

by  Xiaosong Wang

Unsupervised Joint Mining of Deep Features and Image Labels
for Large-scale Radiology Image Categorization and Scene Recognition
XiaosongWang,LeLu,Hoo-changShin,LaurenKim,MohammadhadiBagheri,
IsabellaNogues,JianhuaYao,RonaldM.Summers
DepartmentofRadiologyandImagingSciences,NationalInstitutesofHealthClinicalCenter,
10CenterDrive,Bethesda,MD20892
7
{xiaosong.wang,le.lu,lauren.kim2,mohammad.bagheri,isabella.nogues}@nih.gov,
1
0 [email protected], [email protected]
2

n
a
J Abstract segmentation and scene recognition using public image
datasets (e.g., Pascal VOC [16], ImageNet ILSVRC [12,
3
2 Therecentrapidandtremendoussuccessofdeepconvo- 48], MS COCO [38]), with significantly superior perfor-
lutionalneuralnetworks(CNN)onmanychallengingcom- mancethanpreviousarts, especiallythosenon-deepmeth-

] puter vision tasks largely derives from the accessibility of ods built upon hand-crafted image features. However, the
V
the well-annotated ImageNet and PASCAL VOC datasets. good efficacy of CNNs often comes at the cost of large
C
Nevertheless,unsupervisedimagecategorization(i.e.,with- amounts of annotated training data. ImageNet pre-trained
.
s outtheground-truthlabeling)ismuchlessinvestigated,yet deepCNNmodels[26,30,37]serveanindispensablerole
c critically important and difficult when annotations are ex- to be bootstrapped or fine-tuned [24] for all externally-
[
tremelyhardtoobtainintheconventionalwayof“Google sourceddataexploitationtasks[36,5].

1 Search” and crowd sourcing. We address this problem by In the medical imaging domain, nevertheless, no large-
v
presentingaloopeddeeppseudo-taskoptimization(LDPO) scalelabeledimagedatasetcomparabletoImageNetexists
9
frameworkforjointminingofdeepCNNfeaturesandimage (except the one in [49] which is not directly comparable).
9
5 labels. Our method is conceptually simple and rests upon Modern hospitals store vast amounts of radiological im-
6 thehypothesized“convergence”ofbetterlabelsleadingto ages/reportsintheirPictureArchivingandCommunication
0 bettertrainedCNNmodelswhichinturnfeedmorediscrim- Systems(PACS).Themainchallengenowliesinhowtoob-
.
1 inative image representations to facilitate more meaning- tain or compute the ImageNet-like semantic labels given a
0 ful clusters/labels. Our proposed method is validated in largecollectionofmedicalimages. Conventionalmeansof
7 tackling two important applications: 1) Large-scale med- collectingimagelabels(e.g.,Googleimagesearchusingthe
1
ical image annotation has always been a prohibitively ex- termsfromWordNetontologyhierarchy[40],SUN/PLACE
:
v pensive and easily-biased task even for well-trained ra- databases[61,66]orNEILknowledgebase[6];followedby
Xi diologists. Significantly better image categorization re- crowd-sourcing[12])arenotapplicable,dueto1)unavail-
sults are achieved via our proposed approach compared ability of a high quality or large capacity medical image
r
a to the previous state-of-the-art method. 2) Unsupervised search engine, and 2) the formidable difficulties of medi-
scene recognition on representative and publicly available calannotationtasksforannotatorswithnoclinicaltraining.
datasets with our proposed technique is examined. The Additionally,evenforwell-trainedradiologists,thistypeof
LDPO achieves excellent quantitative scene classification “assigning labels to images” task is not aligned with their
results. OntheMITindoorscenedataset,itattainsaclus- diagnosticroutineworksothatdrasticinter-observervaria-
teringaccuracyof75.3%,comparedtothestate-of-the-art tionsorinconsistencyareexpected. Theprotocolsofdefin-
supervisedclassificationaccuracyof81.0%(whenbothare ing image labels based on visible anatomic structures (of-
basedontheVGG-VDmodel). ten multiple), or pathological findings (possibly multiple)
orbothcuesmayhaveintrinsicallyhighambiguities.
Recentsemi-supervisedimagefeaturelearningandself-
1 Introduction
taught image recognition techniques [53, 46, 35, 27, 11]
Deep Convolutional Neural Networks (CNNs) have have advanced both supervised image classification and
demonstrated remarkable success on many challenging unsupervised clustering processes, demonstrating some
computer vision tasks of object recognition, detection, promising results. Common image patches [53, 34], ob-
ject parts [27], prototypes [11, 10] or spatial context [14] teringon theImageNet-pretrained deepimagefeatures via
canbefirstminedamongstimagesofthesametheme(e.g., AlexNet[30])andisstronglyclosetothefully-supervised
thesameindoorsceneclass[44])andthenconcatenatedto deepclassificationresultof81.0%[8].
serve as the discriminative image representations for clas-
2 RelatedWork
sification. All of these methods, however, require image
labelsinordertolearnclass-specificinformativeimagerep- Image Categorization or Auto-annotation: Image
resentations. auto-annotation task is addressed via multiple instance
learning [59] but the target domain is restricted to a small
Inthispaper, wepresenttheLoopedDeepPseudo-task
subset (only 25 out of 1000 classes) of ImageNet [12]
Optimization framework (LDPO) for joint mining of im-
and SUN [61]. [58] introduces a hierarchical set of un-
agefeaturesandlabels,withnopriorknowledgeoftheim-
labeled data clusters (spanning a spectrum of visual con-
age categories. The “true” image category labels are as-
ceptgranularities)thatcanbeefficientlylabeledtoproduce
sumed to be latent and not directly observable. The main
high performance classifiers (thus less label noises than
idea is to learn and train CNN models using pseudo-task
theinstance-levellabeling). [49]firstextractthesentences
labels (since human-annotated labels are unavailable) and
that depict disease referencing key images (analogous to
iterate this process with the expectation that pseudo-task
“keyframesinvideos”)viaNLPfromatotalcollectionof
labels will gradually resemble the real image categories.
∼ 780K patients’ radiology text reports, and 215,786 key
This looped optimization algorithm flow starts with deep
images of 61,845 unique patients are found. Then, image
CNNfeatureextractionandimageencodingusingdomain-
categorization labels are computed using unsupervised hi-
specifically(e.g.,CNNtrainedonradiologyimagesandtext
erarchicalBayesiandocumentclustering,i.e.,latentDirich-
report-derived labels [49]) or generically initialized CNN
letallocation(LDA)topicmodeling[1],toform80classes.
models. Afterwards, theCNN-encodedimagefeaturevec-
Thetext-computedcategoryinformationofferssomecoarse
torsareclusteredtocomputeandrefineimagelabels, then
levelofradiologysemanticsbutappearstobelimitedintwo
we feed the newly clustered labels to fine-tune the current
aspects:1)Theclassesarehighlyunbalanced,inwhichone
CNN models. Next, the obtained more task-specific and
dominating category contains 113,037 images while other
representative deep CNN will serve as the deep image en-
classescontainafewdozens.2)Someclassescanbehighly
coder in the successive iteration. This looped process will
incoherentamongtheirwithin-the-classimageinstances.
halt until a stopping criterion is met. For medical image
Unsupervised and Semi-supervised Learning: Dai et
annotation, LDPO generated image clusters can be further
al. [11, 10] study the semi-supervised image classifica-
interpreted by a natural language processing (NLP) based
tion and clustering on problems of texture [31], small- to
textminingsystemand/oraclinician[65].
middle-scaleobjectcategories(e.g., Caltech-101[18])and
Our contributions are three-fold. 1), The unsupervised scenerecognition[45]. Ensembleprojections(EP)asarich
joint mining of deep image features and labels via LDPO set of visual prototypes are derived as the new image rep-
isconceptuallysimpleandbasedonthehypothesized“con- resentationforclusteringandrecognition. Graphbasedap-
vergence”ofbetterlabelsleadtobettertrainedCNNmod- proaches [39, 29] are used to link the unlabeled image in-
elswhichinturn,offermoreeffectivedeepimagefeatures stances to labeled ones (which are served as anchors) and
tofacilitatemoremeaningfulclustering/labels. Thislooped propagatelabelsbythegraphtopologyandconnectiveness
property is unique to deep CNN classification-clustering weights. Inanunsupervisedmanner,Coatesetal. [9]em-
models since other types of classifiers do not learn better ployk-meanstomineimagepatchfiltersandutilizethere-
imagefeaturessimultaneously. 2),Weapplyourmethodto sultedfiltersforfeaturecomputation. Surrogateclassesare
thelarge-scalemedicalimageauto-annotation. Tothebest obtainedbyaugmentingeachimagepatchwithitsgeomet-
of our knowledge, this is the first work exploiting to inte- rically transformed versions and a CNN is trained on top
grate unsupervised deep feature clustering and supervised ofthesesurrogateclassestogeneratefeatures,asstudiedin
deeplabelclassificationforself-annotatingalarge-scalera- [15]. [64]integratesthehierarchicalagglomerativecluster-
diology image database where the conventional means of ingprocessintoarecurrentneuralnetworkbymergingthe
imageannotationmaynotbequitefeasible. Ourbestcon- clusters(asgroupsofimages)iterativelytowardtheprede-
verged model obtains the Top-1 classification accuracy of finedclusternumberandsimultaneouslyupdatingtheCNN
0.8109andTop-5accuracy0.9412with270formedimage activationsforimagerepresentation.
categories. 3), LDPO framework is also validated through Our looped optimization method shares a similar con-
thescenerecognitiontaskwheretheground-truthlabelsare cept with [64] in the joint learning of image clusters and
available (only for the validation purpose). We report the image representations. However, it differs significantly in
67-classclusteringaccuracyof75.3%ontheMIT-67indoor thefollowingrespects:1)anunlabeledimagecollectioncan
scene dataset [44] that doubles the performance from the beinitializedwitheitherrandomly-assignedlabelsorlabels
baselinemethods(ofusingk-meansoragglomerativeclus- obtained by a pseudo-task (e.g., text topic modeling gen-
eratedlabels[49]); 2)Ourframeworkhastheflexibilityof Withoutlossofgenerality,ourmethodisfirstemployed
working with any clustering function. Particularly, it em- in the scenario of medical image categorization. We high-
ploys Regularized Information Maximization (RIM [21]) light the problem-specific settings for scene recognition
clustering to perform clustering the image (like k-means) taskwhentheyaredifferent.AsillustratedinFig.1,theiter-
withmodelselectiononfindingtheoptimalnumberofclus- ationbeginsbyextractingthedeepCNNimagefeatureus-
terswhereasonlyagglomerativeclusteringloss[22]canbe ingeitheradomain-specific[49]orgenericImageNet[30]
integrated into the neural network model in [64]. 3) The CNN model (Sec. 3.1). Next, the clustering on deep fea-
empiricalconvergenceprocessofourLDPOmethodisob- turewithk-meansork-meansfollowedbyRIMisexploited
servableandquantifiable,asdescribedinSec. 3.2. (Sec. 3.2). Byevaluatingthepurityandmutualinformation
Mid-level Image Representation: Since the seminal betweenformedclustersinconsecutiverounds,thesystem
work on discriminative image patch discovery [53], mid- eitherterminatesthecurrentiteration(andyieldsconverged
level visual elements based image representation has been clustering outputs); or uses the newly refined image clus-
exploredintensivelyandfoundbeingeffectiveonboosting ter labels to train or fine-tune the CNN model in the next
the performance of many visual computing tasks, particu- iteration. Formedicalimagecategorization(dashedboxin
larlyscenerecognition[53,13,27,54,33,2,11,34,60]. A Fig.1), LDPO-generatedimageclusterscanbefurtherfed
varietyofmid-levelvisualelementscanbeharvested,e.g., into text processing. The system can extract semantically
image patches [53, 13, 34, 60], parts/segments [27, 54, 2], meaningful text words for each formed cluster. Further-
prototypes [11], attributes [51, 7] through different learn- more, the hierarchical category relationship is built using
ingandminingtechniques, e.g., iterativeoptimization[53, the class confusion measures of the final converged CNN
27], classification and co-segmentation [54], Multiple In- classificationmodels(Sec. 3.3).
stance Learning (MIL) [33], random forests [2], ensemble
3.1 DeepCNNImageRepresentation&Encoding
projection[11] and association rule mining [34]. Nonethe-
less,thesemethodsrequirethatimagesaregroupedbefore A variety of CNN models can be used in our method.
theirrepresentationsareminedinsideeachgroup,whichis We analyze the CNN activations from layers of different
aformofweaklysupervisedlearning(WSL). depthsinAlexNet[30],VGGnet[52]andGoogLeNet[55].
Our work is partly related to the iterative optimization Pre-trained models on the ImageNet ILSVRC data are ob-
in [53, 27] that seeks to identify discriminative local vi- tained from Caffe Model Zoo [26]. We also employ the
sual patterns as parts and reject others, while our goal is CaffeCNNimplementation[26]toperformfine-tuningon
to jointly mine better deep image representations and the CNNsusingthekeyimagedatabase[49,50]. AlexNetisa
labelsforallimages,towardsiterativeauto-annotation. We popular7-layerCNNarchitectureandtheextractedfeatures
can integrate the association rule mining technique [34] to from its convolutional or fully-connected layers have been
extractthefrequentimageparts(thatarefurtherusedtoen- broadlyinvestigated[20,47,28,41].Inourexperimentswe
codeimagerepresentation)intoourLDPOpipeline,andre- harness image feature activations of the 5th convolutional
port excellent unsupervised scene recognition accuracy of layerConv5and7thfully-connected(FC)layerFC7,sug-
75.3%onMITindoorscenedataset[60,8,44]. gested by [8, 3]. GoogLeNet [55] is a much deeper CNN
architecture that comprises 9 inception modules and each
3 JointMiningofDeepFeaturesandLabels module is a set of convolutional layers with multiple win-
Supervised or semi-supervised learning paradigms (as dow sizes of 1×1,3×3,5×5. We utilize the deep fea-
described in Sec. 2) usually require (at least partial) im- turesfromthelastinceptionlayerInception5bandthefinal
age labels as a prerequisite. These lines of work, at the pooling layer Pool5. For the scene recognition task, very
eraof“deeplearning”,wouldnecessitateahugeamountof deep VGGNet (VGG-VD) [52] is also employed, in addi-
data annotation efforts. For medical imaging applications, tion to AlexNet. The extracted features from VGG-VD’s
well-trainedclinicalprofessionalsorphysiciansareinneed last fully-connected layers are used for the patch-mining
fordatalabeling,insteadofAmazonMechnicalTurkersin basedimageencoding. Table1illustratesthedetailedCNN
computer vision. Employing and converting the medical layersandtheiractivationdimensions.
records stored in the PACS into image labels or tags is a Deepimagefeaturesextractedfromthelastconvolution
highlynon-trivialandunsolvedNLPproblemwithhighla- layer preserve their overall spatial locations or image lay-
beling uncertainties, observed by [50]. Our approach ex- outs while the fully-connected CNN layer will lose spa-
ploitsunsupervisedcategorydiscoveryusingempiricalim- tialinformation. Weadopttoencodethelastconvolutional
agecuesforgroupingorclustering,throughaniterativeop- layeroutputs(asfeatureactivationmaps)inaformofdense
timization process of 1) deep image feature extraction and poolingviaFisherVector(FV)[43]andVectorLocallyAg-
clustering; and2)deepCNNmodelfine-tuning(i.e.,using gregated Descriptor (VLAD) [25], before feeding them to
newlabelsfromclustering), toupdatedeepfeatureextrac- thefully-connectedlayer. ThedimensionsofFVorVLAD
tioninthenextround. encodeddeepfeaturesaremuchhigherthanthoseoftheFC
Figure1.Theoverviewofloopeddeeppseudo-taskoptimization(LDPO)framework.
Table 1. Configurations of CNN output layers and encoding
train the Gaussian mixture Model(GMM). The dimension
schemes(Outputdimensionis4096,exceptPool5as1024).
ofresultedFVfeaturesissignificantlyhigherthanFC7’s,
CNNmodel Layer Activations Encoding i.e. 32768(2 × 64 × 256) vs 4096. After PCA, the FV
MedicalImageCategorization representation per image is reduced to a 4096-component
AlexNet FC7 4096 − vector. A list of deep image features, the encoding meth-
AlexNet Conv5 (13,13,256) FV+PCA ods and output dimensions are provided in Table 1. To be
AlexNet Conv5 (13,13,256) VLAD+PCA
consistentwiththesettingofFVencoding,weinitializethe
GoogLeNet Pool5 1024 −
VLADencodingofconvolutionalfeaturesbyk-meansclus-
GoogLeNet Inception5b (7,7,1024) VLAD+PCA
tering (k = 64). Thus the resulted dimensions of VLAD
SceneRecognition
descriptorsare16384(64×256)ofConv5inAlexNetand
AlexNet FC7 4096 PM+PCA
65536(64×1024)ofInception5binGoogLeNet,bothre-
VGG-VD FC7 4096 PM+PCA
ducedto4096viaPCA.
layers. Since there is redundant information from the en-
Patch mining based encoding: We adopt a procedure
codeddeepfeatures,PrincipalComponentAnalysis(PCA)
similartothatin[34]toextractmid-levelelementsforim-
is performed to reduce the feature dimensionality to 4096
agerepresentation. Ourmethod,however,unlike[34],does
(same to the FC dimension [30, 55]) that makes different
not require prior knowledge of the image categories. For
encodingschemesmorecomparable.
eachimageI inthedataset,wefirstextractasetofpatches
Minedmid-levelvisualelementsbasedimageencoding
from multiple spatial scales and compute the CNN acti-
has proven to be a more discriminative representation in
vation for each patch. Among all activations (e.g., 4096-
natural scene recognition [53, 13, 27, 34, 60]. Visual ele-
D vectors on FC7), only indexes of top k maximal acti-
mentsareexpectedtobecommonamongsttheimageswith
vations are recorded and used to form a transaction (e.g.,
samelabelbutseldomoccurinothercategories. Theasso-
{1024,3,24,4096},k =4)[34]. Eachimagecontainsaset
ciation rule mining technique is integrated into our looped
oftransactions,whichappearsontheimage. Insteadofre-
optimizationmethodflow(similarto[34])toautomatically
trievingpatchesinaclass-specificfashion([34]withknown
discover mid-level image patches for encoding. We con-
labels),weemployedassociationrulemininginsidethesets
jecture that discriminative patches can be discovered and
ofeitherrandomlygroupedimages(forthefirstiteration)or
graduallyimprovedthroughtheLDPOiterationsevenifthe
imageclusterscomputedby“clusteringonCNNfeatures”.
initializationimagelabelsarenotaccurate.
Thetop50minedpatterns(whichcoverthemaximumnum-
CNN activation based encoding: Given a pre-trained bersofpatches)perimageclusterarefurthermergedacross
(generic or domain-specific) CNN model (e.g., Alexnet or the entire dataset to form a consolidated vocabulary of vi-
GoogLeNet), an input image I is resized to fit the model sualelements.Detailedglobalmergingproceduresareelab-
definitionandfeedintotheCNNmodeltoextractfeatures oratedinAlgorithm1. Comparedto[34], wefindthatour
{fL} (1 (cid:54) i,j (cid:54) sL) from the L-th convolutional layer globalmergingstrategyeffectivelyreducesredundancyand
i,j
withdimensionssL×sL×dL,e.g.,13×13×256ofConv5 offers more discriminative image features for both cluster-
inAlexNetand7×7×1024ofPool5inGoogLeNet. For ingandclassificationtasks(seedetailsinSec. 4). Finally,
the Fisher Vector implementation, we use the settings as the“bag-of-elements”imagerepresentationsarecomputed
suggested in [8]: 64 Gaussian components are adopted to asthesameprocessin[34].
Algorithm1:Globalmergingofpatchclusters andtheregularizationtermaresubsequentlydefinedas
Input:AsetofminedpatternsfromeachofKimage p(c=k|f,W) ∝ exp(wkTf +bk) (2)
(cid:88)
clusters,i.e.V ={vi},|V|=50∗K,setofpatches R(W;λ) = λ wkTwk, (3)
p ∈PforeachpatternandLDAdetectors[23]
i k
d ∈Dtrainedonassociatedpatchsetp .
i i where W = {w ,...,w ,b ,...,b } is the set of param-
Output:AsetofmergedpatternsV(cid:48) ={vn}andassociated eters and w ∈ 1RD,b K∈ R1. MaKximizing the above ob-
patchLDAdetectorsD(cid:48) ={d } k k
n jective function is equivalent to solving a logistic regres-
foreachsetof{v ,p ,d }do
i i i
ComputeS =1/|p |(cid:80) dTx(X isasetof sionproblem. RistheL2 regulatorofweight{wk}andits
ij j x∈Xpj i pj poweriscontrolledbyλ. Largeλvaluesenforcereduction
CNNactivationsofpatchesinp ).
j
ofthetotalnumberofcategoriesorclustersbyimposingno
end
penaltyonunpopulatedcategories [21]. Thischaracteristic
ifbothS andS >apredefinedthresholdthen
ij ji
Merge{v ,p ,d }and{v ,p ,d }andtrainnewLDA enablesRIMtoattaintheoptimalnumberofcategoriesco-
i i i j j j
detectord basedonp ∪p . herent with the data distribution. λ is fixed to 1 in all our
n i j
end experiments.
returnV(cid:48),D(cid:48);
Before using the newly-generated clustering labels of
image to fine-tune the deep CNN model in the next it-
3.2 ImageClusteringandLDPOConvergence eration, the LDPO framework is designed to evaluate the
current clustering quality to decide if a convergence has
Image clustering plays an indispensable role in our
been reached. Two convergence measurements have been
LDPO framework. We hypothesize that the newly gen-
adopted from [56], i.e., Purity and Normalized Mutual In-
erated clusters driven by looped deep pseudo-task opti-
formation (NMI). We take these two criteria as the forms
mization have incrementally improved quality than previ-
ofempiricalsimilarityexaminationbetweentwoclustering
ous ones, in the following measurements: 1) Images in
outcomes from adjacent LDPO iterations. When the simi-
each cluster are visually more coherent and discriminative
laritymeasureisaboveacertainthreshold,weconsiderthat
frominstancesinotherclusters;2)Theimagecountsamong
theoptimalclustering-baseddatacategorizationisreached.
all clusters are approximately balanced; 3) The number of
It has been empirically found that the final category num-
clustersisself-adaptivebymodelselection. Twoclustering
bers (from the RIM process) in later LDPO iterations sta-
methodsareexploited,i.e.,standalonek-means;oranover-
bilizearoundaconstant. Theconvergenceonclassification
segmentedk-means(whereK ismuchlargerthanthefirst
plots is also observable through the increased top-1, top-5
setting, e.g., 1000)followedbyRIM[21]formodelselec-
classificationaccuracyvaluesinthefirstfewLDPOrounds
tionandparameteroptimization.
andeventuallystabilizearoundaconstant.
k-means is an efficient clustering algorithm provided
NLPTextProcessing: Thecategorydiscoveryofmed-
that the number of clusters is known. For scene recog-
icalimagesentailsclinically-semanticlabelingofthemed-
nition application, we use k-means clustering to initialize
ical images. From the optimized clusters (obtained after
thepatchminingprocedureandgeneratenewimagelabels
Sec. 3.2),wecollecttheassociatedtextreportsandassem-
for the next iteration, while the underlying cluster number
ble each image cluster’s text reports together into a group.
is unknown for the medical image categorization problem.
Next, NLP is performed on each unit of radiology reports
Therefore we first utilize k-means clustering to initialize
tofindhighlyrecurringwordsthatmayserveasinformative
the RIM clustering with a considerably large k; then RIM
keywordsperclusterbycountingandrankingthefrequency
willperformmodelselectiontooptimizeonk. RIMworks
of each word. Common words to all clusters are first re-
without the assumption that the cluster number is known
movedfromthelist. Theresultingkeywordsandrandomly
asaprioriandisdesignedfordiscriminativeclustering,by
sampledexemplaryimagesforeachclusterorcategoryare
maximizing the mutual information between data distribu-
compiledforreviewingbyboard-certifiedradiologists.This
tionandtheresultedcategoriesviaaregularizationtermon
processsharessomeanalogytothehuman-machinecollab-
modelcomplexity. Theobjectivefunctionisdefinedas
orativeimagedatabaseconstruction[65,58].
f(W;F,λ)=I {c;f}−R(W;λ), (1)
W
where c ∈ {1,...,K} is a category label, F is the set of 3.3 HierarchicalCategoryRelationship
image features f = (f ,...,f )T ∈ RD. I {c;f} is
i i1 iD W ImageNet[12]isconstructedaccordingtotheWordNet
an estimation of the mutual information between the fea-
ontologyhierarchy[40]. Inthiswork,ourconvergedCNN
ture vector f and the label c under the conditional model classificationmodelcanbefurtherextendedtoexplorethe
p(c|f,W). R(W;λ) is the complexity penalty and spec- hierarchicalclassrelationshipinatreerepresentation.First,
ified according to p(c|f,W). We adopt the unsupervised the pairwise class similarity or affinity score A between
i,j
multilogit regression cost as [21]. The conditional model classes(i,j)ismodeledviaanadaptedmeasurementofCNN
classificationconfusion.
1(cid:16) (cid:17)
A = Prob(i|j)+Prob(j|i) (4)
i,j 2
= 1(cid:16)(cid:80)Im∈CiCNN(Im|j) + (cid:80)In∈CjCNN(In|i)(cid:17)
2 |C | |C |
i j
(5)
whereC , C aretheimagesetsforclassi,j respectively,
i j
|·|isthecardinalityfunction,CNN(I |j)istheCNNclas-
m
sification score of image I (from class C ) according to
m i
classjthatisdirectlyobtainedbytheN-wayCNNsoftmax.
HereA =A issymmetricbyaveragingProb(i|j)and
i,j j,i
Prob(j|i).
TheAffinityPropagationalgorithm[19](AP)isthenin-
vokedtoperforma“tuningparameter-free”partitioningon
thispairwiseaffinitymatrix{A }∈RK×K. Thisprocess Figure2.Sampleimagesoftwounsupervisedlydiscoveredimage
i,j
can be executed recursively to generate a hierarchically- clusters with associated clinically semantic key words, contain-
merged category tree. Without loss of generality, we as- ing(likelyappeared)anatomies,pathologies,theirattributesand
sume that at level L, classes iL,jL are formed by merging imagingprotocolsorproperties.
classesatlevelL-1throughAPclustering. Thenewaffinity Table2.ClassificationAccuracyofConvergedCNNModels
scoreA iscomputedasfollows.
iL,jL CNNsetting Cluster# Top-1 Top-5
1(cid:16) (cid:17) AlexNet-FC7-Topic 270 0.8109 0.9412
A = Prob(iL|jL)+Prob(jL|iL) (6)
iL,jL 2 AlexNet-FC7-ImageNet 275 0.8099 0.9547
(cid:80) (cid:80)
CNN(I |k)
Prob(iL|jL)= Im∈CiL k∈jL m (7) AlexNet-Conv5-FV 712 0.4115 0.4789
|CiL| AlexNet-Conv5-VLAD 624 0.4333 0.5232
GoogLeNet-Pool5 462 0.4109 0.5609
wheretheL-thlevelclasslabeljLincludesallmergedorig-
inalclasses(i.e.,0-thlevelbeforeAPiscalled)k ∈ jL ob- GoogLeNet-Inc.5b-VLAD 929 0.3265 0.4001
tainedthusfar. N-wayCNNclassificationscoresonlyneed LDPO during clustering (Sec. 3.2) for a fair comparison.
tobeevaluatedonceatthebeginningofAP.Theconsequent ThusthemodelselectionRIMmoduleisdropped.
valueofanyAiL,jL atanymergedlevelisthesumofthe0- IneachLDPOround,1)theimageclusteringstep(Sec.
thlevelconfusionscores. Themodeledcategoryhierarchy 3.2)isappliedontheentireimagedatasetinordertoassign
can alleviate the highly uneven visual separability among aclusterlabeltoeachimage,2)forCNNmodelfine-tuning
discoveredimagecategories[63]. (Sec. 3.1), images are randomly reshuffled into three sub-
4 ExperimentalResults sets of training (70%), validation (10%) and testing (20%)
ateachiteration. ThisensuresthatLDPOconvergencewill
Datasets: We experiment on the same medical im-
generalize to the entire image database. The CNN model
age dataset as in [49] that contains totally 215,786 2D
isfine-tunedateachLDPOiterationonceanewsetofim-
key-images and the associated radiology reports of 61,845
age labels is generated from the clustering stage. We use
uniquepatients.Key-imagesareresizedto256×256bitmap
Caffe [26] implementation of CNN models. The softmax
images(from512×512). Theintensityrangesarerescaled
loss layer (i.e., ’FC8’ in AlexNet and ’loss3/classifier’ in
using the default “optimal” window settings stored in the
GoogLeNet)ismoresignificantlymodulatedby1)settinga
DICOMheaderfiles(IntensityrescalingimprovestheCNN
higherlearningratethanallotherCNNlayers;and2)updat-
classification accuracy by ∼ 2% comparing to [49]).
ingthe(varyingbutconverging)numberofcategoryclasses
Patient-sensitive information in radiology reports is re-
fromtheclusteringresults.
movedforprivacyreasons. Furthermore,wequantitatively
evaluate our LDPO framework on three widely-reported 4.1 UnsupervisedMedicalImageCategorization
scene recognition benchmark datasets: 1). I-67 [44] of 67 We first investigate the convergence issue of the LDPO
indoor scene classes with 15620 images; 2). B-25 [62] of method under different system configurations and then re-
25 architectural styles from 4794 images; 3). S-15 [32] of porttheCNNclassificationperformanceonthediscovered
15 outdoor and indoor mixed scene classes with 4485 im- categories.
ages. For scene recognition, the ground truth (GT) labels Clustering Method: As shown in Fig. 3 (a), RIM
are only used to validate the final quantitative LDPO clus- can estimate unsupervised category numbers consistently
tering results (where cluster-purity becomes classification wellunderdifferentimagerepresentations(deepCNNfea-
accuracy). The cluster number is assumed to be known to ture configurations + encoding schemes). Standalone k-
(a) (b) (c) (d)
Figure3.LDPOPerformanceusingRIMclusteringunderdifferentimageencodingmethods(i.e.,FVandVLAD)andCNNArchitectures
(i.e.,AlexNetandGoogLeNet).Thenumberofclustersdiscovered,Top-1accuracyoftrainedCNNs,thepurityandNMImeasurementsof
clustersfromadjacentiterationsareillustratedin(a,b,c,d),respectively.
means clustering enables LDPO to converge quickly with ogyreports)arehighlyunbalanced[49],withthreeclusters
highclassificationaccuracieswhereasRIMbasedmodelse- inhabiting the majority of images. Note that our method
lection module produces more balanced and semantically imposes no explicit constraint on the number of instances
meaningful clustering results (see more in Sec. 4.1). This per cluster. Fig. 2 shows sample images and their top-10
advantageisprobablyduetoRIM’stwouniqueproperties: associated key words from two randomly selected clusters
1) less restricted geometric assumptions in the clustering (more results are provided in the supplementary material).
featurespace; 2)thecapacitytoattaintheoptimalnumber The LDPO clusters are found to be clinically or seman-
ofclustersbymaximizingthemutualinformationbetween tically related to the corresponding key words, which de-
inputdataandtheinducedclustersviaaregularizedterm. scribe presented anatomies, pathologies (e.g., adenopathy,
Pseudo-TaskInitialization: Bothgenericanddomain- mass), their associated attributes (e.g., bulky, frontal) and
imagingprotocolsorproperties.
specificCNNmodels [30,55,49]areemployedforLDPO
initialization. Fig. 3 illustrates the performance of LDPO CategorizationRecognizable? Wevalidatethefollow-
using two CNN variants – AlexNet-FC7-ImageNet and ing hypothesis: a high quality unsupervised image catego-
AlexNet-FC7-Topic. AlexNet-FC7-ImageNet yields no- rizationschemewillgeneratelabelsthatcanbemoreeasily
ticeably slower LDPO convergence than its counterpart of recognized by any supervised CNN model. From Table 2,
AlexNet-FC7-Topic, as the latter has already been fine- AlexNet-FC7-Topic has the Top-1 classification accuracy
tuned by the report-derived category information on the of 0.8109 and Top-5 accuracy 0.9412 with 270 formed
same radiologyimage database[49].. Nevertheless, the fi- image categories while AlexNet-FC7-ImageNet achieves
nalclusteringoutcomesaresimilarafterconvergencefrom the accuracies of 0.8099 and 0.9547, from 275 discov-
AlexNet-FC7-ImageNet or AlexNet-FC7-Topic. At ∼ 10 ered classes. In contrast, [49] reports the Top-1 accura-
iterations,twodifferentinitializationsresultinsimilarclus- cies of 0.6072, 0.6582 and Top-5 as 0.9294, 0.9460 from
ternumbers, purity/NMIscoresandevenclassificationac- only 80 classes using AlexNet [30] or VGGNet-19 [52],
curacies(Table2). respectively. The classification accuracies shown in Ta-
ble 2 are computed using the final LDPO-converged CNN
Deep CNN Feature and Image Encoding: Different
models and the testing dataset. Markedly better accura-
configurationsofimagerepresentationcanaffecttheperfor-
cies(especiallyonTop-1)onclassifyinghighernumbersof
manceofmedicalimagecategorization,asshowninFig.3.
classes ( that are generally more challenging) also demon-
Deep images features are extracted at different layers of
strate the advantages of the LDPO discovered image clus-
depth from two CNN models (i.e., AlexNet, GoogLeNet)
ters or labels over those in [49], under the same radiology
andmaypresentthedepth-specificvisualinformation. Dif-
database. Upon evaluation by two board-certified radiolo-
ferentimagefeatureencodingschemes(FVorVLAD)add
gists,AlexNet-FC7-Topicof270categoriesandAlexNet-
furtheroptionsorvariationsintothisprocess. Thenumbers
FC7-ImageNet of 275 classes are considered the best of
of clusters range from 270 (AlexNet-FC7-Topic with no
totalsixmodel-feature-encodingsetups. Interestingly,both
explicitfeatureencodingscheme)to931(themoresophis-
models have no external feature encoding schemes built-
ticatedGoogLeNet-Inc.5b-VLADwithVLADencoding).
in and preserve gloss image layouts (without spatially un-
ThenumbersofclustersdiscoveredbyRIMareexpectedto
ordered FV or VLAD encoding modules [8, 25]). Refer
reflecttheamountofknowledgeorinformationcomplexity
tosupplementarymaterialformoreresultsonradiologists’
storedinthePACSdatabase.
evaluation.
UnsupervisedCategorization: Ourcategorydiscovery
4.2 SceneRecognition
clusters are generally visually coherent within the cluster
andsize-balancedacrossclusters. However,imageclusters Weusethreescenerecognitiondatasetstoquantitatively
formedonlybasedontextinformation(of∼ 780K radiol- evaluatetheproposedLDPO-PMmethod(withpatchmin-
Table3.ClusteringperformanceofLOMandothermethodson3scenerecognitiondatasets.ThelastColumnpresentsthestate-of-the-art
fully-supervisedsceneClassificationAccuracy(CA)foreachdataset,producedby[8,42,66]respectively.
Dataset KM[57] LSC[4] AC[22] EP[10] MDPM[34] LDPO-A-FC LDPO-A-PM LDPO-V-PM Supervised
ClusteringAccuracy(%) CA(%)
I-67[44] 35.6 30.3 34.6 37.2 53.0 37.9 63.2 75.3 81.0[8]
B-25[62] 42.2 42.6 43.2 43.8 43.1 44.2 59.2 59.5 59.1[42]
S-15[32] 65.0 76.5 65.2 73.6 63.4 73.1 90.2 84.0 91.6[66]
NormalizedMutualInformation
I-67[44] .386 .335 .359 - .558 .389 .621 .759 -
B-25[62] .401 .403 .404 - .424 .407 .588 .546 -
S-15[32] .659 .625 .653 - .596 .705 .861 .831 -
Table4.SceneRecognitionAccuracyonMIT-67Dataset[44]
ing)basedontwometrics: 1)clusteringbasedscenerecog-
nition accuracy and 2) supervised classification (e.g., Li- Method Accuracy(%)
blinear ) on image representations learned in an unsuper- D-patch[53] 38.1
visedfashion. ThepurityandNMImeasurementsarecom- D-parts[54] 51.4
putedbetweenthefinalLDPOclustersandGTsceneclasses DMS[13] 64.0
wherepuritybecomestheclassificationaccuracyagainst MDPM-VGG[34] 77.0
GT. The LDPO cluster numbers are set to match the GT MetaObject[60] 78.9
classnumbersof(67,25,15),respectively. Wecomparethe
FC(VGG) 68.9
LDPO scene recognition performance to those of several
CONV-FV(VGG)[8] 81.0
popular clustering methods, such as KM [57]: k-means;
LDPO-V-PM-LL 72.5
LSC [4]; AC [22]: Agglomerative clustering; EP [10]:
Ensemble Projection + kmeans; and MDPM [34]: Mid- LDPO-V-PM 75.3
levelDiscriminativePatchMining+kmeans. BothEPand
ble4. Theone-versus-allLiblinearclassificationinLDPO-
MDPM use mid-level visual elements based image repre-
V-PM-LL does not noticeably improve upon purely unsu-
sentations. Three variants of our method (i.e., LDPO-A-
pervisedLDPO-V-PM.ThismayindicatethattheLDPO-
FC7: FC7featureonAlexNet,LDPO-A-PM:FC7feature
PM image representation is sufficient to adequately sepa-
onAlexNetwithpatchmining,andLDPO-V-PM:FC7fea-
rate images from different scene classes. Last, we exper-
tureonVGG-VDwithpatchmining)areexploited. Onall
iment the clustering convergence issue with two different
threedatasets,theLDPO-A-PMandLDPO-V-PMachieve
initializations: random initialization or image labels ob-
significantly higher purity and NMI values than the previ-
tainedfromk-meansclusteringonFC7featuresofanIma-
ous clustering methods (cf. Table 3). Especially for the
geNetpretrainedAlexNet. Whiletheclusteringaccuracyof
MIT-67 indoor scene dataset [44], our best model LDPO-
theLDPO-PMwithrandominitializationincreasesrapidly
V-PM achieves the unsupervised scene recognition accu-
duringitsfirstiterations,bothschemesultimatelyconverge
racy of 75.3%, which nearly doubles the performances of
tosimilarperformancelevels. ThissuggeststhattheLDPO
KM and AC on FC7 features of an ImageNet pretrained
convergenceisinsensitivetothechoseninitialization.
AlexNet [30, 3]. Note that the state-of-the-art supervised
classificationaccuracyonMIT-67is81.0%[8]andourun-
5 Conclusion
supervisedmethodiscomparativelyclosetothat.VGG-VD
–adeeperCNNmodel–empiricallybooststherecognition Inthispaper,wepresentaLoopedDeepPseudo-taskOp-
performancefromLDPO-A-PMof63.2%toLDPO-V-PM timizationframeworkforunsupervisedjointminingofim-
at75.3%onMIT-67. Howeverthisperformancegainisnot age features and labels. Our method is validated through
observedontwoothersmallerdatasets. two important problems: 1) discovery and exploration
of semantic categories from a large-scale medical image
Next, we evaluate the supervised discriminative power databaseand2)unsupervisedscenecognitiononthreepub-
of LDPO-PM learned image representation. We measure lic datasets. Extensive experiments demonstrate excellent
itsclassificationaccuracyusingtheMIT-67datasetandits quantitativeandqualitativeresultsonbothtasks. Themea-
standard partition [44], i.e., 80 training and 20 testing im- surable LDPO “convergence” makes the ill-posed image
ages per class. As in [53, 54, 13, 34, 60, 8], we use the auto-annotationproblembetterconstrained.
Liblinear classification toolbox [17] on the LDPO-V-PM Acknowledgements This work was supported by the
image representation (noted as LDPO-V-PM-LL), under Intramural Research Program of the NIH Clinical Cen-
5-fold cross validation. The supervised and unsupervised ter. This work utilized the computational resources of the
scene recognition accuracy results from previous state-of- NIH HPC Biowulf cluster (http://hpc.nih.gov). We thank
the-art work and variants of our method are listed in Ta- NVIDIACorporationfortheGPUdonation.
References [18] L. Fei-Fei, R. Fergus, and P. Perona. Learning generative
[1] D. M. Blei, A. Y. Ng, and M. I. Jordan. Latent dirichlet visual models from few training examples: an incremental
allocation. Journal of machine Learning research, 3:993– bayesianapproachtestedon101objectcategories. Proc.of
1022,2003. IEEECVPRworkshop,2004.
[2] L. Bossard, M. Guillaumin, and L. Van Gool. Food-101– [19] B.FreyandD.Dueck. Clusteringbypassingmessagesbe-
miningdiscriminativecomponentswithrandomforests. In tweendatapoints. Science,315:972–976,2007.
EuropeanConferenceonComputerVision,pages446–461. [20] R.Girshick, J.Donahue, T.Darrell, andJ.Malik. Region-
Springer,2014. based convolutional networks for accurate object detection
and semantic segmentation. IEEE Trans. Pattern Anal.
[3] K. Chatfield, K. Simonyan, A. Vedaldi, and A. Zisserman.
Mach.Intell.,2015.
Returnofthedevilinthedetails: Delvingdeepintoconvo-
lutionalnets. InBritishMachineVisionConference,2014. [21] R.Gomes,A.Krause,andP.Perona. Discriminativecluster-
ingbyregularizedinformationmaximization. NIPS,2010.
[4] X. Chen and D. Cai. Large scale spectral clustering with
landmark-basedrepresentation. InAAAI,2011. [22] K.C.GowdaandG.Krishna. Agglomerativeclusteringus-
ing the concept of mutual nearest neighbourhood. Pattern
[5] X.ChenandA.Gupta. Weblysupervisedlearningofconvo-
recognition,10(2):105–112,1978.
lutionalnetworks. InProc.ofICCV,2015.
[23] B. Hariharan, J. Malik, and D. Ramanan. Discriminative
[6] X. Chen, A. Shrivastava, and A. Gupta. Neil: Extracting
decorrelationforclusteringandclassification. InEuropean
visualknowledgefromwebdata. InProc.ofICCV,2013.
Conference on Computer Vision, pages 459–472. Springer,
[7] J.Choi,M.Rastegari,A.Farhadi,andL.S.Davis. Adding
2012.
unlabeledsamplestocategoriesbylearnedattributes.InPro-
[24] M. Huh, P. Agrawal, and A. A. Efros. What makes im-
ceedings of the IEEE Conference on Computer Vision and
agenet good for transfer learning? In arXiv preprint:
PatternRecognition,pages875–882,2013.
arXiv:1608.08614,2016.
[8] M.Cimpoi, S.Maji, andA.Vedaldi. Deepfilterbanksfor
[25] H.Jegou,F.Perronnin,M.Douze,J.Sanchez,P.Perez,and
texturerecognitionandsegmentation. Proc.ofIEEECVPR,
C.Schmid. Aggregatinglocalimagedescriptorsintocom-
2015.
pactcodes.PatternAnalysisandMachineIntelligence,IEEE
[9] A.Coates, A.Ng, andH.Lee. Ananalysisofsingle-layer
Transactionson,34(9):1704–1716,Sept2012.
networksinunsupervisedfeaturelearning. AIandStatistics,
[26] Y.Jia,E.Shelhamer,J.Donahue,S.Karayev,J.Long,R.Gir-
2011.
shick, S. Guadarrama, and T. Darrell. Caffe: Convolu-
[10] D. Dai and L. Van Gool. Ensemble projection for semi- tionalarchitectureforfastfeatureembedding.arXivpreprint
supervisedimageclassification. InProc.ofICCV,2013. arXiv:1408.5093,2014.
[11] D. Dai and L. Van Gool. Unsupervised high-level feature [27] M.Juneja,A.Vedaldi,C.Jawahar,andA.Zisserman.Blocks
learning by ensemble projection for semi-supervised im- thatshout: Distinctivepartsforsceneclassification. CVPR,
age classification and image clustering. Technical report, pages923–930,2013.
arXiv:1602.00955,2016.
[28] A. Karpathy and L. Fei-Fei. Deep visual-semantic align-
[12] J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei- ments for generating image descriptions. Proc. of IEEE
Fei.Imagenet:Alarge-scalehierarchicalimagedatabase.In CVPR,pages3128–3137,2015.
Computer Vision and Pattern Recognition, pages 248–255. [29] D. Kingma, S. Mohamed, D. Rezende, and M. Welling.
IEEE,2009. Semi-supervised learning with deep generative models.
[13] C. Doersch, A. Gupta, and A. A. Efros. Mid-level visual NIPS,2014.
element discovery as discriminative mode seeking. In Ad- [30] A. Krizhevsky, I. Sutskever, and G. E. Hinton. Imagenet
vances in Neural Information Processing Systems (NIPS), classification with deep convolutional neural networks. In
pages494–502,2013. Advances in neural information processing systems, pages
[14] C. Doersch, A. Gupta, and A. A. Efros. Unsupervised vi- 1097–1105,2012.
sual representation learning by context prediction. In Pro- [31] S.Lazebnik,C.Schmid,andJ.Ponce. Asparsetexturerep-
ceedingsoftheIEEEInternationalConferenceonComputer resentationusinglocalaffineregions. IEEETrans.Pattern
Vision,pages1422–1430,2015. Anal.Mach.Intell.,27(8):1265–1278,2005.
[15] A.Dosovitskiy,J.Springenberg,M.Riedmiller,andT.Brox. [32] S. Lazebnik, C. Schmid, and J. Ponce. Beyond bags of
Discriminativeunsupervisedfeaturelearningwithconvolu- features: Spatial pyramid matching for recognizing nat-
tionalneuralnetworks. NIPS,2014. ural scene categories. In 2006 IEEE Computer Society
[16] M.Everingham,S.M.A.Eslami,L.VanGool,C.Williams, Conference on Computer Vision and Pattern Recognition
J.Winn,andA.Zisserman. Thepascalvisualobjectclasses (CVPR’06),volume2,pages2169–2178.IEEE,2006.
challenge: A retrospective. International journal of com- [33] Q. Li, J. Wu, and Z. Tu. Harvesting mid-level visual con-
putervision,111(1):98–136,2015. ceptsfromlarge-scaleinternetimages.InProceedingsofthe
[17] R.-E.Fan, K.-W.Chang, C.-J.Hsieh, X.-R.Wang, andC.- IEEEConferenceonComputerVisionandPatternRecogni-
J. Lin. Liblinear: A library for large linear classification. tion,pages851–858,2013.
Journal of machine learning research, 9(Aug):1871–1874, [34] Y. Li, L. Liu, C. Shen, and A. van den Hengel. Mid-level
2008. deeppatternmining. InCVPR,pages971–980,2015.
[35] Y. Li and Z. Zhou. Towards making unlabeled data never [54] J.SunandJ.Ponce. Learningdiscriminativepartdetectors
hurt. ICML,2011. forimageclassificationandcosegmentation. InProceedings
[36] X.Liang,S.Liu,Y.Wei,L.Liu,L.Lin,andS.Yan.Compu- of the IEEE International Conference on Computer Vision,
tationalbabylearning. InProc.ofICCV,2015. pages3400–3407,2013.
[37] M.Lin,Q.Chen,andS.Yan. Networkinnetwork. InProc. [55] C. Szegedy, W. Liu, Y. Jia, P. Sermanet, S. Reed,
ofICLR,2015. D.Anguelov, D.Erhan, V.Vanhoucke, andA.Rabinovich.
[38] T. Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ra- Goingdeeperwithconvolutions. IEEEConf.onComputer
manan,P.Dollar,andL.Zitnick. Microsoftcoco: Common VisionandPatternRecognition,arXiv:1409.4842,2015.
objectsincontext. InECCV,2014. [56] T.Tuytelaars,C.H.Lampert,M.B.Blaschko,andW.Bun-
[39] W.Liu, J.He, andS.Chang. Largegraphconstructionfor tine. Unsupervisedobjectdiscovery: Acomparison. Inter-
scalablesemi-supervisedlearning. ICML,2010. nationalJournalofComputerVision,2009.
[40] G.A.Miller. Wordnet: alexicaldatabaseforenglish. Com- [57] A.VedaldiandB.Fulkerson.VLFeat:Anopenandportable
municationsoftheACM,1995. libraryofcomputervisionalgorithms,2008.
[41] J. Y. Ng, F. Yang, and L. S. Davis. Exploiting local [58] M.Wigness,B.Draper,andJ.Beveridge.Efficientlabelcol-
features from deep networks for image retrieval. CoRR, lectionforunlabeledimagedatasets. Proc.ofIEEECVPR,
abs/1504.05133,2015. 2015.
[42] K.-C.PengandT.Chen. Aframeworkofextractingmulti- [59] J.Wu,Y.Yu,C.Huang,andK.Yu. Deepmultipleinstance
scalefeaturesusingmultipleconvolutionalneuralnetworks. learningforimageclassificationandauto-annotation. Proc.
In2015IEEEInternationalConferenceonMultimediaand ofCVPR,pages3460–3469,2015.
Expo(ICME),pages1–6.IEEE,2015. [60] R.Wu,B.Wang,andY.Yu. Harvestingdiscriminativemeta
[43] F.Perronnin,J.Snchez,andT.Mensink.Improvingthefisher objects with deep cnn features for scene classification. In
kernelforlarge-scaleimageclassification. InComputerVi- Proc.ofICCV,2015.
sion ECCV 2010, volume 6314 of Lecture Notes in Com- [61] J.Xiao,J.Hays,K.Ehinger,A.Oliva,andA.Torralba. Sun
puter Science, pages 143–156. Springer Berlin Heidelberg, database: Large-scalescenerecognitionfromabbeytozoo.
2010. InCVPR,pages3485–3492,2010.
[44] A.QuattoniandA.Torralba. Recognizingindoorscenes. In [62] Z. Xu, D. Tao, Y. Zhang, J. Wu, and A. C. Tsoi. Archi-
ComputerVisionandPatternRecognition,IEEEConference tecturalstyleclassificationusingmultinomiallatentlogistic
on,pages413–420.IEEE,2009. regression. In European Conference on Computer Vision,
[45] A. Quattoni and A. Torralba. Recognizing indoor scenes. pages600–615.Springer,2014.
Proc.ofIEEECVPR,2009. [63] Z.Yan,H.Zhang,R.Piramuthu,V.Jagadeesh,D.DeCoste,
[46] R. Raina, A. Battle, H. Lee, B. Packer, and A. Ng. Self- W.Di,andY.Yu. Hd-cnn: Hierarchicaldeepconvolutional
taught learning: transfer learning from unlabeled data. neural network for large scale visual recognition. Proc. of
ICML,2007. ICCV,2015.
[47] A.Razavian,H.Azizpour,J.Sullivan,andS.Carlsson. Cnn [64] J.Yang,D.Parikh,andD.Batra.Jointunsupervisedlearning
features off-the-shelf: an astounding baseline for recogni- of deep representations and image clusters. arXiv preprint
tion. ArXiv:1403.6382,2014. arXiv:1604.03628,2016.
[48] O. Russakovsky, J. Deng, H. Su, J. Krause, S. Satheesh, [65] F.Yu,Y.Zhang,S.Song,A.Seff,andJ.Xiao. Lsun: Con-
S. Ma, Z. Huang, A. Karpathy, A. Khosla, M. Bernstein, structionofalarge-scaleimagedatasetusingdeeplearning
et al. Imagenet large scale visual recognition challenge. withhumansintheloop. arXiv:1506.03365,2015.
arXivpreprintarXiv:1409.0575,2014. [66] B.Zhou, A.Lapedriza, J.Xiao, A.Torralba, andA.Oliva.
[49] H.Shin,L.Lu,L.Kim,A.Seff,J.Yao,andR.Summers.In- Learning deep features for scene recognition using places
terleavedtext/imagedeepminingonalarge-scaleradiology database. InNIPS,pages487–495,2014.
database. Proc.ofIEEECVPR,2015.
[50] H.Shin,L.Lu,L.Kim,A.Seff,J.Yao,andR.Summers.In-
terleavedtext/imagedeepminingonalarge-scaleradiology
imagedatabaseforautomatedimageinterpretation. Journal
ofMachineLearningResearch,pages17(107):1–31,2016.
[51] A.Shrivastava,S.Singh,andA.Gupta. Constrainedsemi-
supervised learning using attributes and comparative at-
tributes.InEuropeanConferenceonComputerVision,pages
369–383.Springer,2012.
[52] K. Simonyan and A. Zisserman. Very deep convolutional
networks for large-scale image recognition. In Proc. Int.
Conf.Learn.Repr.,2015.
[53] S.Singh,A.Gupta,andA.A.Efros.Unsuperviseddiscovery
ofmid-leveldiscriminativepatches.InEuropeanConference
onComputerVision,2012.