Table Of ContentSKETCH-EPITOMES :DISCRIMINATIVELYMINIMALIST REPRESENTATIONS FOR
OBJECTCATEGORIES
RaviKiranSarvadevabhatla,R.Venkatesh Babu
VideoAnalyticsLab, SERC,
Indian InstituteofScience, Bangalore, India.
[email protected],[email protected]
5
1
0
2 ABSTRACT
n Freehandlinesketchesareaninterestinganduniqueformofvisual
a representation. Typically, suchsketchesarestudiedandutilizedas
J anendproductofthesketchingprocess. However,wehavefoundit
Fig.1: Inspiteofminimaldetail,wecanrecognizethelinesketch
1 instructivetostudy the sketches as sequentially accumulated com- easilyandcorrectlyasbelongingtothecategorycup.
3 position of drawing strokes added over time. Studying sketches
in this manner has enabled us to create novel sparse yet discrim-
] inativesketch-basedrepresentationsforobjectcategorieswhichwe
V it apart from other methods of sparse sketch generation: The pro-
termcategory-epitomes.Ourprocedureforobtainingtheseepitomes
cessof epitomeconstructionguarantees thatthemost fundamental
C concurrentlyprovidesanaturalmeasureforquantifyingthesparse-
strokes which enable category recognition (discrimination) are re-
. nessunderlying theoriginal sketch, whichwetermepitome-score.
s Weconstructandanalyzecategory-epitomesandepitome-scoresfor tained. Toquantifythesparsenessofanepitome,weprovideanat-
c ural measure termed epitome-score. We analyze the eptiomes and
freehandsketchesbelongingtovariousobjectcategories. Ouranal-
[
corresponding epitome-scoresfor freehandsketches acrossvarious
ysisprovidesanovelviewpointforstudyingthesemanticnatureof
objectcategories.Ouranalysisprovidesanovelviewpointforstudy-
1 objectcategories.
v ingthesemanticnatureofobjectcategories.
IndexTerms— sketch, object categoryrecognition, sketchse-
2 Therestofthepaperisorganizedasfollows: Webrieflyreview
quenceanalysis,patternrecognition,epitome,sparse,minimalist
8 relatedliteratureinSection2.InSection3,wedescribetheconstruc-
0 tionofasketchclassifier. Thissketchclassifierplaysacrucialrole
0 1. INTRODUCTION inobtainingcategory-epitomeofagivensketch.InSection4,wede-
0 scribehowthecategory-epitomeofasketchisactuallyobtainedand
. Sketches,asaformofvisualrepresentation, exhibitagreatvariety provide a simple, natural measure termed epitome-score to quan-
2
fromrealisticportraitstosparselydrawn,stylisticones.Inparticular, tify its epitomal-ness. Section 5 contains an analysis of category-
0
considerfreehand(i.e.hand-drawn)sketchesofobjects.Aninstance epitomes and epitome-scores across object categories. Section 6
5
ofsuchasketchcanbeseeninFigure1.Thoughcontainingminimal concludes thepaper byoutlining someof thepromising directions
1
: detail,theobjectcategorytowhichitbelongsiseasilydetermined. forfuturework.
v Thissuggestsaninherentsparsenessinthehumanneuro-visualrep-
i resentationoftheobject. Therefore,studyingsuchsparsesketches
X
can aid our understanding of the cognitive processes involved and 2. RELATEDWORK
r spurthedesignofefficientvisualclassifiers.
a
Freehand line sketches are typically formed as a composition Inhisseminal work[5], Marrsuggested anabstract representation
of primitive hand-drawn curves (called strokes) added sequen- called the primal sketch – a sparse, sketch-like representation of
tially over time. A significant body of work has examined such generic images in terms of image primitives. Inspired by his the-
sketchesinthecontextofclassificationandcontent-based retrieval ories,methodsforformalizingthenotionofprimalsketchhavebeen
problems[1][2][3][4]. In these problems, the end product of the proposed [6] and primal sketch representations have been used as
sketching process, i.e. the full sketch, is typically considered in- features for object detection[3], texture characterization[7] and for
toto. However, we believe it can be quite instructive to study the super-resolution[8]. Yet another line of research aims to generate
temporal process of sketch formation itself, starting with the first sketchesfromoneormoresourceimages[4][9]withoutexplicitre-
hand-drawn stroke until the last stroke which finalizes the sketch. coursetotheideaofprimalsketch. Acommonfeatureinallthese
Ourbelieforiginatesinasurprisingdiscoverywehavemade: For worksistheutilizationofphotographicimagesasthestartingpoint.
a given sketch, there exists a minimal discriminative subset of all In contrast, our starting point is the sketch stroke data created by
itsstrokeswhichcontributetothesketch’sidentity(category)being humanbeings.Thisprovidesaglimmerofhopethatthesparseneu-
recognizedconsistentlyandcorrectly.Wetermthesketchcomposed rovisualrepresentationoftheobjectbeingsketchedistransferredto
using this minimal stroke subset as a category-epitome. Figure 2 thesketchintheprocessofdrawing,atleastinpart.
shows examples of freehand linesketches and their corresponding Inadditiontotheartificial(i.e. not generatedbyhumanhand)
sparsecategory-epitomes. natureofsketch generation, theworksmentioned above donotat-
Inthis paper, wedescribe how these sparse category-epitomes tempttoquantifythesparsenessoftheresultingsketch. Theyalso
areobtained. Thecategory-epitomehasauniquefeaturewhichsets donotexaminethetemporalnatureofsketchcomposition. Incon-
Fig.2:Originalsketches(toprow)andcorrespondingcategory-epitomes(bottomrow)forvariousobjectcategories.
trast, recent work by Berger et. al.[10] analyzes the temporal as- 3.3. Sketchfeatureextraction
pectofsketchinginthecontextofmimickingartiststyle. However,
theiremphasisisonsynthesizingabstractfacialsketchesratherthan AsthetoprowofFigure2demonstrates,thespatialdensityofsketch
recognition. Moreover,theirsketchesareproducedbyprofessional strokescanbequitesmall. Moreover,extractingtypicalimagefea-
artists.Incontrast,oursketcheshavebeengeneratedbycrowdsourc- tures (e.g. based on texture) is ruled out and edge information is
ingfromthegeneralpublic. Theideaofidentifyingandutilizinga quitesparse. Nevertheless,anumberofalternativeshavebeenpro-
discriminativesubsetofstrokeswasemployedbyKarteeketal.[11] posedassketchfeatures.Forasurveyofthesemethodsinthecontext
forclassifyingonlinehandwrittendata.Anotherworkcloseinspirit ofsketch-basedimageretrieval,referto[1]. WeextractHistogram
tooursistheunpublished,butpubliclyavailablemanuscriptofJiang of Oriented Gradients(HOG[14])-like sketch descriptors using the
etal.[12]whichexaminesthetemporalevolutionofsketchesfroma pipelinedescribedbyEitzet. al.[13]. Thecollectionofdescriptors
visualizationperspective. Finally,theworkofEitzetal.[13]exam- so obtained are then combined using Fisher image representation
ineshowhumanstendtodrawobjectsbyanalyzingalargenumber approach[15]toobtainafeaturevectorforeachsketch.
ofsketchesspreadacrosscommonlyencounteredobjectcategories.
Weemploytheirdatabaseofsketchesinthispaper.
3.4. SketchClassification
3. BUILDINGTHESKETCHCLASSIFIER
Asaninitialexplorationandforeaseofanalysis,weconsideronly
3.1. Thesketchdatabase
thefirst50alphabeticallysortedsketchcategories.Webuildasketch
The sketches used in our study have been taken from the publicly classifierby utilizing80% of the augmented sketches(Section 3.2)
available freehand line sketch database of Eitz et al.[13]. This from each category for training and the rest for testing. Doing so
database contains a curated set of 20,000 hand-drawn sketches provides2400×0.8×50=96,000sketchesfortrainingand80×
evenly distributedacross 250 object categories. Asmentioned be- 0.2×50=800sketchesfortesting1.
fore, these sketches have been obtained by crowdsourcing across
Fromeach test sketch, itsFisher featurevector(Section 3.3) is
thegeneralpopulation. Assuch, theyareagood startingpointfor
obtained. A multi-class Support Vector Machine (SVM) classifier
analyzingtheunderpinningsofthesketchingprocessbyhumans. A
employingaRadialBasisFunction(RBF)kernelwastrainedonthe
fewexamplesfromthedatabasecanbeseeninthetoprowofFigure
Fisherfeaturevectorsbyemploying5-foldcrossvalidationandgrid-
2. Thedominantappealofthisdatabaseisthatthetemporalstroke
basedparametersearch.Theaccuracyoftheresultingclassifierwas
information(thesequentialorderinwhichthestrokesweredrawn)
60.25%. Forcontext,theaccuracy(forthesamesplitratiooftrain-
for a sketch has been provided. As we will see in Section 4, it is
ingandtestsketches)obtainedbyEitzet. al. [13]is54%withthe
preciselythetemporalstrokeinformationthatformsthebasisforob-
caveatthatthenumberofcategoriesarelargerthanours–250.
taining the category-epitome and the corresponding epitome-score
Intheoverallschemeofthings,thesketchclassifierisonlyuse-
forsketches.
fultotheextentthatitletsusdeterminethecategory-epitomes.Seen
in this light, the accuracy of the classifier assumes secondary im-
3.2. Sketchdataaugmentation portance. However,aclassifierwithgoodperformanceisstilldesir-
ableasitensuresalargercoverageofthetestset. Inaddition,such
ThedatabaseofEitz. et. al. contains80sketchespercategory. To aclassifiercould potentiallyhelpobtainsparser category-epitomes
increasethenumber ofsketchespercategoryavailablefortraining compared to a counterpart whose performance is relatively poor.
the classifier (Section 3.4), we perform data augmentation by ap- Nevertheless,toprogresstowardsourgoalofdeterminingcategory-
plyinggeometricandmorphologicaltransformationstoeachsketch. epitomes,wesettlefortheexistingperformanceofthesketchclas-
Specifically,eachsketchisinitiallysubjectedtoimagedilationus- sifier. Inthenext section, weshall seehow thesketch classifieris
inga5×5squarestructuringelement. Anumberoftransformsare actuallyutilizedinconstructingthecategory-epitome.
applied to this thickened sketch – mirroring (across vertical axis),
rotation(±5,±15degrees),combinationsofhorizontalandvertical
shifts(±5,±15pixels),centralzoom(±3%,±7%ofimageheight).
1Testingisdoneonlyontheoriginalsketchsubjectedtodilationandnot
Asaresult,30newsketchesaregeneratedperoriginalsketch. The
onits subsequently transformed variants generated fordata augmentation.
dataaugmentationprocedureresultsin2400sketchespercategory, Hencethefactorof80foreachtestcategoryinsteadofthefull2400asin
foratotalof600,000sketchesacross250categories. training.
Fig.3: Constructingthecategory-epitomeforanairplanesketch: Thesketchhas9strokes. S1−S9arethecumulativestrokesequence
canvases.Aredcrossmarkindicatesamisclassification(0)whileagreentickmarkindicatescorrectclassification(1).CanvasS4outlinedby
acyanrectangleisthecategory-epitome. NotethateventhoughcanvasS2isclassifiedcorrectly,weconsidercategory-epitomeascanvasS4,
thetemporallyearliest,correctlyclassifiedcanvaswhosesuccessorsareclassifiedcorrectlyaswell.
4. OBTAININGTHECATEGORY-EPITOME aswell. UsingtheexampleinFigure3,theclassificationlabelse-
quenceisgivenbyL={0,1,0,1,1,1,1,1,1}.FromEquation(1),
4.1. ConstructingCumulativeStrokeSequences theproductsequenceiscomputedasP = {0,0,0,1,1,1,1,1,1}.
FromEquation(2),weobtaine=4.Inotherwords,canvasS4(out-
Asthefirststepindeterminingthecategory-epitome, weconstruct
linedbyacyanrectangleinFigure3)correspondstothecategory-
sequences of cumulative strokes derived from correctly classified
epitome: the temporally earliest, correctly classifiedcanvas whose
testsketches. Suppose thesequence of strokes inthetemporal or-
successorsS5...S9areclassifiedcorrectlyaswell.Figure2shows
dertheyweredrawninatestsketchisgivenbyS ={s1,s2...sN} sketchesfromvariouscategoriesandtheircorrespondingcategory-
whereNisthetotalnumberofstrokesinthesketch.Toconstructthe
epitomes.
corresponding cumulative stroke sequence, we begin with a blank
canvas. Strokesfromthegivensketcharesuccessivelyaddedtothe
4.3. Epitome-score:Quantifyingthecategory-epitome
blankcanvasinthetemporalorder.Aseachstrokeisadded,interme-
diatecanvasesS1,S2,...,SNarecreated.Specifically,theinterme- Ourprocedureforobtainingthecategory-epitome,describedabove,
diatecanvasesaregivenbyS1 = {s1},S2 = {s1,s2},...,SN = also provides a natural method for quantifying the “epitome”-ness
{s1,s2,...,sN}. Attheendofthisprocess,weobtainthecumula- oftheoriginal,fullsketch. UsingeobtainedfromEquation(2),we
tivestrokesequenceCSS ={S1,...,SN}.Figure3illustratesthe definetheepitome-scoreE ofasketchas:
creationofcumulativestrokesequenceforasketchfromairplane
category.
e,e6=1
E = N (3)
4.2. Constructingthecategory-eptiome (0, e=1
Havinggeneratedthecumulativestrokesequenceforasketchasde- where N is the total number of strokes in the sketch. e = 1
scribedabove, thecategory-epitomecanbeconstructed. Usingthe corresponds to the situation where merely drawing the first stroke
classifierfromSection3,eachintermediatecanvasofthecumulative conveystheepitome-nessofthesketch. Therefore,wehavedefined
strokesequence isclassifiedtoobtainabinarylabeling–thelabel the corresponding epitome-score E to be 0 for consistency across
is 1 if the sketch category is correctly identified and 0 otherwise. sketches. Ourdefinitionofepitome-scoreE essentiallyconveysthe
Thus,weobtainabinarylabelsequenceL={l1,l2,...,lN}corre- sparseness underlying the sketch – the smaller its value, the more
spondingtoeachcanvasofthecumulativestrokesequenceCSS(see sparserthesketchislikelytobe.Epitome-scoresverycloseto1,on
Figure3). theotherhand,indicatethatveryfewstrokesintheoriginalsketch
Note that the final canvas SN corresponds to the original test are redundant. Fortunately, very high epitome-scores are not the
sketch since all the strokes have been added to the canvas at that norm,aswewillsee–sparsityispervasiveacrosscategories.Refer-
point.Therefore,thefinalclassificationlabellN mustbe1sincewe ringonceagain toFigure3, theepitome-score for theairplane
are working with correctly classified test sketches. Now, consider sketchiscomputedasE = 4 =0.44.
9
theproductsequenceP ={P1,P2.....PN}formedbycumulative
multiplicationoflabelsli ∈L,i=1,2...N: 5. ANALYSIS
5.1. Epitome-scoresacrosscategories
N
Pi = lj (1) Webeginouranalysisbycomputingthemedianofepitome-scores
Yj=i forsketchesonacategory-by-categorybasis. Figure4displaysthe
medianepitome-scores(shownasfilledcirclesinthefigure)fortest
Then,thecategory-epitomecorrespondstocanvasSeofthecu- sketchesacross50objectcategories. Itishearteningtoobservethat
mulativestrokesequencesuchthat
42%(21/50) of the categories have median epitome-scores below
0.5 and 80%(40/50) of the categories have scores below 0.75.
These trends suggest an inherent sparseness for visual represen-
e=1≤mi≤inN{i|Pi =1} (2) tation of object categories because the smaller the epitome-score,
thesparser thecategory-epitome sketch. Ifweexamine the scores
Informally, the category-epitome Se is the temporally earliest, closely,theepitome-scoresforsomeofthecategories(appleand
correctlyclassifiedcanvaswhosesuccessorsareclassifiedcorrectly boomerang)are0. Forthesecategories, thesketches(seeFigure
e
or0.75
−sc
e
m
pito 0.5
e
n
a
edi0.25
M
0
apple baseball bat boomerang candle cactus bed bread bowl camel car (sedan) canoe bookshelf cabinet bell beer−mug blimp cell phone arm bear (animal) binoculars castle backpack bathtub book bus cannon brain calculator cake camera ant barn bench banana bush carrot butterfly basket airplane alarm clock armchair axe angel bee bridge bulldozer cat bicycle ashtray bottle opener
Categories
Fig.4: Medianepitome-scores (y-axis)andcorresponding errorbarsfor50 objectcategories(x-axis). Thestandard errorsareclampedto
[0,1]–therangeofepitome-scores. Themedianscoresareshownasfilledcircles.
and normalize by the number of test sketches in the category. To
facilitateanalysisandavoidvisual clutter, weselect4prototypical
categorieswhoseplotcontoursoccupytheextremities(towardstop-
leftcornerandbottom-rightcorner)oftheoriginalplot.Inaddition,
weselect6otherprototypicalcategorieswhoseplotcontoursliebe-
tweentheextremityplotspreviouslymentioned. Theresultingplot
canbeviewedinFigure5.
Epitome-scorecanbeconsideredasaproxyforsemanticlevel-
of-detail. Viewedinthislight, theplot fromFigure5suggeststhe
varyingepitome-scorebudgetsacrosscategories–somecategories
requireaconsiderablelevelofdetailbeforetheirepitomalavatarsare
revealed(e.g. categorieswithplotstowardsthelowerrightsuchas
bridgeandbicycle). Ontheotherhand, categorieswithplots
towardstheupperleftcorner(apple,boomerang)haverelatively
Fig.5:Demonstratingtheeffectofbudgetingepitome-scoreoncate- lessstringentdemandsonlevelofdetail.
goryrecognitionrateusingdatafrom10selectedcategories.Values To view a larger set of category-epitomes, t-SNE[16] visu-
of various possible epitome-score thresholds are shown on x-axis. alizations of epitomes for select categories and additional re-
The%ofimagesineachcategorywhichexceedaparticularvalue sults which support the analysis presented in this section, visit
of the threshold are shown on y-axis. The 10 category names are http://val.serc.iisc.ernet.in/sketchepitome/se.html
shownadjacenttotheirrespectiveplots. .
6. FUTUREWORK
2)aretypicallydrawnsuchthatadominantstroke(anovalinthever-
ticalplanewithanotchatthetopfortheappleandthechevron-like
A number of directions exist for future work. One obvious di-
contourfortheboomerang)essentiallycapturestheepitome-nessof
rection would be to improve the performance of existing sketch
thecategory(SeealsoEquation3).
classifier in terms of number of categories as well as accuracy. A
The varying lengths of error bars in Figure 4 indicates that
well-performing classifier which utilizes fewer training samples
someobject categoriesmay havemultiplerepresentative category-
translates to a potentially larger set of test category-epitomes for
epitomesrather than asingle, unique representative. For example,
analysis. We also intend to compare our current distribution of
thesketches of the category (human) arm arelikely drawn witha
category-epitomes/epitome-scoresandrelatedanalysiswiththatob-
fairlyconsistentappearancebyhumans.Thisconsistencyinfluences
tained when human subjects are asked to identify the cumulative
thesketchclassifierandcancauseittoproduce category-epitomes
stroke sequences. For this comparison, we plan to use the bench-
whichexhibitaminoramountofvariations,thusresultinginacom-
mark sketch database created by Rosa´lia et al.[17] who employ
pactdistributionofepitome-scores(shortererrorbars). Incontrast,
a human-evaluation based technique to identify a subset of 160
thevarietyinsketchesofacategorysuchascarrotisreflectedin non-ambiguous2 objectcategoriesfromthe250originallyprovided
thecorrespondingcategory-epitomesandbyextension,inthelonger
byEitzetal[13]. However, asthenumber ofcategories increases,
errorbarsofitsepitome-scores.
visualizingtrends(Figures4-5)becomesachallenge. Therefore,
novel visualizationmethodsneed tobeexplored oralternately, the
5.2. Epitome-scoreasaproxyforsemanticlevelofdetail number of categories needs to be curated in a meaningful manner
forrepresentativesketchanalysis.
A different perspective can be gained by examining categories for
a given value of epitome-score. For each category, we count the 2Rosa´liaetal.[17]defineanon-ambiguoussketchasonewhoseidentity
number of test sketches whose epitome-score exceeds a threshold isagreeduponbyatleast2peopleamongthesubjectssurveyed.
7. REFERENCES ferenceonComputerVisionandPatternRecognition(CVPR),
2003,pp.729–736.
[1] R.HuandJ.Collomosse, “Aperformanceevaluationofgradi-
[9] S. Marvaniya, S. Bhattacharjee, V. Manickavasagam, and
entfieldhogdescriptorforsketchbasedimageretrieval,”Com-
A. Mittal, “Drawing an automatic sketch of deformable ob-
put.Vis.ImageUnderst.(CVIU),vol.117,no.7,pp.790–806,
jectsusing only afew images,” inComputer Vision- ECCV
July2013.
2012.WorkshopsandDemonstrations-Florence,Italy,Octo-
[2] R.Hu,M.Barnard,andJ.Collomosse,“Gradientfielddescrip- ber7-13,2012,Proceedings,PartI,2012,pp.63–72.
torforsketchbasedretrievalandlocalization,” inProceedings
[10] I.Berger, A. Shamir, M. Mahler, E. Carter, and J. Hodgins,
ofthe17thIEEEInternationalConferenceonImageProcess-
“Style and abstraction in portrait sketching,” ACM Trans.
ing(ICIP),Washington,DC,USA,2010,pp.1025–1028,IEEE
Graph.(SIGGRAPH),vol.32,no.4,pp.55:1–55:12,July2013.
ComputerSociety.
[11] A. Karteek, P. S. Lahari, and C. V. Jawahar, “Discriminant
[3] I.Kokkinos, P.Maragos, andA.Yuille, “Bottom-upand top-
substrokesforonlinehandwritingrecognition,”inProceedings
downobjectdetectionusingprimalsketchfeaturesandgraph-
oftheEighthInternationalConferenceonDocumentAnalysis
icalmodels,” inIEEEConf.onComputerVisionandPattern
and Recognition, Washington, DC, USA, 2005, ICDAR ’05,
Recognition(CVPR),2006.
pp.499–505,IEEEComputerSociety.
[4] Y.Qi,J.Guo,Y.Li,H.Zhang,T.Xiang,andY.Song,“Sketch-
ing by perceptual grouping,” in IEEE International Confer- [12] B.Jiang and J. Zhu, “Exploring human sketching process,”
enceonImageProcessing, ICIP2013, Melbourne, Australia, 2013.
September15-18,2013,2013,pp.270–274. [13] M.Eitz,J.Hays,andM.Alexa, “Howdohumanssketchob-
[5] DavidMarr, “Chapter2: Representingtheimage,” inVision: jects?,” ACMTrans.Graph.(Proc.SIGGRAPH),vol.31, no.
AComputationalInvestigationintotheHumanRepresentation 4,pp.44:1–44:10,2012.
and Processing of Visual Information, pp. 54–79. The MIT [14] NavneetDalalandBillTriggs, “Histogramsoforientedgradi-
Press,2010. entsforhumandetection,” inInCVPR,2005,pp.886–893.
[6] C.Guo,S.Zhu,andY.N.Wu, “Towardsamathematicaltheory
[15] FlorentPerronninandChristopherDance, “Fisherkernelson
ofprimalsketchandsketchability,”inProceedingsoftheNinth
visualvocabulariesforimagecategorization,” inComputerVi-
IEEEInternational ConferenceonComputerVision-Volume
sionandPatternRecognition, 2007.CVPR’07.IEEEConfer-
2, Washington, DC, USA, 2003, pp. 1228–, IEEE Computer
enceon.IEEE,2007,pp.1–8.
Society.
[16] L.Maaten andG. Hinton, “Visualizingdatausing t-sne,” J.
[7] R.M. Haralick, L. T. Watson, and T. J. Laffey, “The topo- Mach.Learn.Res.,vol.9,pp.2579–2605,Nov.2008.
graphicprimalsketch,” TheInternationalJournalofRobotics
[17] Rosa´liaG.Schneider andTinneTuytelaars, “Sketchclassifi-
Research,vol.2,no.1,pp.50–72,March1983.
cationandclassification-drivenanalysisusingfishervectors,”
[8] J.Sun,N.Zheng,H.Tao,andH.Y.Shum,“Imagehallucination SIGGRAPHAsia,vol.33,no.6,pp.174:1–174:9,Nov.2014.
withprimal sketchpriors,” inProceedingsoftheIEEECon-
