Table Of ContentDynamic Concept Composition for Zero-Example Event Detection
XiaojunChang1,YiYang1,GuodongLong1,ChengqiZhang1 andAlexanderG.Hauptmann2
1CentreforQuantumComputationandIntelligentSystems,UniversityofTechnologySydney.
2LanguageTechnologiesInstitute,CarnegieMellonUniversity.
{cxj273,yee.i.yang}@gmail.com,{guodong.long,chengqi.zhang}@uts.edu.au,[email protected]
6
Abstract more and more research attention in the computer vision
1
andmultimediacommunities(Chenetal.2014;Changetal.
0 In this paper, we focus on automatically detecting 2015b; Cheng et al. 2014; Lai et al. 2014; Li et al. 2013;
2 events in unconstrained videos without the use of any
Chang et al. 2015c; Ma et al. 2012; Yan et al. 2015a;
n visualtrainingexemplars.Inprinciple,zero-shotlearn- 2015b).
a ingmakesitpossibletotrainaneventdetectionmodel
J based on the assumption that events (e.g. birthday Current state-of-the-art systems for event detection first
party) can be described by multiple mid-level seman- seekacompactrepresentationofthevideousingfeatureex-
4
tic concepts (e.g. “blowing candle”, “birthday cake”). traction and encoding with a pre-trained codebook (Lowe
1
Towards this goal, we first pre-train a bundle of con- 2004; Bay, Tuytelaars, and Gool 2006; Wang and Schmid
] ceptclassifiersusingdatafromothersources.Thenwe 2013). With labeled training data, sophisticated statistical
V evaluatethesemanticcorrelationofeachconceptw.r.t. classifiers,suchassupportvectormachines(SVM),arethen
the event of interest and pick up the relevant concept
C applied on top to yield predictions. With sufficient labeled
classifiers, which are applied on all test videos to get
trainingexamples,thesesystemshaveachievedremarkable
. multiplepredictionscorevectors.Whilemostexisting
s performance in the past (Lai et al. 2014; Sun and Nevatia
c systemscombinethepredictionsoftheconceptclassi-
[ fierswithfixedweights,weproposetolearntheoptimal 2014; Li et al. 2013; Cheng et al. 2014). However, MED
weightsoftheconceptclassifiersforeachtestingvideo facestheseveredata-scarcitychallenge:onlyveryfew,per-
1 byexploringasetofonlineavailablevideoswithfree- haps even none, positive training samples are available for
v formtextdescriptionsoftheircontent.Tovalidatethe some events, and the performance degrades dramatically
9 effectiveness of the proposed approach, we have con- once the number of training samples falls short. Reflect-
7 ducted extensive experiments on the latest TRECVID ing this challenge, the National Institute of Standards and
6 MEDTest2014,MEDTest2013andCCVdataset.The
Technology (NIST) hosts an annual competition on a vari-
3 experimentalresultsconfirmthesuperiorityofthepro-
ety of retrieval tasks, of which the Zero-Exemplar Multi-
0 posedapproach.
. media Event Detection (0Ex MED in short) in TRECVID
1
2013 (TRECVID 2013) and 2014 (TRECVID 2014) has
0
Introduction received considerable attention. Promising progress (Dal-
6
ton, Allan, and Mirajkar 2013; Habibian, van de Sande,
1 In multimedia event detection (MED), a large number of
and Snoek 2013; Habibian, Mensink, and Snoek 2014;
v: unseen videos is presented and the the learning algorithm Changetal.2015a)hasbeenmadeinthisdirection,butfur-
Xi must rank them according to their likelihood of contain- therimprovementisstillanticipated.
ing an event of interest, such as rock climbing or attempt-
Inthispaper,weaimtodetectcomplexeventwithoutany
r ing a bike trick. Compared to traditional recognition of
a labeledtrainingdatafortheeventofinterest.Followingpre-
visual concepts (e.g. actions, scenes, objects, etc.), event
vious work on zero-shot learning (Lampert, Nickisch, and
detection is more challenging for the following reasons.
Harmeling2009;Palatuccietal.2009),weregardanevent
First, an event is a higher level abstraction of video se-
as compositions of multiple mid-level semantic concepts.
quences than a concept and consists of multiple concepts.
Thesesemanticconceptclassifiersaresharedamongevents
For example, an event, say birthday part, can be described
and can be trained using other resources. We then learn a
by multiple concepts (e.g. “birthday cake”, “blowing can-
skip-grammodel(Mikolovetal.2013)toassesstheseman-
dle”, etc.) Second, an event spreads over the entire dura-
tic correlation of the event description and the pre-trained
tion of long videos while a concept can be detected in a
vocabulary of concepts, based on which we automatically
shorter video sequence or even in a single frame. As the
select the most relevant concepts to each event of inter-
firstimportantsteptowardsautomaticcategorization,recog-
est.Thisstepiscarriedoutwithoutanyvisualtrainingdata
nition, search, indexing and retrieval, MED has attracted
at all. Such concept bundle view of event also aligns with
Copyright(cid:13)c 2016,AssociationfortheAdvancementofArtificial thecognitivescienceliterature,wherehumansarefoundto
Intelligence(www.aaai.org).Allrightsreserved. conceive objects as bundles of attributes (Roach and Lloyd
1978). The concept prediction scores on the testing videos datasets(namelyMEDTest2014dataset,MEDTest2013
are combined to obtain a final ranking of the presence of dataset and CCV ), and achieve state-of-the-art perfor-
sub
theeventofinterest.However,mostexistingzero-shotevent mances.
detectionsystemsaggregatethepredictionscoresofthecon-
ceptclassifierswithfixedweights.Obviously,thisassumes RelatedWorks
all the predictions of a concept classifier share the same
Complex event detection on unconstrained web videos has
weightandfailstoconsiderthedifferencesoftheclassifier’s
attractedwideattentioninthefieldofmultimediaandcom-
predictioncapabilityonindividualtestingvideos.Aconcept
putervision.Significantprogresshasbeenmadeinthepast
classifier, in fact, does have different prediction capability
(Lai et al. 2014; Li et al. 2013; Sun and Nevatia 2014). A
ondifferenttestingvideos,wheresomevideosarecorrectly
decent video event detection system usually consists of a
predictedwhileothersarenot.Therefore,insteadofusinga
goodfeatureextractionmoduleandahighlyeffectiveclassi-
fixedweightforeachconceptclassifier,apromisingalterna-
ficationmodule(suchaslargemarginsupportmachinesand
tiveistoestimatethespecificweightforeachtestingvideo
kernel methods). Various low-level features (static, audio,
toalleviatetheindividualpredictionerrorsfromtheimper-
etc.) already achieve good performances under the bag-of-
fectconceptclassifiersandachieverobustdetectionresult.
wordsrepresentation.Furtherimprovementsareobtainedby
Theproblemoflearningspecificweightsofalltheseman-
aggregatingcomplementaryfeaturesinthevideolevel,such
ticconceptclassifiersforeachtestingvideoischallengingin
as coding (Boureau et al. 2010; Perronnin, Sa´nchez, and
thefollowingaspects:Firstly,itisunclearhowtodetermine
Mensink2010)andpooling(Caoetal.2012).Itisobserved
thespecificweightsfortheunlabeledtestingvideosinceno
thatwithenoughlabeledtrainingdata,superbperformance
labelinformationcanbeused.Secondly,togetarobustde-
can be obtained. However, when the number of positive
tectionresult,weneedtomaximallyensurepositivevideos
trainingvideosfallsshort,thedetectionperformancedrops
havehigherscoresthannegativevideosinthefinalranking
dramatically. In this work, we focus on the more challeng-
list.Notethatthegoalofeventdetectionistorankthepos-
ing zero-exemplar setting where no labeled training videos
itivevideosabovenegativeones.Tothisend,weproposeto
fortheeventofinterestareprovided.
learntheoptimalweightsforeachtestingvideobyexploring
Our work is inspired by the general zero-shot learn-
asetofonlineavailablevideoswithfree-formtextdescrip-
ing framework (Lampert, Nickisch, and Harmeling 2009;
tions of their content. Meanwhile, we directly enforce that
Palatucci et al. 2009; Mensink, Gavves, and Snoek 2014),
positivetestingvideoshavethehighestaggregatedscoresin
which arises from practical considerations such as the
thefinalresult.
tremendous cost of acquiring labeled data and the constant
The main building blocks of the proposed approach for
needofdealingwithdynamicandevolvingreal-worldobject
zero-example event detection can be described as follows.
categories. On the event detection side, recent works have
Wefirstrankthesemanticconceptsforeacheventofinterest
begun to explore intermediate semantic concepts (Chang
usingtheskip-grammodel,basedonwhichtherelevantcon-
et al. 2015a), and achieved limited success on the zero-
ceptclassifiersareselected.Thenfollowing(Liuetal.2013;
exemplarsetting(Dalton,Allan,andMirajkar2013;Habib-
Lai et al. 2015), we define the aggregation process as an
ian, van de Sande, and Snoek 2013; Habibian, Mensink,
information propagation procedure which propagates the
andSnoek2014)alsoconsideredselectingmoreinformative
weights learned on individual on-line available videos to
concepts.However,noneoftheseworksconsiderdiscover-
the individual unlabeled testing videos, which enforces vi-
ing the optimal weights of different concept classifiers for
suallysimilarvideoshavesimilaraggregatedscoresandof-
eachindividualtestingvideo.
fersthecapabilitytoinferweightsforthetestingvideos.To
stepfurther,weusetheL norminfinitepushconstraintto
∞ TheProposedApproach
minimize the number of positive videos ranked below the
highest-scorednegativevideos,whichensuresmostpositive In this paper, we focus on the challenging zero-exemplar
videos have higher aggregated scores than negative videos. event detection problem. In a nutshell, we are given a se-
In this way, we learn the optimal weights for each testing quenceofunseentestingvideosandalsotheeventdescrip-
videoandpushpositivevideostorankabovenegativevideos tion, but without any labeled training data for the event of
aspossible. interest.Thegoalistorankthetestingvideossothatpositive
Contributions: To summarize, we make the following videos(thosecontaintheeventofinterest)arerankedabove
contributionsinthiswork: negatives.Withthisgoalinmind,wefirstassociateaquery
eventwithrelatedsemanticconceptsthatarepre-trainedus-
1. Weproposeanovelapproachforzeroexampleeventde-
ingothersources.Thenweaggregatetheindividualconcept
tection to learn the optimal weights of related concept
prediction scores using the proposed dynamic composition
classifiersforeachtestingvideobyexploringasetofon-
approach.
line available videos with free-form text descriptions of
theircontent.
SemanticQueryGeneration
2. InfinitypushSVMhasbeenincorporatedtoensuremost
Ourworkisbuiltupontheobservationthateacheventcanbe
positivevideoshavethehighestaggregatedscoresinthe
described by multiple semantic concepts. For example, the
finalpredictionresults.
marriage proposal event can be attributed to several con-
3. We conduct extensive experiments on three real video cepts, such as “ring” (object), “kissing” (action), “kneeling
down” (action) and “cheering” (acoustic). Since semantic individualtestingvideo,andhenceachieverobustaggrega-
concepts are shared among different events and each con- tionresults.
ceptclassifiercanbetrainedindependentlyusingdatafrom
othersources,zero-exampleeventdetectioncanbeachieved DynamicComposition
bycombiningtherelevantconceptpredictionscores.Differ-
Upuntilnow,wehavel videoswithweaklabelandutest-
entfromthepioneerwork(Lampert,Nickisch,andHarmel-
ing videos. We propose to learn an aggregation function
ing2009),whichlargelyreliesonhumanknowledgetode- f (s ) = wTs for each testing video (i = 1,··· ,l +u),
composeclasses(events)intoattributes(concepts),ourgoal wiheire w =i[wi1,··· ,wm]T is a non-negative aggregation
istoautomaticallyevaluatethesemanticsimilaritybetween i i
weight vector with wj being the aggregation weight of sj.
the event of interest and the concepts, based on which we i i
Clearly, it is straightforward for us to learn the optimal ag-
selecttherelevantconceptsforeachevent.
gregation weights for the videos in the collected set based
Events come with textual side informatin, e.g., an event
on the weak label information. However, it is challenging
name or a short description. For example, the event dog
toderivetheoptimalaggregationweightsfortheunlabeled
show in the TRECVID MEDTest 2014 (TRECVID 2014)
videossincenolabelinformationisavailable.
is defined as “a competitive exhibition of dogs”. With the
To achieve this goal, we build our model based on the
availability of a pre-trained vocabulary of concept classi-
local smoothness property in graph-based semi-supervised
fiers, we can evaluate the semantic correlation between the
learning, which assumes visually similar videos have com-
query event and each individual concepts. Specifically, we
parablelabelswithinalocalregionofthesamplespace(Liu
learnaskip-grammodel(Mikolovetal.2013)usingtheEn-
etal.2013;Laietal.2015).Exploringthelocalconnectivity
glish Wikipedia dump1. The skip-gram model infers a D-
of data is a successful strategy for graph construction. The
dimensional vector space representation by fitting the joint
neighborsofvideox canbedefinedasthek-nearestvideos
probabilityoftheco-occurrenceofsurroundingcontextson i
in the collection to x . In this work, we consider the prob-
largeunstructuredtextdata,andplacessemanticallysimilar i
abilistic neighbors. Following the work in (Nie, Wang, and
wordsneareachotherintheembeddingvectorspace.Thus
Huang2014),welearnthedatasimilaritymatrixbyassign-
itisabletocapturealargenumberofprecisesyntacticand
ingtheadaptiveneighborsforeachvideobasedonthelocal
semanticwordrelationships.Forshortphasesconsistingof
connectivity.
multiplewords(e.g.,eventdescriptions),wesimplyaverage
For the i-th video x , all the videos {x ,x ,...,x }
itsword-vectorrepresentations.Afterproperlynormalizing i 1 2 l+u
can be connected to x as a neighbor with probability a .
therespectiveword-vectors,wecomputethecosinedistance i ij
Usually, if (wTs − wTs )2 is smaller, a larger probabil-
of the event description and all individual concepts, result- i i j j
inginacorrelationvectorw∈[0,1]m,wherew measuresa ityaij shouldbeassignedbetweenthevideoxi andxj.So
k
priorirelevanceofthek-thconceptandtheeventofinterest. a natural method to determine the probabilities aij|jl+=u1 is
Basedontherelevancevector,weselectthemostinforma- solvingthefollowingproblem:
tiveconceptclassifiersforeachevent.
l+u
WeakLabelGeneration min (cid:88)(wTs −wTs )2a , (1)
i i j j ij
According to the NIST standard, we utilize the TRECVID W,aTi1=1,0≤ai≤1i,j=1
MEDresearchdatasettoexploretheoptimalweightsforthe
wheres isavectorwiththej-thelementass .
testingvideos.Allthevideosintheresearchsetcomewitha i ij
However,theproblemEquation(1)hasatrivialsolution,
sentenceofdescription,summarizingthecontentscontained
in the videos. Note that all the videos in the research set only the nearest video can be the neighbor of the video xi
with probability 1 and all the other videos can not be the
has no event-level label information. We further collect a
neighbors of x . On the other hand, if the following prob-
set of videos with free-form text descriptions of their con- i
lemissolvedwithoutinvolvinganydistanceinformationbe-
tent, which are widely available online in websites such as
tweenthevideos:
YouTubeandNetFlix.
Asthedescriptionsofthevideosinbothresearchsetand
onlinewebsiteareverynoisy,weapplystandardnaturallan- l+u
(cid:88)
guage processing (NLP) techniques to clean up the anno- min a2 , (2)
ij
tations, including removal of the common stop words and aTi1=1,0≤ai≤1j=1
stemmingtonormalizewordinflections.
SimilartothestepsofSQG,wemeasurethesemanticcor- the optimal solution is that all the data points can be the
relationbetweenthecleanedsentenceandeachconceptde- neighbors of the video xi with the same probability l+1u,
scription,anduseitasaweaklabelforeachconcept.Inthe whichcanbeseenasapriorintheneighborassignment.
nextsection,wewilllearntheoptimalaggregationweights Combining Equation (1) and Equation (2), we can solve
fortheindividualtestingvideobyexploitingthesupervision thefollowingproblem:
informationwithweaklabel,whichaccountsforthediffer-
l+u
ences in the concept classifiers’ prediction abilities on the (cid:88)
min (wTs −wTs )2a +γa2 . (3)
i i j j ij ij
1http://dumps.wikimedia.org/enwiki/ W,aTi1=1,0≤ai≤1j=1
The second term in Equation (3) is a regularization and Experiments
γ is the regularization parameter. Denote ds = (wTs −
ij i i In this section, we conduct extensive experiments to vali-
wjTsj)2, the problem Equation (3) can be written in vector datetheproposedDynamicConceptCompositionforzero-
formas: exemplareventdetectiontask,abbreviatedasDCC.
1 ExperimentSetup
min (cid:107)a + ds(cid:107)2. (4)
aTi1=1,0≤ai≤1 i 2γ i 2 Dataset: To evaluate the effectiveness of the proposed ap-
Itiseasytoverifythisproblemcanbesolvedwithaclosed proach,weconductextensiveexperimentsonthefollowing
formsolution. threelarge-scaleeventdetectiondatasets:
Tostepfurther,weincorporateaninfinitepushlossfunc- — TRECVID MEDTest 2014 dataset (TRECVID 2014):
tion (Rakotomamonjy 2012) to achieve robust aggregation ThisdatasethasbeenintroducedbytheNISTforallpar-
result.Thegoalofinfinitepushlossfunctionistominimize ticipantsintheTRECVIDcompetitionandresearchcom-
the number of positive videos which are ranked below the munity to perform experiments on. There are in total 20
highestscorednegativevideos.Theinfinitepushlossfunc- events, whose description can be found in (TRECVID
tion has shown promising performance for event detection 2014).WeusetheofficialtestsplitreleasedbytheNIST,
problemin(Changetal.2015c).Infact,thenumberofposi- and strictly follow its standard procedure (TRECVID
tivevideosrankedbelowthehighestscorednegativevideos 2014). To be more specific, we detect each event sep-
equals to the maximum number of positive videos ranked arately, treating each of them as a binary classifica-
belowanynegativevideos.Hence,wedefineitasfollows: tion/rankingproblem.
— TRECVID MEDTest 2013 dataset (TRECVID 2013):
1 (cid:88)
(cid:96)({fi}li=1;P,N)=mj∈aNx(pi∈PIfi(s+i)<fj(s−j)), (5) TMhEeDsTeetstitn2g0s14o,fwMithE1D0Toesftth2e0ir1230deavteanstestoivserslaimppilianrg.to
where I is the indicator function whose value is 1 if — Columbia Consumer Video dataset (Jiang et al. 2011):
fi(s+i ) < fj(s−j ) and 0 otherwise. The maximum operator The official Columbia Consumer Video dataset contains
over j equals to calculating the l∞-norm of a vector con- 9,317 videos in 20 different categories, including scenes
sistingofnentries,eachofwhichcorrespondstoonevalue like “beach”, objects like “cat”, and events like “basket-
basedonj intheparenthesesofEquation(5).Byminimiz- ball”and“parade”.Sincethegoalofthisworkistosearch
ing this penalty, positive videos tend to score higher than complexevents,weonlyusethe15eventcategories.
anynegativevideos.Thisessentiallyensurespositivevideos
AccordingtothestandardoftheNIST,eacheventisde-
have higher combined scores than the negatives, leading to
tected separately and the performance of event detection is
moreaccuratecombinedresults.
evaluatedusingthemeanAveragePrecision(mAP).
For computational tractability we upper bound the dis-
Concept Detectors: 3,135 concept detectors are pre-
crete0-1lossI(δ < 0)bytheconvexhingeloss(1−δ) ,
+ trainedusingTRECVIDSINdataset(346categories)(Over
whereasusual(δ) :=max(δ,0)isthepositivepart.Since
+ et al. 2014; Jiang et al. 2014), Google sports (478 cate-
weusuallyparmoreattention,ifnotexclusively,tothetop
gories) (Karpathy et al. 2014; Jiang et al. 2014), UCF101
oftheranklist,wefocusonminimizingthemaximumrank-
dataset (101 categories) (Soomro, Zamir, and Shah 2012;
ingerroramongallnegativeexemplarsj ∈N:
Jiang et al. 2014), YFCC dataset (609 categories) (YFC ;
Jiang et al. 2014) and DIY dataset (1601 categories) (Yu,
1 (cid:88)
(cid:96)({f }l ;P,N)=max( (1−(wTs+−wTs−)) ), Jiang, and Hauptmann 2014; Jiang et al. 2014). The im-
i i=1 j∈N p i i j j + proveddensetrajectoryfeatures(includingtrajectory,HOG,
i∈P
(6) HOFandMBH)arefirstextractedusingthecodeof(Wang
and Schmid 2013) and encode them with the Fisher vec-
Finally,theobjectivefunctioncanbewrittenas:
torrepresentation(Perronnin,Sa´nchez,andMensink2010).
Following(WangandSchmid2013),thedimensionofeach
l+u
(cid:88) descriptorisfirstreducedbyafactorof2andthenuse256
min (wTs −wTs )2a +γa2
i i j j ij ij componentstogeneratetheFishervectors.Then,ontopof
W,aTi1=1,0≤ai≤1j=1 the extracted low-level features, the cascade SVM (Graf et
+λmax(1 (cid:88)(1−(wTs+−wTs−)) ), (7) al.2004)istrainedforeachconceptdetector.
j∈N p i i j j + Competitors: We compare the proposed approach with
i∈P
thefollowingalternatives:1).Prim(Habibian,Mensink,and
s.t.w ≥0,i=1,··· ,l+u.
i Snoek2014):Primitiveconcepts,separatelytrained.2).Sel
Since we are working on large-scale video event detec- (Mazloom et al. 2013): A subset of primitive concepts that
tion, there is a great challenge in the computation cost. are more informative for each event. 3). Bi (Rastegari et
Thankstotheproximalmap(Yu2013),weemploythefaster al.2013):Bi-conceptsdiscoveredin(Rastegarietal.2013).
ADMMproposedin(Changetal.2015c)forefficientsolu- 4).OR(Habibian,Mensink,andSnoek2014):BooleanOR
tion.Forspacelimitation,weomitthedetailedoptimization combinationsofPrimconcepts.5).Fu(Habibian,Mensink,
andwillincludeinourjournalversion. andSnoek2014):BooleanAND/ORcombinationsofPrim
Table1:Experimentresultsfor0ExeventdetectiononMEDTest2014,MEDTest2013,andCCV .Meanaverageprecision
Sub
(mAP),inpercentages,isusedastheevaluationmetric.LargermAPindicatesbetterperformance.
abcd MEDTest2014
ID Prim Sel Bi OR Fu Bor PCF DCC
E021 2.12 2.98 2.64 3.89 3.97 3.12 4.64 6.37
E022 0.75 0.97 0.83 1.36 1.49 1.15 1.48 2.85
E023 33.86 36.94 35.23 39.18 40.87 38.68 41.78 44.26
E024 2.64 3.75 3.02 4.66 4.92 4.11 4.87 6.12
E025 0.54 0.76 0.62 0.97 1.39 0.84 1.01 1.26
E026 0.96 1.59 1.32 2.41 2.96 1.96 2.65 4.23
E027 11.21 13.64 12.48 15.93 16.26 15.12 16.47 19.63
E028 0.79 0.67 1.06 1.57 1.95 1.72 2.25 4.04
E029 8.43 10.68 12.21 14.01 14.85 13.19 14.75 17.69
E030 0.35 0.63 0.48 0.91 0.96 0.36 0.48 0.52
E031 32.78 53.19 45.87 69.52 69.66 67.49 72.64 77.45
E032 3.12 5.88 4.37 8.12 8.45 7.54 8.65 11.38
E033 15.25 20.19 18.54 22.14 22.23 21.53 23.26 26.64
E034 0.28 0.47 0.41 0.71 0.75 0.53 0.76 0.94
E035 9.26 13.28 11.09 16.53 16.68 15.82 18.65 21.78
E036 1.87 2.63 2.14 3.15 3.39 2.88 3.76 5.47
E037 2.16 4.52 3.81 6.84 6.88 5.42 6.83 8.45
E038 0.66 0.74 0.58 0.99 1.16 0.85 1.12 2.89
E039 0.36 0.57 0.42 0.69 0.77 0.64 0.85 2.26
E040 0.65 0.98 0.72 1.57 1.57 1.24 1.76 3.12
mean 6.40 9.55 7.89 10.76 11.05 10.21 11.44 13.37
MEDTest2013
mean 7.07 7.94 6.92 9.45 9.88 8.43 9.96 12.64
CCV
sub
mean 19.05 19.40 20.25 21.16 21.89 23.08 23.87 24.36
concepts, w/o concept refinement. 6). Bor: The Borda rank showninTable1:1).SelsignificantlyoutperformPrimwith
aggregationwithequalweightsonthediscoveredsemantic mAP of 9.55% vs 6.40% on MEDTest 2014, which indi-
concepts.7).PCF(Changetal.2015a):Thepair-comparison catesthatselectingthemostdiscriminativeconceptsgener-
frameworkisincorporatedforzero-shoteventdetection. ally improves detection performance than naively using all
the concepts. 2). Comparing the results of Bi, OR, Fu and
Zero-exemplareventdetection Bor, we verify that the selected informative concept clas-
We report the full experimental results on the TRECVID sifiers are not equally important for event detection. It is
MEDTest2014datasetinTable1andalsoasummaryonthe beneficial to derivate the weights for each concept classi-
MEDTest2013datasetandCCV .Fromtheexperimental fier. By treating the concept classifiers differentially, better
sub
resultsshowninTable1,theproposedalgorithm,DCC,per- performanceisachieved.3).ComparingtheresultsofDCC
forms better than the other approaches with a large margin withtheotheralternatives,weobservethatlearningoptimal
(13.37% vs 11.44% achieved by PCF). The proposed ap- weights of concept classifiers for each testing video signif-
proach gets significant improvement on some vents, such icantlyimprovesperformanceofeventdetection.Thiscon-
as Dog Show (E023), Rock Climbing (E027), Beekeeping firmstheimportanceofdynamicconceptcomposition.
(E031) and Non-motorized Vehicle Repair (E033). By an- We made similar observations from the results on the
alyzing the discovered concepts for these events, we find TRECVIDMEDTest2013datasetandCCVsubdataset.
thattheirclassifiersareverydiscriminativeandreliable.For
Extensiontofew-exemplareventdetection
example, for the event Rock Climbing, we discovered the
concepts “Sport climbing”, “Person climbing” and “Boul- Theproposedzero-exampleeventdetectionframeworkcan
dering”, which are the most informative concepts for Rock alsobeusedforfew-exemplareventdetection:weaggregate
climbing in the concept vocabulary. Figure 1 illustrates the theconceptclassifiersandthesupervisedclassifierusingthe
top retrieved results on the non-motorized vehicle repair dynamicconceptcompositionapproach.Inthissection,ex-
event.Tosavespace,weonlyshowthetop4comparedal- periments are conducted to demonstrate the benefit of this
gorithms(OR,Fu,PCFandDCC).Itisclearthatvideosre- hybridapproach.Table2summarizesthemAPonboththe
trievedbytheproposedDCCaremoreaccurateandvisually MEDTest2014and2013datasets,whileFigure2compares
coherent. theperformanceevent-wise.
We make the following observations from the results According to the NIST standard, we consider the 10 Ex
Figure1:Toprankedvideosfortheeventnon-motorizedvehiclerepair.Fromtoptobelow:OR,Fu,PCFandDCC.True/false
labels(providedbyNIST)aremarkedinthelower-rightofeachframe.
Figure2:PerformancecomparisonofIDT,DCC,andthehybridofIDTandDCC.
Table2:Few-exemplarresultsonMED14andMED13. example event detection result. This improves the perfor-
DCC(0Ex) 13.37 12.64 mance from 13.92% to 16.98% on the MEDTest 2014
IDT(10Ex) 13.92 18.08 dataset. As expected, the gain obtained from such simple
DCC(0Ex)+IDT(10Ex) 16.37 19.24 hybriddiminisheswhencombiningwithmoresophisticated
methods.Overall,theresultsclearlydemonstratetheutility
setting, where 10 positive videos are given for each event ofourframeworkeveninthefew-exemplarsetting.
of interest. The improved dense trajectories feature (Wang
Conclusions
and Schmid 2013) is extracted, on top of which an SVM
To address the challenging task of zero-exemplar or few-
classifier is trained. It is worthwhile to note that our DCC
exemplareventdetection,weproposedtolearntheoptimal
which had no labeled training data can get comparable re-
weights for each testing video by exploring the collected
sultswiththesupervisedclassifier(mAP13.37%vs13.92%
videos from other sources. Data-driven word embedding
on MEDTest 2014). This demonstrates that proper aggre-
models were used to seek the relevance of the concepts to
gation with optimal weights for each testing video can get
theeventofinterest.Tofurtherderivatetheoptimalweights
promisingresultsforeventdetection.
of the concept classifiers for each testing video, we have
WecompareDCCwithIDTevent-wiseinFigure2.From
proposed a novel dynamic concept composition method by
the results we can see that our DCC outperforms the su-
exploiting the textual information of the collected videos.
pervisedIDTonmultipleevents,namelyE023,E024,E031
Extensive experiments are conducted on three real video
and E033. To be specific, on the event Beekeeping (E031),
datasets.Theexperimentalresultsconfirmtheefficiencyof
DCC significantly outperform the supervised IDT (77.45%
theproposedapproach.
vs33.92%).Thisisnotsupervising,sinceDCCsignificantly
benefitedfromthepresenceofinformativeandreliablecon-
Acknowledgment
cepts such as “apiary bee house” and “honeycomb” on the
particularBeekeepingevent. This paper was partially supported by the ARC DECRA
Finally we combine DCC with IDT to get the final few- projectDE130101311,partiallysupportedbytheARCdis-
covery project DP150103008, partially supported by the Liu,D.;Lai,K.;Ye,G.;Chen,M.;andChang,S. 2013. Sample-
US Department of Defense, U. S. Army Research Office specificlatefusionforvisualcategoryrecognition. InCVPR.
(W911NF-13-1-0277)andbytheNationalScienceFounda- Lowe,D.G. 2004. Distinctiveimagefeaturesfromscale-invariant
tionunderGrantNo.IIS-1251187.TheU.S.Governmentis keypoints. International Journal of Computer Vision 60(2):91–
authorized to reproduce and distribute reprints for Govern- 110.
mental purposes notwithstanding any copyright annotation Ma,Z.;Yang,Y.;Cai,Y.;Sebe,N.;andHauptmann,A.G. 2012.
thereon. Knowledgeadaptationforadhocmultimediaeventdetectionwith
fewexemplars. InACMMM.
References Mazloom,M.;Gavves,E.;vandeSande,K.E.A.;andSnoek,C.
2013. Searchinginformativeconceptbanksforvideoeventdetec-
Bay,H.;Tuytelaars,T.;andGool,L.J.V. 2006. SURF:speeded
tion. InICMR.
uprobustfeatures. InECCV.
Mensink, T.; Gavves, E.; and Snoek, C. G. M. 2014. COSTA:
Boureau,Y.;Bach,F.;LeCun,Y.;andPonce,J. 2010. Learning
co-occurrencestatisticsforzero-shotclassification. InCVPR.
mid-levelfeaturesforrecognition. InCVPR.
Mikolov,T.;Sutskever,I.;Chen,K.;Corrado,G.S.;andDean,J.
Cao,L.;Mu,Y.;Natsev,A.;Chang,S.;Hua,G.;andSmith,J.R.
2013. Distributedrepresentationsofwordsandphrasesandtheir
2012. Scene aligned pooling for complex video recognition. In
compositionality. InNIPS.
ECCV,688–701.
Nie,F.;Wang,X.;andHuang,H. 2014. Clusteringandprojected
Chang, X.; Yang, Y.; Hauptmann, A. G.; Xing, E. P.; and Yu, Y. clusteringwithadaptiveneighbors. InKDD.
2015a. Semanticconceptdiscoveryforlarge-scalezero-shotevent
Over,P.;Awad,G.;Michel,M.;Fiscus,J.;Sanders,G.;Kraaij,W.;
detection. InIJCAI.
Smeaton,A.F.;andQueenot,G. 2014. Trecvid2014 anoverview
Chang,X.;Yang,Y.;Xing,E.P.;andYu,Y.-L. 2015b. Complex of the goals, tasks, data, evaluation mechanisms and metrics. In
eventdetectionusingsemanticsaliencyandnearly-isotonicSVM. TRECVID.
InICML.
Palatucci, M.; Pomerleau, D.; Hinton, G. E.; and Mitchell, T. M.
Chang,X.;Yu,Y.;Yang,Y.;andHauptmann,A.G.2015c.Search- 2009. Zero-shotlearningwithsemanticoutputcodes. InNIPS.
ing persuasively: Joint event detection and evidence recounting
Perronnin,F.;Sa´nchez,J.;andMensink,T. 2010. Improvingthe
withlimitedsupervision. InACMMM.
Fisherkernelforlarge-scaleimageclassification. InECCV.
Chen,J.;Cui,Y.;Ye,G.;Liu,D.;andChang,S.2014.Event-driven
Rakotomamonjy,A. 2012. Sparsesupportvectorinfinitepush. In
semantic concept discovery by exploiting weakly tagged internet
ICML.
images. InICMR.
Rastegari,M.;Diba,A.;Parikh,D.;andFarhadi,A. 2013. Multi-
Cheng,Y.;Fan,Q.;Pankanti,S.;andChoudhary,A.N.2014.Tem-
attributequeries:Tomergeornottomerge? InCVPR.
poralsequencemodelingforvideoeventdetection. InCVPR.
Roach,E.,andLloyd,B.B. 1978. Cognitionandcategorization.
Dalton, J.; Allan, J.; and Mirajkar, P. 2013. Zero-shot video re-
Soomro,K.;Zamir,A.R.;andShah,M.2012.UCF101:Adataset
trievalusingcontentandconcepts. InCIKM.
of101humanactionsclassesfromvideosinthewild.
Graf, H. P.; Cosatto, E.; Bottou, L.; Durdanovic, I.; and Vapnik,
Sun,C.,andNevatia,R. 2014. DISCOVER:discoveringimpor-
V. 2004. Parallelsupportvectormachines:ThecascadeSVM. In
tantsegmentsforclassificationofvideoeventsandrecounting. In
NIPS.
CVPR.
Habibian,A.;Mensink,T.;andSnoek,C.G.M. 2014. Composite
TRECVID. 2013. Multimedia event detection.
conceptdiscoveryforzero-shotvideoeventdetection. InICMR.
http://www.nist.gov/itl/iad/mig/med13.cfm.
Habibian,A.;vandeSande,K.E.A.;andSnoek,C.G.M. 2013.
TRECVID. 2014. Multimedia event detection.
Recommendations for video event recognition using concept vo-
http://www.nist.gov/itl/iad/mig/med14.cfm.
cabularies. InICMR.
Wang,H.,andSchmid,C.2013.Actionrecognitionwithimproved
Jiang,Y.;Ye,G.;Chang,S.;Ellis,D.P.W.;andLoui,A.C. 2011.
trajectories. InICCV.
Consumervideounderstanding:abenchmarkdatabaseandaneval-
uationofhumanandmachineperformance. InICMR. Yan,Y.;Yang,Y.;Meng,D.;Liu,G.;Tong,W.;Hauptmann,A.G.;
andSebe,N. 2015a. Eventorienteddictionarylearningforcom-
Jiang, L.; Meng, D.; Yu, S.; Lan, Z.; Shan, S.; and Hauptmann,
plex event detection. IEEE Transactions on Image Processing
A.G. 2014. Self-pacedlearningwithdiversity. InNIPS.
24(6):1867–1878.
Karpathy,A.;Toderici,G.;Shetty,S.;Leung,T.;Sukthankar,R.;
Yan,Y.;Yang,Y.;Shen,H.;Meng,D.;Liu,G.;Hauptmann,A.G.;
andFei-Fei,L. 2014. Large-scalevideoclassificationwithconvo-
andSebe,N. 2015b. Complexeventdetectionviaeventoriented
lutionalneuralnetworks. InCVPR.
dictionarylearning. InAAAI.
Lai, K.; Yu, F. X.; Chen, M.; and Chang, S. 2014. Video event The YFCC dataset. http://webscope.sandbox.yahoo.
detectionbyinferringtemporalinstancelabels. InCVPR. com/catalog.php?datatype=i&did=67.
Lai,K.;Liu,D.;Chang,S.;andChen,M. 2015. Learningsam-
Yu,S.;Jiang,L.;andHauptmann,A.G.2014.Instructionalvideos
plespecificweightsforlatefusion. IEEETransactionsonImage
for unsupervised harvesting and learning of action examples. In
Processing24(9):2772–2783. ACMMM.
Lampert,C.H.;Nickisch,H.;andHarmeling,S. 2009. Learning Yu,Y. 2013. Ondecomposingtheproximalmap. InNIPS.
todetectunseenobjectclassesbybetween-classattributetransfer.
InCVPR.
Li,W.;Yu,Q.;Divakaran,A.;andVasconcelos,N.2013.Dynamic
poolingforcomplexeventrecognition. InICCV.
