Table Of ContentSocio-Affective Computing 6
Rajiv Shah
Roger Zimmermann
Multimodal Analysis
of User-Generated
Multimedia Content
Socio-Affective Computing
Volume 6
SeriesEditor
AmirHussain,UniversityofStirling,Stirling,UK
Co-Editor
ErikCambria,NanyangTechnologicalUniversity,Singapore
This exciting Book Series aims to publish state-of-the-art research on socially
intelligent, affective and multimodal human-machine interaction and systems.
It will emphasize the role of affect in social interactions and the humanistic side
of affective computing by promoting publications at the cross-roads between
engineering and human sciences (including biological, social and cultural aspects
of human life). Three broad domains of social and affective computing will be
covered by the book series: (1) social computing, (2) affective computing, and
(3) interplay of the first two domains (for example, augmenting social interaction
through affective computing). Examples of the first domain will include but not
limitedto:alltypesofsocialinteractionsthatcontributetothemeaning,interestand
richnessofourdailylife,forexample,informationproducedbyagroupofpeople
used to provide or enhance the functioning of a system. Examples of the second
domainwillinclude,butnotlimitedto:computationalandpsychologicalmodelsof
emotions, bodily manifestations of affect (facial expressions, posture, behavior,
physiology), and affective interfaces and applications (dialogue systems, games,
learning etc.). This series will publish works of the highest quality that advance
the understanding and practical application of social and affective computing
techniques. Research monographs, introductory and advanced level textbooks,
volumeeditionsandproceedingswillbeconsidered.
Moreinformationaboutthisseriesathttp://www.springer.com/series/13199
Rajiv Shah • Roger Zimmermann
Multimodal Analysis
of User-Generated
Multimedia Content
RajivShah RogerZimmermann
SchoolofComputing SchoolofComputing
NationalUniversityofSingapore NationalUniversityofSingapore
Singapore,Singapore Singapore,Singapore
ISSN2509-5706 ISSN2509-5714 (electronic)
Socio-AffectiveComputing
ISBN978-3-319-61806-7 ISBN978-3-319-61807-4 (eBook)
DOI10.1007/978-3-319-61807-4
LibraryofCongressControlNumber:2017947053
©TheEditor(s)(ifapplicable)andTheAuthor(s)2017
Thisworkissubjecttocopyright.AllrightsarereservedbythePublisher,whetherthewholeorpartof
the material is concerned, specifically the rights of translation, reprinting, reuse of illustrations,
recitation, broadcasting, reproduction on microfilms or in any other physical way, and transmission
or information storage and retrieval, electronic adaptation, computer software, or by similar or
dissimilarmethodologynowknownorhereafterdeveloped.
The use of general descriptive names, registered names, trademarks, service marks, etc. in this
publicationdoesnotimply,evenintheabsenceofaspecificstatement,thatsuchnamesareexempt
fromtherelevantprotectivelawsandregulationsandthereforefreeforgeneraluse.
Thepublisher,theauthorsandtheeditorsaresafetoassumethattheadviceandinformationinthis
book are believed to be true and accurate at the date of publication. Neither the publisher nor the
authors or the editors give a warranty, express or implied, with respect to the material contained
hereinor for anyerrors oromissionsthat may havebeenmade. Thepublisher remainsneutralwith
regardtojurisdictionalclaimsinpublishedmapsandinstitutionalaffiliations.
Printedonacid-freepaper
ThisSpringerimprintispublishedbySpringerNature
TheregisteredcompanyisSpringerInternationalPublishingAG
Theregisteredcompanyaddressis:Gewerbestrasse11,6330Cham,Switzerland
I dedicate this book to my late father,
Ram Dhani Gupta;
my mother, Girija Devi; and my other family
members for their
continuous support, motivation, and
unconditional love.
I love you all so dearly.
Foreword
We have stepped into an era where every user plays the role of both content
providerandcontentconsumer.Withmanysmartphoneappsseamlesslyconverting
photographs and videos to social media postings, user-generated multimedia con-
tent now becomes the next big data waiting to be turned into useful insights and
applications. The book Multimodal Analysis of User-Generated Multimedia Con-
tent by Rajiv and Roger very carefully selects a few important research topics in
analysingbiguser-generatedmultimediadatainamultimodalapproachpertinentto
manynovelapplicationssuchascontent recommendation, contentsummarization
andcontentuploading.Whatmakesthisbookstandoutamongothersistheunique
focusonmultimodalanalysiswhichcombinesvisual,textualandothercontextual
featuresofmultimediacontenttoperformbettersensemaking.
RajivandRogerhavemadethebookagreatresourceforanyreaderinterestedin
the above research topics and respective solutions. The literature review chapter
givesaverydetailedandcomprehensivecoverageofeachtopicandcomparisonof
state-of-the-artmethodsincludingtheonesproposedbytheauthors.Everychapter
thatfollowsisdedicatedtoaresearchtopiccoveringthearchitectureframeworkof
a proposed solution system and its function components. This is accompanied by
afine-graineddescriptionofthemethodsusedinthefunctioncomponents.Toaid
understanding, the description comes with many relevant examples. Beyond
describing the methods, the authors also present the performance evaluation of
these methods using real-world datasets so as to assess their strengths and weak-
nessesappropriately.
Despiteitsdeeptechnicalcontent,thebookissurprisinglyeasytoread.Ibelieve
the authors have paid extra attention to organizing the content for easy reading,
careful proof editing and good use of figures and examples. The book is clearly
written at the level suitable for reading by computer science students in graduate
and senior years. It is also a good reference reading for multimedia content
analytics researchers in both academia and industry. Whenever appropriate, the
authors show their algorithms with clearly defined input, output and steps with
vii
viii Foreword
comments.Thisfacilitatesanyfurtherimplementationaswellasextensionsofthe
methods.Thisisperhapsthepartofthebookwhichwillattractthe“programming-
type”readersmost.
I would like to congratulate both Rajiv and Roger for their pioneering work in
multimodalanalysisforuser-generatedmultimediacontent.Ibelievethisbookwill
becomewidelyadoptedandreferencedinthemultimediacommunity.Itisagood
guide for anyone who wishes to better understand the challenges and solutions of
analysing multimedia data. I wish the authors all the best in their future research
endeavour.
SchoolofInformationSystems, Ee-PengLim,PhD
SingaporeManagementUniversity,
Singapore,Singapore
May2017
Preface
Theamountofuser-generatedmultimediacontent(UGC)hasincreasedrapidlyin
recentyearsduetotheubiquitousavailabilityofsmartphones,digitalcameras,and
affordablenetworkinfrastructures.Aninterestingrecenttrendisthatsocialmedia
websites such as Flickr and YouTube create opportunities for users to generate
multimedia content, instead of creating multimedia content by themselves. Thus,
capturing UGC such as user-generated images (UGIs) and user-generated videos
(UGVs) anytime and anywhere, and then instantly sharing them on social media
platforms such as Instagram and Flickr, have become a very popular activity.
Hence,user-generatedmultimediacontentisnowanintrinsicpartofsocialmedia
platforms.Tobenefitusersandsocialmediacompaniesfromanautomaticseman-
tics and sentics understanding of UGC, this book focuses on developing effective
algorithms for several significant social media analytics problems. Sentics are
common affective patterns associated with natural language concepts exploited
for tasks such as emotion recognition from text/speech or sentiment analysis.
Knowledge structures derived from the semantics and sentics understanding of
user-generatedmultimediacontentarebeneficialinanefficientmultimediasearch,
retrieval, and recommendation. However, real-world UGC is complex, and
extractingthesemanticsandsenticsfromonlymultimediacontentisverydifficult
becausesuitableconceptsmaypresentindifferentrepresentations.Moreover,due
totheincreasingpopularityofsocialmediawebsitesandadvancementsintechnol-
ogy, it is possible now to collect a significant amount of important contextual
information (e.g., spatial,temporal, and preference information). Thus,it necessi-
tates analyzing the information of UGC from multiple modalities for a better
semantics and sentics understanding. Moreover, the multimodal information is
very useful in a social network based news video reporting task (e.g., citizen
journalism) which allows people to play active roles in the process of collecting
news reports (e.g., CNN iReport). Specifically, we exploit both content and con-
textualinformationofUGIsandUGVstofacilitatedifferentmultimediaanalytics
problems.
ix
x Preface
Further advancements in technology enable mobile devices to collect a signif-
icant amount of contextual information in conjunction with captured multimedia
content.Sincethecontextualinformationgreatlyhelpsinthesemanticsandsentics
understandingofuser-generatedmultimediacontent,researchersexploititintheir
research work related to multimedia analytics problems. Thus, the multimodal
information(i.e.,bothcontentandcontextualinformation)ofUGCbenefitsseveral
diverse social media analytics problems. For instance, knowledge structures
extracted from multiple modalities are useful in an effective multimedia search,
retrieval, and recommendation. Specifically, applications related to multimedia
summarization, tag ranking and recommendation, preference-aware multimedia
recommendation, multimedia-based e-learning, and news video reporting are
built by exploiting the multimedia content (e.g., visual content) and associated
contextualinformation(e.g.,geo-,temporal,andothersensorydata).However,itis
verychallengingtoaddresstheseproblemsefficientlyduetothefollowingreasons:
(i) difficulty in capturing the semantics of UGC, (ii) the existence of noisy meta-
data,(iii)difficultyinhandlingbigdatasets, (iv)difficultyinlearninguserprefer-
ences, (v) the insufficient accessibility and searchability of video content, and
(vi) weak network infrastructures at some locations. Since different information
knowledge structures are derived from different sources, it is useful to exploit
multimodalinformationtoovercomethesechallenges.
Exploiting information from multiple sources helps in addressing challenges
mentioned above and facilitating different social media analytics applications.
Therefore, in this book, we leverage information from multiple modalities and
fuse the derived knowledge structures to provide effective solutions for several
significantsocialmediaanalyticsproblems.Ourresearchfocusesonthesemantics
and sentics understanding of UGC leveraging both content and contextual infor-
mation.First,forabetterunderstandingofaneventfromalargecollectionofUGIs,
wepresenttheEventBuildersystem.Itenablespeopletoautomaticallygeneratea
summary of the event in real-time by visualizing different social media such as
WikipediaandFlickr.Inparticular,weexploitWikipediaastheeventbackground
knowledgetoobtainmorecontextualinformationabouttheevent.Thisinformation
is very useful in an effective event detection. Next, we solve an optimization
problem to produce text summaries for the event. Subsequently, we present the
EventSensor system that aims to address sentics understanding and produces a
multimediasummaryforagivenmood.Itextractsconceptsandmoodtagsfromthe
visual content and textual metadata of UGCs and exploits them in supporting
several significant multimedia analytics problems such as a musical multimedia
summary.EventSensorsupportssentics-based eventsummarizationby leveraging
EventBuilderasitssemanticsenginecomponent.Moreover,wefocusoncomput-
ingtagrelevanceforUGIs.Specifically,weleveragepersonalandsocialcontexts
ofUGIsandfollowaneighborvotingschemetopredictandranktags.Furthermore,
we focus on semantics and sentics understanding from UGVs since they have a
significant impact on different areas of a society (e.g., enjoyment, education, and
journalism).
