Table Of ContentMichail Giannakos · Daniel Spikol ·
Daniele Di Mitri · Kshitij Sharma ·
Xavier Ochoa · Rawad Hammad Editors
The Multimodal
Learning
Analytics
Handbook
The Multimodal Learning Analytics Handbook
Michail Giannakos • Daniel Spikol (cid:129)
Daniele Di Mitri (cid:129) Kshitij Sharma (cid:129)
Xavier Ochoa (cid:129) Rawad Hammad
Editors
The Multimodal Learning
Analytics Handbook
Editors
MichailGiannakos DanielSpikol
NorwegianUniversityofScienceand UniversityofCopenhagen
Technology Copenhagen,Denmark
Trondheim,Norway
KshitijSharma
DanieleDiMitri NorwegianUniversityofScienceand
DIPF,FrankfurtamMain Technology
Germany Trondheim,Norway
XavierOchoa RawadHammad
NewYorkUniversity UniversityofEastLondon
NewYork,NY,USA London,UK
ISBN978-3-031-08075-3 ISBN978-3-031-08076-0 (eBook)
https://doi.org/10.1007/978-3-031-08076-0
©TheEditor(s)(ifapplicable)andTheAuthor(s),underexclusivelicensetoSpringerNatureSwitzerland
AG2022
Chapter11islicensedunderthetermsoftheCreativeCommonsAttribution4.0InternationalLicense
(http://creativecommons.org/licenses/by/4.0/).Forfurtherdetailsseelicenseinformationinthechapter.
Thisworkissubjecttocopyright.AllrightsaresolelyandexclusivelylicensedbythePublisher,whether
thewholeorpartofthematerialisconcerned,specificallytherightsoftranslation,reprinting,reuse
ofillustrations,recitation,broadcasting,reproductiononmicrofilmsorinanyotherphysicalway,and
transmissionorinformationstorageandretrieval,electronicadaptation,computersoftware,orbysimilar
ordissimilarmethodologynowknownorhereafterdeveloped.
Theuseofgeneraldescriptivenames,registerednames,trademarks,servicemarks,etc.inthispublication
doesnotimply,evenintheabsenceofaspecificstatement,thatsuchnamesareexemptfromtherelevant
protectivelawsandregulationsandthereforefreeforgeneraluse.
Thepublisher,theauthors,andtheeditorsaresafetoassumethattheadviceandinformationinthisbook
arebelievedtobetrueandaccurateatthedateofpublication.Neitherthepublishernortheauthorsor
theeditorsgiveawarranty,expressedorimplied,withrespecttothematerialcontainedhereinorforany
errorsoromissionsthatmayhavebeenmade.Thepublisherremainsneutralwithregardtojurisdictional
claimsinpublishedmapsandinstitutionalaffiliations.
ThisSpringerimprintispublishedbytheregisteredcompanySpringerNatureSwitzerlandAG
Theregisteredcompanyaddressis:Gewerbestrasse11,6330Cham,Switzerland
Contents
PartI IntroductiontoMMLA
IntroductiontoMultimodalLearningAnalytics ............................. 3
MichailGiannakos,DanielSpikol,DanieleDiMitri,KshitijSharma,
XavierOchoa,andRawadHammad
PartII MMLAforDesign
MultimodalLearningAnalyticsandtheDesignofLearningSpaces....... 31
Milica Vujovic, Davinia Hernández-Leo, Roberto
Martinez-Maldonado,MutluCukurova,andDanielSpikol
PartIII MMLAforFeedbackandRegulation
MultimodalSystemsforAutomatedOralPresentationFeedback:
AComparativeAnalysis......................................................... 53
XavierOchoa
ModelingtheComplexInterplayBetweenMonitoringEventsfor
RegulatedLearningwithPsychologicalNetworks ........................... 79
Jonna Malmberg, Mohammed Saqr, Hanna Järvenoja, Eetu Haataja,
HéctorJ.Pijeira-Díaz,andSannaJärvelä
The Role of Metacognition and Self-regulation on Clinical
Reasoning: Leveraging Multimodal Learning Analytics to
TransformMedicalEducation.................................................. 105
ElizabethB.Cloude,MeganD.Wiedbusch,DarynA.Dever,DarioTorre,
andRogerAzevedo
v
vi Contents
PartIV MMLAtoSupportTheoryandPedagogy
IntermodalityinMultimodalLearningAnalyticsforCognitive
Theory Development: A Case from Embodied Design for
MathematicsLearning........................................................... 133
Sofia Tancredi, Rotem Abdu, Ramesh Balasubramaniam,
andDorAbrahamson
Bridging the Gap Between Informal Learning Pedagogy
andMultimodalLearningAnalytics........................................... 159
RawadHammad,MohammedBahja,andMohammadAminKuhail
PartV MMLAApproaches,ArchitecturesandMethodologies
MultimodalLearningExperienceforDeliberatePractice .................. 183
DanieleDiMitri,JanSchneider, Bibeg Limbu,KhaleelAsyraafMat
Sanusi,andRolandKlemke
CDM4MMLA: Contextualized Data Model for MultiModal
LearningAnalytics............................................................... 205
Shashi Kant Shankar, María Jesús Rodríguez-Triana, Luis P. Prieto,
AdolfoRuiz-Calleja,andPankajChejara
APhysiology-AwareLearningAnalyticsFramework........................ 231
MelanieBleckandNguyen-ThinhLe
PartVI MMLAandAffectiveStates
OnceMorewithFeeling:EmotionsinMultimodalLearningAnalytics ... 261
MarcusKubsch,DanielaCaballero,andPabloUribe
PartVII PrivacyandEthicsofMMLA
The Evidence of Impact and Ethical Considerations
ofMultimodalLearningAnalytics:ASystematicLiteratureReview...... 289
Haifa Alwahaby, Mutlu Cukurova, Zacharoula Papamitsiou,
andMichailGiannakos
PartVIII ThePast,Present,andFutureofMMLA
Sensor-BasedAnalyticsinEducation:LessonsLearnedfrom
ResearchinMultimodalLearningAnalytics.................................. 329
MichailGiannakos,MutluCukurova,andSofiaPapavlasopoulou
FramingtheFutureofMultimodalLearningAnalytics..................... 359
MarceloWorsley
Index............................................................................... 371
Part I
Introduction to MMLA
Introduction to Multimodal Learning
Analytics
MichailGiannakos ,DanielSpikol ,DanieleDiMitri ,KshitijSharma,
XavierOchoa ,andRawadHammad
Abstract This chapter provides an introduction and an overview of this edited
book on Multimodal Learning Analytics (MMLA). The goal of this book is to
introducethereadertothefieldofMMLAandprovideacomprehensiveoverview
ofcontemporaryMMLAresearch.Thecontributionscomefromdiversecontextsto
supportdifferentobjectivesandstakeholders(e.g.,learningscientists,policymakers,
technologists). In this first introductory chapter, we present the history of MMLA
and the various ongoing challenges, giving a brief overview of the contributions
ofthebook,andconcludebyhighlightingthepotentialemergingtechnologiesand
practicesconnectedwithMMLA.
Keywords Multimodallearninganalytics · Learninganalytics · Sensordata · AI
ineducation
1 Introduction
The intersection of data coming from different modalities (multimodal data) and
advanced computational analyses has the ability to improve our understanding on
how humans learn, but also provide novel affordances that enhance our learning
M.Giannakos((cid:2))·K.Sharma
NorwegianUniversityofScienceandTechnology,Trondheim,Norway
e-mail:[email protected]
D.Spikol
UniversityofCopenhagen,Copenhagen,Denmark
D.DiMitri
DIPF,FrankfurtamMain,Germany
X.Ochoa
NewYorkUniversity,NewYork,NY,USA
R.Hammad
UniversityofEastLondon,London,UK
©TheAuthor(s),underexclusivelicensetoSpringerNatureSwitzerlandAG2022 3
M.Giannakosetal.(eds.),TheMultimodalLearningAnalyticsHandbook,
https://doi.org/10.1007/978-3-031-08076-0_1
4 M.Giannakosetal.
capacities (e.g.,affectivelearning(D’Mello,2017),embodied learning(Abraham-
son et al., 2021)). During the last years we have seen an increasing research
interestinthecollectionandanalysisofrichdataleveragingmultipledatachannels
fromvarioussourcesandindifferentmodalities,i.e.,multimodallearninganalytics
(MMLA) (e.g., Blikstein & Worsley, 2016; Worsley, 2018; Sharma & Giannakos,
2020). MMLA maintains Learning Analytics’ overarching goal of understanding
and improving learning in all the different environments where it occurs, by
exploiting new opportunities once we capture new forms of digital data from
students’ learning activity, and by using computational analysis techniques from
data science and artificial intelligence (AI).1 At the same time MMLA expands
Learning Analytics methodologies, tools and potential implications (Giannakos et
al., 2020a, b), by leveraging advances in machine learning (ML) and affordable
sensor technologies to act as a virtual observer and analyst of learning activities
(Ochoaetal.,2017).
Multimodal data can complement our understanding on how humans learn,
providing more information on (meta)cognitive, affective and behavioural aspects
oflearning(Cukurovaetal.,2020).Multimodaldatacanenrichthedigitalrepresen-
tationofthelearnerinthecomputer(DiMitrietal.,2018).Recentworkshighlight
howMMLAresearchenablesustoextractinsightsfromtext,speech,gesture,affect,
or gaze analysis (e.g., Blikstein & Worsley, 2016; Sharma & Giannakos, 2020;
Spikol et al., 2018), and utilize those insights and indicators to provide automated
feedbacktothelearnerstostimulatetheirawarenessandreflection(e.g.,Ochoaet
al., 2018; Ochoa, 2022; Di Mitri et al., 2022). With today’s increased availability
andcomplexityindata,andadvanceddataanalysistechniquesnewchallengesand
opportunitiesalsoarise.Asintraditionallearningsciences,thegoalofMMLAisto
understand and explain the phenomena of how humans learn. At the same time,
MMLA research holds significant potential for advancing learning sciences and
supporting both empirical and theoretical research by utilizing its capabilities to
observethelearningactivityatthemicro-leveland“sense”humans’cognitiveand
affectiveandpsychomotorfactors.
The introduction of MMLA research also signals the need for advancing our
understanding about how new sources of information can support our understand-
ing. Unlike qualitative data (e.g., interviews, observations) or even mainstream
quantitative data (e.g., questionnaires and clickstreams) where the connection of
these data with their meaning is evident, the original form of multimodal data
(i.e.signalscaptured)aretypicallytoocomplextocontributetotheunderstanding
of the researcher without intensive data analysis and data processing steps. For
instance, the extraction of specific data features is usually needed (e.g., ampli-
tude, temporality, or frequency) to formulate meaningful constructs which can
be employed as input for statistical analyses and future validation or rejection
of hypotheses and research questions. In other words, the process of inspecting,
cleaning, transforming, converting, aggregating, and modeling data in order to
1WhatisLearningAnalytics?:https://www.solaresearch.org/about/what-is-learning-analytics/
IntroductiontoMultimodalLearningAnalytics 5
developmeaningfulinformationthatcanhelpustofurtherourknowledgeanddraw
conclusions(e.g.,respondtoaresearchquestion),includesmorecomplexprocesses
and multidimensional methodological decisions compared to traditional learning
analytics and learning technology research. Those decisions concern different
methodologicalsteps,suchasdatagathering,pre-processing,analysis,annotation,
modelingandsense-making,inawaythatismeaningfulforlearningscientistsand
learningtechnologists,aswellasotherstakeholders(e.g.,studentsorteachers),and
posechallengesandopportunitiesintheemergentfieldofMMLA.Thisbookaims
toserveasauniqueresourceforstate-of-the-artmethods,processesandchallenges,
butalsohighlightpromisingareasforfuturework.
In particular, the goal of this book is to introduce the reader to contemporary
MMLA research and practice. The contributions come from diverse contexts to
support different objectives and stakeholders such as learning scientists, poli-
cymakers or learning technologists. In this opening chapter, we introduce the
reader to the book, by presenting the history of MMLA and the various ongoing
challenges, giving a brief overview of the contributions of the book, and conclude
byhighlightingthepotentialofMMLAresearchtoadvancelearningresearch.
2 Background andBrief History:What HaveWe Learned
andWhere DoWe GoNow?
2.1 MMLA BriefHistoryandCommunity Development
Learningtechnologyresearchfrequentlyutiliseslearningtracestointerpretcomplex
learning phenomena, and to support teaching and learning processes. Learning
traces are left behind while learners’ interact with technologies and the learning
context in general (e.g., other learners, an instructor, and non-digital learning
materials). Such interactions produce data that range from low level constructs
of simple interpretation (e.g., response times and correctness of multiple-choice
questions) to higher level constructs of more complex interpretation (e.g., reading
strategiesandcollaborativelearningpatterns).“Traditional”analyticsrepresentsuch
interactionsasasequenceofactionsandindexeswhichistypicallycontainedina
logfileandvisualisedwithanavigationgraphthatshowsthelearningperformance
andothermeaningfulinformation.Theseanalyticsinformthelearnerortheteacher
through different representations such as graphs and diagrams, and typically are
included as an add-on to the learning system for increasing its functionalities and
intelligence(e.g.,recommendersystems,adaptivesupportofindividual’sprogress).
Thishasimplicationstobothpedagogyinallowinglearnersandteacherstobeaware
of their progress and make informed decisions as well as in learning technology
design,asforexample,whichservicestoaddinalearningsystem.
In recent years, advances in sensor data, social signal processing and compu-
tational analyses have demonstrated the potential to help us understand learning
