Table Of ContentSmart Innovation, Systems and Technologies 199
Md Atiqur Rahman Ahad
Paula Lago
Sozo Inoue Editors
Human Activity
Recognition
Challenge
Smart Innovation, Systems and Technologies
Volume 199
Series Editors
Robert J. Howlett, Bournemouth University and KES International,
Shoreham-by-sea, UK
Lakhmi C. Jain, Faculty of Engineering and Information Technology,
Centre for Artificial Intelligence, University of Technology Sydney,
Sydney, NSW, Australia
The Smart Innovation, Systems and Technologies book series encompasses the
topics of knowledge, intelligence, innovation and sustainability. The aim of the
seriesistomakeavailableaplatformforthepublicationofbooksonallaspectsof
single and multi-disciplinary research on these themes in order to make the latest
resultsavailableinareadily-accessibleform.Volumesoninterdisciplinaryresearch
combining two or more of these areas is particularly sought.
Theseriescoverssystemsandparadigmsthatemployknowledgeandintelligence
inabroadsense.Itsscopeissystemshavingembeddedknowledgeandintelligence,
whichmaybeappliedtothesolutionofworldproblemsinindustry,theenvironment
and the community. It also focusses on the knowledge-transfer methodologies and
innovationstrategiesemployedtomakethishappeneffectively.Thecombinationof
intelligent systems tools and a broad range of applications introduces a need for a
synergy of disciplines from science, technology, business and the humanities. The
series will include conference proceedings, edited collections, monographs, hand-
books, reference books, and other relevant types of book in areas of science and
technologywheresmartsystemsandtechnologiescanofferinnovativesolutions.
Highqualitycontentisanessentialfeatureforallbookproposalsacceptedforthe
series. It is expected that editors of all accepted volumes will ensure that
contributionsaresubjectedtoanappropriatelevelofreviewingprocessandadhere
to KES quality principles.
Indexed by SCOPUS, EI Compendex, INSPEC, WTI Frankfurt eG, zbMATH,
Japanese Science and Technology Agency (JST), SCImago, DBLP.
All books published in the series are submitted for consideration in Web of
Science.
More information about this series at http://www.springer.com/series/8767
Md Atiqur Rahman Ahad Paula Lago
(cid:129) (cid:129)
Sozo Inoue
Editors
Human Activity Recognition
Challenge
123
Editors
Md Atiqur RahmanAhad PaulaLago
University of Dhaka Kyushu Institute of Technology
Dhaka,Bangladesh Fukuoka,Japan
Osaka University
Osaka,Japan
SozoInoue
Kyushu Institute of Technology
Fukuoka,Japan
ISSN 2190-3018 ISSN 2190-3026 (electronic)
Smart Innovation,Systems andTechnologies
ISBN978-981-15-8268-4 ISBN978-981-15-8269-1 (eBook)
https://doi.org/10.1007/978-981-15-8269-1
©TheEditor(s)(ifapplicable)andTheAuthor(s),underexclusivelicensetoSpringerNature
SingaporePteLtd.2021
Thisworkissubjecttocopyright.AllrightsaresolelyandexclusivelylicensedbythePublisher,whether
thewholeorpartofthematerialisconcerned,specificallytherightsoftranslation,reprinting,reuseof
illustrations, recitation, broadcasting, reproduction on microfilms or in any other physical way, and
transmissionorinformationstorageandretrieval,electronicadaptation,computersoftware,orbysimilar
ordissimilarmethodologynowknownorhereafterdeveloped.
The use of general descriptive names, registered names, trademarks, service marks, etc. in this
publicationdoesnotimply,evenintheabsenceofaspecificstatement,thatsuchnamesareexemptfrom
therelevantprotectivelawsandregulationsandthereforefreeforgeneraluse.
The publisher, the authors and the editors are safe to assume that the advice and information in this
book are believed to be true and accurate at the date of publication. Neither the publisher nor the
authors or the editors give a warranty, expressed or implied, with respect to the material contained
hereinorforanyerrorsoromissionsthatmayhavebeenmade.Thepublisherremainsneutralwithregard
tojurisdictionalclaimsinpublishedmapsandinstitutionalaffiliations.
ThisSpringerimprintispublishedbytheregisteredcompanySpringerNatureSingaporePteLtd.
The registered company address is: 152 Beach Road, #21-01/04 Gateway East, Singapore 189721,
Singapore
Preface
Activity Recognition applications include remote monitoring of daily activities for
elders living alone and automatic record creation for nurses in hospitals. Although
these applications only require the recognition of the activity being done, i.e.,
cooking, such complex activities are usually made up of several smaller activities
like “taking something from the fridge”. Recognizing such steps has multiple
advantages. For instance, in the scenario of an elder living alone, recognizing
activity steps enables applications such as reminding of the next steps.
Current activity recognition systems focus on recognizing either the complex
label(macroactivity)orthesmallsteps(microactivities)buttheirjointrecognition
iscriticalforanalysisliketheonesproposed.Infact,inanursingscenario,washing
hands after taking blood is quite different than doing it before, as it is mandatory.
Therefore, with the Cooking Activity Recognition Challenge, we aimed at the
recognitionofthemacro-andmicroactivitiestakingplaceduringcookingsessions.
In this book, there are 10 chapters on activity recognition challenge from
Bangladesh, China, France, Japan, Slovenia, Switzerland, and the USA. As there
werethreedatasourcesavailable,eachsolutionselectedthedatasourcetheyfound
more suitable for the task. Most of the chapters used only the data from
accelerometer sensors. These were placed on both wrists (smartwatches), on the
right forearm and on the left hip (smartphones). The chapters using accelerometer
datahaveexploredawidevarietyofapproaches,whileothershavedevotedtodeep
learningapproaches.Somechaptershaveexploredacombinationofdatasourcesas
well as different models.
All chapters have utilized two separate models for each level of classification.
Theresultsaresummarizedandanalyzedinthechapter“SummaryoftheCooking
Activity Recognition Challenge”.
v
vi Preface
Chapter“ActivityRecognitionfromSkeletonandAccelerationDataUsingCNN
and GCN” used the motion capture and only the accelerometer placed on the right
arm as input to a Convolutional Neural Network.
Chapter “Let’s Not Make It Complicated—Using Only LightGBM and Naive
Bayes for Macro- and Micro-Activity Recognition from a Small Dataset” also
combinestheaccelerometerwithmotioncapturedataandthenaLightGBMmodel
for classification.
Chapter “Deep Convolutional Bidirectional LSTM for Complex Activity
Recognition with Missing Data” also only used three accelerometers, excluding
theleftwristsensor,andadeeplearningapproach.However,theychoseLSTMas
the model for the data.
Chapter “SCAR-Net: Scalable ConvNet for Activity Recognition with Multi-
modal Sensor Data” proposed a convolutional network SCAR-Net which imple-
ments an end-to-end approach. However, they use two models, one for each level.
Chapter “Multi-sampling Classifiers for the Cooking Activity Recognition
Challenge” used all three data sources (accelerometer, motion capture, and
OpenPose) and a classifier based on time-series similarity.
Chapter “Multi-class Multi-label Classification for Cooking Activity
Recognition” used both motion capture and accelerometer data as input to a
multi-class random forest.
Chapter“CookingActivityRecognitionwithConvolutionalLSTMUsingMulti-
label Loss Function and Majority Vote” uses LSTM and also all 4 accelerometer
sensors. All deep learning models use raw data as input, with preprocessing to
account for different sampling rates and missing values.
Chapter “Identification of Cooking Preparation Using Motion Capture Data: A
Submission to theCooking Activity Recognition Challenge” usedonly the motion
capture data. This data source records the 3D position of 25 body parts. They
reconstructed the sequence of data to train over the whole sequence instead of
segments. A random forest proved to be more effective in the classification than
other approaches.
Chapter “Cooking Activity Recognition with Varying Sampling Rates Using
DeepConvolutionalGRUFramework”usedonlythreeaccelerometers,andadeep
learning model combining a Convolutional Neural Network (CNN) and a Gated
Recurrent Unit (GRU).
Preface vii
Wewouldliketothanktheexpertreviewersfortheirtimetoreviewthechapters.
We thank Springer for their prompt cooperation to publish this book. We strongly
feel that this book will enrich the researcher communities in the academia and
industry to move further on human activity and behavior understanding.
Best regards,
Md Atiqur Rahman Ahad, Ph.D., SMIEEE
Professor, University of Dhaka
Dhaka, Bangladesh
Specially Appointed Associate Professor
Osaka University
Osaka, Japan
Paula Lago, Ph.D.
Postdoctoral Researcher
Kyushu Institute of Technology
Fukuoka, Japan
Sozo Inoue, Ph.D.
Professor, Kyushu Institute of Technology
Fukuoka, Japan
Contents
Summary of the Cooking Activity Recognition Challenge . . . . . . . . . . . . 1
Sayeda Shamma Alia, Paula Lago, Shingo Takeda, Kohei Adachi,
Brahim Benaissa, Md Atiqur Rahman Ahad, and Sozo Inoue
Activity Recognition from Skeleton and Acceleration Data
Using CNN and GCN. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15
Donghui Mao, Xinyu Lin, Yiyun Liu, Mingrui Xu, Guoxiang Wang,
Jiaming Chen, and Wei Zhang
Let’s Not Make It Complicated—Using Only LightGBM
and Naive Bayes for Macro- and Micro-Activity Recognition
from a Small Dataset . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27
Ryoichi Kojima, Roberto Legaspi, Kiyohito Yoshihara, and Shinya Wada
Deep Convolutional Bidirectional LSTM for Complex Activity
Recognition with Missing Data. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 39
Swapnil Sayan Saha, Sandeep Singh Sandha, and Mani Srivastava
SCAR-Net: Scalable ConvNet for Activity Recognition
with Multimodal Sensor Data. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 55
Zabir Al Nazi
Multi-sampling Classifiers for the Cooking Activity
Recognition Challenge . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 65
Ninnart Fuengfusin and Hakaru Tamukoh
Multi-class Multi-label Classification for Cooking
Activity Recognition . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 75
Shkurta Gashi, Elena Di Lascio, and Silvia Santini
Cooking Activity Recognition with Convolutional LSTM
Using Multi-label Loss Function and Majority Vote. . . . . . . . . . . . . . . . 91
Atsuhiro Fujii, Daiki Kajiwara, and Kazuya Murao
ix
x Contents
Identification of Cooking Preparation Using Motion Capture Data:
A Submission to the Cooking Activity Recognition Challenge . . . . . . . . 103
Clément Picard, Vito Janko, Nina Reščič, Martin Gjoreski,
and Mitja Luštrek
Cooking Activity Recognition with Varying Sampling Rates
Using Deep Convolutional GRU Framework . . . . . . . . . . . . . . . . . . . . . 115
Md. Sadman Siraj, Omar Shahid, and Md Atiqur Rahman Ahad
