Table Of ContentUniversita` degli Studi di Pisa
Dipartimento di Informatica
Dottorato di Ricerca in Informatica
Ph.D. Thesis
Ambient intelligence in assisted living
environments
Filippo Palumbo
Supervisor
Prof. Stefano Chessa
September 6, 2016
Contents
Acronyms xi
1 Ambient Assisted Living: Concepts, technologies, and applications 3
1.1 Research questions and objectives . . . . . . . . . . . . . . . . . . . . 8
1.1.1 Enabling infrastructure . . . . . . . . . . . . . . . . . . . . . . 9
1.1.2 Context-Awareness . . . . . . . . . . . . . . . . . . . . . . . . 10
1.1.3 Long-term monitoring . . . . . . . . . . . . . . . . . . . . . . 12
1.2 The proposed solution . . . . . . . . . . . . . . . . . . . . . . . . . . 13
1.3 Structure of the thesis . . . . . . . . . . . . . . . . . . . . . . . . . . 14
2 Background and related works 15
2.1 Enabling platforms for AAL . . . . . . . . . . . . . . . . . . . . . . . 16
2.1.1 Middleware for pervasive computing . . . . . . . . . . . . . . 19
2.1.2 Background: The OSGi model and the universAAL project . . 22
2.2 Context-awareness in AAL . . . . . . . . . . . . . . . . . . . . . . . . 27
2.2.1 Indoor localization . . . . . . . . . . . . . . . . . . . . . . . . 28
2.2.2 Activity recognition . . . . . . . . . . . . . . . . . . . . . . . . 32
2.2.3 Background: The EvAAL competition . . . . . . . . . . . . . 35
2.3 Long-term monitoring . . . . . . . . . . . . . . . . . . . . . . . . . . 42
2.3.1 Background: Cognitivist vs emergent approach . . . . . . . . 43
3 The GiraffPlus middleware infrastructure 47
3.1 The GiraffPlus reference scenario . . . . . . . . . . . . . . . . . . . . 49
3.1.1 Hardware Components . . . . . . . . . . . . . . . . . . . . . . 51
3.1.2 The Giraff robotic platform . . . . . . . . . . . . . . . . . . . 54
3.2 The middleware . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 55
3.2.1 Service Discovery and Communication . . . . . . . . . . . . . 58
3.2.2 The Mobile Middleware and the ASIP programming model . . 62
3.3 Performance evaluation . . . . . . . . . . . . . . . . . . . . . . . . . . 72
3.3.1 Mobile/Fixed node interaction . . . . . . . . . . . . . . . . . . 73
3.3.2 Resource-constrained/Fixed node interaction . . . . . . . . . . 76
3.3.3 Evaluation of results with respect to reference requirements . . 83
3.4 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 85
iv CONTENTS
4 Context-awareness: indoor localization and activity recognition 87
4.1 The EvAAL reference scenario . . . . . . . . . . . . . . . . . . . . . . 88
4.2 CEO: a Context Event Only indoor localization technique for AAL . 89
4.2.1 The device-free indoor localization algorithm . . . . . . . . . . 89
4.2.2 Performance analysis of CEO . . . . . . . . . . . . . . . . . . 92
4.3 AReM: Activity Recognition from Multisensor data fusion . . . . . . 101
4.3.1 Sensor data collection and processing . . . . . . . . . . . . . . 102
4.3.2 Decision Tree . . . . . . . . . . . . . . . . . . . . . . . . . . . 105
4.3.3 Echo State Networks . . . . . . . . . . . . . . . . . . . . . . . 107
4.3.4 The EvAAL experience . . . . . . . . . . . . . . . . . . . . . . 110
4.3.5 Performance analysis of AReM . . . . . . . . . . . . . . . . . 113
4.4 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 122
5 Long-term behavioral monitoring 127
5.1 Overall architecture . . . . . . . . . . . . . . . . . . . . . . . . . . . . 127
5.2 Error modeling of indoor localization systems . . . . . . . . . . . . . 129
5.3 The monitoring system . . . . . . . . . . . . . . . . . . . . . . . . . . 133
5.3.1 The marking process . . . . . . . . . . . . . . . . . . . . . . . 134
5.3.2 The perception process . . . . . . . . . . . . . . . . . . . . . . 137
5.3.3 The detection process . . . . . . . . . . . . . . . . . . . . . . . 137
5.4 Performance evaluation . . . . . . . . . . . . . . . . . . . . . . . . . . 141
5.4.1 The experimental setup . . . . . . . . . . . . . . . . . . . . . 141
5.4.2 System assessment . . . . . . . . . . . . . . . . . . . . . . . . 143
5.5 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 147
6 Conclusions and future work 151
6.1 Impact and lessons learned . . . . . . . . . . . . . . . . . . . . . . . . 154
Bibliography 159
List of Figures
1.1 Smart environments scenarios. . . . . . . . . . . . . . . . . . . . . . . 4
1.2 EU27 population by age and sex. . . . . . . . . . . . . . . . . . . . . 5
1.3 Public expenditure on long-term care. . . . . . . . . . . . . . . . . . . 5
1.4 The proposed solution. . . . . . . . . . . . . . . . . . . . . . . . . . . 13
2.1 Reference model for pervasive computing middleware. . . . . . . . . . 18
2.2 The OSGi architecture. . . . . . . . . . . . . . . . . . . . . . . . . . . 23
2.3 The universAAL components. . . . . . . . . . . . . . . . . . . . . . . 24
2.4 The EvAAL 2012 and 2013 paths. . . . . . . . . . . . . . . . . . . . . 36
2.5 The map of the EvAAL Living Lab. . . . . . . . . . . . . . . . . . . . 40
2.6 Long-term monitoring enabling situation-awareness. . . . . . . . . . . 43
3.1 The middleware as a “glue”. . . . . . . . . . . . . . . . . . . . . . . . 48
3.2 The GiraffPlus system architecture . . . . . . . . . . . . . . . . . . . 49
3.3 The Giraff platform . . . . . . . . . . . . . . . . . . . . . . . . . . . . 54
3.4 An in-depth view of the middleware component . . . . . . . . . . . . 56
3.5 Interaction between components and buses . . . . . . . . . . . . . . . 59
3.6 The announce-listen protocol model . . . . . . . . . . . . . . . . . . . 61
3.7 Main interfaces class diagram. . . . . . . . . . . . . . . . . . . . . . . 64
3.8 The Android middleware architecture. . . . . . . . . . . . . . . . . . 65
3.9 The sequence diagram of the service discovery mechanism. . . . . . . 67
3.10 The sequence diagram of the subscription mechanism. . . . . . . . . . 67
3.11 ASIP simplified class diagram . . . . . . . . . . . . . . . . . . . . . . 69
3.12 ASIP Messages: example of syntax for a distance service . . . . . . . 70
3.13 An application scenario exploiting the service discovery functionality. 72
3.14 Middleware latency with 1 producer and 1 consumer varying the re-
quests per seconds. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 73
3.15 Middleware latency with 1 consumer varying the number of producers
transmitting at 5 requests per seconds. . . . . . . . . . . . . . . . . . 75
3.16 Middleware latency varying the number of consumers with 1, 10, 25,
and 50 producers transmitting at 5 requests per seconds. . . . . . . . 76
3.17 Comparison of energy consumptions in one hour of test of the two
different approaches analyzed with respect to the ALL-ON situation. 77
3.18 Serial testing set-up. . . . . . . . . . . . . . . . . . . . . . . . . . . . 78
vi LIST OF FIGURES
3.19 Oscilloscope output. . . . . . . . . . . . . . . . . . . . . . . . . . . . 79
3.20 Throughput for Java clients with various testbed network configura-
tions. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 80
3.21 Throughput for Python clients with various testbed network config-
urations. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 80
3.22 The hardware set-ups used for latency testing with serial connection. 81
3.23 The hardware set-ups used for latency testing with TCP and MQTT
connection. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 82
3.24 Latency for Java clients with various testbed network configurations. 83
3.25 Latency for Python clients with various testbed network configurations. 83
4.1 The EvAAL setting used by CEO. . . . . . . . . . . . . . . . . . . . 91
4.2 The finite state machine representing CEO. . . . . . . . . . . . . . . 92
4.3 Comparing CEO with a “blind” system. . . . . . . . . . . . . . . . . 94
4.4 CDFs comparison of systems accuracy in EvAAL editions 2012 and
2013. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 96
4.5 Algorithm for fusing CEO results with another system’s results. . . . 97
4.6 Error distribution for all competitors in 2012 with and without fusion
with CEO. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 98
4.7 Error distribution for all competitors in 2013 with and without fusion
with CEO. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 99
4.8 The Activity Recognition multisensor processing chain. . . . . . . . . 101
4.9 The bus-based communication middleware integration. . . . . . . . . 103
4.10 A sample output sequence of the sensor data processing block. . . . . 104
4.11 Structure of the decision tree fusion mechanism. . . . . . . . . . . . . 105
4.12 Magnitude plot of the tri-axial embedded accelerometer. . . . . . . . 106
4.13 Acceleration plot for the vertical position. . . . . . . . . . . . . . . . 107
4.14 Acceleration plot for the horizontal position. . . . . . . . . . . . . . . 108
4.15 The two types of bending activity. . . . . . . . . . . . . . . . . . . . . 108
4.16 Graphical representation of the results obtained by the AReM sys-
tem at the EvAAL competition. X-axis represents the progressive
time-slots of 250 milliseconds. Y-axis represents the activity: 0 for
standing, 1 for walking, 2 for sitting, 3 for bending, 4 for cycling, 5
for falling, 6 for lying, and -1 for the not evaluated null class. . . . . . 112
4.17 The sensors setup during the competition. . . . . . . . . . . . . . . . 113
4.18 Confusion matrices of the performance in the multi-classification learn-
ing task. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 118
4.19 Confusion matrices of the performance obtained under the Heteroge-
neous ARS settings. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 122
4.20 Graphical representation of the results. . . . . . . . . . . . . . . . . . 123
5.1 UML activity diagram of the macro activities of the proposed ap-
proach to anomaly detection. . . . . . . . . . . . . . . . . . . . . . . 128
0.0. LIST OF FIGURES vii
5.2 Scatter and histogram bar plots of the CPS localization system . . . . 131
5.3 Scatter and histogram bar plots of the n-Core localization system . . 131
5.4 Scatter and histogram bar plots of the RealTrac localization system . 133
5.5 Quantile-quantile plot of the squared Mahalanobis distance versus the
corresponding quantiles of the chi-square distribution. . . . . . . . . . 134
5.6 Basic scenarios of the marking process. . . . . . . . . . . . . . . . . . 134
5.7 Two scenarios of marking process in a real-world apartment with an
elderly with some risk of disease progression. . . . . . . . . . . . . . . 136
5.8 An illustrative example of Similarity between two consecutive marks. 138
5.9 An illustrative example of Similarity between tracks. . . . . . . . . . 139
5.10 Similarity function for the two marks represented in Figure 5.7 over
a time frame of about 6 hours. . . . . . . . . . . . . . . . . . . . . . . 140
5.11 S-shape activation function. . . . . . . . . . . . . . . . . . . . . . . . 141
5.12 S-shaped similarity . . . . . . . . . . . . . . . . . . . . . . . . . . . . 141
5.13 Outputs of the perception, detection, and of the human observation
when the error model of the CPS localization system is applied on
each day of the observed week. . . . . . . . . . . . . . . . . . . . . . . 147
5.14 Outputs of the perception, detection, and of the human observation
when the error model of the n-Core localization system is applied on
each day of the observed week. . . . . . . . . . . . . . . . . . . . . . . 148
5.15 Outputs of the perception, detection, and of the human observation
when the error model of the REALTrac localization system is applied
on each day of the observed week. . . . . . . . . . . . . . . . . . . . . 148
List of Tables
1.1 Ambient Assisted Living application areas. . . . . . . . . . . . . . . . 6
1.2 Ambient sensors used in Smart Environments. . . . . . . . . . . . . . 7
1.3 Typical wearable and mobile sensors for AAL. . . . . . . . . . . . . . 7
2.1 Comparison of programming abstractions offered by middleware sys-
tems. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19
2.2 Comparison of system architecture for middleware systems. . . . . . . 20
2.3 Comparison of middleware services. . . . . . . . . . . . . . . . . . . . 21
2.4 The capabilities and services offered by universAAL and GiraffPlus. . 27
2.5 Overview of indoor positioning technologies. . . . . . . . . . . . . . . 31
2.6 Scoring criteria for the localization competition . . . . . . . . . . . . 36
2.7 The winning competing systems’ scores (with 10 being the highest
possible score for each metric). . . . . . . . . . . . . . . . . . . . . . . 37
2.8 The winning competing systems’ scores (with 10 being the highest
possible score for each metric). . . . . . . . . . . . . . . . . . . . . . . 41
3.1 Energy consumption in mW . . . . . . . . . . . . . . . . . . . . . . . 75
3.2 Non-functional requirements for AAL middleware. . . . . . . . . . . . 84
4.1 Competitors of the 2012 and 2013 localization track . . . . . . . . . . 93
4.2 Third quartile error of competing systems and its variation after fu-
sion with CEO. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 100
4.3 Number of sequences for each activity in the Activity Recognition
dataset. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 111
4.4 EvAAL 2013 activity recognition track final scores. . . . . . . . . . . 112
4.5 Organization of the computational learning tasks and activities in-
volved in the different ARS settings. . . . . . . . . . . . . . . . . . . 114
4.6 Test set per-class accuracy and F1 score achieved by LI-ESNs and
IDNNs. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 117
4.7 Per-class accuracy and F1 score achieved by LI-ESNs and IDNNs on
task 1. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 119
4.8 Per-class accuracy and F1 score achieved by LI-ESNs and IDNNs on
task 2. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 120
x LIST OF TABLES
4.9 Per-class accuracy and F1 score achieved by LI-ESNs and IDNNs on
task 3. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 120
4.10 Per-class accuracy and F1 score achieved by LI-ESNs and IDNNs. . . 121
5.1 Performance statistics: mean, variance, and percentiles in meters of
the localization error for the selected systems during the EvAAL com-
petition. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 130
5.2 Skewness and kurtosis values of the localization error for the selected
systems during the EvAAL competition. . . . . . . . . . . . . . . . . 132
5.3 The parameters chosen for the bivariate gaussian distributions . . . . 132
5.4 Main parameters set in the tuning session. . . . . . . . . . . . . . . . 142
5.5 Behavioral deviations observed in the testing session. . . . . . . . . . 144
5.6 Confusion Matrices. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 145
5.7 Offline assessment indicators. . . . . . . . . . . . . . . . . . . . . . . 146
5.8 Online assessment indicators. . . . . . . . . . . . . . . . . . . . . . . 146
