using multi-modal bio-digital technologies to support the assessment of cognitive abilities of

289 Pages·2015·6.94 MB·English

by  Hock Chye Gan

USING MULTI-MODAL BIO-DIGITAL TECHNOLOGIES
TO SUPPORT THE ASSESSMENT OF COGNITIVE
ABILITIES OF CHILDREN WITH PHYSICAL AND

NEUROLOGICAL IMPAIRMENTS
Hock Chye Gan

Thesis submitted to the University of Hertfordshire in partial fulfilment of the requirements of the
degree of Doctor of Philosophy (PhD)

August, 2014
Abstract
Current studies done using a learning test for children have problems as they only make evaluations
of Physically and Neurologically Impaired (PNI) children who can succeed in the test and can be
considered as a PASS/FAIL test. This pilot study takes a holistic view of cognitive testing of PNI
children using a user-test-device triad model and provides a framework using non-PNI children and
adults as controls. Comparisons using adapted off-the-shelf novel interfaces to the computer, in
particular, an Electroencephalograph (EEG) head-set, an eye-tracker and a head-tracker and a
common mouse were carried out. In addition, two novel multi-modal technologies were developed
based on the use of brain-waves and eye-tracking as well as head-tracking technologies to support
the study. The devices were used on three tests with increasing cognitive complexity. A self-
developed measure based on success streaks (consecutive outcomes) was introduced to improve
evaluations of PNI children. A theoretical model regarding a fit of ability to devices was initially setup
and finally modified to fit the view of the empirical model that emerged from the outcomes of the
study. Results suggest that while multi-modal technologies can address weaknesses of the individual
component modes, a compromise is made between the user’s ability for multi-tasking between the
modes and the benefits of a multi-modal device but the sample size is very small. Results also show
children failing a test with a mouse but passing it subsequently when direct communication is used
suggesting that a device can affect a test for children who are of a developing age. This study
provides a framework for a more meaningful conversation between educational psychologists as
well as other professionals and PNI parents because it provides more discrimination of outcomes in
cognitive tests for PNI children. The framework provides a vehicle that addresses scientifically the
concerns of parents and schools.
Page 1 of 288
Acknowledgements
I would like to thank the PhD supervision team, Ray Frank, Dr. Farshid Amirabdollahian, Dr.
Austen Rainer for the countless hours they have put in (including weekends) to help me with my
work. In particular, I would like to thank Ray Frank for his unwavering support and efforts. I would
like to wish him an enjoyable retirement as he deserves it after all his hard work for numerous years.
I would also like to thank Dr. Rob Sharp in his role as consultant psychologist to the work we have
been doing.
I would like to express my thanks for all the people in the University of Hertfordshire who
have given me help and support including Noel Taylor and Catherine Khan from the School of
Psychology, people I have worked with in the School of Engineering and various members of staff
from the School of Computer Science for providing me with opportunities, guidance and advice.
I am grateful to the schools that agreed to participate in this study especially the staff of the
special needs schools for their time and effort, and to the participants and their care-givers/families
in assisting us with this study.
I am grateful to my family for their patience, understanding and support they have provided
me throughout my work.
I would like to thank Ian Glasscock and BioDigitalHealth for the considerable support they
have given to this research. They have contributed resources and equipment, finances, their time
and their expertise. This research is supported by the Engineering and Physical Sciences Research
Council (EPSRC UK) under iCASE 09001842.
Page 2 of 288
Table of Contents
1. Introduction to thesis.................................................................................................................. 13
1.1. Research question ............................................................................................................... 13
1.2. Building on previous work ................................................................................................... 13
1.3. The problem ........................................................................................................................ 14
1.3.1. The user-device-test model ......................................................................................... 14
1.3.2. The user....................................................................................................................... 16
1.3.3. The device ................................................................................................................... 16
1.3.5. Research question refinements .................................................................................. 17
1.4. Knowledge required ............................................................................................................ 18
1.5. Contribution to knowledge ................................................................................................. 18
1.6. Ethical issues ....................................................................................................................... 19
1.7. iCASE Work.......................................................................................................................... 20
1.8. Publications ......................................................................................................................... 21
1.9. Thesis structure ................................................................................................................... 21
2. Literature review......................................................................................................................... 24
2.1. Aim ...................................................................................................................................... 24
2.2. User ..................................................................................................................................... 24
2.3. Device ................................................................................................................................. 28
2.3.1. Electroencephalogram (EEG) neuro-headset .............................................................. 29
2.3.2. Eye-tracker .................................................................................................................. 31
2.3.3. Head-tracker ............................................................................................................... 33
Page 3 of 288
2.3.4. Multi-modal fusion ...................................................................................................... 33
2.3.5. Signing ......................................................................................................................... 34
2.3.6. Initial theoretical model for device skills ..................................................................... 36
2.4. Test ..................................................................................................................................... 38
2.4.1. Motor-skills test .......................................................................................................... 39
2.4.2. Light cognitive (Categorization) test ............................................................................ 40
2.4.3. Psychological test (ECDT) ............................................................................................ 42
2.4.4. Development of new measure using Streaks .............................................................. 43
2.5. Conclusions ......................................................................................................................... 45
3. DEVICE: Exploratory investigations of devices ............................................................................ 47
3.1. Introduction ........................................................................................................................ 47
3.2. EEG headset ........................................................................................................................ 47
3.2.1. Expressive indications ................................................................................................. 48
3.2.2. Affective indications .................................................................................................... 49
3.2.3. Cognitive indications ................................................................................................... 50
3.2.4. Aim .............................................................................................................................. 51
3.2.5. Method ....................................................................................................................... 52
3.2.6. Observations and discussion ....................................................................................... 53
3.2.7. Conclusion for EEG headset ........................................................................................ 57
3.3. Eye-tracker and fusion ........................................................................................................ 58
3.3.1. Introduction ................................................................................................................ 58
3.3.2. Aim .............................................................................................................................. 58
Page 4 of 288
3.3.3. Method ....................................................................................................................... 58
3.3.4. Observations and discussions...................................................................................... 60
3.3.5. Conclusion for fusion of EEG headset and eye-tracker ............................................... 63
3.4. Head-tracker ....................................................................................................................... 64
3.4.1. Introduction ................................................................................................................ 64
3.4.2. Aim .............................................................................................................................. 64
3.4.3. Method ....................................................................................................................... 64
3.4.4. Observations and discussion ....................................................................................... 65
3.4.5. Conclusion for head-tracker ........................................................................................ 66
3.5. Conclusion ........................................................................................................................... 66
4. TEST: Design and implementation of the test system ................................................................. 68
4.1. Introduction ........................................................................................................................ 68
4.2. Aim ...................................................................................................................................... 69
4.3. System overview ................................................................................................................. 69
4.4. User interface overview ...................................................................................................... 70
4.4.1. Test programs ............................................................................................................. 70
4.4.2. Test devices ................................................................................................................. 72
4.4.3. Test parameters .......................................................................................................... 72
4.4.4. Device parameters ...................................................................................................... 72
4.5. Software functional components ........................................................................................ 73
4.5.1. System components .................................................................................................... 74
4.5.2. Test components ......................................................................................................... 75
Page 5 of 288
4.5.3. Device components ..................................................................................................... 75
4.6. Scenario for general successful test cycle ........................................................................... 75
4.6.1. Launch of Psyborg User Interface................................................................................ 75
4.6.2. Launch of device training or calibration program ....................................................... 76
4.6.3. Launch of familiarisation program for test .................................................................. 76
4.6.4. Launch of test program used for study ....................................................................... 76
4.6.5. Display of results ......................................................................................................... 77
4.7. Conclusions ......................................................................................................................... 77
5. USER: Exploratory Pilot study with PNI users using the operational model ................................ 78
5.1. Introduction ........................................................................................................................ 78
5.2. Aim ...................................................................................................................................... 78
5.3. Method ............................................................................................................................... 78
5.3.1. Participants ................................................................................................................. 78
5.3.2. Procedure .................................................................................................................... 79
5.3.3. Design.......................................................................................................................... 82
5.4. Results and discussion......................................................................................................... 82
5.4.1. Notation ...................................................................................................................... 83
5.4.2. Description .................................................................................................................. 86
5.5. Conclusion ........................................................................................................................... 90
6. MAIN PILOT STUDY: Overview .................................................................................................... 92
6.1. Introduction ........................................................................................................................ 92
6.1.1. Research questions ..................................................................................................... 92
Page 6 of 288
6.1.2. Definitions ................................................................................................................... 93
6.1.3. Summary of devices used in the tests ......................................................................... 97
6.1.4. Summary of tests ...................................................................................................... 101
6.1.5. The user, device, test triad ........................................................................................ 102
6.1.6. Participants ............................................................................................................... 104
6.2. Experimental protocol....................................................................................................... 109
6.3. Overview to the design ..................................................................................................... 110
6.3.1. Method for ranking devices ...................................................................................... 110
6.3.2. Method for comparing performance of devices ....................................................... 111
6.3.3. Method for detection of interaction effects involving devices.................................. 112
6.4. Standard template for test description ............................................................................. 112
6.5. Conclusion ......................................................................................................................... 113
7. MAIN PILOT STUDY: Motor-skills test (COMPTEST) .................................................................. 114
7.1. Introduction ...................................................................................................................... 114
7.2. Aim .................................................................................................................................... 116
7.3. Method ............................................................................................................................. 117
7.3.1. Participants ............................................................................................................... 117
7.3.2. Procedure .................................................................................................................. 117
7.3.3. Design........................................................................................................................ 119
7.3.4. Data capture.............................................................................................................. 120
7.4. Results ............................................................................................................................... 120
7.4.1. Results from the experiment with PNI Children ........................................................ 122
Page 7 of 288
7.4.2. Results from the experiment with non-PNI children ................................................. 129
7.4.3. Results from the experiment with adults .................................................................. 134
7.5. Discussion ......................................................................................................................... 135
7.5.1. Discussion of results from PNI children, non-PNI children and adults ....................... 135
7.5.2. Use of Signing – a versatile conceptual tool .............................................................. 145
7.5.3. Use of measures ........................................................................................................ 147
7.6. Conclusions ....................................................................................................................... 153
8. MAIN PILOT STUDY: Light cognitive test (CATTEST) .................................................................. 156
8.1. Introduction ...................................................................................................................... 156
8.2. Aim .................................................................................................................................... 157
8.3. Method ............................................................................................................................. 158
8.3.1. Participants ............................................................................................................... 158
8.3.2. Procedure .................................................................................................................. 158
8.3.3. Design........................................................................................................................ 161
8.3.4. Data capture.............................................................................................................. 161
8.4. Results ............................................................................................................................... 162
8.4.1. Results from the experiment with PNI Children ........................................................ 165
8.4.2. Results from the experiment with non-PNI Children ................................................ 172
8.4.3. Results from the experiment with adults .................................................................. 175
8.5. Discussion ......................................................................................................................... 175
8.5.1. Variation of maximum streak distributions between PNI children and non-PNI children
175
Page 8 of 288
8.5.2. Variation of streak size distributions between PNI children and non-PNI children ... 176
8.5.3. Effects of user, test and device between PNI children, non-PNI children and adults 176
8.5.4. Variation of cognitive ability between PNI children, non-PNI children and adults .... 178
8.6. Conclusions ....................................................................................................................... 179
9. MAIN PILOT STUDY: Psychological test (ECDT) ......................................................................... 181
9.1. Introduction ...................................................................................................................... 181
9.1.1. General ...................................................................................................................... 181
9.1.2. Adding to ECDT metrics ............................................................................................. 182
9.2. Aim .................................................................................................................................... 185
9.3. Method ............................................................................................................................. 186
9.3.1. Participants ............................................................................................................... 186
9.3.2. Procedure .................................................................................................................. 186
9.3.3. Design........................................................................................................................ 189
9.3.4. Data capture.............................................................................................................. 189
9.4. Results ............................................................................................................................... 189
9.4.1. Results from the experiment with PNI Children ........................................................ 191
9.4.2. Results from the experiment with non-PNI Children ................................................ 197
9.4.3. Results from the experiment with adults .................................................................. 200
9.5. Discussion ......................................................................................................................... 201
9.5.1. Variation of maximum streak distributions between PNI children and non-PNI children
201
9.5.2. Variation of streak size distributions between PNI children and non-PNI children ... 201
Page 9 of 288