cognitive and haptic modalities of perceptual ability analysis for next generation human computer

164 Pages·2015·3.19 MB·English

by  Tiago Guerreiro

COGNITIVE AND HAPTIC MODALITIES OF
PERCEPTUAL ABILITY ANALYSIS FOR NEXT
GENERATION HUMAN COMPUTER INTERFACES

A Thesis
submitted in partial fulfillment of the requirement for the Degree of
Master of Electronics and Telecommunication Engineering

Jadavpur University
May 2014

By
Shreyasi Datta
Registration No.: 120943 of 2012-2013
Examination Roll No.: M4ETC14-24

Under the Guidance of
Prof. Amit Konar

Department of Electronics and Telecommunication Engineering
Jadavpur University
Kolkata-700032, India
FACULTY OF ENGINEERING AND TECHNOLOGY
JADAVPUR UNIVERSITY

CERTIFICATE

This is to certify that the dissertation entitled “Cognitive and Haptic Modalities of Perceptual
Ability Analysis for Next Generation Human Computer Interfaces” has been carried out by
SHREYASI DATTA (University Registration No.:120943 of 2012-2013) under my guidance and
supervision and be accepted in partial fulfillment of the requirement for the Degree of Master of
Electronics and Telecommunication Engineering. The research results presented in the thesis have
not been included in any other paper submitted for the award of any degree to any other University
or Institute.

-----------------------------------------------
Prof. Amit Konar
Thesis Supervisor
Dept. of Electronics and Telecommunication Engineering
Jadavpur University

-----------------------------------------------
Prof. Iti Saha Misra
Head of the Department
Dept. of Electronics and Telecommunication Engineering
Jadavpur University

----------------------------------------------------
Prof. Sivaji Bandyopadhyay
Dean
Faculty of Engineering and Technology
Jadavpur University

i
FACULTY OF ENGINEERING AND TECHNOLOGY
JADAVPUR UNIVERSITY

CERTIFICATE OF APPROVAL*

The forgoing thesis is hereby approved as a creditable study of an engineering subject and
presented in a manner satisfactory to warrant acceptance as prerequisite to the degree for which it
has been submitted. It is understood that by this approval the undersigned do not necessarily
endorse or approve any statement made, opinion expressed or conclusion drawn there in but
approve the thesis only for which it is submitted.

Committee on final examination for the evaluation of the thesis

----------------------------------------
Signature of the Examiner

----------------------------------------
Signature of the Supervisor

*Only in the case the thesis is approved

ii
FACULTY OF ENGINEERING AND TECHNOLOGY
JADAVPUR UNIVERSITY

DECLARATION OF ORIGINALITY AND COMPLIANCE OF
ACADEMIC THESIS

I hereby declare that this thesis entitled “Cognitive and Haptic Modalities of Perceptual Ability
Analysis for Next Generation Human Computer Interfaces” contains literature survey and
original research work by the undersigned candidate, as part of her Degree of Master of Electronics
and Telecommunication Engineering. All information have been obtained and presented in
accordance with academic rules and ethical conduct. It is hereby declared that, as required by these
rules and conduct, all materials and results that are not original to this work have been properly
cited and referenced.

Candidate Name: Shreyasi Datta

Examination Roll No.:M4ETC14-24

Thesis Title: Cognitive and Haptic Modalities of Perceptual Ability Analysis
for Next Generation Human Computer Interfaces

--------------------------------------------
Signature of the candidate

iii
Acknowledgements
I would like to express my earnest gratitude and sincere thanks to my thesis supervisor Prof. Amit
Konar, Department of Electronics and Telecommunication Engineering, Jadavpur University, for
giving me the opportunity to work under him and inspiring me to explore the interesting fields of
Machine Intelligence and Human Computer Interfaces. I am indebted to him for his patient
guidance, critical and constructive views and untiring support that shaped my work. The past year
has been a remarkable experience in terms of gaining knowledge and skill that I hope to carry on
and develop further.

I am extremely grateful to Prof. Subir Kumar Sarkar, Prof. Kalyan Kumar Mallick and Prof. Iti
Saha Misra, who have acted as the Heads of the Department of Electronics and Telecommunication
Engineering, Jadavpur University, during my Master of Engineering course for their valuable
guidance and support. I convey my regards and respect to all Professors and associates in the
Department of Electronics and Telecommunication Engineering for their help and support. I am
exceedingly grateful to Prof. D. N. Tibarewala, School of Bioscience and Engineering, Jadavpur
University, who has provided great help, guidance and invigorating suggestions for my work. I am
thankful to the Department of Electronics and Telecommunication Engineering and the School of
Bioscience and Engineering, Jadavpur University for their support and assistance with materials,
equipments and subjects for conducting extensive experiments in the laboratories.

I would like to extend my sincere respect and thanks to all the seniors who have provided constant
support and guidance throughout my work, especially, Anwesha Khasnobish, Anwesha Banerjee,
Saugat Bhattacharyya, Anuradha Saha, Pratyusha Rakshit, Sriparna Saha, Arup Kumar Sadhu and
Ramadoss Janarthanan. I consider myself fortunate to have worked with such friendly and
motivating seniors who helped me in cultivating my knowledge and improving my skills. I am
grateful to my friend and classmate Monalisa Pal, for her unfaltering help in clarifying my doubts
at any hour of the day. I would also like to convey my gratitude to my juniors Dwaipayan Sardar
and Debobrota Basu for their help and incitement.

Finally, I am immensely indebted to my parents, family and friends for their continuous support
and encouragement that made me believe in myself and provided the strength to work hard.

Date: Shreyasi Datta
Place: Dept. of Electronics and Telecommunication Engineering
Examination Roll No.: M4ETC14-24
Jadavpur University

iv
Preface
The idea behind the development of next generation Human Computer Interfaces (HCI) is to
provide healthy individuals an easier and better life while providing diseased individuals assistance
and rehabilitation. Such devices are based on some form of input bio-modality that is processed
efficiently to provide the desired results while being interfaced with the computer. HCI based
control of rehabilitative aids, communicative devices and robotic equipments are gaining a
tremendous significance as observed from recent literature. This direction of research is the focus
of this thesis. Different aspects of HCI applications related to cognitive and haptic tasks using three
different bio-modalities, namely, brain signals through Electroencephalogram (EEG), eye
movements through Electrooculogram (EOG) and tactile perception through tactile pressure
images, have been studied using a wide variety of Machine Learning algorithms.

With respect to EEG based Brain Computer Interfacing, the novelty of the current research lie in
the design of artificial pathways bypassing the natural neural pathways of object recognition as
well as visual- motor co-ordination using artificial neural networks, the study of haptic perceptual
abilities from brain responses with respect to recognition of embossed digit/surface texture from
tactile stimulus or vibrotactile stimulus, proposing a modified Artificial Bee Colony Optimization
algorithm for the selection of the optimal number of electrodes for cognitive activity recognition
from EEG, using a novel EEG feature extraction method based on similarity measure between
channels and a novel Fuzzy Logic based classifier for EEG classification, and EEG generation by
differential evolution of the parameters of a non-linear Duffing Oscillator. The goals of these works
are either assistive technology development or proposals of efficient techniques of signal
processing for future applications. Providing touch sensation to artificial/robotic arms is another
area that has been explored. Such applications target robot aided teleoperation, telenavigation etc.,
robotic surgeries or providing touch sensation to a person with a prosthetic arm. This section of
work is devoted to object recognition from tactile pressure images acquired using sophisticated
touch sensors wrapped robot/human hands using image processing tools and pattern recognition
algorithms. EOG signals have been analyzed using intelligent algorithms in the next section
beginning with the classification of eye movements for various possible HCI applications. A
scheme of EOG tracking for detection and assistance to an eye movement characterized disorder is
provided. As eye movements convey a significant amount of information regarding a person’s
cognitive context as assessed from literature, EOG has also been used for cognitive context
recognition for applications in context aware pervasive computing systems. Finally, a bimodal
cognitive context recognition platform is proposed based on the feature level and decision level
fusion of EEG and EOG signals. All necessary data are acquired in the laboratories and each
section of work is supported by sufficient experimental results to validate the proposed
methodologies and algorithms.

v
List of Publications
1. Shreyasi Datta, Anuradha Saha, Amit Konar, “Perceptual Basis of Texture Classification
from Tactile Stimulus by EEG Analysis”, In National Conference in Brain and
Consciousness (NCBC), pp.38-45, 2013.

2. Anwesha Khasnobish, Shreyasi Datta, Monalisa Pal, Amit Konar, D.N. Tibarewala, “EEG
and Tactile Correlates of Object Shape Perception”, In National Conference on Brain and
Consciousness (NCBC), pp. 32-37, 2013.

3. Shreyasi Datta, Anwesha Khasnobish, Amit Konar, D. N. Tibarewala, R. Janarthanan,
“Performance Analysis of Object Shape Classification and Matching from Tactile Images
Using Wavelet Energy Features”, Procedia Technology, vol. 10, pp. 805-812, 2013.

4. Anwesha Banerjee, Shreyasi Datta, Monalisa Pal, Amit Konar, D.N. Tibarewala, R.
Janarthanan, “Classifying Electrooculogram to Detect Directional Eye Movements”,
Procedia Technology, vol. 10, pp. 67-75, 2013.

5. Shreyasi Datta, Anwesha Khasnobish, Amit Konar, D. N. Tibarewala, R. Janarthanan,
“Object Shape and Size Recognition from Tactile Images”, In 2013 IEEE International
Conference on Control, Communication and Computing (ICCC), pp. 16-21, 2013.

6. Monalisa Pal, Anwesha Banerjee, Shreyasi Datta, Amit Konar, D. N. Tibarewala, R.
Janarthanan, “Electrooculography based blink detection to prevent Computer Vision
Syndrome”, In 2014 IEEE International Conference on Electronics, Computing and
Communication Technologies (IEEE CONECCT), pp. 1-6, 2014.

7. Shreyasi Datta, Anwesha Banerjee, Amit Konar, D.N. Tibarewala, “Electrooculogram
Based Cognitive Context Recognition”, In 2014 IEEE International Conference on
Electronics, Communication and Instrumentation (ICECI), pp. 1-4, 2014.

8. Anwesha Khasnobish, Shreyasi Datta, Amit Konar, D.N. Tibarewala, “Neural Encoding of
Rigid Object-Shape Perception from Visually Stimulated EEG Signals”, In IEEE
International Conference on Control, Instrumentation, Energy and Communication
(CIEC), pp. 197-201, 2014.

9. Shreyasi Datta, Anwesha Banerjee, Monalisa Pal, Amit Konar, D. N. Tibarewala, R.
Janarthanan, “Blink Recognition to Detect the Possibility of Eye Dystonia based on
Electrooculogram Analysis”, In IEEE International Conference on Control,
Instrumentation, Energy and Communication (CIEC), pp. 236-240, 2014.

10. Sriparna Saha, Shreyasi Datta, Amit Konar, R. Janarthanan, “A Study of Emotion
Recognition from Body Gestures Using Kinect Sensor”, In IEEE International Conference
on Communication and Signal Processing (ICCSP), pp. 648-652, 2014.

11. Anwesha Khasnobish, Shreyasi Datta, Amit Konar, D.N. Tibarewala, R. Janarthanan,
“Object Shape Recognition from EEG Signals with Tactile,Visuo-Tactile and Audio-
Tactile Stimuli”, In IEEE International Conference on Communication and Signal
Processing (ICCSP), pp. 696-700, 2014.
vi
12. Anwesha Banerjee, Shreyasi Datta, Amit Konar, D. N. Tibarewala, R. Janarthanan,
“Electrooculogram based Detection of Visual Memory Recall Process”, In IEEE
International Conference on Communication and Signal Processing (ICCSP), pp. 711-715,
2014.

13. Shreyasi Datta, Anwesha Khasnobish, Amit Konar, D. N. Tibarewala, Atulya K. Nagar,
“EEG Based Artificial Learning of Motor Coordination for Visually Inspired Task using
Neural Networks”, In 2014 International Joint Conference on Neural Networks (IJCNN),
IEEE WCCI 2014. (accepted)

14. Shreyasi Datta, Pratyusha Rakshit, Amit Konar, Atulya K. Nagar, “Selecting the Optimal
EEG Electrode Positions for a Cognitive Task using an Artificial Bee Colony with
Adaptive Scale Factor Optimization Algorithm”, In 2014 IEEE Congress on Evolutionary
Computation (CEC), IEEE WCCI 2014. (accepted)

15. Anwesha Khasnobish, Shreyasi Datta, Dwaipayan Sardar, Amit Konar, D. N. Tibarewala,
Atulya K. Nagar, “EEG Analysis for Digit Recognition by Tactile and Vibrotactile
Stimulations,” In 11th IASTED International Conference on Biomedical Engineering
(BioMed 2014), 2014. (accepted)

16. Anwesha Banerjee, Monalisa Pal, Shreyasi Datta, Amit Konar, D. N. Tibarewala, “Eye
Movement Sequence Analysis using Electrooculogram to Assist Autistic Children”,
Biomedical Signal Processing and Control, ISSN: 1746-8094, Elsevier, 2014.
(provisionally accepted)

17. Anwesha Banerjee, Shreyasi Datta, Monalisa Pal, Amit Konar, D.N. Tibarewala,
“Voluntary Eye Movement Controlled Multitasking Graphical User Interface based on
Electrooculogram”, Medical and Biological Engineering and Computing, ISSN: 0140-
0118 (Print Version), 1741-0444 (Electronic Version), Springer. (submitted)

18. Anwesha Khasnobish, Shreyasi Datta, Dwaipayan Sardar, D. N. Tibarewala, Amit Konar,
“Interfacing Robotic Tactile Sensation with Human Vibrotactile Perception for Digit
Recognition”, Robotics and Autonomous Systems: Special Issue on Emerging Spatial
Competences: From Machine Perception to Sensorimotor Intelligence, ISSN: 0921-8890,
Elsevier. (submitted)

vii
Contents
Chapter 1: Introduction
1.1 Introduction to Human Computer Interfaces 2
1.2 Perceptual Abilities from Cognitive and Haptic Modalities 3
1.2.1 Perception 3
1.2.2 Cognitive and Haptic Information Processing from Bio-modalities 3
1.2.3 The Electroencephalogram 5
1.2.4 The Electrooculogram 8
1.2.5 Touch Perception in Robotics 10
1.3 Principles of Computational Intelligence Required 10
1.3.1 Pattern Recognition 10
1.3.2 Optimization 12
1.4 Application Areas of the Current Research 12
1.4.1 Brain Computer Interfacing 12
1.4.2 Rehabilitation 13
1.4.3 Robotic Touch Sensation 13
1.4.4 Context Aware Pervasive Computing Systems 13
1.5 Objective of the Thesis 13
1.6 Organization of the Thesis 14

Chapter 2: Standard Tools and Techniques
2.1 Feature Extraction Methods from EEG and EOG 19
2.1.1 Autoregressive Parameters 19
2.1.2 Adaptive Autoregressive Parameters 20
2.1.3 Hjorth Parameters 21
2.1.4 Power Spectral Density/Band Power Estimates 22
2.1.5 Wavelet Decomposition based Features 23
2.1.6 Temporal Differences 25
2.1.7 Feature Processing and Normalization 26
2.2 Classification and Regression Algorithms 26
2.2.1 Support Vector Machine Classifier 26
2.2.2 k-Nearest Neighbour Classifier 28
2.2.3 Linear Discriminant Analysis Classifier 29
2.2.4 Naïve Bayesian Classifier 30
2.2.5 Artificial Neural Networks for Classification and Regression 30
2.2.6 Ensemble Classifiers 33
2.3 Other Tools 34
2.3.1 Principal Component Analysis for Dimension Reduction 34
2.3.2 Independent Component Analysis for Signal Separation 36

viii
Chapter 3: Electroencephalogram Analysis for BCI Applications
3.1 EEG Acquisition Methods and Pre-processing 41
3.1.1 International 10-20 System of Electrode Placement 41
3.1.2 Acquisition Apparatus 41
3.1.3 Pre-processing Tools 43
3.2 Artificial Neural Network based Learning for Visual Tasks 43
3.2.1 Rigid Object Shape Perception 44
3.2.2 Motor Co-ordination 46
3.3 Haptic Perceptual Abilities from Stimulated EEG Analysis 53
3.3.1 Digit Recognition from Tactile and Vibrotactile Stimulus 53
3.3.2 Texture Recognition from Tactile Stimulus 59
3.4 Selecting the Optimal EEG Electrodes by a modified Artificial Bee Colony
Optimization Algorithm 61
3.5 A Novel Cognitive Activity Recognition Platform: Channel Similarity features and
Interval Type 2 Fuzzy System Classifier 72
3.6 EEG Generation using a Duffing Oscillator 77
3.7 Summary 82

Chapter 4: Tactile Perception for Robotics and Prosthetics
4.1 Tactile Data Acquisition Methods 86
4.1.1 PPS TactArray 86
4.1.2 Tekscan Grip Research 87
4.2 Object Recognition from Tactile Data 88
4.2.1 Experimental Design 88
4.2.2 Hierarchical Object Shape and Size Classification with Statistical features 89
4.2.3 Object shape Classification and Similarity Matching with Wavelet Energy
Features 94
4.3 Interfacing Robotic Tactile Sensation for Embossed Digit Recognition with Human
Perception 97
4.3.1 Classification Algorithm: Fuzzy k-NN 98
4.3.2 Embossed Digit Recognition from Tactile Pressure Images 99
4.3.3 Vibrotactile Stimulation for Human Perception Analysis 101
4.3.4 Results and Performance Analysis 102
4.4 Summary 105

Chapter 5: Electrooculogram Analysis for HCI Applications
5.1 Electrooculogram Acquisition and Pre-processing 109
5.2 Classification of Eye Movements 110
5.2.1 Fundamentals 110
5.2.2 Eye Movement Sequence Analysis and Autism Assistance 115
5.3 Reading Speed Recognition from EOG 118
5.4 Eye movements for Cognitive Context Analysis 120
ix