Heuristic Ensembles of Filters for Accurate and Reliable Feature Selection

254 Pages·2016·4.33 MB·English

Heuristic Ensembles of Filters
for Accurate and Reliable
Feature Selection

Ghadah Nasser Aldehim

Submitted for the degree of Doctor of Philosophy

School of Computing Sciences
University of East Anglia
December 2015

This copy of the thesis has been supplied on condition that anyone who consults it is
understood to recognise that its copyright rests with the author and that use of any
information derived there from must be in accordance with current UK Copyright Law.
In addition, any quotation or extract must include full attribution.

I
Abstract

Feature selection has become increasingly important in data mining in recent years.
However, the accuracy and stability of feature selection methods vary considerably
when used individually, and yet no rule exists to indicate which one should be used for
a particular dataset. Thus, an ensemble method that combines the outputs of several
individual feature selection methods appears to be a promising approach to address the
issue and hence is investigated in this research.
This research aims to develop an effective ensemble that can improve the accuracy and
stability of the feature selection. We proposed a novel heuristic ensemble of filters
(HEF). It combines two types of filters: subset filters and ranking filters with a heuristic
consensus algorithm in order to utilise the strength of each type. The ensemble is tested
on ten benchmark datasets and its performance is evaluated by two stability measures
and three classifiers. The experimental results demonstrate that HEF improves the
stability and accuracy of the selected features and in most cases outperforms the other
ensemble algorithms, individual filters and the full feature set.
The research on the HEF algorithm is extended in several dimensions; including more
filter members, three novel schemes of mean rank aggregation with partial lists, and
three novel schemes for a weighted heuristic ensemble of filters. However, the
experimental results demonstrate that adding weight to filters in HEF does not achieve
the expected improvement in accuracy, but increases time and space complexity, and
clearly decreases stability. Therefore, the core ensemble algorithm (HEF) is
demonstrated to be not just simpler but also more reliable and consistent than the later
more complicated and weighted ensembles.
In addition, we investigated how to use data in feature selection, using ALL or PART of
it. Systematic experiments with thirty five synthetic and benchmark real-world datasets
were carried out.

II
Table of Contents
Abstract ......................................................................................................................................... II
Table of Contents ........................................................................................................................ III
List of Figures ............................................................................................................................ VII
List of Tables .............................................................................................................................. IX
List of Abbreviations .................................................................................................................. XI
Publications ............................................................................................................................... XIII
Acknowledgements .................................................................................................................. XIV
Chapter 1 ....................................................................................................................................... 1
Introduction ................................................................................................................................... 1
1.1 Background ......................................................................................................................... 2
1.2 Motivation ........................................................................................................................... 3
1.3 Research Aim and Objectives ............................................................................................. 4
1.4 Research Questions ............................................................................................................. 5
1.5 Contributions....................................................................................................................... 5
1.6 Structure of Thesis .............................................................................................................. 6
Chapter 2 ....................................................................................................................................... 8
Literature Review on Feature Selection Ensemble ....................................................................... 8
2.1 Introduction ......................................................................................................................... 9
2.2 Feature Selection ............................................................................................................... 10
2.2.1 General Procedure of Feature Selection ..................................................................... 11
2.2.1.1 Subset Generation: .............................................................................................. 12
2.2.1.2 Subset Evaluation:............................................................................................... 14
2.2.1.3 The Stopping Criterion: ...................................................................................... 14
2.2.1.4 The Validation Procedure: .................................................................................. 15
2.3 Filter: ................................................................................................................................. 15
2.3.1. Distance Measures (Weight) ..................................................................................... 16
2.3.2. Information Measures ............................................................................................... 18
2.3.3. Dependency Measures (Correlation) ........................................................................ 22
2.3.4. Consistency Measures ............................................................................................... 24
2.3.5 Advantages and Disadvantages of Filters .................................................................. 25
2.4 Wrappers ........................................................................................................................... 26
2.4.1 Sequential Search Techniques ................................................................................... 27
2.4.1.1 Greedy Search ..................................................................................................... 27
2.4.1.2 Floating Search Strategy ..................................................................................... 29
2.4.1.3 Best-first Search Wrapper ................................................................................... 30
2.4.2 Exponential and Randomised Search Algorithms...................................................... 30
2.4.2.1 Beam Search ....................................................................................................... 30
2.4.2.2 Simulated Annealing ........................................................................................... 31
2.4.2.3 Genetic Algorithms ............................................................................................. 31
2.4.3 Advantages and Disadvantages of Wrappers ............................................................. 31
2.5 Hybrid ............................................................................................................................... 32
2.5.1 Sequential Searches with Hybrid Evaluation ............................................................. 32
2.5.2 Random Searches with Hybrid Evaluation ................................................................ 35
2.5.3 Advantages and Disadvantages of Hybrid Methods .................................................. 36
2.6 Introduction to Ensemble .................................................................................................. 37
2.6.1 Methods for Constructing Ensemble .......................................................................... 38
III
2.7 Ensemble of Feature Selection .......................................................................................... 39
2.7.1 The Ensemble Idea for Feature Selection .................................................................. 40
2.7.2 Combination Methods of Ensemble Feature Selection .............................................. 40
2.8 Researches in Feature Selection Ensemble ....................................................................... 41
2.8.1 Ensemble of Single Feature Selection Technique with Instance Level Perturbation. 41
2.8.2 Ensemble of Multiple Feature Selection Techniques ................................................. 43
2.9 Summary ........................................................................................................................... 48
Chapter 3 ..................................................................................................................................... 49
Methodology ............................................................................................................................... 49
3.1 Introduction ....................................................................................................................... 50
3.2 Proposed Ensemble of Feature Selection .......................................................................... 50
3.3 Using Data in Feature Selection ........................................................................................ 53
3.4 Evaluation Methods .......................................................................................................... 53
3.4.1 Stability Methods as an Indicator of Reliability Measure of Feature Selection ......... 54
3.4.2 Classification Performance as Effectiveness Measure of Feature Selection .............. 58
3.4.2.1 Validation Techniques ........................................................................................ 59
3.4.2.2 Classification Performance Measures ................................................................. 60
3.4.2.3 Statistical Tests for Comparison ......................................................................... 61
3.4.2.4 Algorithms for Classification .............................................................................. 62
3.5 Comparison Strategies ...................................................................................................... 63
3.6 System Software Design ................................................................................................... 64
3.6.1 WEKA........................................................................................................................ 64
3.6.2 Java Code ................................................................................................................... 64
3.7 Experiment Design ............................................................................................................ 65
3.7.1 Data ............................................................................................................................ 65
3.7.2 Experiment Procedure ................................................................................................ 66
Chapter 4 ..................................................................................................................................... 68
Heuristic Ensemble of Filters ...................................................................................................... 68
4.1 Introduction ....................................................................................................................... 69
4.2 Heuristic Ensemble of Filters (HEF) ................................................................................ 69
4.2.1 Proposed Heuristic Ensemble of Filters (HEF) .......................................................... 69
4.2.2 Choice of Individual Filters ....................................................................................... 71
4.2.3 The Heuristic Rules .................................................................................................... 74
4.3 Experiments ...................................................................................................................... 75
4.3.1 Data ............................................................................................................................ 75
4.3.2 Experiment Design and Procedure ............................................................................. 76
4.4 Results ............................................................................................................................... 77
4.4.1 Number of Selected Features ..................................................................................... 77
4.4.2 Accuracy Evaluation .................................................................................................. 78
4.5 Conclusion ........................................................................................................................ 81
Chapter 5 ..................................................................................................................................... 83
Determining Appropriate Approaches for Using Data in Feature Selection .............................. 83
5.1 Introduction ....................................................................................................................... 84
5.2 The PART and ALL Methods ........................................................................................... 85
5.3 Related Works about PART and ALL Methods: .............................................................. 86
5.4 Experiments ...................................................................................................................... 88
5.4.1 Data ............................................................................................................................ 88
5.4.1.1 Real world Bench Mark Data .............................................................................. 88
5.4.1.2 Generation of Synthetic Datasets ........................................................................ 88
5.4.2 Experiment Design and Procedure ............................................................................. 95
IV
5.5 Results ............................................................................................................................... 97
5.5.1 Real-World Bench Mark Dataset ............................................................................... 97
5.5.1.1 Number of Selected Features .............................................................................. 97
5.5.1.2. Accuracy Evaluation with Different Classifiers .............................................. 100
5.5.1.3. Stability Evaluation .......................................................................................... 103
5.5.2 Results on Synthetic Datasets .................................................................................. 108
5.5.2.1. Accuracy Evaluation ........................................................................................ 108
5.5.2.2. Stability Evaluation .......................................................................................... 111
5.5.3 Experiment with Benchmark Synthetic Data ........................................................... 121
5.6 Discussion ....................................................................................................................... 123
5.6.1 Real-world Benchmark Datasets .............................................................................. 123
5.6.2 Synthetic Datasets .................................................................................................... 124
5.7 Conclusion ...................................................................................................................... 125
Chapter 6 ................................................................................................................................... 128
Improving the Heuristic Ensemble of Filters ............................................................................ 128
6.1 Introduction ..................................................................................................................... 129
6.2 Adding Wrapper after HEF ............................................................................................. 130
6.2.1 Proposed Hybrid Heuristic Ensemble of Filters (HHEF) ........................................ 130
6.2.2 Choice of Wrappers ................................................................................................. 131
6.3 Adding More Filters in HEF ........................................................................................... 132
6.3.1 Choice of Filters ....................................................................................................... 132
6.3.2 Choice of Number of Filters .................................................................................... 133
6.4 Changing the Aggregation Method for Combining Feature Subsets .............................. 134
6.4.1 Converting Feature Subset to Ranking Subset ......................................................... 134
6.4.2 Dealing with Partial List or (Top-K List) ................................................................ 135
6.4.3 Ranking Aggregation Methods ................................................................................ 136
6.5 Experiments .................................................................................................................... 139
6.5.1 Experiment Design and Procedure ........................................................................... 139
6.6 Results ............................................................................................................................. 140
6.6.1 Hybrid Heuristic Ensemble of Filters (HHEF) ........................................................ 141
6.6.1.1 Accuracy Evaluation ......................................................................................... 141
6.6.1.2 Similarity Evaluation ........................................................................................ 144
6.6.2 Adding More Filters in HEF .................................................................................... 145
6.6.2.1 Accuracy Evaluation ......................................................................................... 146
6.6.1.2 Similarity evaluation ......................................................................................... 149
6.6.2.3 Time Complexity Analysis ............................................................................... 154
6.6.3 Changing the Aggregation Method for Combining Feature Subsets: ...................... 157
6.6.3.1 Accuracy Evaluation ......................................................................................... 158
6.6.3.2 Stability Evaluation ........................................................................................... 161
6.7. Conclusion ..................................................................................................................... 162
Chapter 7 ................................................................................................................................... 164
Weighted Heuristic Ensemble of Filters ................................................................................... 164
7.1 Introduction ..................................................................................................................... 165
7.2 Related Work .................................................................................................................. 165
7.3 Weighted Heuristic Ensemble Filters (WHEF)............................................................... 167
7.3.1 Fixed Weight Methods (FWHEF) ............................................................................ 168
7.3.2 Variable Weight Based on Validation Set (VWHEF).............................................. 170
7.3.3 Selective Filters Based on Validation Set (SFHEF) ................................................ 174
7.4 Experiments .................................................................................................................... 175
7.4.1 Experimental Design Procedure and Evaluation methods ....................................... 175
V
7.5 Results ............................................................................................................................. 177
7.5.1 Accuracy Evaluation with Different Classifiers ...................................................... 177
7.5.2. Stability Evaluation ................................................................................................. 183
7.5.3. Runtime Performance ............................................................................................. 189
7.6. Discussion and Evaluation ............................................................................................. 191
7.7. Conclusion ..................................................................................................................... 192
Chapter 8 ................................................................................................................................... 194
Evaluation and Discussion ........................................................................................................ 194
8.1 Introduction ..................................................................................................................... 195
8.2 Overview of the Research as a Whole ............................................................................ 196
8.3 Heuristic Ensemble of Filters (HEF) .............................................................................. 198
8.4 Use of Data in FS ............................................................................................................ 201
8.5 Aggregation Method ....................................................................................................... 202
8.6 Weighed HEF.................................................................................................................. 206
8.7 Comparison between HEF and Other Research .............................................................. 208
8.8 Summary ......................................................................................................................... 213
Chapter 9 ................................................................................................................................... 214
Conclusions ............................................................................................................................... 214
9.1 General Conclusions ....................................................................................................... 215
9.2 Limitations ...................................................................................................................... 216
9.3 Further Work ................................................................................................................... 216
References ................................................................................................................................. 220
Appendices ................................................................................................................................ 229
Appendix A: Further results from Chapter 5 ........................................................................ 230
Appendix B: Further results from Chapter 7 ........................................................................ 234
Appendix C: Further results from Chapter 8 ........................................................................ 239

VI
List of Figures
Figure 2. 1: Feature Selection Process (Dash and Liu, 1997) .................................................................... 12
Figure 2.2: Illustration of the filter process ................................................................................................ 16
Figure 2.3: Illustration of the wrapper process ........................................................................................... 27
Figure 3.1: The proposed ensemble of feature selection ............................................................................ 51
Figure 4.1: Framework of HEF for feature selection ................................................................................. 70
Figure 5. 1: ALL Method ........................................................................................................................... 86
Figure 5. 2: PART Method ......................................................................................................................... 86
Figure 5.3: Number of selected features by the PART method on the Colon dataset ................................ 99
Figure 5.4: Number of selected features by the PART method on the Leukaemia dataset ........................ 99
Figure 5.5: The similarity measures of IATI with the features selected by the filters, comparing the PART
with the ALL approaches ................................................................................................................ 105
Figure 5.6: The similarity measures of ICW with the features selected by the filters, comparing the PART
with the ALL approaches ................................................................................................................ 105
Figure 5.7 : The difference (∆acc) between the average accuracies of the three classifiers trained by the
ALL and PART approaches as well as the averages of similarity measures ................................... 107
Figure 5.8: Accuracy of NB classifier obtained for S1 to S8 datasets with both methods ....................... 108
Figure 5.9: Accuracy of NB classifier of the S2NR4 to S8NR16 datasets with both methods ................ 109
Figure 5.10: Accuracies of NB classifier of the S2Noise5 to S8Noise10 datasets with both methods .... 110
Figure 5.11: IATI comparison between each filter subset with optimal subset on: (a) S1, S2 and S3 (b) S4,
S5 and S6 ......................................................................................................................................... 111
Figure 5.12: ICW comparison between each filter subset with optimal subset on: (a) S1, S2 and S3 (b) S4,
S5 and S6 ......................................................................................................................................... 111
Figure 5.13: Comparing feature selector's stability (CWrel, ATI) with the PART method on: (a) S1, S2
and S3 (b) S4, S5 and S6 ................................................................................................................. 112
Figure 5.14: IATI comparison between each filter subset with optimal subset on: (a) S1, S4 and S7 (b) S2,
S5 and S8 ......................................................................................................................................... 114
Figure 5.15: ICW comparison between each filters subset with optimal subset on: (a) S1, S4 and S7 (b)
S2, S5 and S8................................................................................................................................... 115
Figure 5.16: Comparing feature selector's stability (CWrel, ATI) with the PART method on: (a) S1, S4
and S7 (b) S2, S5 and S8 ................................................................................................................. 115
Figure 5.17: IATI comparison between each filter subset with optimal subset on: (a) S2NR4, S5NR4 and
S8NR4 (b) S2NR16, S5NR16 and S8NR16 .................................................................................... 116
Figure 5.18: ICW comparison between each filter subset with optimal subset on: (a) S2NR4, S5NR4 and
S8NR4 (b) S2NR16, S5NR16 and S8NR16 .................................................................................... 117
Figure 5. 19: Comparing feature selector stability (CWrel, ATI) with PART method on: (a) S2NR4,
S5NR4 and S8NR4 (b) S2NR16, S5NR16 and S8NR16 ................................................................ 117
Figure 5.20: IATI comparison between each filter subset with optimal subset on: (a) S2Noise5, S5Noise5
and S8Noise5 (b) S2Noise10, S5Noise10 and S8Noise10 .............................................................. 120
Figure 5.21: ICW comparison between each filter subset with optimal subset on: (a) S2Noise5, S5Noise5
and S8Noise5 (b) S2Noise10, S5Noise10 and S8Noise10 .............................................................. 120
Figure 5.22: Comparing feature selector's stability (CWrel, ATI) with the PART method on: (a)
S2Noise5, S5Noise5 and S8Noise5 (b) S2Noise10, S5Noise10 and S8Noise10 ............................ 120
Figure 5.23: IATI comparison between each filter subset with optimal subset over synthetic data which
were widely used. ............................................................................................................................ 122
Figure 5.24: ICW comparison between each filter subset with optimal subset over synthetic data which
were widely used. ............................................................................................................................ 122
Figure 5.25: Comparing feature selection stability (CWrel, ATI) with the PART method. ..................... 122
Figure 5.26: Accuracy of NB classifier over synthetic data widely used on both methods ..................... 122
Figure 6.1: Framework of hybrid ensemble of FS ................................................................................... 131
Figure 6.2: Average number of features selected by HHEF ..................................................................... 141
Figure 6.3: The average test accuracy of NB classifiers trained with 2 HEF and 6 hybrid HEF ............. 142
VII
Figure 6.4: The average test accuracy of KNN classifiers trained with 2 HEF and 6 hybrid HEF .......... 142
Figure 6.5: The average test accuracy of SVM trained with 2 HEF and 6 hybrid HEF ........................... 143
Figure 6.6: Accuracy comparison using SVM of HEF and all hybrid ensemble approaches against each
other with Nemenyi test. .................................................................................................................. 143
Figure 6.7: The stability measures of ATI with the features selected by 2 HEF and 6 hybrid HEF ........ 144
Figure 6.8: Stability comparison using ATI of HEF and all hybrid ensemble approaches against each
other with the Nemenyi test ............................................................................................................. 145
Figure 6.9: Results of the Nemenyi test used to evaluate the accuracy of KNN of each filter and ensemble
approaches against each other ......................................................................................................... 148
Figure 6.10: Stability comparison using ATI of each filters and ensemble approaches against each other
with Nemenyi test ............................................................................................................................ 152
Figure 6.11: Stability comparison using CWrel of each filters and ensemble approaches against each other
with Nemenyi test ............................................................................................................................ 152
Figure 6.12: Average accuracy and stability of HEF+5F and 5 filter members on 10 real datasets, focusing
on each evaluation measure ............................................................................................................. 153
Figure 6. 13: Average accuracy and stability of HEF+5F and 5 filter members on 10 real dataset, focusing
on each FS technique ....................................................................................................................... 153
Figure 6.14: Average runtime performances of 9 real datasets (excluding Ovarian) using three classifiers
......................................................................................................................................................... 157
Figure 6. 15: Results of the Nemenyi test was used to evaluate the accuracy of NB of three different
schemes of mean rank aggregation against each other .................................................................... 160
Figure 6. 16: Average test accuracy over 10 real datasets with three different schemes of mean rank
aggregation focusing on the three classifiers ................................................................................... 161
Figure 7.1: Framework of FWHEF .......................................................................................................... 169
Figure 7.2: Determining the weight by classification accuracy on the validation dataset ........................ 171
Figure 7.3: Framework of VWHEF ......................................................................................................... 172
Figure 7.4: Framework of SFHEF ............................................................................................................ 175
Figure 7.5: The average test accuracy of NB using 10 datasets focusing on different methods .............. 181
Figure 7.6: The average test accuracy of KNN using 10 datasets focusing on different methods ........... 182
Figure 7.7: The average test accuracy of SVM using 10 datasets focusing on different methods ........... 182
Figure 7.8: Comparison of all ensemble approaches against each other by SVM, using 25% of selected
features with Nemenyi test .............................................................................................................. 183
Figure 7.9: The average ATI using 10 datasets focusing on different methods ....................................... 185
Figure 7.10: The average CWrel using 10 datasets focusing on different methods ................................. 186
Figure 7.11: ATI comparison of all ensemble approaches against each other with Nemenyi test using
75%, 50% and 25% of selected features .......................................................................................... 187
Figure 7.12: The mean stability measures of ATI and CWrel with the features selected by proposed
ensemble approaches over 10 runs of 10-fold cross-validation ....................................................... 188
Figure 7.13: Average runtime performance of 9 real-world datasets (excluding Ovarian) using three
classifiers ......................................................................................................................................... 191
Figure 8.1: Naming strategy of each version of HEF ............................................................................... 196
Figure 8. 2: a) Average test accuracy (b) Average stability over 10 real datasets with two different
aggregation methods ........................................................................................................................ 205
Figure 8.3: Average number of features selected using two different aggregation methods. ................... 206

VIII
List of Tables
Table 3. 1: Confusion matrix for a two-class prediction problem .............................................................. 60
Table 3.2: Description of the benchmark datasets ...................................................................................... 65
Table 4. 1: Number of selected features for each dataset by the four filters and two ensembles ............... 77
Table 4.2: The accuracies of NB models trained with all the features and the features selected by filters
and heuristic ensembles. .................................................................................................................... 79
Table 4.3: The accuracies of KNN models trained with all the features and the features selected by filters
and heuristic ensembles ..................................................................................................................... 79
Table 4.4: The accuracies of SVM models trained with all the features and the features selected by filters
and heuristic ensembles. .................................................................................................................... 80
Table 5. 1: Summary of the 9 synthetic datasets from S1 to S9 without noise injection ........................... 91
Table 5.2: Summary of the 6 synthetic dataset with different without noise injection ........................ 92
Table 5.3: Summary of the 6 synthetic datasets after adding noise to the class y ...................................... 94
Table 5.4: Average number of selected features by each filters and ensemble .......................................... 98
Table 5. 5: The accuracies of Naïve Bayesian classifier trained with all the features and the features
selected by filters and heuristic ensembles by the PART method ................................................... 100
Table 5.6: The accuracies of the KNN models trained with all the features and the features selected by
filters and heuristic ensembles by the PART method ...................................................................... 101
Table 5.7: The accuracies of the SVM models trained with all the features and the features selected by
filters and heuristic ensembles by the PART method ...................................................................... 102
Table 5.8: The stability measures of ATI with the features selected by filters and heuristic ensembles over
10 runs of 10-fold cross-validation .................................................................................................. 104
Table 5.9: The stability measures of CWrel with the features selected by filters and heuristic ensembles
over 10 runs of 10-fold cross-validation .......................................................................................... 104
Table 6.1: The accuracies of NB models trained with all the features and the features selected by filters
and heuristic ensembles ................................................................................................................... 147
Table 6.2: The accuracies of KNN models trained with all the features and the features selected by filters
and heuristic ensembles ................................................................................................................... 147
Table 6.3: The accuracies of SVM models trained with all the features and the features selected by filters
and heuristic ensembles ................................................................................................................... 148
Table 6.4: The number of best and worst accuracies summarisation of three classifiers ......................... 149
Table 6.5: The stability measures of ATI with the features selected by 5 filters and 4 heuristic ensembles
......................................................................................................................................................... 150
Table 6.6: The stability measures of CWrel with the features selected by 5 filters and 4 heuristic
ensembles ........................................................................................................................................ 151
Table 6.7: Running time (seconds) for each filter with NB classifier ...................................................... 155
Table 6.8: Running time (seconds) for each filter with KNN classifier ................................................... 156
Table 6.9: Running time (seconds) for each filter with SVM classifier ................................................... 156
Table 6.10: The accuracies of NB models trained with three different schemes of mean rank aggregation
......................................................................................................................................................... 158
Table 6.11: The accuracies of KNN models trained with three different schemes of mean rank
aggregation ...................................................................................................................................... 159
Table 6.12: The accuracies of SVM models trained with three different schemes of mean rank
aggregation ...................................................................................................................................... 159
Table 6.13: Average test accuracy over 10 real datasets with three different schemes of mean rank
aggregation focusing on the three classifiers ................................................................................... 160
Table 6.14: The stability measures of ATI with three different schemes of mean rank aggregation ....... 161
Table 6.15: The stability measures of CWrel with three different schemes of mean rank aggregation ... 162
Table 7.1 : The average test accuracy of NB classifiers trained with the features selected by HEFb,
FWHEF, VWHEF and SFHEF, with 75% of these features being selected .................................... 177
Table 7.2: The average test accuracy of KNN classifiers trained with the features selected by HEFb,
FWHEF, VWHEF and SFHEF, with 75% of these features being selected .................................... 178
IX
Table 7. 3: The average test accuracy of SVM classifiers trained with the features selected by HEFb,
FWHEF, VWHEF and SFHEF, with 75% of these features being selected .................................... 178
Table 7.4: Average test accuracy over 10 real datasets focusing on the classifier ................................... 180
Table 7.5: The stability measures of ATI with the features selected by four ensemble approaches over 10
runs of 10-fold cross-validation ....................................................................................................... 184
Table 7.6: The stability measures of CWrel with the features selected by four ensemble approaches over
10 runs of 10-fold cross-validation .................................................................................................. 184
Table 7.7: Running time (seconds) for each ensemble approach with NB classifier on 10 real datasets . 189
Table 7.8: Running time (seconds) for each ensemble approach with KNN classifier on 10 real datasets
......................................................................................................................................................... 190
Table 7.9: Running time (seconds) for each ensemble approach with SVM classifier on 10 real datasets
......................................................................................................................................................... 190
Table 8.1: List of abbreviations for each version of HEF ........................................................................ 195
Table 8.2: Average test accuracy and stability over 10 real benchmark datasets with two different
aggregation methods ........................................................................................................................ 204
Table 8.3: Comparison of HEFb-75% with other EFS studies. Values given are average accuracy;
parentheses show the number of features selected, and the last column presents the methods of other
FS studies (FS + Classifier + Evaluation Scheme). ......................................................................... 209

X