Classifier performance evaluation and comparison - icmla

183 Pages·2010·3.82 MB·English

Classifier performance evaluation and comparison
Classifier performance evaluation and
comparison
Jose A. Lozano, Guzmán Santafé, Iñaki Inza
Intelligent Systems Group
The University of the Basque Country
International Conference on Machine Learning and Applications (ICMLA 2010)
December 12-14, 2010
logo
- 1 -
Classifier performance evaluation and comparison
Outline of the Tutorial
1 Introduction
2 Scores
3 Estimation Methods
4 Hypothesis Testing
logo
- 2 -
Classifier performance evaluation and comparison
Introduction
Outline of the Tutorial
1 Introduction
2 Scores
3 Estimation Methods
4 Hypothesis Testing
logo
- 3 -
Physical Process
Usually unknown
Classifier performance evaluation and comparison
Introduction
Classification Problem
Data set
logo
- 4 -
Physical Process
Usually unknown
Classifier performance evaluation and comparison
Introduction
Classification Problem
Data set
logo Expert
- 5 -
Physical Process
Usually unknown
Classifier performance evaluation and comparison
Introduction
Supervised Classification
Data set
Learning from Experience
“Automate the work of the expert”
Tries to model ρ(X , C) Classification
Model
logo Expert
- 6 -
Classifier performance evaluation and comparison
Introduction
Supervised Classification
Data set Data set
Classification Model
Classifier labels new data (unknown class value)
Classification
Model
logo
Expert
- 7 -
Naive Bayes
Data set
...
Classifier performance evaluation and comparison
Introduction
Motivation for Honest Evaluation
Many classification paradigms
X4
...
Neural Net
logo
Decis- 8i-on Tree
Naive Bayes
?
?
Data set
...
Classifier performance evaluation and comparison
Introduction
Motivation for Honest Evaluation
Which is the best paradigm for a classification problem?
X4
...
?
Neural Net
logo
Decis- 9i-on Tree
Naive Bayes
Classifier performance evaluation and comparison
Introduction Data set
Motivation for Hon..e.st Evaluation Naive Bayes
Many parameter configurations
...
logo
- 10