anytime algorithms for learning anytime classifiers saher - Technion

192 Pages·2008·1.46 MB·English

by  Saher Esmeir

ANYTIME ALGORITHMS FOR
LEARNING ANYTIME CLASSIFIERS
SAHER ESMEIR
Technion - Computer Science Department - Ph.D. Thesis PHD-2008-12 - 2008
Technion - Computer Science Department - Ph.D. Thesis PHD-2008-12 - 2008
ANYTIME ALGORITHMS FOR LEARNING
ANYTIME CLASSIFIERS
RESEARCH THESIS
SUBMITTED IN PARTIAL FULFILLMENT OF THE
REQUIREMENTS
FOR THE DEGREE OF DOCTOR OF PHILOSOPHY
SAHER ESMEIR
SUBMITTED TO THE SENATE OF THE TECHNION — ISRAEL INSTITUTE OF TECHNOLOGY
ELUL, 5768 HAIFA SEPTEMBER, 2008
Technion - Computer Science Department - Ph.D. Thesis PHD-2008-12 - 2008
Technion - Computer Science Department - Ph.D. Thesis PHD-2008-12 - 2008
THIS RESEARCH THESIS WAS DONE UNDER THE SUPERVISION OF
ASSOC. PROF. SHAUL MARKOVITCH IN THE DEPARTMENT OF
COMPUTER SCIENCE
ACKNOWLEDGMENTS
Words cannot suffice to describe my deep gratitude to my advi-
sor, Shaul Markovitch, who guided me during my studies and was
always available when I needed him. I learned from Shaul how to
become an independent researcher and how to think innovatively,
yet critically. His encouragement and support were crucial in shap-
ing my research. He taught me not to give up and to believe in what
I did. Above all, he perfectly balanced between guiding me profes-
sionally and maintaining a respectful and pleasant relationship.
I owe a notable debt to Evgeniy Gabrilovich for his help and
advice throughout my studies, as well as to Lev Finkelstein for his
contributions to the anytime classification project. I learned a huge
deal from the both of them.
I am also grateful to the following individuals, who shared their
wisdom, enriched my life with their insights, and were always willing
to assist me: Ron Begleiter, Sivan Bercovici, Ofer Egozi, Ran El-
Yaniv, Ariel Felner, Yaniv Hamo, Anat Hashavit, Nela Gurevich,
Sonya Liberman, Kira Radinsky, Ron Rubinstein, Shadi Saba, Sunil
Vadera, Kolman Vornovitsky, Haggai Toledano, Lina Zarivach, and
Shlomo Zilberstein.
My sincere thanks and appreciation to Microsoft Research and
Development Center at Haifa and Medic Vision Brain for hosting
me as summer intern, to the administrative staff at the Computer
Science Department for their help and assistance, and to Sharon
Kessler, who read significant portions of this manuscript and im-
proved it with her editorial comments, excitement and passion for
details.
Technion - Computer Science Department - Ph.D. Thesis PHD-2008-12 - 2008
And there are also those friends who heard me speak about this
project, day and night, during the last couple of years. Their im-
measurable love, support, and care gave me the power to work and
lead me to new paths.
Finally, I am deeply beholden to my loving family, who offered
me unconditional love and support throughout the course of this
thesis: my sister Samera, my brothers Samer and Esmeir, and my
parents, Mona and Nimer, who have raised me to be the person I
am today. Mom and dad: without you, nothing of this would have
been ever possible.
THE GENEROUS FINANCIAL HELP OF THE TECHNION, THE CHE
(COUNCIL FOR HIGHER EDUCATION), AND THE MIRIAM AND
AARON GUTWIRTH MEMORIAL FOUNDATION IS GRATEFULLY
ACKNOWLEDGED
Technion - Computer Science Department - Ph.D. Thesis PHD-2008-12 - 2008
To my parents,
Mona and Nimer Esmeir
Technion - Computer Science Department - Ph.D. Thesis PHD-2008-12 - 2008
Technion - Computer Science Department - Ph.D. Thesis PHD-2008-12 - 2008
Contents
Abstract 1
1 Introduction 5
1.1 Proposed Solution . . . . . . . . . . . . . . . . . . . . . . . . . . 8
1.2 Major Contributions . . . . . . . . . . . . . . . . . . . . . . . . . 10
1.3 Thesis Outline . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11
2 Resource-bounded Learning and Classification 13
3 Contract Anytime Learning of Accurate Trees 19
3.1 Top-down Induction of Decision Trees . . . . . . . . . . . . . . . 19
3.2 Limitation of Greedy Learners . . . . . . . . . . . . . . . . . . . 20
3.3 Tree-size as a Preference Bias . . . . . . . . . . . . . . . . . . . . 22
3.4 Fixed-depth Lookahead . . . . . . . . . . . . . . . . . . . . . . . 24
3.5 A Contract Algorithm for Learning Decision trees . . . . . . . . . 25
3.5.1 Evaluating Continuous Attributes by Sampling . . . . . . 30
3.5.2 Multiway Vs. Binary Splits . . . . . . . . . . . . . . . . . 32
3.5.3 Pruning the LSID3 Trees . . . . . . . . . . . . . . . . . . 32
3.5.4 Mapping Contract Time to Sample Size . . . . . . . . . . 33
3.6 Interruptible Induction of Decision Trees . . . . . . . . . . . . . . 34
3.6.1 Interruptible Learning by Sequencing . . . . . . . . . . . . 35
3.6.2 Iterative Improvement of Decision Trees . . . . . . . . . . 36
3.6.3 Incremental Learning: When New Examples Arrive . . . . 42
3.7 Empirical Evaluation . . . . . . . . . . . . . . . . . . . . . . . . 45
3.7.1 Experimental Methodology . . . . . . . . . . . . . . . . . 45
3.7.2 Fixed Time Comparison . . . . . . . . . . . . . . . . . . . 46
3.7.3 Anytime Behavior of the Contract Algorithms . . . . . . . 54
3.7.4 Anytime behavior of IIDT . . . . . . . . . . . . . . . . . . 59
3.7.5 IIDT as Incremental Learner . . . . . . . . . . . . . . . . 62
3.7.6 Comparison with Modern Decision Tree Learners . . . . . 63
Technion - Computer Science Department - Ph.D. Thesis PHD-2008-12 - 2008
4 Anytime Learning of Cost-sensitive Trees 71
4.1 Biasing the Sample Towards Low-cost Trees . . . . . . . . . . . . 71
4.2 Evaluating Sample Trees . . . . . . . . . . . . . . . . . . . . . . 72
4.2.1 Estimating Test Costs . . . . . . . . . . . . . . . . . . . . 73
4.2.2 Estimating Misclassification Costs . . . . . . . . . . . . . 73
4.3 Choosing a Split . . . . . . . . . . . . . . . . . . . . . . . . . . . 74
4.3.1 Choosing a Split: Illustrative Examples . . . . . . . . . . 75
4.3.2 Choosing a Split when Attributes are Numeric . . . . . . 78
4.4 Cost-sensitive Pruning . . . . . . . . . . . . . . . . . . . . . . . . 78
4.5 Setting the Parameters of ACT . . . . . . . . . . . . . . . . . . . 79
4.6 Empirical Evaluation . . . . . . . . . . . . . . . . . . . . . . . . 80
4.6.1 Datasets . . . . . . . . . . . . . . . . . . . . . . . . . . . 81
4.6.2 Methodology . . . . . . . . . . . . . . . . . . . . . . . . . 82
4.6.3 Fixed-time Comparison . . . . . . . . . . . . . . . . . . . 85
4.6.4 Comparing the Accuracy of the Learned Models . . . . . 91
4.6.5 Comparison of Anytime Behavior . . . . . . . . . . . . . . 92
4.6.6 Random Costs . . . . . . . . . . . . . . . . . . . . . . . . 93
4.6.7 Nonuniform Misclassification Costs . . . . . . . . . . . . . 93
5 Anytime Learning of Anycost Classifiers 99
5.1 Pre-constract: Classification Budget is Known to the Learner . . 100
5.1.1 Top-down Induction of Anycost Trees . . . . . . . . . . . 101
5.1.2 Greedy TDIDT$ Instantiations . . . . . . . . . . . . . . . 102
5.1.3 The Pre-Contract-TATA Algorithm . . . . . . . . . . . . 102
5.1.4 Interruptible Learning of Pre-contract Classifiers . . . . . 104
5.2 Contract: When Allocation is Made Right Before Classification . 105
5.2.1 Repertoire of Trees . . . . . . . . . . . . . . . . . . . . . . 106
5.2.2 Learning a repertoire with Nonuniform Cost Gaps . . . . 107
5.3 Interruptible: No Pre-determined Classification Budget . . . . . . 108
5.4 Empirical Evaluation . . . . . . . . . . . . . . . . . . . . . . . . 110
5.4.1 Pre-Contract Classification . . . . . . . . . . . . . . . . . 111
5.4.2 Contract Classification . . . . . . . . . . . . . . . . . . . . 115
5.4.3 Interruptible Classification . . . . . . . . . . . . . . . . . 118
6 Related Work 121
6.1 Non-greedy Cost-insensitive Decision Tree Inducers . . . . . . . . 121
6.1.1 Skewing . . . . . . . . . . . . . . . . . . . . . . . . . . . . 121
6.1.2 Other Cost-insensitive Decision-Tree Inducers . . . . . . . 124
6.2 Anytime Induction of Other Cost-insensitive Classifiers . . . . . . 127
6.3 Cost-sensitive Classification . . . . . . . . . . . . . . . . . . . . . 128
6.4 Other Learning Components . . . . . . . . . . . . . . . . . . . . 132
Technion - Computer Science Department - Ph.D. Thesis PHD-2008-12 - 2008