Feature Extraction: Foundations and Applications

764 Pages·2006·12.969 MB·English

by  Isabelle Guyon, André Elisseeff (auth.), Isabelle Guyon, Masoud Nikravesh, Steve Gunn, Lotfi A. Zadeh (eds.)

IsabelleGuyon,SteveGunn,MasoudNikravesh,LotfiA.Zadeh(Eds.)
FeatureExtraction
StudiesinFuzzinessandSoftComputing,Volume 207
Editor-in-chief
Prof.JanuszKacprzyk
SystemsResearchInstitute
PolishAcademyofSciences
ul.Newelska6
01-447Warsaw
Poland
E-mail:[email protected]
Furthervolumesofthisseries Vol.199.ZhongLi
canbefoundonourhomepage: FuzzyChaoticSystems,2006
ISBN3-540-33220-0
springer.com
Vol.200.KaiMichels,FrankKlawonn,
RudolfKruse,AndreasNürnberger
Vol.191.MartinV.Butz
FuzzyControl,2006
Rule-BasedEvolutionaryOnlineLearning
ISBN3-540-31765-1
Systems,2006
ISBN3-540-25379-3 Vol.201.CengizKahraman(Ed.)
FuzzyApplicationsinIndustrial
Vol.192.JoseA.Lozano,PedroLarrañaga,
Engineering,2006
IñakiInza,EndikaBengoetxea(Eds.)
ISBN3-540-33516-1
TowardsaNewEvolutionaryComputation,
2006 Vol.202.PatrickDoherty,Witold
ISBN3-540-29006-0 Łukaszewicz,AndrzejSkowron,Andrzej
Szałas
Vol.193.IngoGlöckner
KnowledgeRepresentationTechniques:A
FuzzyQuantifiers:AComputationalTheory,
RoughSetApproach,2006
2006
ISBN3-540-33518-8
ISBN3-540-29634-4
Vol.203.GloriaBordogna,GiuseppePsaila
Vol.194.DawnE.Holmes,LakhmiC.Jain
(Eds.)
(Eds.)
FlexibleDatabasesSupportingImprecision
InnovationsinMachineLearning,2006
andUncertainty,2006
ISBN3-540-30609-9
ISBN3-540-33288-X
Vol.195.ZongminMa
Vol.204.ZongminMa(Ed.)
FuzzyDatabaseModelingofImpreciseand
SoftComputinginOntologiesandSemantic
UncertainEngineeringInformation,2006
Web,2006
ISBN3-540-30675-7
ISBN3-540-33472-6
Vol.196.JamesJ.Buckley
Vol.205.MikaSato-Ilic,LakhmiC.Jain
FuzzyProbabilityandStatistics,2006
InnovationsinFuzzyClustering,2006
ISBN3-540-30841-5
ISBN3-540-34356-3
Vol.197.EnriqueHerrera-Viedma,Gabriella
Vol.206.AshokSengupta(Ed.)
Pasi,FabioCrestani(Eds.)
Chaos,Nonlinearity,Complexity,2006
SoftComputinginWebInformation
ISBN3-540-31756-2
Retrieval,2006
ISBN3-540-31588-8 Vol.207.IsabelleGuyon,SteveGunn,
MasoudNikravesh,LotfiA.Zadeh(Eds.)
Vol.198.HungT.Nguyen,BerlinWu
FeatureExtraction,2006
FundamentalsofStatisticswithFuzzyData,
ISBN3-540-35487-5
2006
ISBN3-540-31695-7
Isabelle Guyon
Steve Gunn
Masoud Nikravesh
Lotfi A. Zadeh
(Eds.)
Feature Extraction
Foundations and Applications
ABC
IsabelleGuyon MasoudNikravesh
Clopinet DepartmentofElectrical
955CrestonRoad Engineering&Computer
94708Berkeley,USA Science–EECS
E-mail:[email protected] UniversityofCalifornia
94720Berkeley,USA
E-mail:[email protected]
SteveGunn
SchoolofElectronics
LotfiA.Zadeh
andComputerSciences
UniversityofSouthampton DivisionofComputerScience
SO171BJSouthampton Lab.ElectronicsResearch
Highfield,UnitedKingdom UniversityofCalifornia
E-mail:[email protected] SodaHall387
94720-1776Berkeley,CA,USA
E-mail:[email protected]
LibraryofCongressControlNumber:2006928001
ISSNprintedition:1434-9922
ISSNelectronicedition:1860-0808
ISBN-10 3-540-35487-5SpringerBerlinHeidelbergNewYork
ISBN-13 978-3-540-35487-1SpringerBerlinHeidelbergNewYork
Thisworkissubjecttocopyright.Allrightsarereserved,whetherthewholeorpartofthematerialis
concerned,specificallytherightsoftranslation,reprinting,reuseofillustrations,recitation,broadcasting,
reproductiononmicrofilmorinanyotherway,andstorageindatabanks.Duplicationofthispublication
orpartsthereofispermittedonlyundertheprovisionsoftheGermanCopyrightLawofSeptember9,
1965,initscurrentversion,andpermissionforusemustalwaysbeobtainedfromSpringer.Violationsare
liableforprosecutionundertheGermanCopyrightLaw.
SpringerisapartofSpringerScience+BusinessMedia
springer.com
(cid:2)c Springer-VerlagBerlinHeidelberg2006
PrintedinTheNetherlands
Theuseofgeneraldescriptivenames,registerednames,trademarks,etc.inthispublicationdoesnotimply,
evenintheabsenceofaspecificstatement,thatsuchnamesareexemptfromtherelevantprotectivelaws
andregulationsandthereforefreeforgeneraluse.
Typesetting:bytheauthorsandtechbooksusingaSpringerLATEXmacropackage
Coverdesign:ErichKirchner,Heidelberg
Printedonacid-freepaper SPIN:10966471 89/techbooks 543210
To our friends and foes
Foreword
Everyonelovesagoodcompetition.AsIwritethis,twobillionfansareeagerly
anticipating the 2006 World Cup. Meanwhile, a fan base that is somewhat
smaller (but presumably includes you, dear reader) is equally eager to read
allabouttheresultsof theNIPS2003 FeatureSelection Challenge, contained
herein. Fans of Radford Neal and Jianguo Zhang (or of Bayesian neural net-
works and Dirichlet diffusion trees) are gloating “I told you so” and looking
forproofthattheirwinwasnotafluke.Butthematterisbynomeanssettled,
and fans of SVMs are shouting “wait ’til next year!” You know this book is
a bit more edgy than your standard academic treatise as soon as you see the
dedication: “To our friends and foes.”
Competition breeds improvement. Fifty years ago, the champion in 100m
butterflyswimmingwas22percentslowerthantoday’schampion;thewomen’s
marathonchampionfromjust30yearsagowas26percentslower.Whoknows
how much better our machine learning algorithms would be today if Turing
in 1950 had proposed an effective competition rather than his elusive Test?
Butwhatmakesaneffectivecompetition?ThefieldofSpeechRecognition
hashadNIST-runcompetitionssince1988;errorrateshavebeenreducedbya
factorofthreeormore,butthefieldhasnotyethadtheimpactexpectedofit.
InformationRetrievalhashaditsTRECcompetitionsince1992;progresshas
been steady and refugees from the competition have played important roles
in the hundred-billion-dollar search industry. Robotics has had the DARPA
GrandChallengeforonlytwoyears,butinthattimewehaveseentheresults
go from complete failure to resounding success (although it may have helped
that the second year’s course was somewhat easier than the first’s).
I think there are four criteria that define effective technical competitions:
1. The task must be approachable. Non-experts should be able to enter, to
see some results, and learn from their better-performing peers.
2. The scoring must be incremental. A pass-fail competition where everyone
always fails (such as the Turing Test) makes for a boring game and dis-
couragesfurthercompetition.OnthisscoretheLoebnerPrize,despiteits
VIII Foreword
faults,isabettercompetitionthantheoriginalTuringTest.Inonesense,
everyone failed the DARPA Grand Challenge in the first year (because
noentrant finished the race),butin another sensetherewereincremental
scores: the distance each robot travelled, and the average speed achieved.
3. The results should be open. Participants and spectators alike should be
able to learn the best practices of all participants. This means that each
participant should describe their approaches in a written document, and
thatthedata,auxiliaryprograms,andresultsshouldbepubliclyavailable.
4. The task should be relevant to real-world tasks. One of the problems
with early competitions in speech recognition was that the emphasis on
reducingworderrorratesdidnotnecessarilyleadtoastrongspeechdialog
system—you could get almost all the words right and still have a bad
dialog,andconverselyyoucouldmissmanyofthewordsandstillrecover.
Morerecentcompetitionshavedoneabetterjobofconcentratingontasks
that really matter.
The Feature Selection Challenge meets the first three criteria easily. Sev-
entyfiveteamsentered,sotheymusthavefounditapproachable.Thescoring
didagoodjobofseparatingthetopperformerswhilekeepingeveryoneonthe
scale. And the results are all available online, in this book, and in the accom-
panying CD. All the data and Matlab code is provided, so the Challenge is
easily reproducible. The level of explication provided by the entrants in the
chapters of this book is higher than in other similar competitions. The fourth
criterion, real-world relevance, is perhaps the hardest to achieve. Only time
will tell whether the Feature Selection Challenge meets this one. In the mean
time,thisbooksetsahighstandardasthepublicrecordofaninterestingand
effective competition.
Palo Alto, California Peter Norvig
January 2006
Preface
Feature extraction addresses the problem of finding the most compact and
informative set of features, to improve the efficiency or data storage and
processing. Defining feature vectors remains the most common and conve-
nient means of data representation for classification and regression problems.
Data can then be stored in simple tables (lines representing “entries”, “data
points, “samples”, or “patterns”, and columns representing “features”). Each
feature results from a quantitative or qualitative measurement, it is an “at-
tribute” or a “variable”. Modern feature extraction methodology is driven by
the size of the data tables, which is ever increasing as data storage becomes
more and more efficient.
After many years of parallel efforts, researchers in Soft-Computing, Sta-
tistics, Machine Learning, and Knowledge Discovery, who are interested in
predictive modeling are uniting their effort to advance the problem of fea-
ture extraction. The recent advances made in both sensor technologies and
machine learning techniques make it possible to design recognition systems,
whicharecapableofperformingtasksthatcouldnotbeperformedinthepast.
Featureextractionliesatthecenteroftheseadvanceswithapplicationsinthe
pharmaco-medical industry, oil industry, industrial inspection and diagnosis
systems, speech recognition, biotechnology, Internet, targeted marketing and
many of other emerging applications.
The present book is organized around the results of a benchmark that
took place in 2003. Dozens of research groups competed on five large feature
selection problems from various application domains: medical diagnosis, text
processing, drug discovery, and handwriting recognition. The results of this
effortpavetheway toanewgeneration ofmethodscapableofanalyzingdata
tables with million of lines and/or columns.
Part II of the book summarizes the results of the competition and gath-
ers the papers describing the methods used by the top ranking participants.
Following the competition, a NIPS workshop took place in December 2003
to discuss the outcomes of the competition and new avenues in feature ex-
traction. The contributions providing new perspectives are found in Part III
X Preface
of the book. Part I provides all the necessary foundations to understand the
recent advances made in Parts II and III. The book is complemented by ap-
pendices and by a web site. The appendices include fact sheets summarizing
themethodsusedinthecompetition,tablesofresultsofthecompetition,and
a summary of basic concepts of statistics.
This book is directed to students, researchers, and engineers. It presents
recent advances in the field and complements an earlier book (Liu and Mo-
toda, 1998), which provides a thorough bibliography and presents methods
of historical interest, but explores only small datasets and ends before the
new era of kernel methods. Readers interested in the historical aspects of the
problemarealsodirectedto(DevijverandKittler,1982).Acompletelynovice
reader will find all the necessary elements to understand the material of the
book presented in the tutorial chapters of Part I. The book can be used as
teachingmaterialforagraduateclassinstatisticsandmachinelearning,Part
I supporting the lectures, Part II and III providing readings, and the CD
providing data for computer projects.
Zu¨rich, Switzerland Isabelle Guyon
Southampton, UK Steve Gunn
Berkeley, California Masoud Nikravesh and Lofti A. Zadeh
November 2005
References
P.A.DevijverandJ.Kittler. PatternRecognition:AStatisticalApproach. Prentice-
Hall, 1982.
H. Liu and H. Motoda. Feature Extraction, Construction and Selection: A Data
Mining Perspective. Kluwer Academic, 1998.
Contents
An Introduction to Feature Extraction
Isabelle Guyon, Andr´e Elisseeff .................................... 1
1 Feature Extraction Basics.................................... 1
2 What is New in Feature Extraction?........................... 7
3 Getting Started............................................. 9
4 Advanced Topics and Open Problems.......................... 16
5 Conclusion ................................................. 22
References ...................................................... 23
A Forward Selection with Gram-Schmidt Orthogonalization ........ 24
B Justification of the Computational Complexity Estimates ........ 25
Part I Feature Extraction Fundamentals
1 Learning Machines
Norbert Jankowski, Krzysztof Grabczewski........................... 29
1.1 Introduction................................................ 29
1.2 The Learning Problem....................................... 29
1.3 Learning Algorithms ........................................ 35
1.4 Some Remarks on Learning Algorithms ........................ 57
References ...................................................... 58
2 Assessment Methods
G´erard Dreyfus, Isabelle Guyon .................................... 65
2.1 Introduction................................................ 65
2.2 A Statistical View of Feature Selection: Hypothesis Tests
and Random Probes......................................... 66
2.3 A Machine Learning View of Feature Selection.................. 78
2.4 Conclusion ................................................. 86
References ...................................................... 86