Estimation of Dependences Based on Empirical Data: Empirical Inference Science Afterword of 2006

105 Pages·2006·0.85 MB·English

by  Vladimir Vapnik (auth.)

Information Science and Statistics
Series Editors:
M.Jordan
J.Kleinberg
B.Schölkopf
Information Science and Statistics
Akaike and Kitagawa:The Practice of Time Series Analysis.
Cowell,Dawid,Lauritzen,and Spiegelhalter:Probabilistic Networks and Expert
Systems.
Doucet,de Freitas,and Gordon:Sequential Monte Carlo Methods in Practice.
Fine:Feedforward Neural Network Methodology.
Hawkins and Olwell:Cumulative Sum Charts and Charting for Quality
Improvement.
Jensen:Bayesian Networks and Decision Graphs.
Marchette:Computer Intrusion Detection and Network Monitoring:A Statistical
Viewpoint.
Rubinstein and Kroese:The Cross-Entropy Method:A Unified Approach to
Combinatorial Optimization,Monte Carlo Simulation,and Machine Learning.
Studeny´:Probabilistic Conditional Independence Structures.
Vapnik:The Nature of Statistical Learning Theory,Second Edition.
Wallace:Statistical and Inductive Inference by Minimum Massage Length.
Vladimir Vapnik
Estimation of
Dependences Based on
Empirical Data
Reprint of 1982 Edition
Empirical
Inference Science
Afterword of 2006
Vladimir Vapnik Samuel Kotz (Translator)
NEC Labs America Department ofEngineering Management
4 Independence Way and Systems Engineering
Princeton,NJ 08540 The George Washington University
[email protected] Washington,D.C.20052
Series Editors: Jon Kleinberg Bernhard Schölkopf
Michael Jordan Department ofComputer Max Planck Institute for
Division ofComputer Science Biological Cybernetics
Science and Cornell University Spemannstrasse 38
Department ofStatistics Ithaca,NY 14853 72076 Tübingen
University ofCalifornia, USA Germany
Berkeley
Berkeley,CA 94720
USA
Library ofCongress Control Number:2005938355
ISBN-10:0-387-30865-2
ISBN-13:978-0387-30865-4
Printed on acid-free paper.
© 2006 Springer Science+Business Media,Inc.
All rights reserved.This work may not be translated or copied in whole or in part without the
written permission of the publisher (Springer Science+Business Media,Inc.,233 Spring Street,
New York,NY 10013,USA),except for brief excerpts in connection with reviews or scholarly
analysis. Use in connection with any form of information storage and retrieval, electronic
adaptation,computer software,or by similar or dissimilar methodology now known or hereafter
developed is forbidden.
The use in this publication oftrade names,trademarks,service marks,and similar terms,even if
they are not identified as such,is not to be taken as an expression of opinion as to whether or
not they are subject to proprietary rights.
Printed in the United States ofAmerica. (MVY)
9 8 7 6 5 4 3 2 1
springer.com
Vladimir Vapnik
Estimation of
Dependences Based on
Empirical Data
Translated by Samuel Kotz
With 22 illustrations
To the students of my students in memory of my violin teacher
Ilia Shtein and PhD advisor Alexander Lerner, who taught me
several important things that are very difficult to learn from
books.
PREFACE
Twenty-fiveyearshavepassedsincethepublicationoftheRussianversionofthebook
Estimation of Dependencies Based on Empirical Data (EDBED for short). Twenty-
five years is a long period of time. During these years many things have happened.
Looking back, one can see how rapidly life and technology have changed, and how
slow and difficult it is to change the theoretical foundation of the technology and its
philosophy.
IpursuedtwogoalswritingthisAfterword:toupdatethetechnicalresultspresented
in EDBED (the easy goal) and to describe a general picture of how the new ideas
developedovertheseyears(amuchmoredifficultgoal).
The picture which I would like to present is a very personal (and therefore very
biased)accountofthedevelopmentofoneparticularbranchofscience,EmpiricalIn-
ferenceScience.
Such accounts usually are not included in the content of technical publications. I
havefollowedthisruleinallofmypreviousbooks.ButthistimeIwouldliketoviolate
itforthefollowingreasons. Firstofall,formeEDBEDistheimportantmilestonein
the development of empirical inference theory and I would like to explain why. Sec-
ond,duringtheseyears,therewerealotofdiscussionsbetweensupportersofthenew
paradigm(nowitiscalledtheVCtheory1)andtheoldone(classicalstatistics). Being
involvedinthesediscussionsfromtheverybeginningIfeelthatitismyobligationto
describethemainevents.
The story related to the book, which I would like to tell, is the story of how it
is difficult to overcome existing prejudices (both scientific and social), and how one
shouldbecarefulwhenevaluatingandinterpretingnewtechnicalconcepts.
Thisstorycanbesplitintothreepartsthatreflectthreemainideasinthedevelop-
mentofempiricalinferencescience: fromthepuretechnical(mathematical)elements
ofthetheorytoanewparadigminthephilosophyofgeneralization.
1VCtheoryisanabbreviationforVapnik–Chervonenkistheory.Thisnameforthecorrespondingtheory
appearedinthe1990safterEDBEDwaspublished.
405
406 Preface
Thefirstpartofthestory,whichdescribesthemaintechnicalconceptsbehindthe
newmathematicalandphilosophicalparadigm,canbetitled
RealismandInstrumentalism: ClassicalStatisticsandVCTheory
InthispartItrytoexplainwhybetween1960and1980anewapproachto empirical
inferencesciencewasdevelopedincontrasttotheexistingclassicalstatisticsapproach
developedbetween1930and1960.
Thesecondpartofthestoryisdevotedtotherationaljustificationofthenewideas
ofinferencedevelopedbetween1980and2000. Itcanbetitled
FalsifiabilityandParsimony: VCDimensionandtheNumberofEntities
ItdescribeswhytheconceptofVCfalsifiabilityismorerelevantforpredictivegener-
alizationproblemsthantheclassicalconceptofparsimonythatisusedbothinclassical
philosophyandstatistics.
Thethirdpartofthestory,whichstartedinthe2000scanbetitled
NoninductiveMethodsofInference: DirectInferenceInsteadofGeneralization
It deals with the ongoing attempts to construct new predictive methods (direct infer-
ence)basedonthenewphilosophythatisrelevanttoacomplexworld, incontrastto
theexistingmethodsthatweredevelopedbasedontheclassicalphilosophyintroduced
forasimpleworld.
I wrote this Afterword with my students’ students in mind, those who just began
theircareersinscience. Tobesuccessfultheyshouldlearnsomethingveryimportant
thatisnoteasytofindinacademicpublications.
Inparticulartheyshouldseethebigpicture: whatisgoingoninthedevelopment
of this science and in closely related branches of science in general (not only about
some technical details). They also should know about the existence of very intense
paradigm wars. They should understand that the remark of Cicero, “Among all fea-
turesdescribinggeniusthemostimportantisinnerprofessionalhonesty”,isnotabout
ethics but about an intellectual imperative. They should know that Albert Einstein’s
observationabouteverydayscientificlifethat“Greatspiritshavealwaysencountered
violentoppositionfrommediocreminds,”isstilltrue. Knowledgeofthesethingscan
helpthemtomaketherightdecisionsandavoidthewrongones. ThereforeIwrotea
fourthparttothisAfterwordthatcanbetitled
TheBigPicture.
This,however,isanextremelydifficultsubject. Thatiswhyitiswisetoavoiditin
technicalbooks,andriskytodiscussitcommentingonsomemoreorlessrecentevents
inthedevelopmentofthescience.
WritingthisAfterwordwasadifficultprojectformeandIwasabletocompleteit
inthewaythatitiswrittenduetothestrongsupportandhelpofmycolleaguesMike
Miller,DavidWaltz,BernhardScho¨lkopf,LeonBottou,andIlyaMuchnik.
Iwouldliketoexpressmydeepgratitudetothem.
Princeton,NewJersey, VladimirVapnik
November2005
CONTENTS
1 REALISMANDINSTRUMENTALISM: CLASSICALSTATISTICSANDVCTHE-
ORY(1960–1980) 411
1.1 TheBeginning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 411
1.1.1 ThePerceptron . . . . . . . . . . . . . . . . . . . . . . . . . 412
1.1.2 UniformLawofLargeNumbers . . . . . . . . . . . . . . . . 412
1.2 RealismandInstrumentalisminStatisticsandthePhilosophyofScience414
1.2.1 TheCurseofDimensionalityandClassicalStatistics . . . . . 414
1.2.2 TheBlackBoxModel . . . . . . . . . . . . . . . . . . . . . 416
1.2.3 RealismandInstrumentalisminthePhilosophyofScience . . 417
1.3 RegularizationandStructuralRiskMinimization . . . . . . . . . . . 418
1.3.1 RegularizationofIll-PosedProblems . . . . . . . . . . . . . 418
1.3.2 StructuralRiskMinimization. . . . . . . . . . . . . . . . . . 421
1.4 TheBeginningoftheSplitBetweenClassicalStatisticsandStatistical
LearningTheory . . . . . . . . . . . . . . . . . . . . . . . . . . . . 422
1.5 TheStoryBehindThisBook . . . . . . . . . . . . . . . . . . . . . . 423
2 FALSIFIABILITY AND PARSIMONY: VC DIMENSION AND THE NUMBER OF
ENTITIES(1980–2000) 425
2.1 SimplificationofVCTheory . . . . . . . . . . . . . . . . . . . . . . 425
2.2 CapacityControl . . . . . . . . . . . . . . . . . . . . . . . . . . . . 427
2.2.1 BellLabs . . . . . . . . . . . . . . . . . . . . . . . . . . . . 427
2.2.2 NeuralNetworks . . . . . . . . . . . . . . . . . . . . . . . . 429
2.2.3 NeuralNetworks: TheChallenge . . . . . . . . . . . . . . . 429
2.3 SupportVectorMachines(SVMs) . . . . . . . . . . . . . . . . . . . 430
2.3.1 StepOne: TheOptimalSeparatingHyperplane . . . . . . . . 430
2.3.2 TheVCDimensionoftheSetofρ-MarginSeparatingHyper-
planes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 431
2.3.3 StepTwo: CapacityControlinHilbertSpace . . . . . . . . . 432
407