Sensitivity Analysis for Neural Networks

88 Pages·2010·1.584 MB·English

by  Daniel S. Yeung, Ian Cloete, Daming Shi, Wing W. Y. Ng (auth.)

Natural Computing Series
SeriesEditors: G.Rozenberg
Th.Bäck A.E.Eiben J.N.Kok H.P.Spaink
LeidenCenterforNaturalComputing
AdvisoryBoard: S.Amari G.Brassard K.A.DeJong C.C.A.M.Gielen T.Head
L.Kari L.Landweber T.Martinetz Z.Michalewicz M.C.Mozer E.Oja
G.Pa˘un J.Reif H.Rubin A.Salomaa M.Schoenauer H.-P.Schwefel C.Torras
D.Whitley E.Winfree J.M.Zurada
Forfurthervolumes:
http://www.springer.com/series/4190
· · ·
Daniel S. Yeung Ian Cloete Daming Shi
Wing W.Y. Ng
Sensitivity Analysis
for Neural Networks
123
Authors
Prof.DanielS.Yeung Prof.IanCloete
SchoolofComputerScience President
andEngineering Campus3
SouthChinaUniversityofTechnology InternationalUniversityinGermany
WushanRd. 76646Bruchsal,Germany
TianHeDistrict [email protected];[email protected]
Guangzhou,China
[email protected] Dr.WingW.Y.Ng
SchoolofComputerScience
Prof.DamingShi
andEngineering
SchoolofElectricalEngineeringandComputer
SouthChinaUniversityofTechnology
Science
WushanRd.
KyungpookNationalUniversity
TianHeDistrict
Buk-gu,Daegu
Guangzhou,China
SouthKorea
[email protected]
[email protected]
SeriesEditors
G.Rozenberg(ManagingEditor)
[email protected]
Th.Bäck,J.N.Kok,H.P.Spaink
LeidenCenterforNaturalComputing
LeidenUniversity
NielsBohrweg1
2333CALeiden,TheNetherlands
A.E.Eiben
VrijeUniversiteitAmsterdam
TheNetherlands
ISSN1619-7127
ISBN978-3-642-02531-0 e-ISBN978-3-642-02532-7
DOI10.1007/978-3-642-02532-7
SpringerHeidelbergDordrechtLondonNewYork
LibraryofCongressControlNumber:2009938187
ACMComputingClassification(1998):I.2.6,F.1.1
©Springer-VerlagBerlinHeidelberg2010
Thisworkissubjecttocopyright.Allrightsarereserved,whetherthewholeorpartofthematerialis
concerned,specificallytherightsoftranslation,reprinting,reuseofillustrations,recitation,broadcasting,
reproductiononmicrofilmorinanyotherway,andstorageindatabanks.Duplicationofthispublication
orpartsthereofispermittedonlyundertheprovisionsoftheGermanCopyrightLawofSeptember9,
1965,initscurrentversion,andpermissionforusemustalwaysbeobtainedfromSpringer.Violations
areliabletoprosecutionundertheGermanCopyrightLaw.
Theuseofgeneraldescriptivenames,registerednames,trademarks,etc.inthispublicationdoesnot
imply,evenintheabsenceofaspecificstatement,thatsuchnamesareexemptfromtherelevantprotective
lawsandregulationsandthereforefreeforgeneraluse.
Coverdesign:KuenkelLopkaGmbH
Printedonacid-freepaper
SpringerispartofSpringerScience+BusinessMedia(www.springer.com)
Preface
Neural networks provide a way to realize one of our human dreams to make
machines think like us. Artificial neural networks have been developed since
RosenblattproposedthePerceptronin1958.Today,manyneuralnetworksarenot
treatedasblackboxesanymore.Issuessuchasrobustnessandgeneralizationabil-
ities have been brought to the fore. The advances in neural networks have led to
moreandmorepracticalapplicationsinpatternrecognition,financialengineering,
automaticcontrolandmedicaldiagnosis,tonamejustafew.
Sensitivity analysis dates back to the 1960s, when Widrow investigated the
probability of misclassification due to weight perturbations, which are caused by
machine imprecision and noisy input. For the purpose of analysis, these pertur-
bations can be simulated by embedding disturbance into the original inputs or
connectionweights.Theinitialideaofsensitivityanalysiswasthenextendedtoopti-
mizationandtoapplicationsofneuralnetworks,suchassamplereduction,feature
selection,activelearningandcriticalvectorlearning.
This text should primarily be of interest to graduate students, academics, and
researchersinbranchesofneuralnetworks,artificialintelligence,machinelearning,
appliedmathematicsandcomputerengineeringwheresensitivityanalysisofneural
networksandrelatedconceptsareused.Wehavemadeanefforttomakethebook
accessibletosuchacross-disciplinaryaudience.
Thebookisorganizedintoeightchapters,ofwhichChap.1givesanintroduction
to the various neural network structures and learning schemes. A literature review
onthemethodologiesofsensitivityanalysisispresentedinChap.2.Differentfrom
thetraditionalhyperspheremodel,thehyper-rectanglemodeldescribedinChap.3
is especially suitable for the most popular and general feedforward network: the
multilayer Perceptron. In Chap. 4, the activation function is also involved in the
calculationofthesensitivityanalysisbyparameterizing.Thesensitivityanalysisof
radialbasisfunctionnetworksisdiscussedinChaps.5and6,withtheformergiv-
ingageneralizationerrormodelwhereasthelatterconcernsoptimizingthehidden
neurons.InChap.7,sensitivityismeasuredinordertoencodepriorknowledgeinto
a neural network. In Chap. 8, sensitivity analysis is applied in many applications,
suchasdimensionalityreduction,networkoptimizationandselectivelearning.
We would like to express our thanks to many colleagues, friends and students
whoprovidedreviewsofdifferentchaptersofthismanuscript.TheyincludeMinh
v
vi Preface
Nhut Nguyen, Xiaoqin Zeng, Patrick Chan, Xizhao Wang, Fei Chen and Lu He.
Weoftenfindourselvesstrugglingwithmanycompetingdemandsforourtimeand
effort. As a result, our families, especially our beloved spouses, are the ones who
suffer the most. We are delighted to dedicate this work to Foo-Lau Yeung, Wilma
CloeteandJianLiu.
ItiswithgreathumilitythatwewouldliketoacknowledgeourGoodLordasthe
truecreatorofallknowledge.Thisworkistheresultofourborrowingasmallpiece
ofknowledgefromHim.
8July2009 DanielS.Yeung
IanCloete
DamingShi
WingW.Y.Ng
Contents
1 IntroductiontoNeuralNetworks . . . . . . . . . . . . . . . . . . . . 1
1.1 PropertiesofNeuralNetworks . . . . . . . . . . . . . . . . . . . 3
1.2 NeuralNetworkLearning . . . . . . . . . . . . . . . . . . . . . . 5
1.2.1 SupervisedLearning . . . . . . . . . . . . . . . . . . . . 5
1.2.2 UnsupervisedLearning . . . . . . . . . . . . . . . . . . . 5
1.3 Perceptron . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6
1.4 AdalineandLeastMeanSquareAlgorithm. . . . . . . . . . . . . 8
1.5 MultilayerPerceptronandBackpropagationAlgorithm . . . . . . 9
1.5.1 OutputLayerLearning . . . . . . . . . . . . . . . . . . . 11
1.5.2 HiddenLayerLearning . . . . . . . . . . . . . . . . . . . 11
1.6 RadialBasisFunctionNetworks . . . . . . . . . . . . . . . . . . 12
1.7 SupportVectorMachines . . . . . . . . . . . . . . . . . . . . . . 13
2 PrinciplesofSensitivityAnalysis . . . . . . . . . . . . . . . . . . . . 17
2.1 PerturbationsinNeuralNetworks. . . . . . . . . . . . . . . . . . 17
2.2 NeuralNetworkSensitivityAnalysis . . . . . . . . . . . . . . . . 18
2.3 FundamentalMethodsofSensitivityAnalysis . . . . . . . . . . . 21
2.3.1 GeometricalApproach . . . . . . . . . . . . . . . . . . . 21
2.3.2 StatisticalApproach . . . . . . . . . . . . . . . . . . . . . 23
2.4 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24
3 Hyper-RectangleModel . . . . . . . . . . . . . . . . . . . . . . . . . 25
3.1 Hyper-RectangleModelforInputSpaceofMLP. . . . . . . . . . 25
3.2 SensitivityMeasureofMLP . . . . . . . . . . . . . . . . . . . . 26
3.3 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27
4 SensitivityAnalysiswithParameterizedActivationFunction . . . . 29
4.1 ParameterizedAntisymmetricSquashingFunction . . . . . . . . . 29
4.2 SensitivityMeasure . . . . . . . . . . . . . . . . . . . . . . . . . 30
4.3 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31
5 LocalizedGeneralizationErrorModel . . . . . . . . . . . . . . . . . 33
5.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33
5.2 TheLocalizedGeneralizationErrorModel . . . . . . . . . . . . . 35
5.2.1 TheQ-NeighborhoodandQ-Union . . . . . . . . . . . . . 36
vii
viii Contents
5.2.2 TheLocalizedGeneralizationErrorBound . . . . . . . . . 36
5.2.3 StochasticSensitivityMeasureforRBFNN . . . . . . . . 38
5.2.4 CharacteristicsoftheErrorBound . . . . . . . . . . . . . 40
5.2.5 ComparingTwoClassifiersUsingtheErrorBound . . . . 42
5.3 ArchitectureSelectionUsingtheErrorBound . . . . . . . . . . . 42
5.3.1 ParametersforMC2SG . . . . . . . . . . . . . . . . . . . 44
5.3.2 RBFNNArchitectureSelectionAlgorithmforMC2SG . . 44
5.3.3 AHeuristicMethodtoReducetheComputational
TimeforMC2SG . . . . . . . . . . . . . . . . . . . . . . 45
5.4 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 45
6 CriticalVectorLearningforRBFNetworks . . . . . . . . . . . . . . 47
6.1 RelatedWork . . . . . . . . . . . . . . . . . . . . . . . . . . . . 47
6.2 ConstructionofRBFNetworkswithSensitivityAnalysis . . . . . 48
6.2.1 RBFClassifiers’SensitivitytotheKernelFunctionCenters 49
6.2.2 OrthogonalLeastSquareTransform . . . . . . . . . . . . 51
6.2.3 CriticalVectorSelection . . . . . . . . . . . . . . . . . . 52
6.3 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 52
7 SensitivityAnalysisofPriorKnowledge . . . . . . . . . . . . . . . . 55
7.1 KBANNs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 56
7.2 InductiveBias . . . . . . . . . . . . . . . . . . . . . . . . . . . . 58
7.3 SensitivityAnalysisandMeasures . . . . . . . . . . . . . . . . . 59
7.3.1 Output-PatternSensitivity . . . . . . . . . . . . . . . . . . 59
7.3.2 Output-WeightSensitivity. . . . . . . . . . . . . . . . . . 60
7.3.3 Output-HSensitivity . . . . . . . . . . . . . . . . . . . . 61
7.3.4 EuclideanDistance . . . . . . . . . . . . . . . . . . . . . 61
7.4 PromoterRecognition . . . . . . . . . . . . . . . . . . . . . . . . 61
7.4.1 DataandInitialDomainTheory . . . . . . . . . . . . . . 62
7.4.2 ExperimentalMethodology . . . . . . . . . . . . . . . . . 63
7.5 DiscussionandConclusion . . . . . . . . . . . . . . . . . . . . . 64
8 Applications . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 69
8.1 InputDimensionReduction . . . . . . . . . . . . . . . . . . . . . 69
8.1.1 SensitivityMatrix . . . . . . . . . . . . . . . . . . . . . . 70
8.1.2 CriteriaforPruningInputs . . . . . . . . . . . . . . . . . 70
8.2 NetworkOptimization . . . . . . . . . . . . . . . . . . . . . . . 71
8.3 SelectiveLearning. . . . . . . . . . . . . . . . . . . . . . . . . . 74
8.4 HardwareRobustness . . . . . . . . . . . . . . . . . . . . . . . . 76
8.5 MeasureofNonlinearity . . . . . . . . . . . . . . . . . . . . . . 77
8.6 ParameterTuningforNeocognitron . . . . . . . . . . . . . . . . 78
8.6.1 ReceptiveField . . . . . . . . . . . . . . . . . . . . . . . 79
8.6.2 Selectivity . . . . . . . . . . . . . . . . . . . . . . . . . . 80
8.6.3 SensitivityAnalysisoftheNeocognitron . . . . . . . . . . 80
Bibliography . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 83
Chapter 1
Introduction to Neural Networks
The human brain consists of ten billion densely interconnected nerve cells, called
neurons; each connected to about 10,000 other neurons, with 60 trillion connec-
tions,synapses,betweenthem.Byusingmultipleneuronssimultaneously,thebrain
canperformitsfunctionsmuchfasterthanthefastestcomputersinexistencetoday.
On the other hand, a neuron can be considered as a basic information-processing
unit,whereasourbraincanbeconsideredasahighlycomplex,nonlinearandpar-
allelbiologicalinformation-processingnetwork,inwhichinformationisstoredand
processedsimultaneously.Learningisafundamentalandessentialcharacteristicof
biological neural networks. The ease with which they can learn led to attempts to
emulateabiologicalneuralnetworkinacomputer.
Inthe1940s,McCullochandPittsproposedamodelforbiologicalneuronsand
biological neural networks. A stimulus is transmitted from dendrites to a soma
via synapses, and axons transmit the response of one soma to another, as shown
in Fig. 1.1. Inspired by the mechanism for learning in biological neurons, artifi-
cial neurons and artificial neural networks can perform arithmetic functions, with
cells corresponding to neurons, activations corresponding to neuronal firing rates,
connections corresponding to synapses, and connection weights corresponding to
synaptic strengths, as shown in Fig. 1.1. The analogy between biological neurons
and artificial neurons is made in Table 1.1. However, neural networks are far too
simple to serve as realistic brain models on the cell level, but they might serve as
verygoodmodelsfortheessentialinformationprocessingtasksthatorganismsper-
form.Thisremainsanopenquestionbecausewehavesolittleunderstandingofhow
thebrainactuallyworks(Gallant,1993).
Inaneuralnetwork,neuronsarejoinedbydirectedarcs–connections.Theneu-
ronsandarcsconstitutethenetworktopology.Eacharchasanumericalweightthat
specifies the influence between two neurons. Positive weights indicate reinforce-
ment;negativeweightsrepresentinhibition.Theweightsdeterminethebehaviorof
the network, playing somewhat the same role as in a conventional computer pro-
gram.Typically,therearemanyinputsforasingleneuron,andasubsequentoutput
of an activation function (or transfer function). Some frequently used activation
functionsinclude:
D.S.Yeungetal.,SensitivityAnalysisforNeuralNetworks,NaturalComputingSeries, 1
DOI10.1007/978-3-642-02532-7_1,(cid:2)C Springer-VerlagBerlinHeidelberg2010
2 1 IntroductiontoNeuralNetworks
a
b
Fig. 1.1 Biological motivations of neural networks. (a) Neuroanatomy of living animals. (b)
Connectionsofanartificialneuron
• LinearFunction:(cid:2)(u)=u
1
• Log-SigmoidFunction:(cid:2)(u)=
(cid:2) 1+e−u
1ifu≥0
• HardLimitfunction:(cid:2)(u)=
0otherwise
Rosenblatt(1958)devisedthePerceptron,whichisnowwidelyused.Widrowand
Hoff (1960) proposed the Adaline at the same time. The Perceptron and Adaline
were the first practical networks and learning rules that demonstrated the high
potentialofneuralnetworks.
Table1.1 Analogybetween
BiologicalNeurons ArtificialNeurons
biologicalandartificial
neurons
Soma Sum+ActivationFunction
Dendrite Input
Axon Output
Synapse Weight
1.1 PropertiesofNeuralNetworks 3
Minsky and Papert (1969) criticized the Perceptron, because they found it is
not powerful enough to do tasks such as parity and connectedness. As long as a
neural network consists of a linear combiner followed by a nonlinear element, a
single-layerPerceptroncanperformpatternclassificationonlyonlinearlyseparable
patterns, regardless of the form of nonlinearity used. Minsky and Papert demon-
strated the limitations of the Perceptron with the simplest XOR problem, and as
a consequence the research in the field was suspended due to the lack of funding.
Because of Minsky and Papert’s conclusion, people lost confidence in neural net-
works.Inthe1970s,theprogressonneuralnetworkscontinuedatamuchreduced
pace,althoughsomeresearchers,suchasAmari,Anderson,Fukushima,Grossberg
andKohonen,keptworkingonit.
In1985,Ackley,HintonandSejnowskidescribedtheBoltzmannmachine,which
is more powerful than a Perceptron, and demonstrated a successful application –
NETtalk.
Rumelhart, Hinton and Williams (1986) are recognized for their milestone
work – the Multilayer Perceptron (MLP) with backpropagation, which remained
thedominantneuralnetworkarchitectureformorethantenyears.Asufficientlybig
MLPhasbeenproventobeabletolearnanyfunction,andmanyvariantsofMLP
havebeenproposedsincethen.In1988,RadialBasisFunctions(RBFs),introduced
asanalternativetoMLPs,speededupMLPtraining(fast-prop).
Fromthe1990stodate,moreandmoreresearchhasbeendonetoimproveneural
networks.Theresearchagendaincludesregularizers,probabilistic(Bayesian)infer-
ence,structuralriskminimizationandincorporationwithevolutionarycomputation.
1.1 PropertiesofNeuralNetworks
Neural networks can be described according to their network, cell, dynamic, and
learningpropertiesasfollows:
NetworkProperties.Aneuralnetworkisanarchitectureconsistingofmanyneu-
rons, which work together to respond to the inputs. We sometimes consider a
networkasablack-boxfunction.Heretheexternalworldpresentsinputstotheinput
cellsandreceivesnetworkoutputsfromtheoutputcells.Intermediatecellsarenot
seen externally, and for this reason they are often called hidden units. We classify
networks as either feedforward networks if they do not contain directed cycles or
recurrentnetworksiftheydocontainsuchcycles.Itisoftenconvenienttoorganize
thecellsofanetworkintolayers.
CellProperties.Eachcellcomputesasingle(numerical)celloutputoractivation.
Cellinputs and activations may bediscrete, taking values {0, 1}or {−1, 0,1},or
theymaybecontinuous,assumingvaluesintheinterval[0,1]or[−1,+1].Typically,
every cell uses the same algorithm for computing its activation. The activation for
a cell must be computed from the activations of cells directly connected to it and
thecorrespondingweightsforthoseconnections.Everycell(exceptforinputcells)
computes its new activation as a function of the weighted sum of the inputs from
directlyconnectedcells.