Neural Networks for Applied Sciences and Engineering: From Fundamentals to Complex Pattern Recognition

596 Pages·2006·6.12 MB·English

by  Sandhya Samarasinghe

Dedication
To Don
My husband
For yourconstant love, support, and encouragement
To do the best I can do in all
My endeavors as a
Woman and a
Scholar!

Contents
Preface...................................................................................................... xvii
Acknowledgments..................................................................................... xxi
About the Author.................................................................................... xxiii
1 From Data to Models: Complexity and Challenges
in Understanding Biological, Ecological, and
Natural Systems................................................................................. 1
1.1: Introduction 1
1.2: LayoutoftheBook 4
References 7
2 Fundamentals of Neural Networks and Models
for Linear Data Analysis ................................................................ 11
2.1: IntroductionandOverview 11
2.2: NeuralNetworksandTheirCapabilities 12
2.3: InspirationsfromBiology 16
2.4: ModelingInformationProcessinginNeurons 18
2.5: NeuronModelsandLearningStrategies 19
2.5.1: ThresholdNeuronasaSimpleClassifier 20
2.5.2: LearningModelsforNeuronsandNeuralAssemblies 23
2.5.2.1: HebbianLearning 23
2.5.2.2: UnsupervisedorCompetitiveLearning 26
2.5.2.3: SupervisedLearning 26
2.5.3: PerceptronwithSupervisedLearningasaClassifier 27
2.5.3.1: PerceptronLearningAlgorithm 28
2.5.3.2: APracticalExampleofPerceptrononaLarger
RealisticDataSet:IdentifyingtheOrigin
ofFishfromtheGrowth-RingDiameterofScales 35
2.5.3.3: ComparisonofPerceptronwithLinear
DiscriminantFunctionAnalysisinStatistics 38
viii &
2.5.3.4: Multi-OutputPerceptronforMulticategory
Classification 40
2.5.3.5: Higher-DimensionalClassificationUsingPerceptron 45
2.5.3.6: PerceptronSummary 45
2.5.4: LinearNeuronforLinearClassificationandPrediction 46
2.5.4.1: LearningwiththeDeltaRule 47
2.5.4.2: LinearNeuronasaClassifier 51
2.5.4.3: ClassificationPropertiesofaLinearNeuron
asaSubsetofPredictiveCapabilities 53
2.5.4.4: Example:LinearNeuronasaPredictor 54
2.5.4.5: APracticalExampleofLinearPrediction:
PredictingtheHeatInfluxinaHome 61
2.5.4.6: ComparisonofLinearNeuronModelwith
LinearRegression 62
2.5.4.7: Example:MultipleInputLinearNeuron
Model—ImprovingthePredictionAccuracy
ofHeatInfluxinaHome 63
2.5.4.8: ComparisonofaMultiple-InputLinearNeuron
withMultipleLinearRegression 63
2.5.4.9: MultipleLinearNeuronModels 64
2.5.4.10: ComparisonofaMultipleLinearNeuron
NetworkwithCanonicalCorrelationAnalysis 65
2.5.4.11: LinearNeuronandLinearNetworkSummary 65
2.6: Summary 66
Problems 66
References 67
3 Neural Networks for Nonlinear Pattern Recognition.............. 69
3.1: OverviewandIntroduction 69
3.1.1: MultilayerPerceptron 71
3.2: NonlinearNeurons 72
3.2.1: NeuronActivationFunctions 73
3.2.1.1: SigmoidFunctions 74
3.2.1.2: GaussianFunctions 76
3.2.2: Example:PopulationGrowthModelingUsing
aNonlinearNeuron 77
3.2.3: ComparisonofNonlinearNeuronwithNonlinear
RegressionAnalysis 80
3.3: One-InputMultilayerNonlinearNetworks 80
3.3.1: ProcessingwithaSingleNonlinearHiddenNeuron 80
3.3.2: Examples:ModelingCyclicalPhenomenawith
MultipleNonlinearNeurons 86
3.3.2.1: Example1:ApproximatingaSquareWave 86
3.3.2.2: Example2:ModelingSeasonalSpeciesMigration 94
3.4: Two-InputMultilayerPerceptronNetwork 98
3.4.1: ProcessingofTwo-DimensionalInputsby
NonlinearNeurons 98
3.4.2: NetworkOutput 102
& ix
3.4.3: Examples:Two-DimensionalPrediction
andClassification 103
3.4.3.1: Example1:Two-DimensionalNonlinear
FunctionApproximation 103
3.4.3.2: Example2:Two-DimensionalNonlinear
ClassificationModel 105
3.5: MultidimensionalDataModelingwithNonlinear
MultilayerPerceptronNetworks 109
3.6: Summary 110
Problems 110
References 112
4 Learning of Nonlinear Patterns by Neural Networks............ 113
4.1: IntroductionandOverview 113
4.2: SupervisedTrainingofNetworksforNonlinear
PatternRecognition 114
4.3: GradientDescentandErrorMinimization 115
4.4: BackpropagationLearning 116
4.4.1: Example:BackpropagationTraining—AHandComputation 117
4.4.1.1: ErrorGradientwithRespecttoOutput
NeuronWeights 120
4.4.1.2: TheErrorGradientwithRespecttothe
Hidden-NeuronWeights 123
4.4.1.3: ApplicationofGradientDescentin
BackpropagationLearning 127
4.4.1.4: BatchLearning 128
4.4.1.5: LearningRateandWeightUpdate 130
4.4.1.6: Example-by-Example(Online)Learning 134
4.4.1.7: Momentum 134
4.4.2: Example:BackpropagationLearning
ComputerExperiment 138
4.4.3: Single-InputSingle-OutputNetworkwith
MultipleHiddenNeurons 141
4.4.4: Multiple-Input,Multiple-HiddenNeuron,and
Single-OutputNetwork 142
4.4.5: Multiple-Input,Multiple-HiddenNeuron,
Multiple-OutputNetwork 143
4.4.6: Example:BackpropagationLearningCase
Study—SolvingaComplexClassificationProblem 145
4.5: Delta-Bar-DeltaLearning(AdaptiveLearningRate)Method 152
4.5.1: Example:NetworkTrainingwithDelta-Bar-Delta—
AHandComputation 154
4.5.2: Example:Delta-Bar-DeltawithMomentum—
AHandComputation 157
4.5.3: NetworkTrainingwithDelta-BarDelta—
AComputerExperiment 158
4.5.4: ComparisonofDelta-Bar-DeltaMethodwith
Backpropagation 159