Table Of ContentStatistical Learning
Using Neural
Networks
Statistical Learning
Using Neural
Networks
A Guide for Statisticians and Data
Scientists
By
Basilio de Bragança Pereira
Federal University of Rio de Janeiro (UFRJ)
and
National Laboratory for Scientific Computing
(LNCC)
Calyampudi Radhakrishna Rao
Pennsylvania State University
and
University at Buffalo
Fábio Borges de Oliveira
National Laboratory for Scientific Computing
(LNCC)
First edition published 2020
by CRC Press
6000 Broken Sound Parkway NW, Suite 300, Boca Raton, FL 33487-2742
and by CRC Press
2 Park Square, Milton Park, Abingdon, Oxon, OX14 4RN
© 2020 Taylor & Francis Group, LLC
CRC Press is an imprint of Taylor & Francis Group, LLC
Reasonable efforts have been made to publish reliable data and information, but the author and
publisher cannot assume responsibility for the validity of all materials or the consequences of their
use. The authors and publishers have attempted to trace the copyright holders of all material repro
duced in this publication and apologize to copyright holders if permission to publish in this form
has not been obtained. If any copyright material has not been acknowledged please write and let us
know so we may rectify in any future reprint.
Except as permitted under U.S. Copyright Law, no part of this book may be reprinted, reproduced,
transmitted, or utilized in any form by any electronic, mechanical, or other means, now known or
hereafter invented, including photocopying, microfilming, and recording, or in any information
storage or retrieval system, without written permission from the publishers.
For permission to photocopy or use material electronically from this work, access www.copyright.
com or contact the Copyright Clearance Center, Inc. (CCC), 222 Rosewood Drive, Danvers, MA
01923, 978-750-8400. For works that are not available on CCC please contact mpkbookspermis
[email protected]
Trademark notice: Product or corporate names may be trademarks or registered trademarks, and
are used only for identification and explanation without intent to infringe.
Library of Congress Cataloging-in-Publication Data
Names: Pereira, Basilio de Bragança, author. | Rao, C. Radhakrishna
(Calyampudi Radhakrishna), 1920-author. | Borges de Oliveira, Fábio, author.
Title: Statistical learning using neural networks : a guide for
statisticians and data scientists / by Basilio de Bragança Pereira,
Calyampudi Radhakrishna Rao, Fábio Borges de Oliveira.
Description: First edition. | Boca Raton : CRC Press, 2020. | Includes
bibliographical references and index.
Identifiers: LCCN 2019056045 (print) | LCCN 2019056046 (ebook) | ISBN
9781138364509 (hardback) | ISBN 9780429431296 (ebook)
Subjects: LCSH: Statistics--Data processing. | Multivariate analysis--Data
processing. | Neural networks (Computer science) | Computational
learning theory. | Statistics--Methodology. | Python (Computer program language)
Classification: LCC QA276.4 .P466 2020 (print) | LCC QA276.4 (ebook) |
DDC 519.50285/632--dc23
LC record available at https://lccn.loc.gov/2019056045
LC ebook record available at https://lccn.loc.gov/2019056046
ISBN: 978-1-138-36450-9 (hdk)
ISBN: 978-0-429-43129-6 (ebk)
Typeset in Computer Modern font
by Cenveo Publisher Services
To our families.
Contents
Preface xi
Acknowledgments xiii
1 Introduction 1
2 Fundamental Concepts on Neural Networks 3
2.1 Artificial Intelligence: Symbolist and Connectionist . . . . . . . . . 3
2.2 TheBrainandNeuralNetworks . . . . . . . . . . . . . . . . . . . 4
2.3 Artificial Neural Networks and Diagrams . . . . . . . . . . . . . . 5
2.4 ActivationFunctions . . . . . . . . . . . . . . . . . . . . . . . . . 8
2.5 NetworkArchitectures . . . . . . . . . . . . . . . . . . . . . . . . 8
2.6 NetworkTraining . . . . . . . . . . . . . . . . . . . . . . . . . . . 12
2.7 KolmogorovTheorem . . . . . . . . . . . . . . . . . . . . . . . . 13
2.8 ModelChoice . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15
2.8.1 Generalization . . . . . . . . . . . . . . . . . . . . . . . . 16
2.8.2 Bias-variance Trade-off: Early Stopping Method of Training 16
2.8.3 ChoiceofStructure . . . . . . . . . . . . . . . . . . . . . . 17
2.8.4 NetworkGrowing . . . . . . . . . . . . . . . . . . . . . . 18
2.8.5 NetworkPruning . . . . . . . . . . . . . . . . . . . . . . . 18
2.9 McCulloch-PittNeuron . . . . . . . . . . . . . . . . . . . . . . . 19
2.10 RosenblattPerceptron . . . . . . . . . . . . . . . . . . . . . . . . 20
2.11 Widrow’sAdalineandMadaline . . . . . . . . . . . . . . . . . . . 22
2.12 Terminology . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23
2.13 RunningPythoninaNutshell . . . . . . . . . . . . . . . . . . . . 28
3 Some Common Neural Network Models 33
3.1 MultilayerFeedforwardNetworks . . . . . . . . . . . . . . . . . . 33
3.2 AssociativeandHopfieldNetworks . . . . . . . . . . . . . . . . . 37
3.3 RadialBasisFunctionNetworks . . . . . . . . . . . . . . . . . . . 45
3.4 WaveletNeuralNetworks . . . . . . . . . . . . . . . . . . . . . . 46
3.4.1 Wavelets . . . . . . . . . . . . . . . . . . . . . . . . . . . 48
3.4.2 Wavelet Networks and Radial Basis Wavelet Networks. . . . 51
3.5 Mixture-of-ExpertsNetworks . . . . . . . . . . . . . . . . . . . . 51
3.6 Neural Network and Statistical Model Interfaces . . . . . . . . . . 55
3.7 SomeCommonNeuralNetworksinPython . . . . . . . . . . . . . 56
3.7.1 FittingData . . . . . . . . . . . . . . . . . . . . . . . . . . 56
vii
viii Contents
3.7.2 Classification . . . . . . . . . . . . . . . . . . . . . . . . . 58
3.7.3 HopfieldNetworks . . . . . . . . . . . . . . . . . . . . . . 62
4 Multivariate Statistics Neural Network Models 67
4.1 ClusterandScalingNetworks . . . . . . . . . . . . . . . . . . . . 67
4.1.1 CompetitiveNetworks . . . . . . . . . . . . . . . . . . . . 67
4.1.2 Learning Vector Quantization (LVQ) . . . . . . . . . . . . 70
4.1.3 Adaptive Resonance Theory (ART) Networks . . . . . . . . 72
4.1.4 Self-Organizing Maps (SOM) Networks . . . . . . . . . . . 78
4.2 DimensionalReductionNetworks . . . . . . . . . . . . . . . . . . 87
4.2.1 BasicStructureofDataMatrix . . . . . . . . . . . . . . . . 88
4.2.2 Mechanics of Some Dimensional Reduction Techniques . . 90
4.2.2.1 Principal Components Analysis (PCA) . . . . . . 90
4.2.2.2 Nonlinear Principal Components . . . . . . . . . 91
4.2.2.3 FactorAnalysis(FA) . . . . . . . . . . . . . . . 91
4.2.2.4 Correspondence Analysis (CA) . . . . . . . . . . 91
4.2.2.5 Multidimensional Scaling . . . . . . . . . . . . . 92
4.2.2.6 Independent Component Analysis (ICA) . . . . . 93
4.2.3 PCANetworks . . . . . . . . . . . . . . . . . . . . . . . . 97
4.2.4 NonlinearPCANetworks . . . . . . . . . . . . . . . . . . 102
4.2.5 FANetworks . . . . . . . . . . . .. . . . . . . . . . . . . 102
4.2.6 Correspondence Analysis (CA) Networks . . . . . . . . . . 103
4.2.7 Independent Component Analysis (ICA) Networks . . . . . 106
4.3 ClassificationNetworks . . . . . . . . . . .. . . . . . . . . . . . 108
4.4 Multivariate Statistics Neural Network Models with Python . . . . 120
4.4.1 Clustering . . . . . . . . . . . . . . . . . . . . . . . . . . . 121
4.4.2 FittingData . . . . . . . . . . . . . . . . . . . . . . . . . . 126
5 Regression Neural Network Models 131
5.1 Generalized Linear Model Networks (GLIMNs) . . . . . . . . . . 131
5.1.1 LogisticRegressionNetworks . . . . . . . . . . . . . . . . 132
5.1.2 RegressionNetworks . . . . . . . . . . .. . . . . . . . . . 136
5.2 Nonparametric Regression and Classification Networks . . . . . . . 139
5.2.1 Probabilistic Neural Networks (PNNs) . . . . . . . . . . . . 139
5.2.2 General Regression Neural Networks (GRNNs) . . . . . . . 140
5.2.3 Generalized Additive Model Networks . . . . . . . . . . . 141
5.2.4 Regression and Classification Tree Networks . . . . . . . . 143
5.2.5 Projection Pursuit and Feedforward Networks . . . . . . . . 145
5.2.6 Example . . . . . . . . . . . . .. . . . . . . . . . . . . . 146
5.3 Regression Neural Network Models with Python . . . . . . . . . . 147
6 Survival Analysis and Other Networks 151
6.1 SurvivalAnalysisNetworks . . . . . . . . . . . .. . . . . . . . . 151
6.2 TimeSeriesForecasting . . . . . . . . . . . . . .. . . . . . . . . 158
6.2.1 Forecasting with Neural Networks . . . . . . . . . . . . . . 163
Contents ix
6.3 ControlChartNetworks . . . . . . . . . . . . . . . . . . . . . . . 167
6.4 SomeStatisticalInferenceResults . . . . . . . . . . . . . . . . . . 170
6.4.1 EstimationMethods . . . . . . . . . . . . . . . . . . . . . 171
6.4.2 BayesianMethods . . . . . . . . . . . . . . . . . . . . . . 172
6.4.3 IntervalEstimation . . . . . . . . . . . . . . . . . . . . . . 175
6.4.4 StatisticalTests . . . . . . . . . . . . . . . . . . . . . . . . 176
6.5 ForecastingwithPython . . . . . . . . . . . . . . . . . . . . . . . 177
A Command Reference 183
Bibliography 215
Index 233
