Neural Networks and Computing. Learning Algorithms and Applications

322 Pages·2016·2.36 MB·English

by  Chow T.W.S.

Tommy W S Chow
Siu-Yeung Cho
Neural Networks
and Computing
Learning Algorithms and Applications
Imperial
Neural Networks
and Computing
Learning Algorithms and Applications
SERIES IN ELECTRICAL AND COMPUTER ENGINEERING
Editor: Wai-Kai Chen (University of Illinois, Chicago, USA)
Published:
Vol. 1: Net Theory and Its Applications
Flows in Networks
by W. K. Chen
Vol. 2: A Mathematical Introduction to Control Theory
by S. Engelberg
Vol. 5: Security Modeling and Analysis of Mobile Agent Systems
by Lu Ma and Jeffrey J. P. Tsai
Vol. 6: Multi-Objective Group Decision Making: Methods Software and
Applications with Fuzzy Set Techniques
by Jie Lu, Guangquan Zhang, Da Ruan and Fengjie Wu
Steven - Neural Networks and Computing.pmd2 9/20/2007, 1:51 PM
SERIES IN ELECTRICAL AND
VOL.7
COMPUTER ENGINEERING
Neural Networks
and Computing
Learning Algorithms and Applications
TommyW S Chow
City University of Hong Kong, Hong Kong
Siu-Yeung Cho
Nanyang Technological University, Singapore
----(Dr---~~~~~~~~~-Im~p_en_._al_C_o_l_le_g_e_P_r_e_ss~
Published by
Imperial College Press
57 Shelton Street
Covent Garden
London WC2H 9HE
Distributed by
World Scientific Publishing Co. Pte. Ltd.
5 Toh Tuck Link, Singapore 596224
USA office: 27 Warren Street, Suite 401-402, Hackensack, NJ 07601
UK office: 57 Shelton Street, Covent Garden, London WC2H 9HE
British Library Cataloguing-in-Publication Data
A catalogue record for this book is available from the British Library.
NEURAL NETWORKS AND COMPUTING
Series in Electrical and Computer Engineering — Vol. 7
Copyright © 2007 by Imperial College Press
All rights reserved. This book, or parts thereof, may not be reproduced in any form or by any means,
electronic or mechanical, including photocopying, recording or any information storage and retrieval
system now known or to be invented, without written permission from the Publisher.
For photocopying of material in this volume, please pay a copying fee through the Copyright
Clearance Center, Inc., 222 Rosewood Drive, Danvers, MA 01923, USA. In this case permission to
photocopy is not required from the publisher.
ISBN-13978-1-86094-758-2
ISBN-101-86094-758-1
Printed in Singapore.
Steven - Neural Networks and Computing.pmd1 9/20/2007, 1:51 PM
Preface

The area of Neural computing that we shall discuss in this
book represents a combination of techniques of classical
optimization, statistics, and information theory. Neural network
was once widely called artificial neural networks, which
represented how the emerging technology was related to artificial
intelligence. It once was a topic that had captivated the interest of
most computer scientists, engineers, and mathematicians. Its
charm of being an adaptive system or universal functional
approximator has compelling appeal to most researchers and
engineers. The Backpropagation training algorithm was once the
most popular keywords used in most engineering conferences.
There is an interesting history on this area dated back from the late
fifties which we saw the advent of the Mark I Perceptron. But the
real intriguing history started from the sixties that we saw Minsky
and Papert’s book “Perceptrons” discredited the early neural
research work. For all neural researchers, the late eighties are well
remembered because the research of neural networks was
reinstated and repositioned. From the nineties to the new
millennium is history to be made by all neural researchers. We saw
the flourish of this topic and its applications stretched from
rigorous mathematical proof to different physical science and even
business applications. Researchers now tend to use the term
“neural networks” instead of “artificial neural networks” when we
have understood the theoretical background more. There have
been volumes of research literature published on the new
development of neural theory and applications. There have been
many attempts to discuss this topic from either a very
mathematical way or a very practical way. But to most users
including students and engineers, how to employ an appropriate
neural network learning algorithm and the selection of model for a
given physical problem appear to be the main issue.
This book, written from a more application perspective,
provides thorough discussions on neural network learning
v
vi Neural Networks and Computing: Learning Algorithms and Applications
algorithms and their related issues. We strive to find the balance
in covering the major topics in neurocomputing, from learning
theory, learning algorithms, network architecture to applications.
We start the book from the fundamental building block “neuron”
and the earliest neural network model, McCulloh and Pitts Model.
In the beginning, we treat the learning concept from the well-
known regression problem which shows how the idea of data
fitting can be used to explain the fundamental concept of neural
learning. We employ an error convex surface to illustrate the
optimization concept of learning algorithm. This is important as it
shows readers that the neural learning algorithm is nothing more
than a high dimensional optimization problem. One of the beauties
of neural network is being a soft computing approach in which the
selection of a model structure and initial settings may not have
noticeable effect on the final solution. But neural learning process
also suffers from its problem of being slow and stuck in local
minima especially when it is required to handle a rather complex
problem. These are the two main issues addressed in the later
chapters of this book. We study the neural learning problem from
a new perspective and offer several modified algorithms to
enhance the learning speed and its convergence ability. We also
show initializations of a network have significant effect on the
learning performance. Different initialization methods are then
discussed and elaborated.
Later chapters of the book deal with Basis function
network, Self-Organizing map, and Feature Selection. These are
interesting and useful topics to most engineering and science
researchers. The Self-Organizing map is the most widely used
unsupervised neural network. It is useful for performing
clustering, dimensional reduction, and classification. The SOM is
very different from the feedforward neural network in the sense of
architecture and learning algorithm. In this book, we have
provided thorough discussions and newly developed extended
algorithms for readers to use. Classification and Feature selection
is discussed in Chapter 6. We include this topic in the book
because bioinformatics has recently become a very important
Preface vii

research area. Gene selection using computational methods, and
performing cancer classification computationally have become the
21st Century research. This book provides a detail discussion on
feature selection and how different methods be applied to gene
selection and cancer classification. We hope this book will
provide useful and inspiring information to readers. A number of
software algorithms written in MATLAB are available for readers
to use. Although the authors have gone through the book for few
times checking typos and errors, we would appreciate readers
notifying us about any typos found.
At last, the authors would thank the support and help from
his colleagues and students. The author must thank his students
Mr. Gaoyang Dai, Dr. Di Huang, Dr. Jim Y F Yam, Dr. Sitao Wu,
and Dr. M K M Rahman for their help in preparing the material.
The authors would also thank Prof. Guanrong Chen for introducing
us to the very helpful publisher. The authors must thank the World
Scientific Publishing Co for publishing this book.
Last but not least, Tommy Chow would like to thank Irene,
Adrian and Ian for their tolerance and support.

.
TThhiiss ppaaggee iinntteennttiioonnaallllyy lleefftt bbllaannkk
Contents
Preface v

1. Introduction 1
1.1 Background . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1
1.2 Neuron Model . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2
1.3 Historical Remarks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4
1.4 Network architecture . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6
1.4.1 Supervised Neural Networks . . . . . . . . . . . . . . . . . . . . . . . . . 6
1.4.1.1 McCulloh and Pitts Model . . . . . . . . . . . . . . . . . . . . . 7
1.4.1.2 The Perceptron Model. . . . . . . . . . . . . . . . . . . . . . . . . 11
1.4.1.3 Multi-layer Feedforward Network . . . . . . . . . . . . . . . 14
1.4.1.4 Recurrent Networks . . . . . . . . . . . . . . . . . . . . . . . . . . 15
1.4.2 Unsupervised Neural Networks . . . . . . . . . . . . . . . . . . . . . . . 17
1.5 Modeling and Learning Mechanism . . . . . . . . . . . . . . . . . . . . . . . . 19
1.5.1 Determination of Parameters . . . . . . . . . . . . . . . . . . . . . . . . . . 20
1.5.2 Gradient Descent Searching Method . . . . . . . . . . . . . . . . . . . 26
Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28

2. Learning Performance and Enhancement 31
2.1 Fundamental of Gradient Descent Optimization . . . . . . . . . . . . . . . . 32
2.2 Conventional Backpropagation Algorithm . . . . . . . . . . . . . . . . . . . . 35
2.3 Convergence Enhancement . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42
2.3.1 Extended Backpropagation Algorithm . . . . . . . . . . . . . . . . . . 44
2.3.2 Least Squares Based Training Algorithm . . . . . . . . . . . . . . . . 47
2.3.3 Extended Least Squares Based Algorithm . . . . . . . . . . . . . . . . 55
2.4 Initialization Consideration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 59
2.4.1 Weight Initialization Algorithm I . . . . . . . . . . . . . . . . . . . . . . . 61
2.4.2 Weight Initialization Algorithm II . . . . . . . . . . . . . . . . . . . . . . 64
2.4.3 Weight Initialization Algorithm III . . . . . . . . . . . . . . . . . . . . . . 67
2.5 Global Learning Algorithms . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 69
2.5.1 Simulated Annealing Algorithm . . . . . . . . . . . . . . . . . . . . . . . . 70
ix