VLSI Artificial Neural Networks Engineering

334 Pages·1994·7.59 MB·English

by  Waleed Fakhr, Mohamed I. Elmasry (auth.), Mohamed I. Elmasry (eds.)

VLSI ARTIFICIAL NEURAL
NETWORKS ENGINEERING
VLSI ARTIFICIAL NEURAL
NETWORKS ENGINEERING
EDITED BY
Mohamed 1. Elmasry
VLSI Research Group, Universify of Waterloo
....
"
SPRINGER SCIENCE+BUSINESS MEDIA, LLC
Library of Congress Cataloging-in-Publication Data
VLSI artificial neural networks engineering / edited by Mohamed
1. E1masry.
p. em.
Includes bib1iographical references and index.
ISBN 978-1-4613-6194-7 ISBN 978-1-4615-2766-4 (eBook)
DOI 10.1007/978-1-4615-2766-4
1. Neural networks (Computer scienee) 2. Integrated eireuits
-Very 1arge scale integration. 1. E1masry, Mohamed 1., 1943-
QA76.87.V57 1994
621.39'9--dc20 94-26768
CIP
Copyright © 1994 by Springer Science+Business Media New York
Originally published by Kluwer Academic Publishers in 1994
Softcover reprint ofthe hardcover Ist edition 1994
AH rights reserved. No part of this publieation may be reprodueed, stored in
a retrieval system or transmitted in any form or by any means, mechanical,
photo-copying, recording, or otherwise, without the prior written permis sion of
the publisher. Springer Science+Business Media, LLC.
Printed an acidlree pap er.
To
Elizabeth, Carmen, Samir, Nadia and Hassan Elmasry
CONTENTS
1 AN OVERVIEW
Waleed Fakhr and Mohamed I. Elmasry 1
1.1 Introduction 1
1.2 Biological Neural Networks
1.3 Artificial Neural Networks (ANNs) 4
1.4 Artificial Neural Network Algorithms 4
1.5 Supervised Neural Networks 5
1.6 Unsupervised Neural Networks 17
1.7 Neural Network Architectures and Implementations 19
1.8 Book Overview 22
2 A SAMPLED-DATA CMOS VLSI IMPLEMENTATION OF A
MULTI· CHARACTER ANN RECOGNITION SYSTEM
Sameh E. Rehan and Mohamed I. Elmasry 33
2.1 Introduction 33
2.2 ANN Implementation Techniques 34
2.3 Developed CMOS Circuits for ANNs 46
2.4 The Prototype MLP ANN Model Architecture 60
2.5 MLP ANN Model Simulations 62
2.6 ANN Circuit Simulations 72
2.7 The Developed VLSI Architectures 72
2.8 The Developed Two-Character ANN Recognizer 77
2.9 The Proposed Multi-Character ANN Recognition System 79
2.10 Conclusions 81
viii VLSI ARTIFICIAL NEURAL NETWORKS ENGINEERING
3 A DESIGN AUTOMATION ENVIRONMENT FOR MIXED
ANALOGIDIGITAL ANNS
Arun Achyuthan and Mohamed l. Elmasry 91
3.1 Introduction 91
3.2 Mixed Analog/Digital ANN Hardware 93
3.3 Overview of the Design Automation Environment 97
3.4 Data Flow Graph 101
3.5 The Analyzer 102
3.6 The Design Library 112
3.7 The Synthesizer 114
3.8 Design Examples 122
3.9 Conclusions 132
4 A COMPACT VLSI IMPLEMENTATION OF NEURAL
NETWORKS
Liang-Yong Song, Anthony Vannelli and Mohamed I. Elmasry 139
4.1 Introduction 139
4.2 The Building Blocks 140
4.3 The Circuit Implementation Example 148
4.4 Expanding the Network with Multiple Chips 153
4.5 Conclusions 155
5 AN ALL-DIGITAL VLSI ANN
Brian White and Mohamed I. Elmasry 157
5.1 Introduction 157
5.2 Neocognitron Neural Network Model 160
5.3 Digi-Neocognitron (DNC) : A Digital Neural Network Model
forVLSI 164
5.4 Character Recognition Example 177
5.5 Advantages for VLSI Implementation 183
5.6 Conclusions 185
6 A NEURAL PREDICTIVE HIDDEN MARKOV MODEL
ARCHITECTURE FOR SPEECH AND SPEAKER
RECOGNITION
Khaled Hassanein, Li Deng and Mohamed l. Elmasry 191
Contents ix
6.1 Introduction 191
6.2 Automatic Speech Recognition Methodologies 192
6.3 An ANN Architecture for Predictive HMMs 203
6.4 Discriminative Training of the Neural Predictive HMM 223
6.5 Speaker Recognition Using the Neural Predictive HMM 231
6.6 Conclusions 238
7 MINIMUM COMPLEXITY NEURAL NETWORKS FOR
CLASSIFICATION
Waleed Fakhr, Mohamed Kamel and Mohamed I. Elmasry 247
7.1 Introduction 247
7.2 Adaptive Probabilistic Neural Networks: APNN and ANNC 251
7.3 Bayesian PDF Model Selection 255
7.4 Maximum Likelihood and Maximum Mutual Information Train-
ing 260
7.5 Experimental Results . 263
7.6 The Adaptive Feature Extraction Nearest Neighbor Classifier
"AFNN" 268
7.7 Conclusions 279
8 A PARALLEL ANN ARCHITECTURE FOR FUZZY
CLUSTERING
Dapeng Zhang, Mohamed Kamel and Mohamed I. Elmasry 283
8.1 Introduction 283
8.2 Fuzzy Clustering and Neural Networks 284
8.3 Fuzzy Competitive Learning Algorithm 288
8.4 Mapping Algorithm Onto Architecture 291
8.5 Fuzzy Clustering Neural Network (FCNN) Architecture: Pro-
cessing Cells 292
8.6 Comparison With The Fuzzy C-Mean (FCM) Algorithm 295
8.7 Conclusions 299
9 A PIPELINED ANN ARCHITECTURE FOR SPEECH
RECOGNITION
Dapeng Zhang, Li Deng and Mohamed I. Elmasry 303
9.1 Introduction 303
x VLSI ARTIFICIAL NEURAL NETWORKS ENGINEERING
9.2 Definition and Notation 304
9.3 PNN Architecture: Processing Stages 306
9.4 Case Studies 312
9.5 Perfonnance Analysis 316
9.6 Conclusions 320
INDEX 323
CONTRIBUTORS
Sameh E. Rehan
Arun Achyuthan
Department of Electrical and Computer Department of Electrical and Computer
Engineering Engineering
University of Waterloo University of Waterloo
Waterloo, Ontario, Canada Waterloo, Ontario, Canada
LiDeng Liang-Yong Song
Department of Electrical and Computer Department of Electrical and Computer
Engineering Engineering
University of Waterloo University of Waterloo
Waterloo, Ontario, Canada Waterloo, Ontario, Canada
Mohamed I. Elmasry Anthony Vannelli
Department of Electrical and Computer Department of Electrical and Computer
Engineering Engineering
University of Waterloo University of Waterloo
Waterloo, Ontario, Canada Waterloo, Ontario, Canada
Waleed Fakhr Brian White
Department of Electrical and Computer Department of Electrical and Computer
Engineering Engineering
University of Waterloo University of Waterloo
Waterloo, Ontario, Canada Waterloo, Ontario, Canada
Khaled Hassanein Dapeng Zhang
Department of Electrical and Computer Department of Electrical and Computer
Engineering Engineering
University of Waterloo University of Waterloo
Waterloo, Ontario, Canada Waterloo, Ontario, Canada
Mohamed Kamel
Systems Design Engineering
University of Waterloo
Waterloo, Ontario, Canada
Preface
Engineers have long been fascinated by how efficient and how fast biological
neural networks are capable of performing such complex tasks as recognition.
Such networks are capable of recognizing input data from any oft he five senses
with the necessary accuracy and speed to allow living creatures to survive.
Machines which perform such complex tasks as recognition, with similar ac
curacy and speed, were difficult to implement until the technological advances
of VLSI circuits and systems in the late 1980's. Since then, the field of VLSI
Artificial Neural Networks (ANNs) have witnessed an exponential growth and
a new engineering discipline was born. Today, many engineering curriculums
have included a course or more on the subject at the graduate or senior under
graduate levels.
Since the pioneering book by Carver Mead; "Analog VLSI and Neural Sys
tems", Addison-Wesley, 1989; there were a number of excellent text and ref
erence books on the subject, each dealing with one or two topics. This book
attempts to present an integrated approach of a single research team to VLSI
ANNs Engineering.
Chapter 1 serves as an introduction. Chapters 2, 3, 4 and 5 deal with VLSI
circuit design techniques (analog, digital and sampled data) and automated
VLSI design environment for ANNs. Chapter 2 reports on a sampled data
approach to the implementation of ANNs with application to character recog
nition. Chapter 2 also contains an overview of the different approaches ofVLSI
implementation of ANNs; explaining the advantage and disadvantage of each
approach. In Chapter 3, the topic of design exploration of mixed analog/digital
ANNs at the high level of the design hierarchy is addressed. The need for cre
ating such a design automation environment, with its supporting CAD tools, is
a necessary condition for the widespread use of application-specific chips for
ANN implementation. In Chapter 4, the same topic of design exploration is
discussed, but at the low level of the hierarchy and targeting analog implemen
tation. Chapter 5 reports on an all-digital implementation of ANNs using the
Neocognitron as the ANN model.
Chapters 6,7,8 and 9 deal with the application of ANNs to a number of fields.
Chapter 6 addresses the topic of automatic speech recognition using neural
predictive hidden Markov Models. Chapter 7 deals with the topic of classi
fication using minimum complexity ANNs. Chapter 8 addresses the topic of
pattern recognition using a fuzzy clustering ANNs. Finally, Chapter 9 deals
with speech recognition using pipelined ANNs.