Table Of Content2278/*FM/frame Page 1 Friday, January 17, 2003 7:35 AM
S
UPERVISED AND
U P
NSUPERVISED ATTERN
R :
ECOGNITION
F E
EATURE XTRACTION
C
AND OMPUTATIONAL
I
NTELLIGENCE
Evangelia Micheli-Tzanakou
Editor/Author
E-mail:[email protected]
CRC Press
1999
2278/*FM/frame Page 2 Friday, January 17, 2003 7:35 AM
Library of Congress Cataloging-in-Publication Data
Micheli-Tzanakou, Evangelia, 1942-
Supervised and unsupervised pattern recognition: feature extraction and computational
intelligence /Evangelia Micheli-Tzanakou, editor/author
p. cm.-- (Industrial electronics series)
Includes bibliographical references and index.
ISBN 0-8493-2278-2
1. Pattern recognition systems. 2. Neural networks (Computer science) I. Title. II.
Series.
TK7882.P3 M53 1999
006.4--dc21
99-043495
CIP
This book contains information obtained from authentic and highly regarded sources. Reprinted
material is quoted with permission, and sources are indicated. A wide variety of references are listed.
Reasonable efforts have been made to publish reliable data and information, but the author and the
publisher cannot assume responsibility for the validity of all materials or for the consequences of their use.
Neither this book nor any part may be reproduced or transmitted in any form or by any means,
electronic or mechanical, including photocopying, microfilming, and recording, or by any information
storage or retrieval system, without prior permission in writing from the publisher.
The consent of CRC Press does not extend to copying for general distribution, for promotion, for
creating new works, or for resale. Specific permission must be obtained in writing from CRC Press for
such copying.
Direct all inquiries to CRC Press LLC., 2000 Corporate Blvd., N.W., Boca Raton, Florida 33431.
© 2000 by CRC Press LLC
No claim to original U.S. Government works
International Standard Book Number 0-8493-2278-2
Library of Congress Card Number 99-043495
Printed in the United States of America 1 2 3 4 5 6 7 8 9 0
Printed on acid-free paper
2278/*FM/frame Page 3 Friday, January 17, 2003 7:35 AM
Contents
Section I — Overviews of Neural Networks, Classifiers, and
Feature Extraction Methods—Supervised
Neural Networks
Chapter 1 Classifiers: An Overview
1.1 Introduction.......................................................................................................3
1.2 Criteria for Optimal Classifier Design.............................................................3
1.3 Categorizing the Classifiers..............................................................................4
1.3.1 Bayesian Optimal Classifiers................................................................4
1.3.2 Exemplar Classifiers.............................................................................5
1.3.3 Space Partition Methods.......................................................................6
1.3.4 Neural Networks...................................................................................7
1.4 Classifiers..........................................................................................................7
1.4.1 Bayesian Classifiers..............................................................................7
1.4.1.1 Minimum ECM Classifers.....................................................8
1.4.1.2 Multi-Class Optimal Classifiers.............................................9
1.4.2 Bayesian Classifiers with Multivariate Normal Populations..............11
1.4.2.1 Quadratic Discriminant Score..............................................11
1.4.2.2 Linear Discriminant Score...................................................11
1.4.2.3 Linear Discriminant Analysis and Classification................12
1.4.2.4 Equivalence of LDF to Minimum TPM Classifier..............14
1.4.3 Learning Vector Quantizer (LVQ)......................................................14
1.4.3.1 Competitive Learning...........................................................14
1.4.3.2 Self-Organizing Map............................................................15
1.4.3.3 Learning Vector Quantization..............................................15
1.4.4 Nearest Neighbor Rule.......................................................................18
1.5 Neural Networks (NN)....................................................................................19
1.5.1 Introduction.........................................................................................19
1.5.1.1 Artificial Neural Networks...................................................19
1.5.1.2 Usage of Neural Networks...................................................19
1.5.1.3 Other Neural Networks........................................................20
1.5.2 Feed-Forward Neural Networks.........................................................20
1.5.3 Error Backpropagation........................................................................22
1.5.3.1 Madaline Rule III for Multilayer Network with
Sigmoid Function.................................................................25
1.5.3.2 A Comment on the Terminology ‘Backpropagation’..........25
2278/*FM/frame Page 4 Friday, January 17, 2003 7:35 AM
1.5.3.3 Optimization Machines with Feed-Forward
Multilayer Perceptrons.........................................................25
1.5.3.4 Justification for Gradient Methods for Nonlinear
Function Approximation......................................................26
1.5.3.5 Training Methods for Feed-Forward Networks...................27
1.5.4 Issues in Neural Networks..................................................................28
1.5.4.1 Universal Approximation.....................................................28
1.5.5 Enhancing Convergence Rate and Generalization of an
Optimization Machine.........................................................................29
1.5.5.1 Suggestions for Improving the Convergence......................30
1.5.5.2 Quick Prop...........................................................................31
1.5.5.3 Kullback-Leibler Distance...................................................32
1.5.5.4 Weight Decay.......................................................................33
1.5.5.5 Regression Methods for Classification Purposes................34
1.5.6 Two-Group Regression and Linear Discriminant Function...............34
1.5.7 Multi-Response Regression and Flexible Discriminant Analysis......36
1.5.7.1 Powerful Nonparametric Regression Methods for
Classification Problems........................................................37
1.5.8 Optimal Scoring (OS).........................................................................37
1.5.8.1 Partially Minimized ASR.....................................................39
1.5.9 Canonical Correlation Analysis..........................................................40
1.5.10 Linear Discriminant Analysis.............................................................41
1.5.10.1 LDA Revisited......................................................................41
1.5.11 Translation of Optimal Scoring Dimensions into
Discriminant Coordinates...................................................................42
1.5.12 Linear Discriminant Analysis via Optimal Scoring...........................44
1.5.12.1 LDA via OS.........................................................................45
1.5.13 Flexible Discriminant Analysis by Optimal Scoring.........................46
1.6 Comparison of Experimental Results.............................................................48
1.7 System Performance Assessment...................................................................49
1.7.1 Classifier Evaluation...........................................................................50
1.7.1.1 Hold-Out Method.................................................................51
1.7.1.2 K-Fold Cross-Validation......................................................51
1.7.2 Bootstrapping Method for Estimation................................................52
1.7.2.1 Jackknife Estimation............................................................53
1.7.2.2 Bootstrap Method.................................................................54
1.8 Analysis of Prediction Rates from Bootstrapping Assessment......................54
References.......................................................................................................56
Chapter 2 Artificial Neural Networks: Definitions, Methods, Applications
2.1 Introduction.....................................................................................................61
2.2 Definitions.......................................................................................................62
2.3 Training Algorithms........................................................................................64
2278/*FM/frame Page 5 Friday, January 17, 2003 7:35 AM
2.3.1 Backpropagation Algorithm................................................................65
2.3.2 The ALOPEX Algorithm....................................................................69
2.3.3 Multilayer Perceptron (MLP) Network Training with ALOPEX......71
2.4 Some Applications..........................................................................................72
2.4.1 Expert Systems and Neural Networks................................................72
2.4.2 Applications in Mammography..........................................................73
2.4.3 Chromosome and Genetic Sequences Classification.........................74
References.......................................................................................................75
Chapter 3 A System for Handwritten Digit Recognition
3.1 Introduction.....................................................................................................79
3.2 Preprocessing of Handwritten Digit Images..................................................79
3.2.1 Optimal Size of the Mask for Dilation..............................................85
3.2.2 Bartlett Statistic...................................................................................85
3.3 Zernike Moments (ZM) for Characterization of Image Patterns...................87
3.3.1 Reconstruction by Zernike Moments.................................................90
3.3.2 Features from Zernike Moments........................................................92
3.4 Dimensionality Reduction...............................................................................96
3.4.1 Principal Component Analysis...........................................................96
3.4.2 Discriminant Analysis ........................................................................98
3.5 Analysis of Prediction Error Rates from Bootstrapping Assessment..........100
3.6 Summary.......................................................................................................105
Acknowledgments.........................................................................................105
References.....................................................................................................105
Chapter 4 Other Types of Feature Extraction Methods
4.1 Introduction...................................................................................................109
4.2 Wavelets........................................................................................................110
4.2.1 Discrete Wavelet Series.....................................................................111
4.2.2 Discrete Wavelet Transform (DWT).................................................112
4.2.3 Spline Wavelet Transform.................................................................112
4.2.4 The Discrete B-Spline Wavelet Transform.......................................114
4.2.5 Design of Quadratic Spline Wavelets...............................................114
4.2.6 The Fast Algorithm...........................................................................117
4.3 Invariant Moments........................................................................................119
4.4 Entropy..........................................................................................................122
4.5 Cepstrum Analysis........................................................................................122
4.6 Fractal Dimension.........................................................................................123
4.7 SGLD Texture Features................................................................................126
References.....................................................................................................130
2278/*FM/frame Page 6 Friday, January 17, 2003 7:35 AM
Section II Unsupervised Neural Networks
Chapter 5 Fuzzy Neural Networks
5.1 Introduction...................................................................................................135
5.2 Pattern Recognition.......................................................................................135
5.2.1 Theory and Applications...................................................................135
5.2.2 Feature Extraction.............................................................................137
5.2.3 Clustering..........................................................................................138
5.3 Optimization..................................................................................................138
5.3.1 Theory and Objectives......................................................................138
5.3.2 Background.......................................................................................139
5.3.3 Modified ALOPEX Algorithm..........................................................141
5.4 System Design...............................................................................................144
5.4.1 Feature Extraction.............................................................................144
5.4.1.1 The Karhunen-Loève Expansion.......................................145
5.4.1.2 Application by a Neural Network.....................................147
5.5 Clustering......................................................................................................153
5.5.1 The Fuzzy c-Means (FCM) Clustering Algorithm...........................153
References.....................................................................................................159
Chapter 6 Application to Handwritten Digits
6.1 Introduction to Character Recognition.........................................................163
6.2 Data Collection.............................................................................................165
6.2.1 Preprocessing....................................................................................166
6.2.2 Noise Thresholding...........................................................................166
6.2.3 Center of Mass Adjustment..............................................................168
6.2.4 Line Thinning....................................................................................168
6.2.5 Fixing to Size....................................................................................168
6.2.6 Rotation.............................................................................................168
6.2.7 Reducing Resolution.........................................................................169
6.2.8 Blurring.............................................................................................170
6.3 Results...........................................................................................................170
6.4 Discussion.....................................................................................................177
6.5 Summary.......................................................................................................181
References.....................................................................................................182
Chapter 7 An Unsupervised Neural Network System for Visual Evoked
Potentials
7.1 Introduction...................................................................................................185
7.2 Data Collection and Preprocessing...............................................................186
7.3 System Design...............................................................................................187
7.4 Results...........................................................................................................188
2278/*FM/frame Page 7 Friday, January 17, 2003 7:35 AM
7.5 Discussion.....................................................................................................191
References.....................................................................................................194
Section III Advanced Neural Network Architectures/Modular
Neural Networks
Chapter 8 Classification of Mammograms Using a Modular Neural Network
8.1 Introduction.....................................................................................................197
8.2 Methods and System Overview......................................................................199
8.2.1 Data Acquisition................................................................................199
8.2.2 Feature Extraction by Transformation..............................................200
8.3 Modular Neural Networks............................................................................202
8.4 Neural Network Training..............................................................................203
8.5 Classification Results....................................................................................203
8.6 The Process of Obtaining Results................................................................207
8.7 ALOPEX Parameters....................................................................................209
8.8 Generalization...............................................................................................213
8.9 Conclusions...................................................................................................218
Acknowledgments.........................................................................................218
References.....................................................................................................218
Chapter 9 Visual Ophthalmologist: An Automated System for Classification
of Retinal Damage
9.1 Introduction...................................................................................................221
9.2 System Overview..........................................................................................221
9.2.1 Image Processing..............................................................................223
9.2.2 Feature Extraction Methods..............................................................223
9.2.3 Image Classification..........................................................................223
9.3 Modular Neural Networks............................................................................223
9.4 Application to Ophthalmology.....................................................................224
9.5 Results...........................................................................................................226
9.6 Discussion.....................................................................................................227
References.....................................................................................................227
Chapter 10 A Three-Dimensional Neural Network Architecture
10.1 Introduction...................................................................................................229
10.2 The Neural Network Architecture................................................................229
10.3 Simulations....................................................................................................230
10.3.1 Visual Receptive Fields.....................................................................231
10.3.2 Modeling of Parkinson’s Disease.....................................................235
10.4 Discussion.....................................................................................................238
References.....................................................................................................238
2278/*FM/frame Page 8 Friday, January 17, 2003 7:35 AM
Section IV General Applications
Chapter 11 A Feature Extraction Algorithm Using Connectivity Strengths
and Moment Invariants
11.1 Introduction...................................................................................................241
11.2 ALOPEX Algorithms....................................................................................242
11.2.1 Original Algorithm............................................................................242
11.2.2 Reinforcement Rules.........................................................................242
11.2.3 A Generalized ALOPEX Algorithm.................................................243
11.2.3.1 Process I.............................................................................244
11.2.3.2 Process II............................................................................245
11.3 Moment Invariants and ALOPEX................................................................246
11.4 Results and Discussion.................................................................................249
Acknowledgments.........................................................................................262
References.....................................................................................................262
Chapter 12 Multilayer Perceptrons with ALOPEX: 2D-Template Matching
and VLSI Implementation
12.1 Introduction...................................................................................................265
12.1.1 Multilayer Perceptrons .....................................................................265
12.2 Multilayer Perceptron and Template Matching............................................268
12.3 VLSI Implementation of ALOPEX..............................................................270
References.....................................................................................................275
Chapter 13 Implementing Neural Networks in Silicon
13.1 Introduction...................................................................................................277
13.2 The Living Neuron........................................................................................278
13.3 Neuromorphic Models..................................................................................280
13.4 Neurological Process Modeling....................................................................292
References.....................................................................................................299
Chapter 14 Speaker Identification through Wavelet Multiresolution
Decomposition and ALOPEX
14.1 Introduction...................................................................................................301
14.2 Multiresolution Analysis through Wavelet Decomposition..........................303
14.3 Pattern Recognition with ALOPEX..............................................................306
14.4 Methods.........................................................................................................306
14.4.1 Data Acquisition................................................................................306
14.4.2 Data Preprocessing............................................................................307
14.4.3 Representing the Wavelet Coefficients for Template Matching.......308
14.5 Results...........................................................................................................310
14.6 Discussion.....................................................................................................313
Acknowledgments.........................................................................................314
2278/*FM/frame Page 9 Friday, January 17, 2003 7:35 AM
References.....................................................................................................314
Chapter 15 Face Recognition in Alzheimer’s Disease: A Simulation
15.1 Introduction...................................................................................................317
15.2 Methods.........................................................................................................317
15.3 Results...........................................................................................................318
15.4 Discussion.....................................................................................................321
References.....................................................................................................321
Chapter 16 Self-Learning Layered Neural Network
16.1 Introduction...................................................................................................323
16.2 Neocognitron and Pattern Classification......................................................325
16.2.1 Training Algorithm............................................................................328
16.3 Objectives......................................................................................................329
16.4 Methods.........................................................................................................329
16.5 Study A..........................................................................................................330
16.5.1 Network Description..........................................................................330
16.5.2 Results from Study A........................................................................331
16.6 Study B..........................................................................................................332
16.6.1 Results from Study B........................................................................332
16.7 Summary and Discussion.............................................................................334
References.....................................................................................................346
Chapter 17 Biological and Machine Vision
17.1 Introduction...................................................................................................347
17.2 Distributed Representation............................................................................347
17.3 The Model.....................................................................................................348
17.4 A Modified ALOPEX Algorithm..................................................................348
17.5 Application to Template Matching...............................................................350
17.6 Brain to Computer Link................................................................................351
17.6.1 Global Receptive Fields in the Human Visual System....................351
17.6.2 The Black Box Approach..................................................................353
17.7 Discussion.....................................................................................................355
References.....................................................................................................358
Index......................................................................................................................359
