Supervised and Unsupervised Pattern Recognition: Feature Extraction and Computational

367 Pages·3.55 MB·english

by  Micheli-Tzanakou E.

Supervised
and

unsupervised

Pattern
Recognition
Feature Extraction and
Computational
© 2000 by CRC Press LLC
Industrial Electronics Series
Series Editor
J. David Irwin, Auburn University

Titles Included in the Series

Supervised and Unsupervised Pattern Recognition:
Feature Extraction and Computational Intelligence
Evangelia Micheli-Tzanakou, Rutgers University
Handbook of Applied Computational Intelligence
Mary Lou Padgett, Auburn University
Nicholas Karayiannis, University of Houston
Lofti A. Zaden, University of California Berkeley
Handbook of Applied Neurocontrols
Mary Lou Padgett, Auburn University
Charles C. Jorgensen, NASA Ames Research Center
Paul Werbos, National Science Foundation
Handbook of Power Electronics
Tim L. Skvarenina, Purdue University
© 2000 by CRC Press LLC
Industrial Electronics Series
Supervised
and
unsupervised
Pattern
Recognition
Feature Extraction and
Computational
Evangelia Micheli-Tzanakou
Rutgers University
Piscataway, New Jersey
CRC Press
Boca Raton London New York Washington, D.C.
Library of Congress Cataloging-in-Publication Data
Micheli-Tzanakou, Evangelia, 1942-
Supervised and unsupervised pattern recognition: feature extraction and computational
intelligence /Evangelia Micheli-Tzanakou, editor/author
p. cm.-- (Industrial electronics series)
Includes bibliographical references and index.
ISBN 0-8493-2278-2
1. Pattern recognition systems. 2. Neural networks (Computer science) I. Title. II.
Series.
TK7882.P3 M53 1999
006.4--dc21
99-043495
CIP
This book contains information obtained from authentic and highly regarded sources. Reprinted
material is quoted with permission, and sources are indicated. A wide variety of references are listed.
Reasonable efforts have been made to publish reliable data and information, but the author and the
publisher cannot assume responsibility for the validity of all materials or for the consequences of their use.
Neither this book nor any part may be reproduced or transmitted in any form or by any means,
electronic or mechanical, including photocopying, microfilming, and recording, or by any information
storage or retrieval system, without prior permission in writing from the publisher.
The consent of CRC Press does not extend to copying for general distribution, for promotion, for
creating new works, or for resale. Specific permission must be obtained in writing from CRC Press for
such copying.
Direct all inquiries to CRC Press LLC., 2000 Corporate Blvd., N.W., Boca Raton, Florida 33431.
© 2000 by CRC Press LLC
No claim to original U.S. Government works
International Standard Book Number 0-8493-2278-2
Library of Congress Card Number 99-043495
Printed in the United States of America 1 2 3 4 5 6 7 8 9 0
Printed on acid-free paper
© 2000 by CRC Press LLC
Dedication
To my late mother for never being satisfied with my progress
and for always pushing me to better things in life.
© 2000 by CRC Press LLC
PREFACE
This volume describes the application of supervised and unsupervised pattern rec-
ognition schemes to the classification of various types of waveforms and images.
An optimization routine, ALOPEX, is used to train the network while decreasing
the likelihood of local solutions. The chapters included in this volume bring together
recent research of more than ten authors in the field of neural networks and pattern
recognition. All of these contributions were carried out in the Neuroelectric and
Neurocomputing Laboratories in the Department of Biomedical Engineering at Rut-
gers University. The chapters span a large variety of problems in signal and image
processing, using mainly neural networks for classification and template matching.
The inputs to the neural networks are features extracted from a signal or an image
by sophisticated and proven state-of-the-art techniques from the fields of digital
signal processing, computer vision, and image processing. In all examples and
problems examined, the biological equivalents are used as prototypes and/or simu-
lations of those systems were performed while systems that mimic the biological
functions are built.
Experimental and theoretical contributions are treated equally, and interchanges
between the two are examined. Technological advances depend on a deep under-
standing of their biological counterparts, which is why in our laboratories, experi-
ments on both animals and humans are performed continuously in order to test our
hypotheses in developing products that have technological applications.
The reasoning of most neural networks in their decision making cannot easily
be extracted upon the completion of training. However, due to the linearity of the
network nodes, the cluster prototypes of an unsupervised system can be reconstructed
to illustrate the reasoning of the system. In these applications, this analysis hints at
the usefulness of previously unused portions of the spectrum.
The book is divided into four parts. The first part contains chapters that introduce
the subjects of neural networks, classifiers, and feature extraction methods. Neural
networks are of the supervised type of learning. The second part deals with
unsupervised neural networks and fuzzy neural networks and their applications to
handwritten character recognition, as well as recognition of normal and abnormal
visual evoked potentials. The third part deals with advanced neural network archi-
tectures, such as modular designs and their applications to medicine and three-
dimensional neural networks architectures simulating brain functions. Finally, the
fourth part discusses general applications and simulations in various fields. Most
importantly, the establishment of a brain-to-computer link is discussed in some detail,
and the findings from these human experiments are analyzed in a new light.
All chapters have either been published in their final form or in a preliminary
form in conference proceedings and presentations. All co-authors to these papers
were mostly students of the editor. Extensive editing has been done so that repetitions
© 2000 by CRC Press LLC
of algorithms, unless modified, are avoided. Instead, where commonality exists, parts
have been placed into a new chapter (Chapter 4), and references to this chapter are
made throughout.
As is obvious from the number of names on the chapters, many students have
contributed to this compendium. I thank them from this position as well. Others
contributed in different ways. Mrs. Marge Melton helped with her expert typing of
parts of this book and with proofreading the manuscript. Mr. Steven Orbine helped
in more than one way, whenever expert help was needed. Dr. G. Kontaxakis,
Dr. P. Munoz, and Mr. Wei Lin helped with the manuscripts of Chapters 1 and 3.
Finally, to all the current students of my laboratories, for their patience while this
work was compiled, many thanks. I will be more visible—and demanding—now.
Dr. D. Irwin was instrumental in involving me in this book series, and I thank
him from this position as well. Ms. Nora Konopka I thank for her patience in waiting
and for reminding me of the deadlines, a job that was continued by Ms. Felicia
Shapiro and Ms. Mimi Williams. I thank them as well.
Evangelia Micheli-Tzanakou, Ph.D.
Department of Biomedical Engineering
Rutgers University
Piscataway, NJ
© 2000 by CRC Press LLC
Contributors
Ahmet Ademoglu, Ph.D. Timothy J. Dasey, Ph.D.
Assistant Professor MIT Lincoln Labs
Institute of Biomedical Engineering Weather Sensing Group
Bogazici University Lexington, MA
Bebek, Istanbul, Turkey
Cynthia Enderwick, M.S.
Sergey Aleynikov, M.S.
Hewlett Packard
IDT
Palo Alto, CA
Hackensack, NJ
Jeremy Bricker, Ph.D. Candidate Faiq A. Fazal, M.S.
Environmental Fluid Mechanics Lucent Technologies
Laboratory Murray Hill, NJ
Department of Civil and Environmental
Engineering
Stanford, CA Raymond Iezzi, M.D.
Kresge Institute
Tae-Soo Chon, Ph.D. Detroit, Michigan
Professor
Department of Biology
Francis Phan, M.S.
College of Natural Sciences
Harmonix Music Systems, Inc.
Pusan National University
Cambridge, MA
Pusan, Korea
Woogon Chung, Ph.D. Seth Wolpert, Ph.D.
Assistant Professor Associate Professor
Department of Control and Pennsylvania State University —
Instrumentation Harrisburg
Sung Kyun Kwan University Middletown, PA
Kyung Gi-Do, South Korea
Lt. Col. Timothy Cooley, Ph.D. Daniel Zahner, M.S.
USAF Academy Data Scope Co.
Department of Mathematical Sciences Paramus, NJ
Colorado Springs, CO
© 2000 by CRC Press LLC
Contents
Section I — Overviews of Neural Networks, Classifiers, and
Feature Extraction Methods—Supervised
Neural Networks
Chapter 1 Classifiers: An Overview
1.1 Introduction
1.2 Criteria for Optimal Classifier Design
1.3 Categorizing the Classifiers
1.3.1 Bayesian Optimal Classifiers
1.3.2 Exemplar Classifiers
1.3.3 Space Partition Methods
1.3.4 Neural Networks
1.4 Classifiers
1.4.1 Bayesian Classifiers
1.4.1.1 Minimum ECM Classifers
1.4.1.2 Multi-Class Optimal Classifiers
1.4.2 Bayesian Classifiers with Multivariate Normal Populations
1.4.2.1 Quadratic Discriminant Score
1.4.2.2 Linear Discriminant Score
1.4.2.3 Linear Discriminant Analysis and Classification
1.4.2.4 Equivalence of LDF to Minimum TPM Classifier
1.4.3 Learning Vector Quantizer (LVQ)
1.4.3.1 Competitive Learning
1.4.3.2 Self-Organizing Map
1.4.3.3 Learning Vector Quantization
1.4.4 Nearest Neighbor Rule
1.5 Neural Networks (NN)
1.5.1 Introduction
1.5.1.1 Artificial Neural Networks
1.5.1.2 Usage of Neural Networks
1.5.1.3 Other Neural Networks
1.5.2 Feed-Forward Neural Networks
1.5.3 Error Backpropagation
1.5.3.1 Madaline Rule III for Multilayer Network with
Sigmoid Function
1.5.3.2 A Comment on the Terminology ‘Backpropagation’
© 2000 by CRC Press LLC
1.5.3.3 Optimization Machines with Feed-Forward
Multilayer Perceptrons
1.5.3.4 Justification for Gradient Methods for Nonlinear
Function Approximation
1.5.3.5 Training Methods for Feed-Forward Networks
1.5.4 Issues in Neural Networks
1.5.4.1 Universal Approximation
1.5.5 Enhancing Convergence Rate and Generalization of an
Optimization Machine
1.5.5.1 Suggestions for Improving the Convergence
1.5.5.2 Quick Prop
1.5.5.3 Kullback-Leibler Distance
1.5.5.4 Weight Decay
1.5.5.5 Regression Methods for Classification Purposes
1.5.6 Two-Group Regression and Linear Discriminant Function
1.5.7 Multi-Response Regression and Flexible Discriminant Analysis
1.5.7.1 Powerful Nonparametric Regression Methods for
Classification Problems
1.5.8 Optimal Scoring (OS)
1.5.8.1 Partially Minimized ASR
1.5.9 Canonical Correlation Analysis
1.5.10 Linear Discriminant Analysis
1.5.10.1 LDA Revisited
1.5.11 Translation of Optimal Scoring Dimensions into
Discriminant Coordinates
1.5.12 Linear Discriminant Analysis via Optimal Scoring
1.5.12.1 LDA via OS
1.5.13 Flexible Discriminant Analysis by Optimal Scoring
1.6 Comparison of Experimental Results
1.7 System Performance Assessment
1.7.1 Classifier Evaluation
1.7.1.1 Hold-Out Method
1.7.1.2 K-Fold Cross-Validation
1.7.2 Bootstrapping Method for Estimation
1.7.2.1 Jackknife Estimation
1.7.2.2 Bootstrap Method
1.8 Analysis of Prediction Rates from Bootstrapping Assessment
References
Chapter 2 Artificial Neural Networks: Definitions, Methods, Applications
2.1 Introduction
2.2 Definitions
2.3 Training Algorithms
© 2000 by CRC Press LLC