Table Of ContentNew Learning Paradigms in Soft Computing
Studies in Fuzziness and Soft Computing
Editor-in-chief
Prof. Janusz Kacprzyk
Systems Research Institute
Polish Academy of Sciences
ul. Newelska 6
01-447 Warsaw, Poland
E-mail: [email protected]
http://www.springer.de/cgi-bin/search_book.pl ?series= 2941
Further volumes of this series can Vol. 73. Y. Yoshida (Ed.)
be found at our homepage. Dynamical A~pects in Fuzzy Decision, 2001
ISBN 3-7908-1397-4
Vol. 62. N. Baba and L. C. Jain (Eels.) Vol. 74. H.-N. Teodorescu, L. C. Jain and
Computational Intelligence in Games, 2001 A. Kandel (Eels.)
ISBN 3-7908-1348-6 Hardware lmplemellfation of lnrelligenr Systems,
2001
Vol. 63. 0. Castillo and P. Melin ISBN 3-7908-1399-0
Soft Computing for Control of Non-Linear
Dynamical Systems, 2001 Vol. 75. Y. Loia and S. Sessa (Eds.)
ISBN 3-7908-1349-4 Soft Computing Agents, 2001
ISBN 3-7908-1404-0
Vol. 64. I. Nishizaki and M. Sakawa
Fuzzy and Multiobjective Games for Conjlicr Vol. 76. D. Ruan, J. Kacprzyk and M. Fedrizzi (Eds.)
Re~·olution, 2001 Soft Computing for Risk EvaluaTion and
ISBN 3-7908-1341-9 Management. 2001
ISBN 3-7908-1406-7
Vol. 65. E. Or!owska and A. Szalas (Eels.)
Vol. 77. W. Liu
Relational Methods for Computer Science
Propositional, Probabilistic and Evidemial
Applications, 2001
Reasoning, 2001
ISBN 3-7908-1365-6
ISBN 3-7908-1414-8
Vol. 66. R. J. Howlett and L. C. Jain (Eds.) Vol. 78. U. Seiffen and L.C. Jain (Eds.)
Radial Basis Function Networks I, 200 I Self-Organizing Neural Nerworks, 2002
ISBN 3-7908-1367-2 ISBN 3-7908-1417-2
Vol. 67. R.J. Howlett and L.C. Jain (Eds.) Vol. 79. A. Osyczka
Radial Basis Function Networks 2, 2001 Evolutionary Algorithms for Single and Multicriter
ISBN 3-7908-1368-0 ia Design OpTimization, 2002
ISBN 3-7908-1418-0
Vol. 68. A. Kandel. M. Last and H. Bunke (Eds.)
Data Mining and Compwational Intelligence, 2001 Vol. 80. P. Wong, F. Aminzadeh and M. Nikravesh
ISBN 3-7908-1371-0 (Eds.)
Soft Compwing for Reservoir ClwmcTerization
Vol. 69. A. Piegat and Modeling, 2002
Fuzzy Modeling and Control, 2001 ISBN 3-7908-1421-0
ISBN 3-7908-1385-0
Vol. 81. V. Dimitrov and V. Korotkich (Eds.)
Vol. 70. W. Pedrycz (Ed.) Fuzzy Logic, 2002
Granular Computing, 2001 ISBN 3-7908-1425-3
ISBN 3-7908-1387-7
Vol. 82. Ch. Carlsson and R. Fuller
Vol. 71. K. Leiviska (Ed. . ) Fuzzy Reasoning in Decision Making and
Industrial ApplicaTions of SofT Computing, 2001 Optimization, 2002
ISBN 3-7908-1388-5 ISBN 3-7908-1428-8
Vol. 72. M. Mares Vol. 83. S. Barro and R. Marin (Eds.)
Fuzzy Cooperative Games, 2001 Fuzzy Logic in Medicine, 2002
ISBN 3-7908-1392-3 ISBN 3-7908-1429-6
Lakhmi C. Jain · Janusz Kacprzyk
Editors
New Learning Paradigms
in Soft Computing
With 140 Figures
and 44 Tables
Springer-Verlag Berlin Heidelberg GmbH
Professor Lakhmi C. Jain
University of South Australia
Knowledge-Based Intelligent
Engineering Systems Centre
Adelaide, Mawson Lakes
South Australia 5095
[email protected]
Professor Janusz Kacprzyk
Polish Academy of Sciences
Systems Research Institute
uI. Newelska 6
01-447 Warsaw
Poland
[email protected]
ISSN 1434-9922
ISBN 978-3-7908-2499-5 ISBN 978-3-7908-1803-1 (eBook)
DOI 10.1 007/978-3-7908-1803-1
Cataloging-in-Publication Data applied for
Die Deutsche Bibliothek - CIP-Einheitsaufnahme
New leaming paradigms in soft computing: with 47 tab les / Lakhmi C. Jain; Janusz Kacprzyk (ed.). -
Heidelberg; New York: Physica-Verl.. 2002
(Studies in fuzziness and soft computing; VoI. 84)
This work is subject to copyright. AII rights are reserved, whether the whole or part of thc material is
concerned, specifically the rights of translation, reprinling, reuse of illustrations, recitation, broadcasting,
reproduction on microfilm or in any other way, and storage in data banks. Duplication of this publication
or parts thereof is permitted only under the provisions of the German Copyright Law of September 9,
1965, in its current version, Viola-
tions are Iiable for prosecution undcr the German Copyright Law.
© Sprioger-Verlag Berlin Heidelberg 2002
Originally publisbed by Physica-Verlag Heidelberg in 2002
Softcover reprint oft he hardcover I st edition 2002
The use of general descriptive names, registered names, trademarks, etc. in this publication does not imply,
cven in the absence of a specitic statement, that such names are exempt from the relevant protective laws
and regulations and therefore free for general use.
Hardcover Design: Erich Kirchner, Heidelberg
Preface
Learning is a key issue in the analysis and design of all kinds of intelligent
systems. In recent time many new paradigms of automated (machine) learning
have been proposed in the literature. Soft computing, that has proved to be an
effective and efficient tool in so many areas of science and technology, seem to
offer new qualities in the realm of machine learning too. The purpose of this
volume is to present some new learning paradigms that have been triggered, or at
least strongly influenced by soft computing tools and techniques, mainly related to
neural networks, fuzzy logic, and evolutionary computations. Below, we will
briefly review the contents of the particular contributions included in this volume.
In "Statistical Learning by Natural Gradient Descent" by H.H. Yang and S. Amari,
single-layer and two-layer perceptrons are trained by natural gradient descent,
based on stochastic perceptron models and statistical inference. An efficient
scheme to represent the Fisher information matrix of a stochastic two-layer
perceptron is found. Based on this scheme, an algorithm to compute the natural
gradient is proposed. When the input dimension n is much larger than the number
of hidden neurons, the complexity of this algorithm is of order O(n). It is
confirmed by simulations that the natural gradient descent learning rule is not only
efficient but also robust.
In "Granular Networks and Granular Computing" by W. Pedrycz, fundamentals of
granular computing and its application to neural networks are concerned. Granular
computing, as the name itself stipulates, deals with representing information in the
form of some aggregates (embracing a number of individual entitites) and their
ensuing processing. The author elaborates on the rationale behind granular
computing. Next, a number of formal frameworks of information granulation are
discussed including several alternatives such as fuzzy sets, interval analysis, rough
sets, and probability. The notion of granularity itself is defined and quantified. A
design agenda of granular computing is formulated and the key design problems
are raised. A number of granular architectures are also discussed with an objective
of dealinating the fundamental algorithmic and conceptual challenges. It is shown
that the use of information granules of different size (granularity) lends itself to
general pyramid architectures of information processing. The role of encoding and
decoding mechanisms visible in this setting is also discussed in detail along with
some particular solutions. It is shown that the granularity of information introduces
VI
another dimension to neurocomputing, and the role of granular constructs in the
design of neural networks and knowledge representation therein is discussed.
In "Learning and Decision-Making in the Framework of Fuzzy Lattices" by V.G.
Kaburlassos and V. Petridis, a novel theoretical framework is delineated for
supervised and unsupervised learning. It is called a framework of fuzzy lattices, or
an FL-framework for short, and it suggests mathematically sound tools for dealing
separately and/or jointly with disparate types of data including vectors of numbers,
fuzzy sets, symbols, etc. Specific schemes are proposed for clustering and
classification that can deal with both missing and 'don't care' type data values; the
schemes in question can be implemented as neural networks. The proposed
learning schemes are employed for pattern recognition on seven data sets including
known benchmark sets, and the results are compared with those obtained by using
various learning techniques known from the literature. Finally, aiming at a mutual
cross-fertilization, the FL-framework is associated with established theories for
learning and/or decision-making including probability theory, fuzzy set theory,
Bayesian decision-making, theory of evidence, and adaptive resonance theory.
In "Lazy Learning: A Logical Method for Supervised Learning" by G. Bontempi,
M. Birattari and H. Bersini, first, the traditional approach to supervised learning
i.e, global modeling, that describes the relationship between the input and the
output with an analytical function over the whole input domain - is reviewed, and
shown that it has a nice property that even for huge data sets, a parametric model
can be stored in a small memory. Also, the evaluation of the parametric model
requires a short program that can be executed in a reduced amount of time. Since
the modeling of complex input/output relations often requires the adoption of
global nonlinear models, in recent years interest has grown in pursuing alternatives
to global modeling techniques. For instance, approaches based on the divide-and
conquer strategy have gained popularity in the research community. Instances of
the divide-and-conquer approach are local modeling techniques. These techniques
do not fit the whole data set but perform the prediction of the output for specific
test input values, also called queries. This chapter presents a set of novel local
modeling techniques, called lazy learning techniques, and their application to a
number of experimental problems. We show that, once a lazy learning perspective
is adopted, the use of conventional linear techniques in nonlinear settings is easy
and effective. In terms of applications, we show that promising areas of application
for these algorithms are regression modeling and data mining.
In "Active Learning in Neural Networks" by M. Hasenjliger and H. Ritter, a new
paradigm for supervised learning is discussed, called active learning, that aims at
improving the efficiency of neural network training procedures. The starting point
for active learning is the observation that the traditional approach of randomly
selecting training samples leads to large, highly redundant training sets. This
redundancy is not always desirable. Especially, if the acquisition of training data is
expensive, one is rather interested in small, informative training sets. Such training
VII
sets can be obtained if the Ieamer can select those training data that he or she
expects to be most informative. In this case, the Ieamer is no longer a passive
recipient of information but takes an active role in the selection of the training
data. Recent research on active learning is reviewed, and main approaches are
discussed. Experimental results to demonstrate the power of active learning are
shown.
In "Knowledge Extraction from Reinforcement Learning" by R. Sun, knowledge
extraction from reinforcement learners is concerned. Two approaches to
knowledge extraction: (I) the extraction of explicit, symbolic rules from neural
reinforcement learners, and (2) the extraction of complete plans from such
learners are dealt with. The advantages of such knowledge extraction include: (I)
the improvement of learning (especially with the rule extraction approach), and (2)
the improvement of the usability of results of learning.
In "Reinforcement Learning for Fuzzy Agents: Application to a Pighouse
Environment Control" by L. Jouffe, the fuzzy actor-critic learning and fuzzy Q
learning, two reinforcement learning methods based on dynamic programming, are
discussed. They are used to tune on line the conclusion part of fuzzy inference
systems. The only information available for learning is the system feedback which
describes in terms of reward and punishment the task the fuzzy agent has to
realize. At each time step, the agent receives a reinforcement signal according to
the last action it has performed in the previous state. The problem involves
optimizing not only the direct reinforcement but also the total amount of
reinforcements the agent can receive in the future. To illustrate the use of these
two learning methods, they are applied to a problem in which one looks for a fuzzy
controller to drive a boat from one bank to another, across a river with a strong
non-linear current. Then, the well-known cart-pole balancing and mountain-car
problems are discussed to compare the methods with other reinforcement learning
methods, and focus on important characteristic aspects of the fuzzy actor-critic
learning and fuzzy Q-learning. The experimental results show the superiority of
these methods with respect to the other related methods known from the literature.
Moreover, the method proposed makes it possible to learn every kind of
reinforcement learning problem (continuous states, discrete/continuous actions,
various type of reinforcement functions). Thanks to this flexibility, these learning
methods have been applied successfully in an industrial problem, to discover a
policy for pighouse environment control.
In "Performance Comparison of Neural Networks and Machine Learning
Techniques: A Cirtical Assessment of the Methodology" by A. Hoffmann, the
practice of empirical studies in neural network and machine learning research, that
has gained an increasing popularity in recent years, is scrutinized. A review of
increasingly sophisticated approaches to the statistical evaluation of experiments is
given. Then, this chapter presents empirical results suggesting that the achievable
statistical validity of many studies of the style done in the past is rather limited. A
VIII
study based on 13 popular data sets from the UCI Machine Learning repository is
shown, which demonstrates how careful one has to be when drawing conclusions
drawn from such empirical studies.
In "Digital Systems Design Through Learning" by W. Pedrycz and P.
Szczepaniak, the authors are concerned with the design and analysis of digital
systems using fuzzy neurocomputing. Some basic definitions of fuzzy-logic
neurons, fundamental neural architectures, learning strategies and interpretation of
their results are presented. The concept of embedding principle, i.e. due to which
an original Boolean problem is represented in the language of fuzzy sets,
afterwards solved through learning, and, finally, the result of learning re
interpreted in terms of two-valued logic, is presented. The use of fuzzy neurons in
a standard design of combinational systems is shown including the minimization of
incompletely specified Boolean functions (viz. those involving the 'don't care'
conditions), and of Boolean functions with many outputs. It is also claimed that
this approach supports reverse engineering in the sense that once the architecture
of the Boolean circuit has been learned, one can interpret the fuzzy logic network
in order to gain an insight into the nature of rules governing the data.
In "Hybrid Inductive Machine Learning: An Overview of CLIP Algorithms" by
K.J. Cios and L.A. Kurgan, inductive machine learning methods for generating
hypotheses about given training data are described. The discussion focuses on
hybrid algorithms that generate hypotheses in the form of production (if. .. then ... )
rules that constitute the model of the data. A machine learning process consists of:
(I) the learning phase in which the system analyzes the data and generates the rules
by fmding some similarities among the data, and (2) the validation phase in which
the generated rules must be verified by computing some performance evaluation
function on new set of data. Machine learning algorithms can be categorized in
several ways. Most importantly they are divided into supervised and unsupervised
algorithms. They can use inductive vs. deductive types of learning, incremental vs.
non-incremental learning modes, etc. Two basic techniques for inferring the
information from data are deduction and induction. Deduction infers information
that is a logical consequence of the information in the database. The deduction
technique can be used if the data describing some domain is proven to be correct.
Induction infers generalized information, or knowledge, by searching for
regularities among the data. Inductive learning produces results that are always
correct for the data but only plausible outside of the data. Learning by induction is
a search for a correct hypothesis/rule, or a set of them, which is guided by the
given examples. Majority of the machine learning algorithms are inductive.
In "An Integer Programming Approach to Inductive Learning Using Genetic and
Greedy Algorithms" by J. Kacprzyk and G. Szkatula, an improved inductive
learning method, IP2, is proposed to derive classification rules that correctly
describe (at least) most of the examples belonging to a class, and do not describe
(up to) most of the examples not belonging to this class. A pre-analysis of data is
IX
included that assigns higher weights to those values of attributes which occur more
often in the positive than in the negative examples. The inductive learning problem
is represented as a modification of the set covering problem that is solved by an
integer programming based algorithm using elements of a greedy algorithm or a
genetic algorithm. The results are very encouraging and are illustrated on thyroid
cancer and coronary heard disease problems.
In "Using Unlabeled Data for Learning Classification Problems" by A. Verikas,
A. Gelzinis and K. Malmquist, an approach of using unlabeled data for learning
classification problems is presented. The chapter consists of two parts. In the first
part, an approach of using both labeled and unlabeled data to train a multilayer
perceptron is shown. The approach banks on the assumption that regions of low
pattern density usually separate data classes. The unlabeled data are iteratively pre
processed by a perceptron being trained to obtain the soft class label estimates. It
is demonstrated that substantial gains in classification performance may be
achieved by using the approach when the labeled data do not adequately represent
the ,entire class distributions. In the second part, a quality function for learning
decision ·boundary between data clusters from unlabeled data is proposed. The
function is based on third order polynomials. The objective of the quality function
is to find a place in the input space sparse in data points. By maximizing the
quality function, a decision boundary between data clusters is found. A superiority
of the proposed quality function over the other similar functions as well as the
conventional clustering algorithms tested is confirmed in the experiments.
In "Problems of Rule Induction from Preterm Birth Data" by J.W. Grzymala
Busse, L.K. Goodwin, W.J. Grzymala-Busse, and X. Zheng, the problem of a
correct diagnosis of preterm birth is discussed. This is a complex and difficult
problem. The preterm birth data, in the experiments discussed, were collected at
the Duke University Medical Center. The data set is large, has many missing
attribute values, is unbalanced, many attributes are numerical, and the data set is
inconsistent. The main objective was to induce a rule set from the preterm birth
data with a high sensitivity as a primary criterion, and a high specificity as the
secondary criterion. Results of the experiments show that the original split for
training and testing data sets was biased, and that the many missing attribute values
significantly degrade prediction accuracy.
In "Reduction of Discriminant Rules Based on Frequent Item Set Calculation" by
M.C. Fernfmdez-Baizan, E. Menasalvas Ruiz and J.F. Martinez Sarrias, the
problem of reduction of the number of attributes to calculate rules in large
databases is concerned. This problem is of great interest in data mining. In this
paper, a method for reducing the number of attributes in rules using the frequent
item sets calculation is proposed. The method is based on a basic step model. In
the approach proposed, algorithms are divided into atomic operations called basic
steps so that it is easier to optimize their execution. The implementation of this
approach in Damisys is shown that demonstrates that the approach is
implementable and is effective for dealing with large data sets.
X
ln "Deriving a Concise Description of Non-Self Patterns in an Artificial Immune
System" by S.T. Wierzchon, the immune system is concerned. From a computer
science perspective, the immune system is a complex, self organizing and highly
distributed system that has no centralized control and uses learning and memory
when solving particular tasks. The learning process does not require negative
examples and the acquired knowledge is represented in explicit form. The main
actors of the immune systems are lymphocytes equipped with a set of receptors
recognizing intruders, or pathogens (i.e. viruses, bacteria, etc.). Because the
receptors on a surface of a single lymphocyte are of identical structure, and they
recognize onJy a narrow class of pathogens, they can be treated as a single receptor
from an abstract point of view. In this paper the focus is on a binary artificial
immune system in which all information is represented by the bit strings of a fixed
length. A receptor activates if it matches a bit string, and as the matching rule the
k-contiguous bits rule, i.e. two strings match if they have the same bits in at least k
contiguous positions, is employed. Under such a method of receptor activation, the
discriminative power of the receptors repertoire is studied. First, a method for
calculating the number of strings recognized by a single receptor is proposed, and
next it is investigated how this number decreases when the size of the repertoire
increases. Second, a method making it possible to determine the number of strings
that cannot be detected by an ,ideal" repertoire in the presence of a set of self
strings (i.e. strings representing normal behavior of a system) is given. The results
obtained are of importance when constructing an optimal receptors repertoire.
The editors wish to the thank the contributors for their original papers that should
be useful and helpful for a alrge audience. An excellent work of reviewers
deserves many thanks too. Finally, the kind support and cooperation of the
publishers, mainly Dr. Martina Bihn and Mrs. Gabriele Keidel, has made this
project much easier to run and complete.
July, 2001
Adelaide, Australia L.C. Jain
Warsaw, Poland J. Kacprzyk
