Table Of ContentTo appear in (cid:0) Proceedings of(cid:0) Applications of Soft Computing(cid:1) SPIE International Sym(cid:2)
posium on Optical Science(cid:1) Engineering and Instrumentation(cid:1) San Diego(cid:1)
(cid:3)(cid:4) July (cid:2) (cid:5) August (cid:5)(cid:6)(cid:6)(cid:4)(cid:7)invited paper(cid:8)
Incremental neuro(cid:0)fuzzy systems
B(cid:0) Fritzke
Systembiophysik(cid:1) Institut fu(cid:2)r Neuroinformatik
Ruhr(cid:3)Universita(cid:2)t Bochum(cid:1) Germany
ABSTRACT
Thepoorscalingbehaviorofgrid(cid:0)partitioningfuzzysystemsin caseof increasingdatadimensionality suggestsusing
fuzzy systems with a scatter(cid:0)partitionof the input space(cid:1) Jang has shown thatzero(cid:0)orderSugenofuzzy systems are
equivalenttoradialbasisfunctionnetworks(cid:2)RBFNs(cid:3)(cid:1) Methodsfor(cid:4)ndingscatterpartitionsforRBFNsareavailable(cid:5)
and it is possible to use them for creating scatter(cid:0)partitioning fuzzy systems(cid:1) A fundamental problem(cid:5) however(cid:5) is
the structure identi(cid:4)cation problem(cid:5) i(cid:1)e(cid:1)(cid:5) the determination of the number of fuzzy rules and their positions in the
input space(cid:1) The supervised growing neural gas method uses classi(cid:4)cation or regression error to guide insertions of
new RBF units(cid:1) This leads to a more e(cid:6)ective positioning of RBF units (cid:2)fuzzy rule IF(cid:0)parts(cid:5) resp(cid:1)(cid:3) than achievable
with the commonly used unsupervised clustering methods(cid:1) Example simulations of the new approach are shown
demonstratingsuperiorbehavior comparedwith grid(cid:0)partitioningfuzzy systems and the standardRBF approachof
Moody and Darken(cid:1)
Keywords(cid:0) RBFN(cid:5) fuzzy system(cid:5) incrementallearning(cid:5) normalization(cid:5) generalization
(cid:0)(cid:1) GRID(cid:2)PARTITIONING FUZZY SYSTEMS
In this introductorysectionwe remind of some fundamental propertiesof fuzzy systems(cid:1) Moreover(cid:5) we describethe
standardapproach of partitioningthe input space of a fuzzy system using a rectangulargrid(cid:1)
Fuzzy systems can model continuous input(cid:7)output relationships(cid:1) The purpose can be(cid:5) for example(cid:5) function
approximation (cid:2)regression(cid:3)(cid:5) non(cid:0)linear control(cid:5) or pattern classi(cid:4)cation(cid:1) In the latter case the output of a fuzzy
system can sometimesbe interpretedas posteriorclass probabilitieswhich can be used to achieve a classi(cid:4)cationby
means of the standardBayes rule(cid:1)
A basic component of a fuzzy system is a fuzzy rule(cid:1) Sometimes these rules are expressed using linguistic labels
such as the rule
IF (cid:2)pressureis low(cid:3) AND (cid:2)temperatureis medium(cid:3) THEN (cid:2)valve openingis (cid:8)(cid:1)(cid:9)(cid:3)(cid:0) (cid:2)(cid:10)(cid:3)
Fuzzymembership functions (cid:2)MFs(cid:3)associatelinguisticlabels(cid:2)e(cid:1)g(cid:1) low(cid:3)withaparticularareaofoneoftheinput
or outputvariables (cid:2)e(cid:1)g(cid:1) (cid:11)pressure(cid:12)(cid:3)(cid:1)
In the example shown above the THEN(cid:0)part of the rule does not consist of a membership variable but of the
(cid:11)crisp(cid:12)value(cid:8)(cid:1)(cid:9)(cid:1) Fuzzysystemsconsistingofthiskindofrulesarecalledzero(cid:0)order Sugeno fuzzysystems(cid:1) Inann(cid:0)th
order Sugeno fuzzy system the THEN(cid:0)part of each rule consists of a polynomial of degree n in the input variables(cid:1)
In this article we will concentrate on zero(cid:0)order Sugeno fuzzy systems since they have the interesting property of
being equivalent to radial basis functionnetworks (cid:2)RBFNs(cid:3) as is discussed in section (cid:13)(cid:1)
Di(cid:6)erent shapes of the MFs have been proposed such as triangular(cid:5) trapezoidal(cid:5) or Gaussian(cid:1) For the following
discussion we assume the MFs to have the form of a Gaussian
(cid:0)
(cid:2)ci(cid:0)xj(cid:3)
mi(cid:2)xj(cid:3)(cid:14)exp (cid:0) (cid:0) (cid:0)
(cid:15)(cid:1)i
(cid:0) (cid:1)
Thereby(cid:5) i denotes the index among the di(cid:6)erent MFs de(cid:4)ned for this variable(cid:1) The variables ci and (cid:1)i are the
centerandthewidthoftheGaussian(cid:5) respectively(cid:1) UsuallyonlyamoderatenumberofMFsisde(cid:4)nedforeachinput
variable (cid:2)see Fig(cid:1) (cid:10) for examples(cid:3)(cid:1)
Otherauthorinformation(cid:0) World(cid:1)Wide(cid:1)Web(cid:0) http(cid:0)(cid:2)(cid:2)www(cid:3)neuroinformatik(cid:3)ruhr(cid:1)uni(cid:1)bochum(cid:3)de(cid:2)ini(cid:2)PEOPLE(cid:2)fritzke(cid:2)top(cid:3)html
1 1
0.9 low low
0.8 medium 0.8 medium low
0.7 high medium
0.6 0.6
medium high
0.5
high
0.4 0.4
0.3
0.2 0.2
0.1
0 0
0 0.5 1 1.5 2 2.5 3 3.5 4 0 1 2 3 4 5 6
a(cid:3) pressureMF(cid:5) (cid:1) (cid:14)(cid:8)(cid:0)(cid:13) b(cid:3) temperatureMF(cid:5) (cid:1) (cid:14)(cid:8)(cid:0)(cid:15)
Figure (cid:1)(cid:2) Gaussian MFs(cid:1) Di(cid:6)erent values of (cid:1) lead to di(cid:6)erently strong overlap between neighboring MFs(cid:1) The
spacing b1etween MFs is set to (cid:10)(cid:1)(cid:8) in both examples(cid:1) 1
0.9 low low
0.8 medium 0.8 medium low
0.7 high medium
0.6 0.6
medium high
0.5
high
0.4 0.4
0.3
0.2 0.2
0.1
0 0
0 0.5 1 1.5 2 2.5 3 3.5 4 0 1 2 3 4 5 6
a(cid:3) pressureMF(cid:5) normalized b(cid:3) temperatureMF(cid:5) normalized
Figure (cid:3)(cid:2) Normalized Gaussian MFs (cid:2)all parameters as in Fig(cid:1) (cid:10)(cid:3)(cid:1) Due to the normalization the sum of all
MFs equals one at each position(cid:1) This leads (cid:2)in case of equal width parameters (cid:1)(cid:5) see text(cid:3) to sigmoidal shapes
for the (cid:11)outer(cid:12) MFs which now enclose all smaller or larger values(cid:5) respectively(cid:1) If the MFs overlap only slightly(cid:5)
normalization leads to a (cid:16)at plateau in the center of the MF which falls o(cid:6) steeply at the border to the next MF
(cid:2)b(cid:3)(cid:1) In the limiting case (cid:2)for (cid:1) (cid:1)(cid:8)(cid:3) the MFs get step bordersand in fact become crisp (cid:2)non(cid:0)fuzzy(cid:3)variables(cid:1)
Often it is desired that the membership values for one input variable add to unity everywhere(cid:1) This can be
achieved by dividing the membership values mi(cid:2)x(cid:3) by the sum of all membership values leading to normalized MFs
Mi(cid:2)x(cid:3)(cid:14)mi(cid:2)x(cid:3)(cid:2) mj(cid:2)x(cid:3)(cid:0)
j
X
Fig(cid:1) (cid:15) illustrates how normalization transforms the MFs shown in Fig(cid:1) (cid:10)(cid:1) The (cid:11)outer(cid:12)MFs now extend to very
negative and very positive values(cid:5) respectively(cid:1) For the pressure MF(cid:5) e(cid:1)g(cid:1)(cid:5) the linguistic labels low(cid:0) medium(cid:0) and
high are distinguished(cid:1) Interpreting all pressure values smaller than the center of the small MF also as small is
consistentwith everyday languageuse(cid:1)
One should note(cid:5) however(cid:5) that only for equal values (cid:1)i it is always the nearest normalized MF which has the
highestvalue(cid:1) If thevaluesofthe(cid:1)i di(cid:6)er(cid:5) thennormalizationleadstoexponentiallyfastdominanceofthebroadest
Gaussian when the input value leaves the convex hull of the Gaussian centers in any direction(cid:1) The mathematical
reason for this is that for any two Gaussians centered at c(cid:1) and c(cid:0) with width parameters (cid:1)(cid:1) and (cid:1)(cid:0) the following
statementholds (cid:2)as can be shown easily(cid:3)(cid:17)
(cid:1)
(cid:2)x(cid:2)c(cid:0)(cid:3)
exp (cid:0) (cid:0)(cid:0)(cid:0)(cid:1)
lim (cid:1) (cid:1) (cid:14)(cid:10) for (cid:1)(cid:1) (cid:3)(cid:1)(cid:0)
jxj(cid:0)(cid:1) (cid:2)x(cid:2)c(cid:0)(cid:3)(cid:2) (cid:3) (cid:2)x(cid:2)c(cid:1)(cid:3)
exp (cid:0) (cid:0)(cid:0)(cid:0)(cid:1) (cid:18)exp (cid:0) (cid:0)(cid:0)(cid:1)(cid:1)
(cid:2) (cid:3) (cid:2) (cid:3)
Thus(cid:5) in caseof di(cid:6)erently wide Gaussians and normalizationthe broadestGaussian dominates all other ones if the
input value x gets very large or very small(cid:1)
1
1
0.5
0.5
0
0
6
6
5
5
4
0 0.5 1 p1r.5essu2re2.5 3 3.5 4 0 1 2 3temperature 0 0.5 1 p1r.5essu2re2.5 3 3.5 4 0 1 2 3tem4perature
a(cid:3) IF(cid:0)part of fuzzy rule(cid:5) non(cid:0)normalized
b(cid:3) IF(cid:0)part of fuzzy rule(cid:5) normalized
Figure (cid:4)(cid:2) IF(cid:0)part(cid:2)orrule patch(cid:3) oflinguisticfuzzyrulein (cid:2)(cid:10)(cid:3)(cid:1) Forthenormalizationit isassumedthatall possible
(cid:19)(cid:2)(cid:20)(cid:14)(cid:10)(cid:20) fuzzy rule positions are actually occupiedin the fuzzy system(cid:1)
After the de(cid:4)nition of MFs one can formulate rules in terms of linguistic values(cid:1) In order to combine sub(cid:0)
expressions concerned with di(cid:6)erent input variables(cid:5) fuzzy operators such as fuzzy AND (cid:2)a(cid:1)k(cid:1)a(cid:1) T(cid:0)norm(cid:3) or fuzzy
OR (cid:2)a(cid:1)k(cid:1)a(cid:1) T(cid:0)conorm(cid:3) are used(cid:1)
We focus our attention on rules which combine their sub(cid:0)expressions by fuzzy AND(cid:1) In the case of a Gaussian
MF the fuzzy AND can be realized by the arithmetic product(cid:1) In this case the IF(cid:0)part of each fuzzy rule can be
describedby an n(cid:0)dimensional Gaussian(cid:17)
(cid:10) T (cid:2)(cid:1)
g(cid:2)x(cid:3)(cid:14)exp (cid:0) (cid:2)x(cid:0)c(cid:3) C (cid:2)x(cid:0)c(cid:3) (cid:2)(cid:15)(cid:3)
(cid:15)
(cid:4) (cid:5)
(cid:2)(cid:1) (cid:0) (cid:0) (cid:0)
Thereby(cid:5) x is the n(cid:0)dimensional input vector(cid:5) C (cid:14) diag(cid:2)(cid:10)(cid:2)(cid:1)(cid:1)(cid:4)(cid:10)(cid:2)(cid:1)(cid:0)(cid:4)(cid:0)(cid:0)(cid:0)(cid:4)(cid:10)(cid:2)(cid:1)n(cid:3) is the inverse of the covari(cid:0)
(cid:0)
ance matrix C corresponding to the product of n one(cid:0)dimensional Gaussians with variances (cid:1)i(cid:1) The vector c (cid:14)
T
(cid:2)c(cid:1)(cid:4)(cid:0)(cid:0)(cid:0)(cid:4)cn(cid:3) is the center of the n(cid:0)dimensional product Gaussian and has the corresponding centers of the one(cid:0)
dimensional factorGaussians as components(cid:1)
As performed for the one(cid:0)dimensional MFs(cid:5) also the IF(cid:0)parts of all fuzzy rules in the fuzzy system can be
normalized(cid:1) If the fuzzy system consists of N rules(cid:5) then the normalized Gaussians can be written as
N
Gi(cid:2)x(cid:3)(cid:14)gi(cid:2)x(cid:3)(cid:2) gj(cid:2)x(cid:3)(cid:0) (cid:2)(cid:19)(cid:3)
j(cid:4)(cid:1)
X
The normalizationensures that for each input signal x the sum of all rule activations adds up to unity(cid:17)
N
Gj(cid:2)x(cid:3)(cid:14)(cid:10)
j(cid:4)(cid:1)
X
Eachrule is considerablyactivatedonly in a particularareaof theinput space(cid:1) Onecanthink oftheinput space
as being covered by small patches(cid:5) each of them corresponding to one fuzzy rule(cid:1) We will denote the IF(cid:0)parts of
fuzzy rules as rule patches(cid:1) In Fig(cid:1) (cid:19) the rule patch of the example fuzzy rule (cid:2)(cid:10)(cid:3) is shown depending on whether
normalization takes place or not(cid:1)
In general(cid:5) a fuzzy system can have a multi(cid:0)dimensional output(cid:1) The i(cid:0)th output component of a fuzzy system
without normalization is given as
N
Oi(cid:2)x(cid:3)(cid:14) wijgj(cid:2)x(cid:3)(cid:0)
j(cid:4)(cid:5)
X
Thereby(cid:5) wij is the THEN(cid:0)part (cid:2)or outputweight(cid:3) of the fuzzy rule j with respectto the output componenti(cid:1) The
extra term g(cid:5)(cid:2)x(cid:3) is constantlyset to (cid:10)(cid:1)(cid:8) and its associatedoutputweight wi(cid:5) usually codesthe average value of the
componenti(cid:5) so that the otherweights only have to encodedeviations from this average value(cid:1) It is also possible to
leave out the bias term in which case the remaining terms have to encodethe completeinput(cid:7)outputmapping(cid:1)
If normalized n(cid:0)dimensional Gaussians are used(cid:5) the i(cid:0)th output componentof the fuzzy system is
N N N
Oi(cid:2)x(cid:3)(cid:14) wijgj(cid:2)x(cid:3) gj(cid:2)x(cid:3)(cid:14) wijGj(cid:2)x(cid:3)(cid:0)
j(cid:4)(cid:1) (cid:6)j(cid:4)(cid:1) j(cid:4)(cid:1)
X X X
In this case no bias unit is neededand the output is a weighted average of the output weights(cid:1)
The outputweights wij can be set manually by domain experts(cid:1) Alternatively(cid:5) a given training data set
i i i n i m
D (cid:14)(cid:2)(cid:0) (cid:4)(cid:1) (cid:3)(cid:4) i(cid:14)(cid:10)(cid:4)(cid:0)(cid:0)(cid:0)(cid:4)M(cid:4) (cid:0) (cid:3)R (cid:4)(cid:1) (cid:3)R
T
couldbeusedtosetthemappropriately(cid:1) Theoutputvectors(cid:1) (cid:14)(cid:2)(cid:5)(cid:1)(cid:4)(cid:0)(cid:0)(cid:0)(cid:4)(cid:5)m(cid:3) have di(cid:6)erentmeaningdependingon
the general kind of the problem(cid:1) For classi(cid:4)cation problems the outputvector often contains a (cid:10)(cid:0)out(cid:0)of(cid:0)m encoding
of the class associated with an input data point (cid:0)(cid:1) In this case it usually consists of a single (cid:21)(cid:10)(cid:21) and (cid:2)m(cid:0)(cid:10)(cid:3) (cid:21)(cid:8)(cid:21)s(cid:1)
For regression problems (cid:1) contains the desired m(cid:0)dimensional continuous output(cid:1) A common goal in both cases is
to (cid:4)nd output weights that minimize the summed squared error
M m
(cid:10) i i (cid:0)
E (cid:14) (cid:2)Oj(cid:2)(cid:0) (cid:3)(cid:0)(cid:5)j(cid:3) (cid:0) (cid:2)(cid:13)(cid:3)
(cid:15)
i(cid:4)(cid:1)j(cid:4)(cid:1)
XX
If the IF(cid:0)parts of the fuzzy rules (cid:2)the Gaussian centers and covariance matrices(cid:3) are (cid:4)xed(cid:5) the determination of
suitable weights wij reduces to a linear least squares problem which can be readily solved by standard matrix
(cid:1)
techniques like the singular value decomposition(cid:1)
(cid:3)(cid:1) CRITIQUE OF GRID(cid:2)PARTITIONING APPROACH
The grid(cid:0)partitioning approach to fuzzy systems has the advantage that all fuzzy rules can be formulated with a
relatively small number of linguistic labels(cid:1) Theselabels are heavily re(cid:0)used in di(cid:6)erent rules(cid:1) If each input variable
n
is characterizedby k MFs andif theinputis n(cid:0)dimensional(cid:5) thentherearek possiblefuzzyIF(cid:0)partsbutonly k(cid:2)n
di(cid:6)erent linguistic labels are neededto generatethem(cid:1)
There is(cid:5) however(cid:5) a serious disadvantage of the grid(cid:0)partitioning approach(cid:17) the very regular partition of the
input space may be unable to producea rule set of acceptablesize which is able to handle a given data set well(cid:1) If(cid:5)
for example(cid:5) the data contains regions with several small clusters of di(cid:6)erent classes(cid:5) then small rule patches have
to be created to classify the data in this region correctly(cid:1) Due to the grid partition(cid:5) however(cid:5) the (cid:4)ne resolution
neededinthisparticularareaispropagatedalsotoareaswhicharemucheasiertohandle(cid:5)perhapsbecausetheyonly
contain data belonging to a single class(cid:1) The grid(cid:0)partitioning approach enforces a large number of small identical
rule patches(cid:5) although one large patch would theoretically be able to classify the data in this region correctly(cid:1) The
problem is illustrated in Fig(cid:1) (cid:20)(cid:1) It becomeseven more serious as the dimension of the input data increases(cid:1)
The (cid:4)ne partition needed for some problems requires the speci(cid:4)cationof correspondinglymany linguistic labels(cid:5)
too(cid:1) There is a limit of understandability achieved at some point(cid:5) e(cid:1)g(cid:1) when more then (cid:10)(cid:15) linguistic labels have
to be de(cid:4)ned for one class(cid:1) Eventually one might be forced to number the labels which would make it completely
impossible to describethe fuzzy system in rules which are intuitively comprehensible(cid:1)
(cid:4)(cid:1) SCATTER(cid:2)PARTITIONING FUZZY SYSTEMS
To eliminatetheproblemsassociatedwithgrid(cid:0)partitioning(cid:5)otherways ofdividing theinputspaceinto rulepatches
have been proposed(cid:1) One approach(cid:5) known as scatter partitioning(cid:5) is to allow the IF(cid:0)parts of the fuzzy rules to
be positioned at arbitrary locations in input space(cid:1) If the rules are represented by n(cid:0)dimensional Gaussians (cid:2)(cid:15)(cid:3) or
normalized Gaussians (cid:2)(cid:19)(cid:3)(cid:5) this means that the centers of the Gaussians are not anymore con(cid:4)ned to corners of a
rectangular grid(cid:1) Rather(cid:5) they can be chosen freely(cid:5) e(cid:1)g(cid:5) by a clustering algorithm working on the training data(cid:1)
If the covariance matrices Ci of the Gaussians are diagonal(cid:5) then they can still be thought of being generated as
a(cid:3) c(cid:3) e(cid:3)
b(cid:3) d(cid:3) f(cid:3)
Figure (cid:5)(cid:2) Classi(cid:4)cation with a grid(cid:0)partitioning fuzzy system(cid:1) a(cid:3) Two dimensional data set in the unit square
consisting of two classes A (cid:2)white(cid:3) and B (cid:2)black(cid:3)(cid:1) b(cid:3) Fuzzy system with (cid:10)(cid:15) rules arranged in a (cid:13)(cid:2)(cid:19) grid(cid:1) c(cid:3) Non(cid:0)
normalized fuzzy system(cid:17) estimated a posteriori probability that the data belongs to class A(cid:1) The box shown is the
scaled unit cube(cid:1) One can note that some output values are higher than one thereby violating the properties of a
probability(cid:1) Inregionswithnotrainingdatathefuzzysystemoutputfallso(cid:6)toanon(cid:0)zeroplateauwhichcorresponds
to the bias(cid:1) Since the training data contains more exemplars from class B(cid:5) the bias has a value smaller than (cid:8)(cid:1)(cid:20)(cid:1)
d(cid:3) Normalized fuzzy system(cid:17) estimated a posteriori probability that the data belongs to class A(cid:1) e(cid:3) Classi(cid:4)cation
according to the non(cid:0)normalized fuzzy system(cid:1) The system is unable to classify all training examples correctly(cid:1) f(cid:3)
Classi(cid:4)cation according to the normalized fuzzy system(cid:1) Also here some training examples are misclassi(cid:4)ed(cid:1) The
probable reason for the misclassi(cid:4)cations is the prede(cid:4)ned resolution of the fuzzy rules which is slightly too coarse
for the given data set(cid:1)
a product of n one(cid:0)dimensional Gaussian MFs(cid:1) Each MF(cid:5) however(cid:5) is now only used by a single rule so that the
numberofdi(cid:6)erentMFsforoneinputvariablemaybeashighasthetotalnumberoffuzzyrules(cid:1) Therefore(cid:5)assigning
meaningful linguistic labels to theseMFs is impossible in all but trivial cases(cid:1)
A problem to be solved with scatter partitions is to (cid:4)nd a suitable number of rules and suitable positions and
width of therule patchesin input space(cid:1) In this contextit is helpful to note(cid:5) that theoverall architectureof a fuzzy
system with Gaussian MFs and scalar THEN(cid:0)parts of the fuzzy rules is equivalent to an RBFN as has been noted
(cid:0)
by Jang(cid:1) We recapitulatethe(cid:4)ndingsof Jang shortlyand proceedthenwith methodstodeterminetheparameters
of a scatter(cid:0)partitioningfuzzy system(cid:1)
a(cid:3) c(cid:3) e(cid:3)
b(cid:3) d(cid:3) f(cid:3)
Figure (cid:6)(cid:2) Classi(cid:4)cation with a grid(cid:0)partitioning fuzzy system(cid:1) a(cid:3) Two dimensional data set in the unit square
consisting of two classes A (cid:2)white(cid:3) and B (cid:2)black(cid:3)(cid:1) b(cid:3) Fuzzy system with (cid:22)(cid:13) rules arrangedin an (cid:23)(cid:2)(cid:23) grid(cid:1) c(cid:3) Non(cid:0)
normalized fuzzy system(cid:17) estimated a posteriori probability that the data belongs to class A(cid:1) d(cid:3) Normalized fuzzy
system(cid:17) estimated a posteriori probability that the data belongs to class A(cid:1) e(cid:3) Classi(cid:4)cation according to the non(cid:0)
normalized fuzzy system(cid:1) f(cid:3) Classi(cid:4)cation accordingto the normalized fuzzy system(cid:1)
(cid:5)(cid:1) EQUIVALENCE OF FUZZY SYSTEMS AND RADIAL BASIS FUNCTION
NETWORKS
(cid:0)
Jang showed that under certain mild restrictions a fuzzy inference is equivalent to an RBFN(cid:1) The following must
be ful(cid:4)lled for the equivalence to be valid(cid:17)
(cid:4) The number of RBF units is equal to the number of fuzzy IF(cid:0)THEN(cid:0)rules(cid:1)
(cid:4) The output of each fuzzy rule is a constant (cid:2)thefuzzy system is a zero(cid:0)orderSugeno fuzzy system(cid:3)(cid:1)
(cid:4) The MFs within each rule are chosen as Gaussian functions with the same variance(cid:1)
(cid:4) The T(cid:0)norm operatorused to computethe activation of each rule is multiplication(cid:1)
(cid:4) BoththeRBFNandthefuzzyinferencesystemunderconsiderationusethesamemethodtoderivetheiroverall
output(cid:5) i(cid:1)e(cid:1) weighted average (cid:2)with normalization(cid:3)or weighted sum (cid:2)withoutnormalization(cid:3)(cid:1)
The generalization from strictly radial basis functions to ones with a diagonal covariance matrix with possibly
(cid:0)
di(cid:6)erent elements in the diagonal is straightforward(cid:1) In this case the squared Euclidean distance kx(cid:0)ck used for
T (cid:2)(cid:1)
computing the Gaussian activations must be replaced by the Mahalanobis distance (cid:2)x(cid:0)c(cid:3) C (cid:2)x(cid:0)c(cid:3) using the
inverse of the covariancematrix C of the respectiveGaussian unit(cid:1)
a(cid:3) b(cid:3) c(cid:3)
Figure (cid:7)(cid:2) Classi(cid:4)cation with a scatter(cid:0)partitioningfuzzy system (cid:2)with normalized MFs(cid:3) constructedby an RBFN
accordingto Moodyand Darken(cid:1) The training data set is that of Fig(cid:1) (cid:13)a(cid:1) a(cid:3) Fuzzy system with (cid:15)(cid:8) rules positioned
with the LBG clustering algorithm(cid:1) b(cid:3) Estimated a posteriori probability that the data belongs to class A(cid:1) c(cid:3)
Classi(cid:4)cation result(cid:1) Since classes in the training data seem to have very little overlap(cid:5) it would seem reasonableto
require that the fuzzy system maps nearly all training data points to the correctclass(cid:1) However(cid:5) the distributionof
rulecentersfoundby LBG containsonly (cid:19)rulesin theupperrightcorneroftheshownpartof theinputspaceand(cid:13)
would benecessaryin this case(cid:1) SinceLBG is initialized at randomfrom the dataset(cid:5) onecan expecta distribution
of the centers proportional to the probability density(cid:1) Considering this(cid:5) the fraction of (cid:19)(cid:2)(cid:15)(cid:8) (cid:14) (cid:8)(cid:0)(cid:10)(cid:20) positioned in
the upper right is already more than the value one can expect looking at the given data(cid:5) namely (cid:13)(cid:2)(cid:19)(cid:22)(cid:14) (cid:8)(cid:0)(cid:10)(cid:10)(cid:1) The
underlying problem is(cid:5) that LBG does not take the desired outputsfor the data into account(cid:1)
Keeping the equivalence between Sugeno fuzzy systems and RBFNs in mind we now discuss a standard RBFN
training method(cid:1) Thereafter(cid:5) we describe an incremental RBFN which(cid:5) according to Jang(cid:21)s results(cid:5) is also an
incremental neuro(cid:0)fuzzysystem(cid:1)
(cid:6)(cid:1) RADIAL BASIS FUNCTION NETWORKS ACCORDING TO MOODY AND
DARKEN
(cid:6)
Moody and Darken have proposed a multi(cid:0)phase approach to RBFNs(cid:1) First(cid:5) a pre(cid:0)de(cid:4)ned number of centers is
(cid:7) (cid:8)
distributedininputspacewithaclustermethod(cid:2)e(cid:1)g(cid:1)(cid:5)LBG ork(cid:0)means (cid:3)(cid:1) Thewidthparameters(cid:1)oftheGaussians
aresetby alocalheuristic(cid:5)e(cid:1)g(cid:1)(cid:5) settingthe(cid:1) ofeachunitequaltothedistancetothenearestotherunits(cid:1) Moreover(cid:5)
Moodyand Darken proposeto use normalized activationsaccordingto (cid:2)(cid:19)(cid:3)(cid:1) In terms of fuzzy systems thesteps just
describedcorrespondto the identi(cid:4)cationof the IF(cid:0)parts of a Sugeno fuzzy rule(cid:1)
The THEN(cid:0)partsof the fuzzy rules or(cid:5) alternatively(cid:5) the output weights of the RBFN(cid:5) are set by pseudo(cid:0)inverse
computationsuchthatthesummedsquareerror(cid:2)(cid:13)(cid:3)foragiventrainingdatasetisminimized(cid:1) Itisalsopossible(cid:5)but
(cid:9)
hasusuallynoadvantage(cid:5)tocomputetheoutputweightsiterativelythroughgradientdescentontheerrorfunction(cid:1)
Thismulti(cid:0)phaseapproachisstraight(cid:0)forwardandisoftenreportedtobemuchfasterthan(cid:5)e(cid:1)g(cid:1)(cid:5)thebackpropaga(cid:0)
tiontrainingofmulti(cid:0)layerperceptronsforthesamedata(cid:1) Apossibleproblemoftheapproach(cid:5)whichhasforexample
(cid:10)
beennotedby Bishop(cid:5) is thatthe clusteringis completelyunsupervisedand doesnottake thegiven desiredoutput
information(cid:2)class labels or continuousoutputvalues(cid:3)into account(cid:1) Clusteringmethodsusually try to minimize the
mean distance between the centers they distribute and the given data (cid:2)which is only the input part of the training
data(cid:3)(cid:1) This error(cid:5) however(cid:5) is of little relevance to many supervised learning problems(cid:1) The resulting distribution
of RBF centers(cid:2)or rule patches(cid:3)may(cid:5) therefore(cid:5)be poorfor the classi(cid:4)cation or regressionproblem at hand(cid:1) Fig(cid:1) (cid:22)
shows an example where this is the case(cid:1)
Ifonevisuallyanalyzesthegeneratedneuro(cid:0)fuzzysysteminFig(cid:1)(cid:22)(cid:5)itbecomesobviousthatatseveralplacesthere
are rules covering neighboringareasand having basically the same output(cid:1) Such rules could becombined into fewer
rules each covering a correspondingly larger area of the input space(cid:1) This would set free resources which could be
used in places where thesystem can bene(cid:4)tfrom them more(cid:5) for example in the upperright cornerof the displayed
part of the input space(cid:1)
Instead of (cid:4)rst constructing a possibly poor neuro(cid:0)fuzzy system and then improving it later on(cid:5) it had some
advantagesif onecould immediatelybuild a goodsystemfor theproblemathand(cid:1) This is thegoalof themethodin
the following section(cid:1)
(cid:7)(cid:1) CREATING SCATTER(cid:2)PARTITIONING FUZZY SYSTEMS WITH THE
SUPERVISED GROWING NEURAL GAS MODEL
The approach to fuzzy systems described in this section has its origin in a neural network model from the area of
(cid:11)
competitivelearning(cid:5)theso(cid:0)calledunsupervisedgrowingneuralgasmodel(cid:1) Thismethodcanbedescribedasfollows
(cid:2)for a broader discussion see the referencegiven(cid:3)(cid:17)
(cid:4) Itisavectorquantizationmethodwhichaimsatdistributingalimitednumberofcenterswiinthen(cid:0)dimensional
input space(cid:1) The goal is to minimize the mean quantization error for a given data set
(cid:1) M i n
DU (cid:14)(cid:2)(cid:0) (cid:4)(cid:0)(cid:0)(cid:0)(cid:4)(cid:0) (cid:3)(cid:4)(cid:0) (cid:3)R (cid:0)
(cid:4) Centers are adapted on(cid:0)line by soft competitive learning(cid:1) In particular(cid:5) for each presented input signal the
winner s (cid:2)the center nearestto the input signal(cid:3) is moved towards the input signal(cid:17)
ws (cid:17)(cid:14)ws(cid:18)(cid:6)(cid:2)(cid:0)(cid:0)ws(cid:3)
To a smaller degree also the direct neighbors within the network topology (cid:2)see below(cid:3) are moved towards the
input signal(cid:1)
(cid:4) Ateachadaptationstepthesquareddistancebetweencurrentinputsignalandwinnerisaddedtoalocalerror
variable of the winner(cid:17)
(cid:0)
Es (cid:17)(cid:14)Es(cid:18)k(cid:0)(cid:0)wsk (cid:2)(cid:20)(cid:3)
All error variables undergo slow exponential decay to eventually forget earlier accumulatederrors(cid:1)
(cid:4) Among the centersa topology consisting of neighborhoodedges is createdthrough competitive Hebbian learn(cid:0)
(cid:12)
ing(cid:1) The principle of this method is to always insert an edge between the winner and the second nearest
center for the current input signal(cid:1) If such an edge does already exist(cid:5) its age parameter is set to zero(cid:1) Edges
with an age larger than a maximum value are removed again(cid:1) Competitive Hebbian learning has been shown
to generate a subgraph of the Voronoi tessellation corresponding to the current set of centers(cid:1) The Voronoi
(cid:1)(cid:5)
tessellation(cid:5) however(cid:5) is among all triangulationsthe best for functioninterpolation(cid:1)
(cid:4) A new center is always inserted after a (cid:4)xed numbers of input signals has been processed(cid:1) It is positioned at
the center of one of the edges emanatingfrom the centerwith the largest accumulatederror(cid:1) The error values
are reducedand are redistributedlocally after each insertion (cid:2)for details see Ref(cid:1) (cid:23)(cid:3)(cid:1)
This strategyleads to a growth processwhich reducesthe mean distortionerrore(cid:6)ectively since new centersare
inserted where a large distortion error has occurred in the past(cid:1) At the same time a topology is created describing
the sub(cid:0)area of input space where the input data is located(cid:1) Fig(cid:1) (cid:9) shows an example simulation(cid:1)
The growth principle of the unsupervised growing neural gas model can be applied in a very similar fashion
(cid:1)(cid:1)(cid:13)(cid:1)(cid:6)
to supervised learning(cid:1) In particular(cid:5) growing RBFNs have been investigated in the past(cid:1) The principle is to
associateoneRBF unit witheach centerand toaccumulatetheclassi(cid:1)cation error or theregression error insteadof
the quantization error(cid:1) Therefore(cid:5) equation (cid:20) is replaced by
(cid:0)
Es (cid:17)(cid:14)Es(cid:18)(cid:2)O(cid:2)(cid:0)(cid:3)(cid:0)(cid:1)(cid:3) (cid:0)
Thereby(cid:5) O(cid:2)(cid:0)(cid:3) is the m(cid:0)dimensional outputvector of the network and (cid:1) is the m(cid:0)dimensional desired output for the
currentinput vector (cid:0)(cid:1)
a(cid:3) (cid:8) signals b(cid:3) (cid:22)(cid:8)(cid:8) signals c(cid:3) (cid:10)(cid:23)(cid:8)(cid:8) signals
d(cid:3) (cid:20)(cid:8)(cid:8)(cid:8) signals e(cid:3) (cid:10)(cid:20)(cid:8)(cid:8)(cid:8)signals f(cid:3) (cid:15)(cid:8)(cid:8)(cid:8)(cid:8) signals
Figure (cid:8)(cid:2) Theunsupervisedgrowingneuralgasmodeladaptstoasignaldistributionwhichhasdi(cid:6)erentdimension(cid:0)
alities in di(cid:6)erent areas of the input space(cid:1) Shown are intermediate stages after the indicated number of processed
input signals(cid:1) The last network shown is not necessarilythe (cid:4)nal one since the growth processcould in principle be
continued inde(cid:4)nitely(cid:1) (cid:2)from Ref(cid:1) (cid:23)(cid:3)
Interpretedverbally(cid:5) theclassi(cid:4)cationorregressionerrorforthecurrentinputsignalisaccumulatedatthenearest
Gaussian(cid:1) Afterawhile(cid:5)thoseGaussianscanbeidenti(cid:4)edwhichareunabletocorrectlymapthedataintheirvicinity(cid:1)
Inserting new Gaussians nearby increasesthe chance that this data is correctlymapped in the future(cid:1)
Further details of the methodare(cid:17)
(cid:9)
(cid:4) The output weights are trained on(cid:0)line with the delta(cid:0)rule(cid:1) Due to the on(cid:0)line nature of the growing neural
gasmodelthisismoreappropriate(cid:2)computationallylesscomplex(cid:3)thantorepeatedlysolvethecompletelinear
system with matrix techniques(cid:1) At the end of the growth process(cid:5) however(cid:5) it is advantageousto computethe
optimal weights with least squares methods(cid:5) once(cid:1)
(cid:4) Thewidthparameter(cid:1)i ofeachGaussianissettoafraction(cid:2)e(cid:1)g(cid:1) (cid:8)(cid:1)(cid:20)(cid:3)ofthemeanlengthofalledgesemanating
from this unit(cid:1) This leads to a similar relative overlap of neighboringGaussians(cid:1)
(cid:4) The growth process has to be terminated by a suitable criterion to prevent over(cid:0)(cid:4)tting(cid:1) One possibility is to
split the available data into training and validation data(cid:1) Only the training data is used to determine the
network parameters(cid:1) In parallel(cid:5) however(cid:5) the performanceon the validation data is observed and the growth
processis stoppedwhen this measure does not improve anymore(cid:1)
In Fig(cid:1) (cid:23) the results of the describedmethodare shown for the datafrom Fig(cid:1) (cid:13)a(cid:1) It can be seen thatthe insertions
based on accumulated classi(cid:4)cation error led to a small network (cid:2)fuzzy system(cid:5) resp(cid:1)(cid:3) which classi(cid:4)es the training
data well(cid:1)
Theproposedmethodis equallywell applicabletoregressionproblems(cid:1) Fig(cid:1) (cid:24)showsadatasetwhichstemsfrom
sampling a Gabor wavelet in a square region of the input space(cid:1) In Fig(cid:1) (cid:10)(cid:8) the results for grid(cid:0)partitioning fuzzy
systems and scatterpartitioning fuzzy systems constructedwith a supervised growing neural gas are shown(cid:1) Fuzzy
systemsofthreedi(cid:6)erentsizeswerecreated(cid:1) Theapproximationerrorfortheincrementallygeneratedfuzzysystems
was consistently lower(cid:1) The obvious reason for this is the non(cid:0)uniform distribution of rules found by the growing
neuralgasnetworkwhich matchesthenon(cid:0)linearityin thetrainingdatawell(cid:1) One shouldnotethatthedatadensity
in the example is perfectly uniform so that any reasonable clustering method (cid:2)like LBG(cid:3) would have distributed a
given number of rules in a uniform fashion(cid:5) too(cid:1) Accordingly(cid:5) for this problem the results of Moody and Darken(cid:21)s
approachare very similar to those of a grid(cid:0)partitioningfuzzy system and are not displayed separately(cid:1)
a(cid:3) b(cid:3) c(cid:3)
Figure (cid:9)(cid:2) Classi(cid:4)cation with a scatter(cid:0)partitioning fuzzy system (cid:2)with normalized MFs(cid:3) constructed by the su(cid:0)
pervised growing neural gas approach according to the author(cid:1) The training data set is that of Fig(cid:1) (cid:13)a(cid:1) a(cid:3) Fuzzy
system with (cid:10)(cid:22) rules positionedby the classi(cid:4)cation(cid:0)errordriven growing neural gasinsertionscheme(cid:1) b(cid:3) Estimated
a posteriori probabilitythatthedatabelongstoclassA(cid:1)c(cid:3)Classi(cid:4)cationresult(cid:1) Eventhoughthesystemhasonly(cid:10)(cid:22)
rules(cid:5) it handles the region in the upperright well(cid:1) One can comparethis with the fuzzy system in Fig(cid:1) (cid:22) which has
(cid:15)(cid:8) rules but is unable to map all data in the mentioned region correctly(cid:1) As was discussed in the caption of Fig(cid:1) (cid:22)(cid:5)
thefailureof MoodyandDarken(cid:21)sapproachin thatsimulation is noexception(cid:1) Rather(cid:5)thismust beexpectedgiven
the data distribution and the random initialization of the rule centers(cid:1)
a(cid:3) data distribution in input space b(cid:3) data in combined input(cid:7)outputspace
Figure (cid:10)(cid:2) Sampled Gabor wavelet as regressiontraining data set(cid:1) (cid:20)(cid:8)(cid:2)(cid:20)(cid:8)(cid:14)(cid:15)(cid:20)(cid:8)(cid:8) points were generated(cid:1)
(cid:8)(cid:1) DISCUSSION AND OUTLOOK
In this articlewe (cid:4)rsttriedto illustratesomeproblems associatedwith thetraditionalgrid(cid:0)partitioningapproachto
fuzzy systems(cid:1) Doing this(cid:5) we also discussed some aspectsrelatedto normalization of MFs(cid:1)
Scatter(cid:0)partitioningfuzzy systems are a possible way to overcome the outlined disadvantages(cid:5) in particular the
poor scaling behavior with increasing input dimensionality(cid:1) The well(cid:0)known equivalence of zero(cid:0)order Sugeno fuzzy
systems and RBFNs suggests to use methods developed in the context of RBFNs to construct scatter(cid:0)partitioning
fuzzysystems(cid:1) ThestandardmethodofMoodyandDarkenhasthedisadvantagethatitusesunsupervisedclustering
methods to position the RBF units (cid:2)fuzzy rule IF(cid:0)parts(cid:5) resp(cid:1)(cid:3)(cid:1) This tends to minimize the quantization error for
the given input data(cid:5) but the resulting network con(cid:4)gurationscan be poor for supervised learning(cid:1)
The supervised growing neural gas method is able to incrementally build up a network structure during the
training process(cid:1) Since it positions new units on the basis of accumulated classi(cid:4)cation or regression error(cid:5) it can
(cid:4)nd con(cid:4)gurationswell adaptedto the given data(cid:1) In the examples investigated this resultedin betterperformance
for networks(cid:7)fuzzysystems of the same size(cid:5) or in smaller networks to reach a given performance(cid:1)
