Table Of ContentSpringer Series in Information Sciences 8
Springer Series in Information Sciences
Editors: Thomas S. Huang Manfred R. Schroeder
Volume 1 Content-Addressable Memories By T. Kohonen 2nd Edition
Volume 2 Fast Fourier Transform and Convolution Algorithms
By H.J. Nussbaumer 2nd Edition
Volume 3 Pitch Determination of Speech Signals Algorithms and Devices
ByW. Hess
Volume 4 Pattern Analysis By H. Niemann
Volume 5 Image Sequence Analysis Editor: T. S. Huang
Volume 6 Picture Engineering Editors: King-sun Fu and T. L. Kunii
Volume 7 Number Theory in Science and Communication
With Applications in Cryptography, Physics, Digital Information,
Computing, and Self-Similarity By M. R. Schroeder 2nd Edition
Volume 8 Self-Organization and Associative Memory By T. Kohonen
2nd Edition
Volume 9 Digital Picture Processing An Introduction By L. P. Yaroslavsky
Volume 10 Probability, Statistical Optics and Data Testing
A Problem Solving Approach By B. R. Frieden
Volume 11 Physical and Biological Processing ofImages
Editors: O. J. Braddick and A. C. Sleigh
Volume 12 Multiresolution Image Processing and Analysis
Editor: A. Rosenfeld
Volume 13 VLSI for Pattern Recognition and Image Processing
Editor: King-sun Fu
Volume 14 Mathematics of Kalman-Bucy Filtering
By P. A. Ruymgaart and T. T. Soong 2nd Edition
Volume 15 Fundamentals of Electronic Imaging Systems
Some Aspects of Image Processing By W. F. Schreiber
Volume 16 Radon and Projection Transform-Based Computer Vision
Algorithms, A Pipeline Architecture, and Industrial Applications
By J.L.C. Sanz, E.B. Hinkle, and A.K. Jain
Volume 17 Kalman Filtering with Real-Time Applications
By C.K. Chui and G. Chen
Volume 18 Linear Systems and Optimal Control
By C. K. Chui and G. Chen
Teuvo Kohonen
Self-Organization and
Associative Memory
Second Edition
With 99 Figures
Springer-Verlag Berlin Heidelberg New York
London Paris Tokyo
Professor Teuvo Kohonen
Laboratory of Computer and Information Sciences, Helsinki University of Technology
SF-02150 Espoo 15, Finland
Series Editors:
Professor Thomas S. Huang
Department of Electrical Engineering and Coordinated Science Laboratory,
University of Illinois, Urbana, IL 61801, USA
Professor Dr. Manfred R. Schroeder
Drittes Physikalisches Institut, Universitat Gottingen, Burgerstra13e 42-44,
D-3400 Gottingen, Fed. Rep. of Germany
ISBN 978-3-540-18314-3 ISBN 978-3-662-00784-6 (eBook)
DOI 10.1007/978-3-662-00784-6
Library of Congress Cataloging-in-Publication Data. Kohonen, Teuvo. Self-organization and associative memory. (Springer
series in information sciences; 8). Bibliography: p. Includes index. 1. Self-organizing systems. 2. Memory. 3. Associative
storage. I. Title. II. Series. Q325.K64 1987 001.53'9 87-26639
This work is subject to copyright. All rights are reserved, whether the whole or part of the material is concerned, specifically
the rights of translation, reprinting, reuse of illustrations, recitation, broadcasting, reproduction on microfilms or in other
ways. and storage in data banks. Duplication of this publication or parts thereof is only permitted under the provisions of the
German Copyright Law of September 9, 1965, in its version of June 24, 1985, and a copyright fee must always be paid.
Violations fall under the prosecution act of the German Copyright Law.
© Springer-Verlag Berlin Heidelberg 1984 and 1988
The use of registered names, trademarks, etc. in this publication does not imply even in the absence of a specific statement.
that such names are exempt from the relevant protective laws and regulations and therefore free for general use.
Typesetting: K & V Fotosatz, Beerfelden
2154/3150-54321
Preface to the Second Edition
Two significant things have happened since the writing of the first edition in
1983. One of them is recent arousal of strong interest in general aspects of
"neural computing", or "neural networks", as the previous neural models are
nowadays called. The incentive, of course, has been to develop new com
puters. Especially it may have been felt that the so-called fifth-generation
computers, based on conventional logic programming, do not yet contain in
formation processing principles of the same type as those encountered in the
brain. All new ideas for the "neural computers" are, of course, welcome. On
the other hand, it is not very easy to see what kind of restrictions there exist to
their implementation. In order to approach this problem systematically, cer
tain lines of thought, disciplines, and criteria should be followed. It is the pur
pose of the added Chapter 9 to reflect upon such problems from a general
point of view.
Another important thing is a boom of new hardware technologies for dis
tributed associative memories, especially high-density semiconductor circuits,
and optical materials and components. The era is very close when the parallel
processors can be made all-optical. Several working associative memory archi
tectures, based solely on optical technologies, have been constructed in recent
years. For this reason it was felt necessary to include a separate chapter
(Chap. 10) which deals with the optical associative memories. Part of its con
tents is taken over from the first edition.
The following new items have been included in this edition, too: more
accurate measures for symbol strings (Sect. 2.2.3), and the Learning Vector
Quantization for pattern recognition (Sect. 7.5). In addition, I have made a
few other revisions to the text.
I would like to emphasize that this monograph only concentrates on a
restricted area of neural computing, namely, different aspects of memory, in
particular associative memory. It may not be justified to expect that the
models which have been set up to illustrate the memory functions will solve
all the practical problems connected with neural computing. Nonetheless,
VI Preface to the Second Edition
memory seems to playa rather central role in thinking, as well as in sensory
perception.
I am very much obliged to Mrs. Leila Koivisto for her invaluable help in
making this extensive revision.
Otaniemi, Finland T. Kohonen
August 1987
Preface to the First Edition
A couple of years ago the Publisher and I agreed upon a revision of Associa
tive Memory - A System-Theoretical Approach (Springer Series in Com
munication and Cybernetics, CC 17). It turned out that this field had grown
rather rapidly. On the other hand, there were some classical publications
which had motivated newer works and which, accordingly, should have been
reviewed in the context of present knowledge. It was therefore felt that CC 17
should be completely reorganized to embed both classical and newer results in
the same formalism.
The most significant contribution of this work with respect to the earlier
book, however, is that while CC 17 concentrated on the principles by which a
distributed associative memory is implementable, the present book also tries
to describe how an adaptive physical system is able to automatically form re
duced representations of input information, or to "encode" it before storing
it. Both of these aspects are, of course, essential to the complete explanation
of memory.
Although the scope is now much wider than earlier, it was felt unnecessary
to alter some rather independent parts of the old edition.· Sections 2.1, 6.1 - 7,
7.1, 2, and 8.1 can be recognized as similar to the corresponding sections of
CC 17, except for some editing and reorganization. On the other hand, about
2/3 of the present contents are completely new.
The book now concentrates on principles and mechanisms of memory and
learning by which certain elementary "intelligent" functions are formed adap
tively, without externally given control information, by the effect of received
signals solely. A significant restriction to the present discussion is set by the
stipulated property that the systems underlying these principles must be
physical; accordingly, the basic components cannot implement arbitrary
arithmetic algorithms although this would be very easy to define even by the
simplest computer programs. The signal transformations must be as simple as
possible, and the changes in the system variables and parameters must be con
tinuous, smooth functions of time. This clearly distinguishes the present ideas
VIII Preface to the First Edition
from the conventional Artificial Intelligence approaches which are totally
dependent on the use of digital computers and their high-level programming
languages.
It is frequently believed that it is impossible to implement higher informa
tion processes without components the characteristics of which are very
nonlinear. It may thereby be thought that all decision processes must be based
on inherently nonlinear operations. If, however, the system properties are
time-variable, then this requirement can be alleviated. In fact, even a linear
system with time-variable parameter values already behaves in a nonlinear
way. Another remark is that although nonlinearities are needed in some places
for decision operations, it is not mandatory that every component and
elementary processing operation be nonlinear; there are many functions, espe
cially those performing statistical averaging of signals, which can best be
realized by linear circuits. We shall revert to this argumentation in Chapter 4.
Finally it may be remarked that if nonlinearities are needed, they may best be
included in more or less fixed preprocessing circuits, especially on the sensory
level.
The early phases of Artificial Intelligence research around 1960 were char
acterized by an enthusiastic attempt to implement learning functions, i.e., a
kind of elementary intelligence, using networks built of simple adaptive com
ponents. In spite of initial optimistic expectations, progress was never par
ticularly rapid which led to an almost opposite reaction; until quite recent
years, very few researchers believed in the future of such principles. My per
sonal view is that the first modelling approaches to learning machines were
basically sound; however, at least a couple of important properties were mis
sing from the models. One of them is a genuine memory function, especially
associative memory which can reconstruct complete representations such as
images and signal patterns from their fragments or other cues; and the second
flaw of the early models was that the importance of the spatial order of the
processing units was never fully realized. It seems, however, that in the biolog
ical brains a significant amount of information representation is encoded in
the spatial location of the processing elements. We shall later see in Chapter 5
that a meaningful spatial encoding can result in a simple self-organizing
physical process which uses similar components to those applied in the early
learning machines; the only new feature to be introduced is a characteristic
local interaction between neighbouring elements.
Associative memory is a very delicate and complex concept which often
has been attributed to the higher cognitive processes, especially those taking
place in the human brain. A statement of this concept can be traced back to
the empiricist philosophers of the 16th century who, again, inherited their
views from Aristotle (384 ~ 322 B.C.). It is nowadays a big challenge to launch
Preface to the First Edition IX
a discussion on associative memory since its characteristics and nature have
been understood in widely different ways by different scientists. Perhaps the
most high-browed one is the approach made in psycholinguistics where the
aim is to relate conceptual items structurally to produce bases of knowledge.
Another specialized view is held in computer engineering where, traditionally,
associative memory is identified with certain searching methods named con
tent-addressing. The only task has thereby been to locate a data set on the
basis of a matching portion in a keyword. Between these extremal concep
tions, there is a continuum of various associative memory paradigms.
The contents of this book may be seen as a progression, starting from a
systematic analysis of the most natural basic units, and ending with the inter
nal representations and associative memory. Thus the main purpose of this
representation is to demonstrate the gradual evolution of intelligent functions
in physical systems, and to reveal those conditions under which a passive
memory medium switches over into an active system that is able to form
meaningful compressed representations of its input data, i.e., abstractions
and generalizations which are often believed to be the basic constituents of
intelligence.
Some material presented in this book has resulted from collaboration with
my colleagues Pekka LehtiO and Erkki Oja, to whom I am very much obliged.
Several pictures relating to computer simulations have been prepared by Kai
Makisara. Thanks are also due to Eija Dower for typing the manuscript.
This work has been done under the auspices of the Academy of Finland.
Otaniemi, Finland T. Kohonen
August 1983
Contents
1. Various Aspects of Memory. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1
1.1 On the Purpose and Nature of Biological Memory . . . . . . . . . . . . 1
1.1.1 Some Fundamental Concepts ........................ 1
1.1.2 The Classical Laws of Association. . . . . . . . . . . . . . . . . . . . 3
1.1. 3 On Different Levels of Modelling .................... 4
1.2 Questions Concerning the Fundamental Mechanisms of Memory 4
1.2.1 Where Do the Signals Relating to Memory Act Upon? ... 5
1.2.2 What Kind of Encoding is Used for Neural Signals? ..... 6
1.2.3 What are the Variable Memory Elements? ............. 7
1.2.4 How are Neural Signals Addressed in Memory? ........ 8
1.3 Elementary Operations Implemented by Associative Memory .. 14
1.3.1 Associative Recall ................................. 14
1.3.2 Production of Sequences from the Associative Memory. . 16
1.3.3 On the Meaning of Background and Context ........... 20
1.4 More Abstract Aspects of Memory. . . . . . . . . . . . . . . . . . . . . . . . . 21
1.4.1 The Problem ofInfinite-State Memory. . . . . . . . . . . . . . . . 21
1.4.2 Invariant Representations ........................... 22
1.4.3 Symbolic Representations ........................... 24
1.4.4 Virtual Images .................................... 25
1.4.5 The Logic of Stored Knowledge ...................... 27
2. Pattern Mathematics ........................................ 30
2.1 Mathematical Notations and Methods .... . . . . . . . . . . . . . . . . . . 30
2.1.1 Vector Space Concepts ............................. 30
2.1.2 Matrix Notations .................................. 41
2.1.3 Further Properties of Matrices ....................... 44
2.1.4 Matrix Equations .................................. 48
2.1.5 Projection Operators. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 54
2.1.6 On Matrix Differential Calculus . . . . . . . . . . . . . . . . . . . . . . 57
2.2 Distance Measures for Patterns. . . . . . . . . . . . . . . . . . . . . . . . . . . . 59
