Table Of ContentLEARNINGINNATURALANDCONNECTIONISTSYSTEMS
Learning in Natural
and Connectionist Systems
Experiments and a Model
by
R. Hans Phaf
University ofA msterdam,
PsychononUcsL>eparnnen~
Amsterdam, The Netherlands
SPRINGER SCIENCE+BUSINESS MEDIA, B.V.
A C.I.P. CataJogue record for this book is available from the Library of Congress.
ISBN 978-94-010-4362-5 ISBN 978-94-011-0840-9 (eBook)
DOI 10.1007/978-94-011-0840-9
Printed an acid-free paper
AII Rights Reserved
© 1994 Springer Science+Business Media Dordrecht
Originally published by Kluwer Academic Publishers in 1994
Softcover reprint ofthe hardcover Ist edition 1994
No part of the material protected by this copyright notice may be reproduced or
utilized in any form or by any means, electronic or mechanical,
including photocopying, recording or by any information storage and
retrieval system, without written permission from the copyright owner.
'1 never satisfy myself until I can make a mechanical model of a thing. If I can make
a mechanical model I can understand it. As long as I cannot make a mechanical model
all the way through I cannot understand...•
Baron Kelvin of Largs (W. Thomson, 1824-1907)
in: Notes of Lectures on molecular dynamics and the wave theory of light. Lecture XX.
Delivered at Johns Hopkins University, Baltimore, MD, 1884, Pp. 270-271.
v
Contents
Preface. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. xi
Acknowledgements . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. xv
Part I: Introduction . . . . . . . . I
1.I. The importance of learning I
1.2. The biological role of learning 4
1.3. Connectionist models as learning systems. 9
1.4. Models and languages. . . . . . . . . . 14
1.5. Aligning connectionist learning with natural learning . 19
Part 2: A connectionist approach to learning. 21
2.1. The connectionist language . . 21
2.1.1. Non-learning networks. 21
2.1.2. Learning networks. . . 31
2.2. Problems with connectionist models 39
2.2.1. Limitations on connectivity: modularity 43
2.2.2. The organization of excitatory and inhibitory connections 47
2.2.3. Attention in neural networks . . . 48
2.2.4. Implementation of the constraints. 55
2.3. CALM: Categorizing and Learning Module. 59
2.3.1. Categorization 66
2.3.2. Separation of correlated patterns 69
2.4. Single module simulations. . . . . . . 72
2.4.I. Convergence time and discrimination time. 74
2.4.2 Discrimination and generalization 79
2.5. Multiple module simulations 83
2.5.1 Learning the EXOR . 83
2.5.2 Learning the word superiority effect 85
2.6. Discussion . . . . . . . . . . . . . . 89
2.6.1. Unsupervised learning in CALM 89
2.6.2. CALM architectures. . . 91
2.6.3. Extension and applications 94
vii
viii Contents
Part 3: Psychological constraints on learning. 96
3.1. Attention and memory storage 96
3.1.1. A dissociation. . . . . 98
3.1.2. Multiple memory systems explanations 100
3.1.3. Multi-process explanations. . . . . 101
3.1.4. Study-test compatibility explanations 102
3.2. Elaborative shifts during rehearsal. 103
3.2.1. Experiment I 107
3.2.2. Experiment 2 110
3.2.3. Experiment 3 117
3.2.4. Experiment 4 123
3.2.5. A continuum of rehearsal operations 129
3.3. Attention and study-test compatibility as dissociative factors 131
3.3.1. Experiment 5 132
3.3.2. Experiment 6 137
3.3.3. Experiment 7 144
3.3.4. Experiment 8 150
3.3.5. Experiment 9 157
3.3.6. The dissociation in a single, modularly organized, memory system 162
3.4. Divided attention and implicit and explicit memory tasks. 167
3.4.1. Experiment 10. . . . . . . . . . . . . . . . . . 169
3.5. Towards a memory model incorporating attentional effects 177
Part 4: A connectionist model for implicit and explicit memory 180
4.1. ELAN: a family of models 180
4.2. Architecture of ELAN-I . 182
4.3. Simulations with ELAN-I. 187
4.3.1. General procedure. 187
4.3.2. Simulation of the word frequency effect 191
4.3.2.1. Mixed list simulation. . . . . . . 199
4.3.2.2. Interpretation of the model's behavior 201
4.3.3. Simulation of anterograde amnesia . . . . . 202
4.3.4. Interference in explicit and implicit memory 209
4.3.4.1 Simulation of retroactive interference 210
4.3.4.2. Simulation of proactive interference . 213
4.3.5. Simulation of divided attention in explicit and implicit tasks 216
Contents ix
4.4. Higher ELAN models. . . . . . . . . . . . . . . 222
4.4.1. ELAN-2: category learning . 223
4.4.2. ELAN-10: sequential recurrency in networks 226
4.4.2.1. List learning in ELAN-tO .. 234
4.4.2.2. A memory span in ELAN-tO 239
Part 5. Evaluation . . . . . . . . . . . . . . . . . . . . . . . 245
5.1. Comparison to other models. . . . . . . . . . . . . . 246
5.2. The connectionism vs. symbol-manipulation controversy 252
5.3. Conclusion . 258
References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 265
Appendix. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28t
Index. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 284
Preface
Learning, or selecting relevant information for storage in memory, has been
neglected in many formal models of human information processing. In the so-caned
artificial intelligence approach this process is often replaced by the analysis by the
modeler of the knowledge required in the model. Even specific models for human
memory often pay more attention to the structure of the memory representations than
to how they are acquired. Two recent developments in Cognitive Psychology have led
to a change in this situation. On the one hand, connectionism, a new formalism for
information processing models inspired by brain processes, proved to be very wen
suited for implementing learning. The observation of differential effects (i.e,
dissociations) of many experimental manipulations on classical and new memory
measures, on the other hand, led to a renewed interest in experimental psychology for
the processes involved in memory storage. Though, for instance, no effect of learning
during complete anaesthesia is found with classical explicit memory tests (which refer
explicitly to the presentation), implicit (unconscious) memory tests often provide clear
evidence for changes in memory during anaesthesia. Similar effects were observed with
patients who suffered from anterograde amnesia. In contrast to explicit recall, little
attention is required during learning for finding a facilitation in implicit memory
performance. One of the three alternative explanations for these phenomena, in terms
of anatomically separate memory systems, for the two kinds of memory performance,
appears to offer little starting points for simulating these experimental results in a
computer model.
It is the aim of this study to bring the research into explicit and implicit
memory performance, which usually cares little for the concrete realization of its
abstract theoretical assumptions, into further contact with connectionist modeling,
which, in turn, takes little notice of the large body of experimental results on human
memory. Many memory psychologists are often more interested in showing that some
hypothesis can be falsified on some, sometimes minor, detail, than in looking for the
common ground in all these hypotheses. Connectionists on the other hand are often so
much blinded by the mathematical and engineering aspects of their network models to
bother to ask how the human system actually performs these tasks. They do not seem
to be aware of the fact that the applications they are studying all involve human
functions and capabilities. This book attempts to help bridge a gap that should not be
there in the first place, not between the behavioral sciences and the neural sciences
(as connectionism is often accused of doing), but between psychologists and
connectionists.
xi
xii Preface
Because most connectionist or neural network models only have a passive,
externally controlled, learning mode, a new learning procedure with some attentional
capabilities has been developed. The novelty of the input determines how much learning
with attention (Elaboration learning) and to what degree learning without attention
(Activation learning) can take place. The learning procedure has been implemented in
a biologically inspired form: the CALM module(CategorizingAnd LearningModule). Due
to its psychologically inspired learning and attentional processes, the module is capable
of learning without supervision and categorizing more quickly and efficiently than many
other current learning procedures. The module forms a building block for different
network models that may be applied to various domains of information processing. The
domain that is addressed in this book is the field of explicit and implicit memory
performance.
Ten laboratory experiments are presented that further explore the boundary
conditions for a model of the two kinds of memory performance. These experiments
have been described in the format that is usually required for publications in a
scientific journal, because it is then more easy to see how an experimental procedure
can be translated in a simulation protocol. It is probably possible to skip these 'boring'
Method sections and still understand the general procedure of the experiments. It is
hard to see how these results can be reconciled with a separate memory systems
explanation, without also assuming different memory processes. In fact, many of the
leading memory psychologists entertaining multiple memory systems views also seem to
assume implicitly a form of multi process view. Some of the most recent publications in
this field show staggering examples of such double explanations. It is argued here that
the separate memory systems view is not necessary but that most results can be
explained by assuming a unitary memory representation distributed over a large number
of modules that can be addressed by different pathways and that can be affected
differentially by the two kinds of (Activation vs. Elaboration) learning. If a particular
memory test requires the formation of new representational components during
presentation, then this test will be more sensitive to learning with attention
(elaboration) than when only existing representational components are addressed. The
importance of the correspondence between storage and test is further strengthened by
the experimental results. Both the connectionist and the experimental ingredients are
then mixed in a model for implicit and explicit memory performance that is composed
of CALM modules. Though very simple and therefore insufficient for simulating all
aspects involved, the model is nevertheless capable of simulating some dissociative
effects, such as the disruption of explicit recall (after specific damage to the network)
Preface xiii
accompanied by a preservation of implicit performance. Due to its simplicity the model
is, as most models are, clearly insufficient, but it provides a consistent manner for
extending it to cover a larger range of phenomena. Of these extensions, a model with
a rehearsal loop and some short-term memory capacity is treated in somewhat more
detail, because it may, eventually, be helpful in simulating control processes and
symbol manipulation phenomena.
It is concluded that human memory functions in a different manner than a
computer memory, which needs an explicit instruction to store information from a
central processor. In the human system, as follows from the implicit memory results,
all forms of processing are accompanied by some form of memory change. The lack of
separation between processor and memory and the distributed nature of
representations, which directly reflect the operations that give rise to them, make
neural network models eminently suited for simulating human memory processes. Due to
their concrete formulation, connectionist models may, moreover, contribute to a quicker
development of psychological theory. For connectionists the psychological approach is
useful because it provides the best and presumably the only example of how human
functions may be performed. The optimal solution to an information processing problem
can probably be found by looking at evolution's answer to it. It thus appears that the
interaction between learning and attention is best studied by combining experimental
research with connectionist modeling. Attentional influences on learning in such a view
are not associated, as in the computer metaphor, with the central control of storage,
but enable the formation of new representational components, whereas without
attention storage may take place in the form of strengthening of existing components.
