Table Of ContentPhysical Models of
Neural Networks
This page is intentionally left blank
Physical Models of
Neural Networks
Tamas Geszti
Department of Atomic Physics
Eotvos University
Budapest, Hungary
fe World Scientific
m Singapore • New Jersey • London • Hong Kong
Published by
World Scientific Publishing Co. Pte. Ltd.,
P O Box 128, Farrer Road, Singapore 9128
USA office: 687 Hartwell Street, Teaneck, NJ 07666
UK office: 73 Lynton Mead, Totteridge, London N20 8DH
Library of Congress Cataloging-in-Publication Data
Geszti, Tamas.
Physical models of neural networks/by Tamas Geszti.
p. cm.
ISBN 9810200129
1. Neural circuitry — Models. 2. Neural computers. I. Title.
QP363.3.G47 1990
612.8'2'011--dc20 89-48133
CIP
Copyright © 1990 by World Scientific Publishing Co. Pte. Ltd.
All rights reserved. This book, or parts thereof, may not be reproduced
in any form or by any means, electronic or mechanical, including photo
copying, recording or any information storage and retrieval system now
known or to be invented, without written permission from the Publisher.
Printed in Singapore by JBW Printers and Binders Pte. Ltd.
PREFACE
You all know what your heart is. It is a pump. It also looks like a pump,
with its muscles capable to make it expand and contract and its valves letting the
blood in and out when necessary. It is good mechanical engineering, well done for
the purpose.
The same is true about your stomach: it is a chemical reactor, correctly
devised and implemented for that.
But what would you say about your brain? You learn at school that it is
the organ of thinking and regulation, but what should such an organ look like?
Although many other metaphores have been mentioned in this context, probably
the piece of engineering closest in function to a brain is a computer. However,
nothing less similar in appearance than a human brain and a man-made computer;
nothing like the straightforward connection between shape and function familiar
from the examples of heart and stomach.
One of the less transparent metaphores says that in some respects - at least as
far as associative memory storage and recall is concerned, but probably quite a bit
beyond that - the brain is analogous to a spin-glass. This suggestion was advanced
by the solid-state physicist John Hopfield in 1982, and marked the beginning of an
important development in creating neural network models which are tractable by
tools of theoretical physics, in particular, statistical physics. The results seem to
have relevance to both brain research and the so-called neural computers, providing
building principles for a class of the latter.
Neural network modelling is an enormous field of which physicists' models
cover only a little but fascinating corner. Those interested in the lively develop
ment of the whole area should consult recent volumes of the journal "Biological
Cybernetics" and the recently launched journal "Neural Networks".
The intention of this lecture notes volume is to offer the Reader an introduc
tion to the subject from the physicist's viewpoint. Although the main subject is
the Hopfield model and its descendants, feed-forward networks and their appli
cations to computing tasks are also reviewed. The text is based on the author's
v
vi PREFACE
course given for physics and biophysics students at Eotvos University, Budapest,
in 1987 and 1988.
It is a pleasure to thank here Jean-Pierre Nadal and John Hertz to whom I
owe much of my insight to the subject, and my students Ferenc Pazmandi, Ferenc
Czako and Istvan Csabai who helped me with many discussions and criticism on
some parts of the manuscript. I thank Zsolt Frei for help in the final prepara
tion of the manuscript. Whenever I felt uneasy about complications concerning
spin-glasses, I had the comfort to make use of Imre Kondor's understanding of
the subject (of course he is not responsible for cases when I ought to have felt
uneasy but I did not). This book would not have been written without the inspir
ing atmosphere of the Physics Department of Limburgs Universitair Centrum in
Diepenbeek (Belgium), with the kind hospitality of Professors Roger Serneels and
Marc Bouten and their inexhaustible insistence on clear understanding. Most re
cently, Dr. Christian Van den Broeck of the same department and Professor Janusz
J^drzejewski visiting there from the University of Wroclaw read the manuscript
very carefully, detected a number of errors and asked questions suggesting im
provements. Going back in time, when I started learning the subject, I profited
from learning together and discussing with Robert Nemeth. Last but not least, it
cannot be left unmentioned that my interest in the subject was aroused by George
Marx asking a couple of years ago, "What's all the excitement about brains and
spin-glasses?"
T. Geszti
November 1989
CONTENTS
PREFACE v
1 ASSOCIATIVE MEMORY AND ATTRACTORS 1
2 THE NEURON AND ITS MODEL 4
2.1 A quick look at the neuron 4
2.2 Modelling the neuron 8
2.3 Feedback dynamical network 10
2.4 Connectivism 13
3 HEBBIAN LEARNING 15
4 THE HOPFIELD MODEL 19
4.1 Definition of the model and basic facts about it 19
4.2 Noise analysis 24
4.3 Mean-field theory of the Hopfield net: derivation
of the Amit-Gutfreund-Sompolinsky equations . . 30
4.4 Mean-field theory: solution of the equations 36
5 DESCENDANTS OF THE HOPFIELD MODEL 40
5.1 The Hopfield model is robust against 40
5.2 The projection rule 43
5.3 Low activity and biased networks 45
5.4 Hierarchical storage 50
5.5 Time sequences 54
5.6 Invariant pattern recognition 58
5.7 Learning within bounds and short-term memory . . .. 60
6 PATIENT LEARNING 64
6.1 Setting the aim 64
6.2 Iterative learning algorithms 66
6.3 The Perceptron Convergence Theorem 70
6.4 The Gardner capacity of a network 72
vii
viii CONTENTS
7 DYNAMICS OF RETRIEVAL 76
7.1 The asymmetrically diluted model 78
7.2 General asymmetric models 83
8 FEED-FORWARD NETWORKS 87
8.1 The simple perceptron 87
8.2 Layered feed-forward networks 91
9 THE BOLTZMANN MACHINE 98
10 SELF-ORGANIZED FEATURE MAPS:
THE KOHONEN MODEL 101
10.1 Preliminaries 101
10.2 Topological ordering: local and global 103
10.3 The Kohonen algorithm 105
10.4 Question marks on theory Ill
11 OUTLOOK 113
APPENDIX A:
THE REPLICA TRICK WAY TO MEAN FIELD 115
APPENDIX B:
DYNAMICS OF A TWO-PATTERN NETWORK 125
REFERENCES 133
SUBJECT INDEX 141
1 ASSOCIATIVE MEMORY AND ATTRACTORS
Have you ever been thinking about how enormous amount of computing it
takes, to see something? There is no optical communication system in your brain:
the images received in your eyes are coded by your nervous system in a fantastic
richness of details, then processed and restored in your consciousness, correcting
for changes of illumination, mood and many other circumstances.
Visual processing is very fast. However most mental processes take a manifest
amount of time. To mention a task just a bit more involved than just vision, to
recognize an old friend you haven't seen for ages may be a difficult decision problem
about whether the one you see now is the same person you used to know before.
For a little while you may feel pending between yes and no. Then you feel attracted
by one of the two answers. The decision being taken, your thinking comes to a
short rest.
This tendency of evolving towards a distinguished state carrying some signif
icant meaning, usually a solution to some part of a task, which - if reached - is
marked by a little rest, seems to be a very widespread scenario of mental processes
on very different levels. Its dominating presence is reflected even in the traditional
layout of opera music: scenes and recitatives visualize the dramatic evolution to
wards points of rest, which are then marked by arias. In the context of physical
models of neural networks it has been clearly stated for the first time by Parisi
(1986).
Simpler to model than a temporary halt is a single configuration or a lim
ited set of them which is approached by the evolution of the system, and once
reached, it is not left any more. Such a set of configurations is called an attractor
of the dynamics. The simplest case is a single attracting configuration. That has
a particular name: it is called an attractive fixed point. Sequences of configura
tions are also possible attractors; they can be repeated cyclically: then they are
called limit cycles] or they can continue chaotically. More complex formations,
where the evolution takes a little rest - possibly to mark something meaningful -
and then goes on as mentioned above, can be hyperbolic fixed points attracting
1
