Table Of ContentFrom Statistical Physics to
Statistical Inference and Back
NATO ASI Series
Advanced Science Institutes Series
A Series presenting the results of activities sponsored by the NATO Science Committee,
which aims at the dissemination of advanced scientific and technological knowledge,
with a view to strengthening links between scientific communities.
The Series is published by an international board of publishers in conjunction with the
NATO Scientific Affairs Division
A Life Sciences Plenum Publishing Corporation
B Physics London and New York
C Mathematical Kluwer Academic Publishers
and Physical Sciences Dordrecht, Boston and London
D Behavioural and Social Sciences
E Applied Sciences
F Computer and Systems Sciences Springer-Verlag
G Ecological Sciences Berlin, Heidelberg, New York, London,
H Cell Biology Paris and Tokyo
I Global Environmental Change
NATO-PCO-DATA BASE
The electronic index to the NATO ASI Series provides full bibliographical references
(with keywords and/or abstracts) to more than 30000 contributions from international
scientists published in all sections of the NATO ASI Series.
Access to the NATO-PCO-DATA BASE is possible in two ways:
- via online FILE 128 (NATO-PCO-DATA BASE) hosted by ESRIN,
Via Galileo Galilei, I-00044 Frascati, Italy.
- via CD-ROM "NATO-PCO-DATA BASE" with user-friendly retrieval software in
English, French and German (© WTV GmbH and DATAWARE Technologies Inc.
1989).
The CD-ROM can be ordered through any member of the Board of Publishers or
through NATO-PCO, Overijse, Belgium.
Series C: Mathematical and Physical Sciences - Vol. 428
From Statistical Physics
to Statistical Inference
and Back
edited by
Peter Grassberger
Department of Theoretical Physics,
University of Wuppertal,
Wuppertal, Germany
and
Jean-Pierre Nadal
Laboratory of Statistical Physics,
Ecole Normale Superieure,
Paris, France
w
Springer-Science+Business Media, B.V.
Proceedings of the NATO Advanced Study Institute on
From Statistical Physics to Statistical Inference and Back
Cargese (Corsica), France
August 31-September 12,1992
A C.I.P. Catalogue record for this book is available from the Library of Congress.
ISBN 978-94-010-4465-3 ISBN 978-94-011-1068-6 (eBook)
DOI 10.1007/978-94-011-1068-6
Printed on acid-free paper
All Rights Reserved
© 1994 Springer Science+Business Media Dordrecht
Originally published by Kluwer Academic Publishers in 1994
Softcover reprint of the hardcover 1st edition 1994
No part of the material protected by this copyright notice may be reproduced or
utilized in any form or by any means, electronic or mechanical, including photo
copying, recording or by any information storage and retrieval system, without written
permission from the copyright owner.
Contents
Preface ............................................................... vii
In place of an Introduction
G. Toulouse .......................................... Some remarks on 1
Principles for Inference
R. Balian ..... Statistical mechanics and the maximum entropy method 11
A. J. M. Garrett ................ Irreversibility, probability and entropy 45
N. Rivier ............ Maximum entropy for random cellular structures 77
J. Rissanen .... Minimal Description Length modeling: an introduction 95
G. Parisi .............. An introduction to learning and generalization 105
S. I. Amari ... Information geometry and manifolds. of neural networks 113
G. J. Klir ......... Uncertainty as a resource for managing complexity 139
Coding and Statistical Physics of Disordered Systems
S. Verdu .................... The development of Information Theory 155
J. Stern ... Statistical inference, zero-knowledge and proofs of identity 169
M. Mezard .............................. Spin glasses: an introduction 183
N. Sourlas ........... Statistical Mechanics and error-correcting codes 195
Learning
N. Tishby Learning and generalization with undetermined architecture 205
M. A. Virasoro .......... Confronting neural network and human behavior
in a quasi regular environment 225
R. Linsker ................ Sensory processing and information theory 237
H. U. Bauer, T. Geisel, K. Pawelzik, and F.Wolf ........ The formation of
representations in the visual cortex 249
D. A. Lane ............. Classifier systems: models for learning agents 263
v
vi
Dynamical Systems
R. Lima .. Space time dynamics and biorthogonal analysis: mementum 281
A. Politi .................... Symbolic encoding in dynamical systems 293
N. B. Tufillaro ... Topological organization of (low-dimensional) chaos 311
J. Rissanen . Noise Separation and MD L modeling of chaotic processes 317
Quantum Mechanics
R. Omnes .......................... Inference in Quantum Mechanics 331
W. H. Zurek ............ Decoherence and the existential interpretation of
quantum theory or "no information without representation" 341
List of Contributors .................................................. 351
Index ................................................................ 353
PREFACE
This book is based on lectures given at a NATO "Advance Study
Institute" in Cargese (France), September 1992. This meeting was
also supported by the C.N.R.S. (Centre National de la Recherche
Scientifique), program "Cognisciences", and by D.R.E.T. We are
very grateful to the NATO Science Committee, the Direction of the
CNRS program "Cognisciences", and the Scientific Direction of
DRET, for having made this meeting possible. It is a ple~ure to
thank Marie-France Hanseler for the local organization in Cargese.
The Institute dealt with the notion of inference at the interface between
physics, information theory, neural sciences and statistics. The word "infer
ence" denotes the derivation of general rules from particular sets of obser
vations. It is thus the basis of all learning from instances, as opposed to
learning by formal deduction and to mere data collection. More specifically,
the subject was statistical inference, in which case on~ cannot hope to get
strict deterministic rules but only statistical ones.
Physicists, for modeling physical systems with a large number of degrees
of freedom, and statisticians, for performing data analysis, have developped
their own concepts and methods for making the "best" inference. There was
thus a need for a clarification: are all these methods equivalent or not? What
is the state of ar~ in making inferences? The Institute was an attempt to
answer these questions.
Two recent developments contributed further to the feeling in the physics
community that a better understanding of inference is needed. The steadily
rising interest in neural computation made it more and more clear that a
deeper understanding of neural systems needs also a better insight of how
they make inferences. And the theory of chaotic non-linear systems is more
and more applied to practical time series analysis, where it could profit enor
mously from the experience assembled by statisticians using mostly linear
models. Finally, there is a long-standing conjecture (by E.T. Jaynes and
others) that some of the puzzles of quantum mechanics are due to our in
complete understanding of how we make inferences.
The talks as well as the round table discussions made clear the bridges
vii
viii
and division lines between the different methods and schools.
Statistical physics is based on the concept of entropy which is also the
central concept in Shannon's information theory. Moreover, the Maximum
Entropy Method (MEM) is related to the Bayesian approach to inference.
These aspects are reviewed in detail in these proceedings, as well as recent
progress in relating them to the theory of neural networks - and to learning
in general. Other lectures dealt with the use of the MEM data analysis and
in condensed matter physics.
However, during the last decades mathematicians have developed alter
natives both to Shannon information theory and to Bayesian inference: al
gorithmic information theory and the Minimum Description Length (MDL)
principle of Rissanen. In particular the latter aims at quantifying the old Oc
cam's strategy ('the best model is the simplest one'). Some lecturers showed
how this approach can be usefull in analysing physical systems.
It became clear at the Institute that the MDL and Bayesian approaches
cOIncide for a particular choice of the Bayesian prior distribution. Otherwise
strong disagreements persist, but the strongest 'Concern more the motivation
than the actual techniques. In particular, for a physicist a model should
always have some "physical content", which seems too vague a notion for
mathematicians and engineers. As was to be expected, no agreement was
reached, but we hope that the procedings reflect some of the controversies
which can only be helpful in stimulating further research.
More specific problems treated in the proceedings include the following:
A possible relationship bewteen the "complexity" of a model and the
"Vapnik-Chervonenskis dimension", a crucial parameter that is known
to characterize the ability of the model to describe the data.
A review of different observables which have been proposed for measur
ing the "complexity" of a system.
Applications of formal language theory and of topological methods to
dynamical systems.
Classifyer systems as an alternative to neural networks, and
Fuzzy logic and related concepts as alternatives to classical probability
theory.
Lastly it was pointed out that the understanding of human inference, as
compared to optimal inference ("is the mind a Bayesian?") may require a
better cooperation between theoreticians and psychologists.
In addition to the main lectures, the program of the Institute was rounded
off by a number of seminars which dealt with more specific questions. Even
if not included in the proceedings, they certainly contributed to the stimu
lating atmosphere. We hope that some of this atmosphere can be fealt by
the reader on the following pages.
Peter Grassberger,
Jean-Pierre Nadal.
CONCLUDING REMARKS
Gerard Toulouse
Laboratoire de Physique Statistique
Ecole Normale Superieure
24 rue Lhomond
75231 Paris Cedex 05 France
ABSTRACT. Here follow some remarks, composed in real time during this meeting and
presented at its end. They are due to a statistical physicist, under the influence of lecturers
experts in many aspects of statistical inference.
1. INTRODUCTION
A reason, I believe, for much of the excitement and the success of this meet
ing is that it touched on the three things that are of interest in this world:
matter, life, mind. As a reminder, let me list some of the topics evoked,
during this trip from statistical physics to statistical inference, and back.
Matter: classical gases, quantum mechanics, hydrodynamics and dy
namical systems, disordered systems, neural nets.
Life: evolution, perception, reasoning, neurobiology.
Mind: intelligence, science, logic, computer science, signal analysis.
Now at least, some of us have acquired a better sense of the chronology
of landmarks, within our various disciplines: Turing machine (30's), prin
cipal component analysis (45), information theory (48), cellular automata
(50's), algorithmic information complexity (60's), computational complex
ity (70's), chaos (70's), spin glass theory (circa 80), Vapnik (VC) dimension
(82), simulated annealing (83), minimum description length (MDL) (83),
probably almost correct theory (PAC) (84), statistical physics of neural net
works (80's). And I am not listing fuzzy logic, ARMA, AIC, embedding
theory, and so forth.
It seems clear that the biggest excitement during this conference came
from the confluence between the statistical physics of neural nets and the
statistical inference concepts of VC dimension and PAC theory. Another
bridge between worst case analysis (traditionally favoured among mathe-
P. Grassberger and J. -Po Nadal (eds.).
From Statistical Physics to Statistical Inference and Back 1-9.
© 1994 Kluwer Academic Publishers.
2
maticians) and statistical estimates (commonly used by physicists) came
also under focus during the presentation of cryptography by J. Stern.
But I will have little to add here to these intense exchanges, and what
I will rather do is to evoke some ideas scattered allover, in order to stress
some aspects and links, that might have otherwise escaped the attention of
some among you. And, first of all, I wish to give a list of references that will
be found useful by those who wish to get a sense of historical perspectives
on some of our interdisciplinary topics.
Barlow, Horace: 1983, 'Intelligence, guesswork, language', Nature 304,
207-209; 1990, 'Conditions for versatile learning, Helmholtz's uncon
scious inference, and the task of perception', Vision Research 30, 1561-
1571
Delbriick, Max: 1986, 'Perception', in Mind from Matter?, Blackwell:
Oxford, 109-119.
Gregory, Richard: 1986, 'Whatever happened to information theory?',
in Odd Perceptions, Methuen: London, 187-194.
Brillouin, Leon: 1954, 'Life, thermodynamics and cybernetics', Ameri
can Scientist 37, 554-568.
Hopfield, John J.: 1986, 'Physics, biological computation and comple
mentarity', in The lesson of quantum theory, North-Holland: Amster
dam, 295-314.
Barlow is a neurobiologist, Delbriick a physicist who became one of the
founders of molecular biology, Gregory surveys the impact of information
theory in biology, Brillouin describes scientific attitudes and helps to ponder
on guesswork about science future, Hopfield touches soberly on quantum
mechanics and consciousness. (These six articles are reprinted, among others,
in Gutfreund, H. and Toulouse, G. eds.: 1993, Biology and Computation: a
Physicists' Choice, World Scientific: Singapore
2. INTELLIGENCE
One possible definition of 'intelligence' is 'ability to solve complex problems'.
At first such a definition may appear as hopelessly circular as the definition
presented by Peter Grassberger (with all due apologies to our scientific di
rector ... ): 'complexity' defined as 'difficulty of a task'. But wait.
In his book 'The mismeasure of man', S.G. Gould describes the recur
rent efforts of scientists (often good ones, pathetically) to measure human
intelligence on one scale. It started, a century and a half ago, with measures
of skull volume, then brain weight, then LQ. tests, etc. The temptation re
mains dangerous. My claim here is that if we had a universal definition for
complexity, and a one-dimensional scale to measure it, then we would have
a measure of intelligence. Ab absurdo, I infer (admittedly, this argument is
