Neural Networks and Analog Computation: Beyond the Turing Limit

192 Pages·1999·6.57 MB·English

by  Hava T. Siegelmann (auth.)

Progress in Theoretical Computer Science
Editor
Ronald V. Book, University of California
Editorial Board
Erwin Engeler, ETH Zentrum, Zurich, Switzerland
Jean-Pierre Jouannaud, Universite de Paris-Sud, Orsay, France
Robin Milner, Cambridge University, England
Martin Wirsing, Universitat Passau, Passau, Germany
Hava T. Siegelmann
Neural Networks
and Analog Computation
Beyond the Turing Limit
Springer Science+Business Media, LLC
Hava T. Siegelmann
Department of Information Systems Engineering
Faculty of Industrial Engineering and Management
Technion
Haifa 3200, Israel
Cover iIlustration by Pepi Marzel.
Library ofCongre5S CataIogmg-in-Publication Data
Siegelmann, Hava T.
Neural networks and analog computation : beyond the Turing limit I Hava T.
Siegelmann.
p. cm. - (Progress in theoretical computer science)
Includes bibliographical references and index.
ISBN 978-1-4612-6875-8 ISBN 978-1-4612-0707-8 (eBook)
DOI 10.1007/978-1-4612-0707-8
1. Neural networks (Computer science) 2. Computational
complexity. 1. Title. II. Series.
QA76.87.S565 1999
006.3'2-dc21 98-29446
CIP
AMS Subject Classifications: 68Q05, 68Q15, 68T
Printed on acid-free paper.
©1999 Springer Science+Business Media New York
Originally published by Birkhiiuser Boston in 1999
Softcover reprint of the hardcover 1s t edition 1999
AlI rights reserved. This work may not be translated or copied in whole or in part without
the written permission of the publisher Springer Science+Business Media, LLC,
except for brief excerpts in
connection with reviews or scholarly analysis. Use in connection with any form of
information storage and retrieval, electronic adaptation, computer software, or by similar
or dissimilar methodology now known or hereafter developed is forbidden.
The use of general descriptive names, trade names, trademarks, etc., in this publication,
even ifthe former are not especially identified, is not to be taken as a sign that such names,
as understood by the Trade Marks and Merchandise Marks Act, may accordingly be used
freely by anyone.
ISBN 978-1-4612-6875-8
Formatted from the author's Ji\.T~ files.
9 8 765 432 1
Contents
Introduction ix
1 Computational Complexity 1
1.1 Neural Networks ..... 2
1.2 Automata: A General Introduction. 4
1.2.1 Input Sets in Computability Theory 5
1.3 Finite Automata .............. . 6
1.3.1 Neural Networks and Finite Automata. 6
1.4 The Turing Machine . . . . . . . . . . . . . . 8
1.4.1 Neural Networks and Turing Machines. 10
1.5 Probabilistic Turing Machines ......... . 11
1.5.1 Neural Networks and Probabilistic Machines 12
1.6 Nondeterministic Turing Machines .... 12
1.6.1 Nondeterministic Neural Networks 13
1. 7 Oracle Turing Machines . . . . . . . . . . 13
1.7.1 Neural Networks and Oracle Machines 13
1.8 Advice Turing Machines ........... . 14
1.8.1 Circuit Families ............ . 15
1.8.2 Neural Networks and Advice Machines. 16
1.9 Notes 17
2 The Model 19
2.1 Variants of the Network 20
2.1.1 A "System Diagram" Interpretation 22
2.2 The Network's Computation. 23
2.3 Integer Weights ...... . 25
vi CONTENTS
3 Networks with Rational Weights 29
3.1 The TUring Equivalence Theorem . 30
3.2 Highlights of the Proof . . . . . . . 33
3.2.1 Cantor-like Encoding of Stacks 33
3.2.2 Stack Operations . . . . . . . . 35
3.2.3 General Construction of the Network. 36
3.3 The Simulation . . . . . . 36
3.3.1 1'-Stack 11achines 37
3.4 Network with Four Layers 38
3.4.1 A Layout Of The Construction 43
3.5 Real-Time Simulation ....... 44
3.5.1 Computing in Two Layers . 45
3.5.2 Removing the Sigmoid From the 11ain Layer 48
3.5.3 One Layer Network Simulates T11 54
3.6 Inputs and Outputs ...... 54
3.7 Universal Network ....... 56
3.8 Nondeterministic Computation 57
4 Networks with Real Weights 59
4.1 Simulating Circuit Families 61
4.1.1 The Circuit Encoding 61
4.1.2 A Circuit Retrieval . . 63
4.1.3 Circuit Simulation By a Network 66
4.1.4 The Combined Network 67
4.2 Networks Simulation by Circuits 68
4.2.1 Linear Precision Suffices . 68
4.2.2 The Network Simulation by a Circuit 70
4.3 Networks versus Threshold Circuits. 72
4.4 Corollaries. . . . . . . . . . . . . . . 75
5 Kolmogorov Weights: Between P and P /poly 77
5.1 Kolmogorov Complexity and Reals . . . . 78
5.2 Tally Oracles and Neural Networks .... 81
5.3 Kolmogorov Weights and Advice Classes . 85
5.4 The Hierarchy Theorem .......... 87
CONTENTS vii
6 Space and Precision 91
6.1 Equivalence of Space and Precision 91
6.2 Fixed Precision Variable Sized Nets. 94
7 Universality of Sigmoidal Networks 97
7.1 Alarm Clock Machines ....... . 100
7.1.1 Adder Machines ...... . 100
7.1.2 Alarm Clock and Adder Machines 102
7.2 Restless Counters .......... . 103
7.3 Sigmoidal Networks are Universal .. 106
7.3.1 Correctness of the Simulation 110
704 Conclusions . . . . . . . 112
8 Different-limits Networks 115
8.1 At Least Finite Automata 115
8.2 Proof of the Interpolation Lemma . 118
9 Stochastic Dynamics 121
9.1 Stochastic Networks 122
9.1.1 The Model 124
9.2 The Main Results .. 125
9.2.1 Integer Networks 125
9.2.2 Rational Networks 126
9.2.3 Real Networks .. 126
9.3 Integer Stochastic Networks 127
904 Rational Stochastic Networks 128
904.1 Rational Set of Choices 129
904.2 Real Set of Choices . 129
9.5 Real Stochastic Networks ... 134
9.6 Unreliable Networks ...... 136
9.7 Nondeterministic Stochastic Networks 138
10 Generalized Processor Networks 141
10.1 Generalized Networks: Definition 142
10.2 Bounded Precision ....... . 143
10.3 Equivalence with Neural Networks 145
lOA Robustness ............ . 146
viii CONTENTS
11 Analog Computation 147
11.1 Discrete Time Models 148
11.2 Continuous Time Models 149
11.3 Hybrid Models . . 150
11.4 Dissipative Models . . . . 150
12 Computation Beyond the Turing Limit 153
12.1 The Analog Shift Map ..... 156
12.2 Analog Shift and Computation 158
12.3 Physical Relevance 163
12.4 Conclusions 164
Bibliography 165
Index 179
Introduction
Humanity's most basic intellectual quest to decipher nature and master it has
led to numerous efforts to build machines that simulate the world or communi
cate with it [Bus70, Tur36, MP43, Sha48, vN56, Sha41, Rub89, NK91, Nyc92].
The computational power and dynamic behavior of such machines is a central
question for mathematicians, computer scientists, and occasionally, physicists.
Our interest is in computers called artificial neural networks. In their most
general framework, neural networks consist of assemblies of simple processors,
or "neurons," each of which computes a scalar activation function of its input.
This activation function is nonlinear, and is typically a monotonic function
with bounded range, much like neural responses to input stimuli. The scalar
value produced by a neuron affects other neurons, which then calculate a new
scalar value of their own. This describes the dynamical behavior of parallel
updates. Some of the signals originate from outside the network and act
as inputs to the system, while other signals are communicated back to the
environment and are thus used to encode the end result of the computation.
Unlike the von Neumann computer model, the structure of a neural net
work cannot be separated into a memory region and a processing unit. In
the neural model, memory and processing are strongly coupled. Each neuron
is part of the processing unit, and the memory is implicitly encoded in the
mutual influence between any pair of neurons. The influence may be excita
tory; its strength, then, will be measured by a positive real weight of the link.
Inhibitory influence will be represented by a negative real number weight.
The status of the weights, whether perceived as unknown parameters or
fixed constants, prompts two different views of the neural model. When the
weights are considered unknown parameters, the network is a semi-parametric
statistical model, able to approximate input-output mappings by means of
parameter estimation (also called learning or adaptation). When the weights
are considered constant (after or without a process of adaptation) the networks
can perform exact computations rather than mere approximations. Due to the
rich and numerous adaptation processes available, neural networks are of great
use as nonlinear adaptive controllers and function approximators in industry.
The· first computational model of a biological neuron was suggested by
x INTRODUCTION
McCulloch and Pitts. In their paper, entitled "A Calculus of Ideas Immanent
to Nervous Activity" [MP43], they argued that because of the all-or-nothing
characteristic of action potentials (as seen by Hodgkin and Huxley), the ner
vous system could be thought of as a finite interconnection of logical devices.
Nervous activity could thus be interpreted as forming relations between sym
bolic propositions, which then proceed to build propositions of yet greater
complexity. The McCulloch-Pitts neuron is still considered a classical digital
abstraction of a simplified brain cell.
This work by McCulloch and Pitts was first presented in 1943, in the sec
ond annual meeting of the Josiah Macy Jr. scientific conference. Attended by
biologists, mathematicians, engineers, and social scientists, the goal of the con
ference was to identify applicable parallels between brain and computer. The
theory of McCulloch and Pitts was controversially received: many stressed
that while digital behavior is a recognized component of cerebral function
ing, any theory that did not take into account continuous variables (i.e., the
underlying chemical and physical phenomena) could not be considered an ac
curate model. From the debates and discussions, two different approaches
emerged. The first, lead by Gerard and Bateson, emphasized the significance
of continuous-valued computation; the second, headed by McCulloch, Pitts,
and von Neumann, stressed the advantages in the simplicity of digital com
putation [Nyc92].
Following the development of von Neumann's universal model of com
putation based on the principle of the McCulloch-Pitts neuron, the digital
approach prevailed in the field of cybernetic research. The digital model laid
the groundwork both for the 20th century's computer paradigm and for an
enhanced understanding of computational processes in the brain.
With a backward glance to the 1940's, this book returns to the less popular
principle of analog computation, from which we develop the foundations of
continuous valued neural networks. When considering analog computers, the
model of the neural network is not unnatural: while in 1943 neural networks
were envisioned as digital, the adaptive neural models used for industrial
purposes in the 1980's were, in fact, analog.
The theoretical foundations presented herein concern a particular kind of
neural model: an analog recurrent neural network (ARNN). It consists of a
finite number of neurons arranged in a general architecture. The network
is analog in that it updates continuously and utilizes a continuous configu
ration space (as opposed to the discrete models of digital computers). In
our analog model, each neuron is characterized by real constant weights and
can hold a real value updated by means of a continuous function of its com
bined inputs. The adjective "recurrent" emphasizes that the interconnection
is general, rather than layered or symmetrical. Such an architecture allows
for internal state representation and describes its evolution over time.