The Computational Beauty of Nature: Computer Explorations of Fractals, Chaos, Complex Systems, and Adaptation

482 Pages·1998·32.51 MB·English

by  Gary William Flake

Preface
The scientist does not study nature becausei t is useful; he studies it becauseh e delights
in it, and he delights in it becausei t is beautiful. If nature were not beautiful, it would not
be worth knowing, and if nature were not worth knowing, life would not be worth living.
Of course I do not here speak of that beauty that strikes the senses, the beauty of qualities
and appearance;s not that I undervalue such beauty, far from it, but it has nothing to do
with science; I mean that profounder beauty which comesf rom the harmonious order of
the parts, and which a pure intelligence can grasp.
- Henri Poincare
A VARIATIONO N an old joke goes as follows:
Engineers study interesting real-world problems but fudge their results.
Mathematicians get exact results but study only toy problems. But
computer scientists, being neither engineersn or mathematicians, study
toy problems and fudge their results.
Now, since I am a computer scientist, I have taken the liberty of altering the joke
to make myself and my colleaguest he butt of it. This joke examines a real problem
found in all scientific disciplines. By substituting experimentalist, theorist, and
simulationist for engineer, mathematician, and computer scientist, respectively, the
joke becomesg eneralized for almost all of the sciencesa nd gets to the heart of a
very real division.
A theorist will often make many simplifying assumptions in order to get to the
essenceo f some physical process. Particles do not necessarilyl ook like billiard balls,
but it often helps to think in this way if you are trying to understand how classical
mechanicss ays things should behave. Likewise, experimentalists often have to deal
with messy proces ses that are prone to measurement error. So if a physicist finds
that the surface temperature of an object is between 100,000 and 200,000 degrees, it
'
doesnt matter if the units are Celsius degreeso r Kelvin degrees, becauset he margin
of error is orders of magnitude larger than the difference in the two measuring units.
A simulationist is a relatively new breed of scientist who attempts to understand
how the world works by studying computer simulations of phenomena found in
xiv Preface
nature. A simulationist will always have to make some assumptions when building
a computer model but will also find that the simulated results are not always a
perfect match for what exists in the real world. Hence, the simulationist, having to
incorporate principles from theory and experimental methodologies, straddles the
fence and must deal with limiting factors found in both extreme approac hes. But
this is not always a bad thing.
Consider an economist who builds a simplified model of the world economy, runs
the model on a computer, and reachest he conclusion that interest rates, unemployment
, inflation, and growth will all reach a constant level at the end of the year
and stay that way forever. For this one case, the simplified ,model tells us very little
about how the real world works becauset he simplified model has failed to capture
an important aspect of the real world. On the other hand, supposet hat a simplified
model is such that it never reachese quilibrium, turns out to be extremely sensitive
to the starting conditions, and displays surprisingly complex behavior. Even if this
model fails to make actual predictions about the real economy, it still has some
predictive power since it may reveal a deeper truth about the inherent difficulty of
predicting the economy or similar systems. In other words, the model is predictable
in its unpredictability. Hence, if a simplified model can behave in a sophisticated
manner, then it is not too great a leap to conclude that the real-world economy can
display an even greater form of sophistication.
For the first case, if after simplifying a natural processw e find behavior that is
profoundly simpler than the original phenomenon, then it is likely that the model
failed to capture some essential piece of the real-world counterpart. On the other
hand, if an analogous form of complexity is still found even in a simplified model,
then it is highly possible that a key feature of the natural system has been isolated.
This illustrates that simulations- especially if they are simplifications- can yield
insight into how things work in the real world.
All of this boils down to a simple but deep idea: simple recurrent rules can
produce extremely rich and complicated behaviors. Pure theory often fails to make
accurate predictions of complicated natural proces ses becauset he world does not
always obey equations with analytical solutions. Similarly, experiments with complicated
observations are often uselessb ecauset hey fail to bridge things from a reverse
direction and correlate complex effects from simple causes. It is only through
the marriage of theory and experimentation that many claims of the complexity
of nature can withstand reasonablet ests. Simulation, then, becomesa form of experimentation
in a universe of theories. The primary purpose of this book is to
celebrate this fact.
How to ReadT his Book
While writing this book I had three different goals in mind . Thus, I think that
there are three different ways that one can read this book.
Preface
On one level, most of the chapters stand on their own and can be appreciated in
'
isolation from the others. I ve spent the better part of ten years collecting interesting
examples of computer simulations . If you are the sort of person who always wanted
a crisp and simple description of how to make a fractal , chaotic system, cellular
automaton , or neural network , then you may wish to skip around , dive right into
the example programs , and play with the simulations that look appealing .
The second way that you can read this book relates to how I learned about all
of the different topics covered in the chapters. While I am trained as a computer
scientist , I spent several years working among physicists and other scientists . Thus,
'
for better or worse, I ve seen many of the covered topics approached in different
manners. In this context , I believe that there is a captivating and interdisciplinary
connection between computation , fractals , chaos, complex systems, and adaptation .
'
I ve tried to explain each in terms of the others, so if you are interested in how one
part of the book relates to the others, then a more sequential reading of the book
may payoff for you.
For the third way, I believe that there is an overall pattern that ties all of the
part topics together into one coherent theme. This preface, the introduction , the
five postscript chapters, and the epilogue are my attempt to thread everything in
the book into one overall message.
If this book is a forest, then the first way of reading it is akin to poking at
individual trees. The second way is analogous to observing how nearby trees relate
to each another . The third way would equate to standing back and taking in the
'
whole forest at once. It doesn t really matter which path you take in exploring
these ideas, as I expect that most readers will stick to one path in preference to
the others. But if you happen to try all three paths, you may be rewarded with a
special type of understanding that not only relates each topic to the others but also
each topic as it is viewed from different perspectives.
Dealingw ith Difficult Subjects
The topics covered in this book demand varying amounts of sophistication from
you. Some of the ideas are so simple that they have formed the basis of lessons for
a third grade class. Other chapters should give graduate students a headache. This
is intentional . If you are confused by a sentence, section, or chapter , first see if the
'
glossary at the end of the book helps; if it doesn t, then by all means move on.
Regarding mathematical equations , there are many ways of looking at them.
If a picture is worth a thousand words (as well as a million pixels), then- as is
shown later- an equation can be worth a billion pictures . An equation can often
describe something so completely and compactly that any other type of description
becomes cumbersome by comparison . Nevertheless, I have done my best to make
every equation somewhat secondary to the supporting text .
Preface
If a particular equation confusesy ou, the first thing you should do is see if the
surrounding text gives you a little more insight into what the symbols mean. If
'
that doesnt work, then look to see if there is a picture that corresponds to the
'
equation. Finally, if all else fails, you should do what I do when I dont understand
" "
an equation: mentally substitute the sounds blah blah blah for the equation and
keep going. You would be surprised how often this works.
' ' '
Dont be discouragedi f you dont understand something. And pleased ont write
off a topic as being beyond your capabilities. If you are genuinely not interested in
a difficult section, so it goes. But if you are interested, it may happen that you will
retroactively understand something, given enough time an~ effort.
Personal Motivation
At the risk of being self-indulgent , I must admit that writing this book has been a
selfish labor of love. Every writer is told at one time or another to pick a target
reader and write for that one person. I wrote this book for the person I was around
ten or fifteen years ago.
I always wished for a book that combined all of these topics within one cover,
gave sufficient information to enable one to duplicate all of the programs , and at
the same time gave enough motivation to appreciate the more fundamental themes.
I also wanted this hypothetical book to have parts that could be understood on a
first reading, but additionally have sections that would be beyond my capabilities
for years to come. This book could be sporadically opened at a random page or
read sequentially . It would serve as a cookbook of computer recipes, be mostly
self-contained , be a basic primer on some common mathematics , and also serve as
a pointer to more fundamental texts.
Whether or not the book you are holding lives up to the idealized version that
'
I ve had in my head is mostly a personal matter . My second most important goal for
the book was for it to be a spark for those, like myself, who hack simply for the joy
" "
of hacking. Thus, while my idealized version of the book has been my bullseye,
'
I ll be happy just to hit the target on most counts since, in the end, the programs
are the heart of the book.
Ackno wledgments
Over the years many friends and colleagues have either directly or indirectly influenced
the outcome of this book. The only way who I can even come close to
acknowledging all that deserve it is to start from the beginning of my academic
career and work up to the present.
When I was an undergrad at Clemson University, Ed Page and Gene Tagliarini
brought me into their research group, gave me a slice of the funding pie, and in-
~ ~ Preface
troduced me to neural networks. During these years, Roy Pargas, Hal Grossman,
Steve Steve nson, Steven Hedetneimi, and Eleanor Hare all had a hand in nudging
'
me in the direction of graduate school and research. It s a rare department that
encouragesu ndergrads to pursue research to the point of publication, and without
this early exposure I am fairly certain that things would have turned out very
different for me.
During the late 1980s, the Center for Nonlinear Studies at Los Alamos National
Laboratory was a hotbed of researcha ctivity that attracted some of the most original
thinkers in the world. Chris Barnes and Roger Jones not only acted as mentors
but also managed to slip me through the back door at CNL~, which allowed me to
work with a truly amazing group of people. I would also like to single out Peter
Ford for inspiring me as a hacker. During this period Chris and Roger introduced
me to Y.-C. Lee, then the Senior Scientist of CNLS, whom I would later follow to
Maryland for graduate school.
At the University of Maryland I was a graduate student in the Computer Science
Department, a research assistant at the Institute for Advanced Computer Studies,
and indirectly funded, along with Y.-C. Lee and his group, in the Laboratory for
Plasma Research. I greatly benefited from TV orking with H.-H. Chen, G.-Z. Sun,
and the rest of the LPR group, which had a lot to do with honing my mathematical
skills. Y.-C. served as a c oadvis er on my thesis along with Jim Reggia. The other
'
members of my thesis committee, Bill Gasarch, Dianne OLeary, and Laveen Kanal,
all had a significant impact on my graduate studies.
While at Siemens Corporate ResearchI have had the good fortune to work on
exciting applications that directly relate to some of the topics in this book. I would
like to thank Thomas Grandke, our CEO, and Wolf-Ekkehard Blanz, the former
head of the Adaptive Information and Signal Processing Department, for creating
a laboratory environment that maintains the delicate balance between researcha nd
development, and for allowing me to pursue external academic activities such as
'
this book. SCRs librarian extraordinare, Ruth Weitzenfeld, kindly helped me track
down many of the referencesm entioned in this book, for which I am most grateful.
I would also like to acknowledgem y colleaguesa t SCR who have helped me to refine
my thoughts on learning theory. I offer a tip of the hat to Frans Coetzee, Chris
Darken, Russell Greiner, Stephen Judd, Gary Kuhn, Mike Miller, Tom Petsche,
Bharat Rao, Scott Rickard, Justinian Rosca, Iwan Santoso, Geoff Towell, and Ray
Watrous.
I am extremely grateful to the friends, family, and colleaguesw ho either have
read preliminary portions of this book or have indirectly contributed to it through
thoughtful discussions. While it is not possible to mention all of the people who
have influenced this book, I would specifically like to thank Marilyn and Stanford
Apseloff, David Bader, Bill Flake, Lee Giles, Sara Gottlieb, Stephen Hanson, Bill
Horne, Barry Johnson, Gary Kuhn, Barak Pear lmutter, Bill Regli, Scott Rickard,
Mark Rosenblum, Pat Vroom, Ray Watrous, and Tony Zador. Lee Giles initially
XV.1~~ Preface
tried to talk me out of writing this book, but his other good qualities more than
make up for this. Special thanks go to Barak Pearl mutter , who went well beyond
the call of duty by giving very detailed critiques of multiple drafts of this book. I
am also grateful for all of the comments provided by the anonymous reviewers. It
goes without saying that any remaining errors are all mine.
Since this book forms a core for the type of course that I have always wanted
to teach, I am grateful that several educators have given me a chance to tryout
some of the topics in this book on their students. In particular , Yannis Kevrekidis
graciously allowed me to lecture to his students at Princeton University on discrete
dynamical systems, and Laura Slattery and I built a set .of lessons for her third
grade class that demonstrated some of the principles of self-organization .
Very special thanks go to Scott Rickard , who has easily been my most eager
reader. He has read more versions of this book than anyone other than myself. I
am deeply grateful for his comments- which often provided a much-needed sanity
check on my writing - and for his encouragement .
Harry Stanton of the MIT Press was a very early supporter of this book. His
enthusiasm was so genuine and contagious that after talking with him, I couldn't
imagine going with any other publisher . Unfortunately , Harry did not live to see this
book into print , but his influence on the final product is still substantial . Bob Prior
and Deborah Cantor-Adams provided much critical assistance during the editorial
and production stages. Beth Wilson carefully copyedited the manuscript . And the
excellent production staff of the MIT Press made the project as painless as possible
by doing their jobs with the highest level of skill and professionalism . To all of
them, I give my thanks .
I have only the deepest thanks for my family . My parents, to whom this book
is dedicated , and my brother and sister, Greg Flake and Vicki Merchel, have a long
history of encouraging me to do crazy things , which has had a lot to do with giving
me the endurance to work through this project to the end.
" "
Finally , a simple thank you does not begin to express the gratitude that I owe
my darling wife, Lynn . So for the thousands of little things , the careful readings,
the lost weekends and evenings, and especially the pep talks, let me just say that I
am deeply and forever in her debt.
Introduction
1
The point of philoso Jpjhis to start with something so simple as not to seem worth stating ,
and to end with something so pamdoxical that no one will believe it.
- Bertrand Russell
Things should be as simple aspossible, but not simpler.
- Albert Einstein
REDUCTIONISM IS THE idea that a system can be understood by examining its
individual parts. Implicit in this idea is that one may have to examine something
at multiple levels, looking at the parts, then the parts of the parts, and so on. This
is a natural way of attempting to understand the universe. Indeed, the hierarchy
of sciencei s recognizably organized in this manner. For example, take the so-called
hard sciences. Biologists study living things at various levels ranging from the
whole body, to organ systems, down to cellular structure. Beyond the cellular level
lie chemical interactions and agents such as enzymes, which are organic chemical
catalysts, and amino acids, which are building blocks for proteins. This is the
domain of the chemist. To reduce things further, one would have to start looking
into how atoms and moleculesi nteract through chemical bonds that are dependent
upon the number of electrons in the outermost electron shell of an atom. But
what, exactly, are atoms? The physicist probes further into the nature of things by
shattering atoms into their constituent parts, which brings us to protons, neutrons,
and finally quarks. Ironically, at this level of understanding, scientists are dependent
on mathematical techniques that often bear little resemblancet o the reality that
we are familiar with.
To be sure, there is some overlap among scientific and mathematical fields that
is exemplified by disciplines that use tools common to other areas (e.g., organic
chemistry, biophysics, and quantum mathematics), but even these hybrid disciplines
have well-defined niches. Reductionism is a powerful way of looking at the universe.
But this begs a somewhat silly question: Since everything ultimately breaks down
'
to the quantum level, why arent all scientists mathematicians at the core? In such a
Introduction
'
world physicians would make diagnoses based on the patient s bodily quarks, which
makes about as much sense as building a house particle by particle . Nevertheless,
every scientist must possess some knowledge of the level one step more fundamental
than his or here specialty, but at some point reductionism must stop for science to
be effective.
Now, suppose that we wished to describe how the universe works. We could take
a reductionist 's approach and catalog all of the different types of objects that exist-
perhaps starting with galaxy clusters, hitting terrestrial life forms about midway
through , and then ending with subatomic particles - but would this approach really
" "
succeed in describing the universe? In making a large list <;>f things it is easy to
" "
forget that the manner in which things work more often than not depends on the
environment in which they exist. For example, we could describe the form of a duck
in excruciating detail , but this gives us only half of the story. To really appreciate
what a duck is, we should look at ducks in the air, in water, in the context of
what they eat or what eats them, how they court , mate, and reproduce , the social
structures they form, how they flock, and their need to migrate .
Looking back at the organization of the sciences, we find that at each level of understanding
, traditional scientists study two types of phenomena: agents (molecules,
cells, ducks, and species) and interactions of agents (chemical reactions , immune
system responses, duck mating , and evolution ). Studying agents in isolation is a
fruitful way of discovering insights into the form and function of an agent, but doing
so has some known limitations . Specifically , reductionism fails when we try to use it
in a reverse direction . As we shall see throughout this book, having a complete and
perfect understanding of how an agent behaves in no way guarantees that you will
be able to predict how this single agent will behave for all time or in the context of
other agents.
We have, then, three different ways of looking at how things work. We can take a
purely reductionist approach and attempt to understand things through dissection.
We also can take a wider view and attempt to understand whole collections at once
by observing how many agents, say the neurons in a brain , form a global pattern ,
such as human intelligence . Or we can take an intermediate view and focus attention
on the interactions of agents. Through this middle path, the interactions of agents
can be seen to form the glue that binds one level of understanding to the next level.
(cid:0)
1.1 Simplicity and Complexity
' '
Let s take this idea further by examining a single ant. By itself, an ant s behavior
is not very mysterious. There is a very small number of tasks that any ant has to
do in the course of its lifetime. Depending on its caste, an ant may forage for food,
'
care for the queens brood, tend to the upkeep of the nest, defend against enemieso f
the nest, or, in the special caseo f the queen, lay eggs. Yet when we consider the ant
colony as a whole, the behavior becomesm uch more complex. Army ant colonies
Introduction
often consist of millions of workers that can sweep whole regions clean of animal life.
The fungus-growing ants collect vegetable matter as food for symbiotic fungi and
then harvest a portion of the fungi as food for the colony. The physical structures
that ants build often contain thousands of passageways and appear mazelike to
human eyes but are easily navigated by the inhabitants . The important thing to
realize is that an ant colony is more than just a bunch of ants. Knowing how each
caste in an ant species behaves, while interesting , would not enable a scientist to
magically infer that ant colonies would possess so many sophisticated patterns of
behavior .
Instead of examining ants, we could have highlighted a number of interesting
examples: economic markets that defy prediction , the pattern recognition capabilities
'
of any of the vertebrates , the human immune system s response to viral and
bacterial attack , or the evolution of life on our planet . All of these examples are
emergent in that they contain simple units that , when combined , form a more complex
whole. This is a case of the whole of the system being greater than the sum of
the parts, which is a fair definition of holism- the very opposite of reductionism .
We also know that agents that exist on one level of understanding are very different
from agents on another level: cells are not organs, organs are not animals,
and animals are not species. Yet surprisingly the interactions on one level of understanding
are often very similar to the interactions on other levels. How so? Consider
the following :
. Why do we find self-similar structure in biology, such as trees, ferns, leaves,
and twigs? How does this relate to the self-similarity found in inanimate
objects such as snowflakes, mountains , and clouds? Is there some way of gen-
eralizing the notion of self-similarity to account for both types of phenomena?
'
. Is there a common reason why it s hard to predict the stock market and also
hard to predict weather? Is unpredictability due to limited knowledge or is it
somehow inherent in these systems?
. How do collectives such as ant colonies, human brains, and economic markets
self-organize to create enormously complex behavior that is much richer than
the behavior of the individual component units?
. What is the relationship between evolution , learning , and the adaptation
found in social systems? Is adaptation unique to biological systems? What is
the relationship between an adaptive system and its environment ?
The answers to all of these questions are apparently related to one simple fact:
Nature is frugal . Of all the possible rules that could be used to govern the interactions
among agents, scientists are finding that nature often uses the simplest . More
than that , the same rules are repeatedly used in very different places. To see why,
consider the three attributes below that can be used to describe the interactions of
agents.
Introduction
Collections , Multiplicity , and Parallelism Complex systems with emergent
properties are often highly parallel collections of similar units. Ant colonies owe
much of their sophistication to the fact that they consist of many ants. This is
obvious, but consider the implications. A parallel system is inherently more efficient
than a sequential system, since tasks can be performed simultaneously and more
readily via specialization. Parallel systems that are redundant have fault tolerance.
If some ants die, a task still has a good chance of being finished since similar ants
can substitute for the missing ones. As an added bonus, subtle variation among
the units of a parallel system allows for multiple problem solutions to be attempted
simultaneously. For example, gazelles as a species activel):' seek a solution to the
problem of avoiding lions. Someg azellesm ay be fast, others may be more wary and
timid, while others may be more aggressivea nd protective of their young. A single
gazelle cannot exploit all of the posed solutions to the problem of avoiding lions
simultaneously, but the speciesa s a whole can. The gazelle with the better solution
stands a better chanceo f living to reproduce. In such a case, the speciesa s a whole
can be thought of as having found a better solution through natural selection.
Iteration , Recursion , and Feedback For living things, iteration correspondst o
reproduction. We can also expand our scope to include participants of an economic
system, antibodies in an immune system, or reinforcement of synapsesi n the human
brain. While parallelism involves multiplicity in a space, iteration involves a form
of persistence in time. Similarly, recursion is responsible for the various types of
self-similarity seen in nature. Almost all biological systems contain self-similar
structures that are made through recurrent proces ses, while many physical systems
contain a form of functional self-similarity that owes its richness to recursion. We
will also seet hat systemsa re often recurrently coupled to their environment through
feedback mechanisms. While animals must react according to their surroundings,
they can also change this environment, which means that future actions by an
animal will have to take these environmental changesi nto account.
Adaptation , Learning , and Evolution Interesting systems can change and
adapt. Adaptation can be viewed as a consequence of parallelism and iteration in
a competitive environment with finite resources. In this case the combination of
multiplicity and iteration acts as a sort of filter . We see this when life reproduces
because it is fit, companies survive and spawn imitations because they make money,
antibodies are copied because they fight infections , and synapses are reinforced
because of their us efulness to the organism . With feedback mechanisms in place
between an agent and an environment , adaptation can be seen as forming a loop in
the cause and effect of changes in both agents and environments .
There are certainly many more ways to describe the interactions of agents; however
, multiplicity , iteration , and adaptation by themselves go a long way indescrib -