Table Of ContentSpringer Series in Synergetics
Editor: Hermann Haken
Synergetics, an interdisciplinary field of research, is concerned with the cooperation of individual
parts of a system that produces macroscopic spatial, temporal or functional structures. It deals with
deterministic as well as stochastic processes.
Volume 40 Information and Self-Organization
A Macroscopic Approach to Complex Systems
By H. Haken
Volume 41 Propagation in Systems Far from Equilibrinm
Editors: J. E. Wesfreid, H. R. Brand,
P. Manneville, G. Albinet, N. Boccara
Volume 42 Neural and Synergetic Computers
Editor: H. Haken
Volume 43 Cooperative Dynamics in Complex Physical Systems
Editor: H. Takayama
Volume 44 Optimal Structures in Heterogeneous Reaction Systems
Editor: P. J. Plath
Volume 45 Synergetics of Cognition
Editors: H. Haken, M. Stadler
Volume 46 Theories of Immune Networks
Editors: H. Atlan, 1. R. Cohen
Volume 47 Relative Information Theories and Applications
By G. Jumarie
Volumes 1-39 are listed on the back inside cover
Guy Jumarie
Relative Information
Theories and Applications
With Forewords by O. J. Klir and H. Haken
Springer-Verlag Berlin Heidelberg New York
London Paris Tokyo Hong Kong
Professor Guy Jumarie
Department of Mathematics and Computer Science,
University of Quebec at Montreal, P. O. Box 8888, St. A,
Montreal (Quebec), Canada, H3C 3P8
Series Editor:
Prof. Dr. Dr. h. c. Hermann Haken
lnstitut ftir Theoretische Physik und Synergetik der Universitat Stuttgart,
Pfaffenwaldring 57 flV, D-7000 Stuttgart 80, Fed. Rep. of Germany and
Center for Complex Systems, Florida Atlantic University,
Boca Raton, FL 33431, USA
ISBN -13: 978-3-642-84019-7 e-ISBN -13: 978-3-642-84017-3
DOl: 10.1007/978-3-642-84017-3
Library of Congress Cataloging-in-Publication Data. Jumarie, Guy. Relative information: theories and applications
1 Guy Jumarie. p. cm. - (Springer series in synergetics; v. 47) Includes bibliographical references. ISBN
-13:978-3-642-84019-71. Information theory. I. Title. II. Series. Q360.J84 1990 003'.54 - dc20 89-26347
This work is subject to copyright. All rights are reserved, whether the whole or part of the material is concerned,
specifically the rights of translation, reprinting, re-use of illustrations, recitation, broadcasting, reproduction on
microfilms or in other ways, and storage in data banks. Duplication of this publication or parts thereof is only per
mitted under the provisions of the German Copyright Law of September 9, 1965, in its current version, and a
copyright fee must always be paid. Violations fall under the prosecution act of the German Copyright Law.
© Springer-Verlag Berlin Heidelberg 1990
Softcover reprint of the hardcover 1st edition 1990
The use of registered names, trademarks, etc. in this publication does not imply, even in the absence of a specific
statement, that such names are exempt from the relevant protective laws and regulations and therefore free for general
use.
1YPesetting: ASCO Trade lYpesetting Limited, Hong Kong
2154/3150(3011)-543210 - Printed on acid-free paper
So all my best is dressing old words new,
Spending again what is already spent.
Shakespeare, Sonnet LXXVI
As far as the laws of mathematics refer to reality,
they are not certain, and as far as they are certain,
they do not refer to reality.
Einstein
Foreword
For four decades, information theory has been viewed almost exclusively as a
theory based upon the Shannon measure of uncertainty and information, usually
referred to as Shannon entropy. Since the publication of Shannon's seminal paper
in 1948, the theory has grown extremely rapidly and has been applied with varied
success in almost all areas of human endeavor.
At this time, the Shannon information theory is a well established and
developed body of knowledge. Among its most significant recent contributions
have been the use of the complementary principles of minimum and maximum
entropy in dealing with a variety of fundamental systems problems such as predic
tive systems modelling, pattern recognition, image reconstruction, and the like.
Since its inception in 1948, the Shannon theory has been viewed as a restricted
information theory. It has often been argued that the theory is capable of dealing
only with syntactic aspects of information, but not with its semantic and
pragmatic aspects. This restriction was considered a v~rtue by some experts and
a vice by others. More recently, however, various arguments have been made that
the theory can be appropriately modified to account for semantic aspects of in
formation as well. Some of the most convincing arguments in this regard are in
cluded in Fred Dretske's Know/edge & Flow of Information (The M.LT. Press,
Cambridge, Mass., 1981) and in this book by Guy lumarie.
This book is a comprehensive presentation of lumarie's approach to informa
tion theory, which he developed during the last decade or so. His principal goal
is to develop an information theory that incorporates not only syntactic aspects
of information but also its semantic aspects. Contrary to other authors, however,
he attempts to achieve this goal by an appropriate modification of the Shannon
theory. lumarie's approach is characterized by introducing observer's viewpoints
into the formulation of the theory. He proposes in the book two ways of ac
complishing this, one based upon a model of observation with informational in
variance (Chap. 4) and one based on a slight modification of the concept of trans
information in the Shannon theory (Chap. 5).
One of the most significant contributions of this book is a careful comparison
of discrete and continuous entropies (Chap. 6), which one cannot find in such
detail anywhere in the literature. lumarie argues that the continuous entropy
(often called the Boltzmann entropy) is more fundamental than the discrete (or
Shannon) entropy. Although this issue will probably remain controversial for
some time, his arguments are well developed and convincing.
In the rest of the book, lumarie develops a unified formalism for information
theory that covers both syntax and semantics and integrates both discrete and
continuous entropies (Chap. 7), and shows how this formalism can be applied for
VIII Foreword
characterizing the entropy of form and pattern in a way which is quite different
from the Kolmogorov entropy (Chap. 8), and for generalizing the classical statisti
cal decision theory (Chap. 9).
In spite of the fact that some topics in the book may be viewed as controver
sial, the book consists of highly original and thought provoking material and,
consequently, it should be of interest to anyone interested in the concept of infor
mation.
Jumarie restricts his formulation primarily to information conceptualized in
terms of probability theory, even though there are some hints in the book about
other possible mathematical frameworks, such as possibility theory or fuzzy set
theory. The significance of these new frameworks (in particular fuzzy set theory
and the Dempster-Shafer theory of evidence) is that they allow us to distinguish
several different types of uncertainty and the associated information. This implies
that information must be viewed as a multidimensional entity, as discussed in
detail in Fuzzy Sets, Uncertainty, and Information (Prentice-Hall, Englewood
Cliffs, N.J., 1988), which I coauthored with Tina Folger.
The conception of information as a multidimensional entity opens many new
avenues. At this time, however, it captures only syntactic aspects of information.
In order to develop a genuine multidimensional information theory, semantic
aspects will have to be incorporated in some manner. Jumarie shows us in this
book how this can be accomplished for the classical, one-dimensional informa
tion theory based upon the Shannon entropy. His key ideas will undoubtedly be
important as guiding principles for accomplishing the same within the larger
framework of multidimensional information theory. This, perhaps, is the most
significant contribution of Jumarie's work.
George 1 Klir
Foreword
In modern science we witness an enormous production of new theories and ex
perimental results. In such a situation, unifying ideas become more and more im
portant. One of these ideas is the concept of information that was originally in
troduced by Shannon to deal with the capacity of communication channels.
Because of its close connection with entropy, it soon became an important tool
for the discussion of problems of statistical mechanics. More recently, in my book
"Information and Self-Organization", I showed that the concept of information
plays a fundamental role in the spontaneous formation of spatial, temporal, or
functional structures.
The present book by Prof. Guy Jumarie, who has made important contribu
tions to the field of information, will provide the reader with an in-depth presen
tation of this field. Professor Jumarie carefully develops the theory of informa
tion and its various manifestations, taking into account both discrete and con
tinuous random variables. His book shows in an impressive way how the various
concepts can be mathematized, interpreted, and related to one another. He also
makes important contributions to the problem of subjectivity. In addition, he
makes novel proposals on how to define entropy of trajectories, of deterministic
functions, and of forms. These chapters lead us to the forefront of research in a
field where quite surprising results can surely still be expected.
I am certain that this book will be most useful, both to university faculty
members and to students, and not only in the field of classical information
theory; all those interested in the phenomena of self-organization and in the
mathematics dealing with complex systems, should benefit from this stimulating
treatment.
H. Haken
Preface
My first paper (in English) on relative information appeared in 1975, and since
this date, I have endeavoured to promote and to develop this idea. The present
work follows my 1986 essay, Subjectivity, Information, Systems: Introduction to
a Theory of Relativistic Cybernetics which itself had followed and largely replac
ed my 1980 essay in French, Subjectivite, Information, Systems. Synthese pour
une cybernetique relativiste. The French publication was on the verge of being
philisophy of science; its English counterpart, however, was technical for the most
part but directed towards problems related to cybernetics and general systems; and
the present book is frankly technical in the sense of engineering and physics.
My central concern along this trajectory has been always the same: it should
be possible to derive a theory of information which explicity takes account of the
viewpoint of the observer, and which, furthermore, is consistent with the
mathematical theory of communication as initiated by Shannon. In the maze of
the possible solutions, I continually came across paths which led me to the same
answer: a local model of relative information is quite suitably defined by the so
called Lorentz-Poincare equations of relativistic physics.
At that time, my unorthodox ideas puzzled and sometimes shocked many
scientists, but as time elapsed, it appeared that many new results, derived in more
classical ways, were bringing indirect support to this approach. For instance,
several generalized informational entropies which have been obtained recently by
mathematicians can be thought of as merely special cases of my concept of
relative entropy.
As a matter of fact, the thesis which we shall advocate here is that the theory
of Shannon contains the key to derive its generalization in the form of the
Lorentz-Poincare equation.
To many mathematicians, information theory is mainly a motivation for
mathematical exercises in the form of axioms, solution, uniqueness of solution.
It is not so here. Indeed, we would like to address as large a readership as possible
and consequently the mathematical prerequisites for understanding the book are
small: calculus, elementary probability, the basic elements of the Shannon infor
mation theory ... and much critical reflection. In order to facilitate easy reading,
each chapter is almost self-contained and could be read separately much like a
technical paper. Indeed, it is the relative independence of the various topics that
enabled the book to be written in this way.
It is a very great pleasure for me to thank Professor H. Haken who accepted
the present monograph in the Springer Series in Synergetics and invited me to the
Institute of Theoretical Physics at Stuttgart.
XII Preface
Professor G. J. Klir was kind enough to read the manuscript and to make some
comments in the form of a foreword, and I am pleased to thank him.
I have had a large number of discussions with Professor G. Battail of the
"Ecole Nationale Superieure des Telecommunications, Paris" and I acknowledge
with thanks this indirect contribution.
Other invaluable help since my arrival in Canada in 1971, was provided by the
National Research Council of this country which regularly made research grants
to me. Many thanks to this institution.
I express my gratitude to Dr. Angela Lahee for her patience in understanding
and rectifying my latin English (indeed we are natives of Guadeloupe in the West
Indies, and thus French speaking and thinking!).
And last but not least, many thanks to a fine person (Nadia) who has had to
suffer from a very strange husband obsessed by the relativity of truth.
G. Jumarie
