Codes, Systems, and Graphical Models

520 Pages·2001·17.558 MB·English

by  Tom Richardson, Rüdiger Urbanke (auth.), Brian Marcus, Joachim Rosenthal (eds.)

The IMA Volumes
in Mathematics
and its Applications
Volume 123
Series Editor
Willard Miller, Jf.
Springer
New York
Berlin
Heidelberg
Barcelona
Hong Kong
London
Milan
Paris
Singapore
Tokyo
Institute for Mathematics and
its Applications
IMA
The Institute for Mathematics and its Applications was estab
lished by a grant from the National Science Foundation to the University
of Minnesota in 1982. The IMA seeks to encourage the development and
study of fresh mathematical concepts and questions of concern to the other
sciences by bringing together mathematicians and scientists from diverse
fields in an atmosphere that will stimulate discussion and collaboration.
The IMA Volumes are intended to involve the broader scientific com
munity in this process.
Willard Miller, J r., Professor and Director
* * * * * * * * * *
IMA ANNUAL PROGRAMS
1982-1983 Statistical and Continuum Approaches to Phase Transition
1983-1984 Mathematical Models for the Economics of Decentralized
Resource Allocation
1984-1985 Continuum Physics and Partial Differential Equations
1985-1986 Stochastic Differential Equations and Their Applications
1986-1987 Scientific Computation
1987-1988 Applied Combinatorics
1988-1989 Nonlinear Waves
1989-1990 Dynamical Systems and Their Applications
1990-1991 Phase Transitions and Free Boundaries
1991-1992 Applied Linear Algebra
1992-1993 Control Theory and its Applications
1993-1994 Emerging Applications of Probability
1994-1995 Waves and Scattering
1995-1996 Mathematical Methods in Material Science
1996-1997 Mathematics of High Performance Computing
1997-1998 Emerging Applications of Dynamical Systems
1998-1999 Mathematics in Biology
1999-2000 Reactive Flows and Transport Phenomena
2000-2001 Mathematics in Multimedia
2001-2002 Mathematics in the Geosciences
2002-2003 Optimization
Continued at the back
Brian Marcus Joachim Rosenthal
Editors
Codes, Systems, and
Graphical Models
With 93 Illustrations
Springer
Brian Marcus Joachim Rosenthal
IBM Almaden Research Center, K65-802 Department of Mathematics
650 Harry Rd. University of Notre Dame
San Jose, CA 95120 Notre Dame, IN 46556-5683
USA USA
e-mail: [email protected] e-mail: [email protected]
Mathematics Subject Classification (2000): lIT, 37B, 65F, 68Q, 93B, 93C, 94A, 94B
Library of Congress Cataloging-in-Publication Data
Codes, systems, and graphical models I editors, Brian Marcus, Joachim Rosenthal.
p. cm. - (The IMA volumes in mathematics and its applications; 123)
Includes bibliographical references and index.
ISBN 0-387-95173-3
1. Coding theory--Congresses. 2. System theory--Congresses. 3. Symbolic
dynamics--Congresses. I. Marcus, Brian, 1949- II. Rosenthal, Joachim, 1961- III.IMA
Workshop on Codes, Systems, and Graphical Models (1999) IV. Series.
QA268 .C63 2001
003'.54--dc21 00-052274
Printed on acid-free paper.
© 2001 Springer-Verlag New York, Inc.
All rights reserved. This work may not be translated or copied in whole or in part without the
written permission of the publisher (Springer-Verlag New York, Inc., 175 Fifth Avenue, New York,
NY 10010, USA), except for brief excerpts in connection with reviews or scholarly analysis. Use
in connection with any form of information storage and retrieval, electronic adaptation, computer
software, or by similar or dissimilar methodology now known or hereafter developed is forbidden.
The use of general descriptive names, trade names, trademarks, etc., in this publication, even if the
former are not especially identified, is not to be taken as a sign that such names, as understood by
the Trade Marks and Merchandise Marks Act, may accordingly be used freely by anyone.
Authorization to photocopy items for internal or personal use, or the internal or personal use of
specific clients, is granted by Springer-Verlag New York, Inc., provided that the appropriate fee is
paid directly to Copyright Clearance Center, 222 Rosewood Drive, Danvers, MA 01923, USA (Tele
phone: [phone]), stating the ISBN number, the title of the book, and the first and last page
numbers of each article copied. The copyright owner's consent does not include copying for general
distribution, promotion, new works, or resale. In these cases, specific written permission must first
be obtained from the publisher.
Production managed by A. Orrantia; manufacturing supervised by Joe Quatela.
Camera-ready copy prepared by the IMA.
Printed and bound by Sheridan Books, Inc., Ann Arbor, MI.
Printed in the United States of America.
9 8 765 432 1
ISBN 0-387-95173-3 SPIN 10789096
Springer-Verlag New York Berlin Heidelberg
A member of BertelsmannSpringer Science+Business Media GmbH
FOREWORD
This IMA Volume in Mathematics and its Applications
CODES, SYSTEMS, AND GRAPHICAL MODELS
is based on the proceedings of a very successful 1999 IMA Summer Program
with the same title.
I would like to thank G. David Forney, Jr. (Massachusetts Institute
of Technology), Brian Marcus (IBM Almaden Research Center), Joachim
Rosenthal (University of Notre Dame), and Alexander Vardy (University
of California, San Diego) for their excellent work as organizers of the two
weeks summer program. Special thanks to Brian Marcus and Joachim
Rosenthal for their role as editors of the proceedings.
I also take this opportunity to thank the National Science Foundation
(NSF) and the National Security Agency (NSA), whose financial support
made the workshop possible.
Willard Miller, Jr., Professor and Director
Institute for Mathematics and its Applications
University of Minnesota
400 Lind Hall, 207 Church St. SE
Minneapolis, MN 55455-0436
612-624-6066, FAX [phone]
[email protected]
World Wide Web: http://www.ima.umn.edu
v
PREFACE
Codes and systems based on graphical models form a prominent area
of research in each of the following subjects: coding theory, systems theory,
symbolic dynamics and automata theory. The aim of the IMA Workshop
on Codes, Systems and Graphical Models (August 2-13, 1999) was to bring
together mathematicians, computer scientists, and electrical engineers in
these subjects to learn how techniques from one area might be applied to
problems in the other areas.
The workshop was divided into two weeks, each with a different focus.
The first week, organized by Dave Forney and Alex Vardy, focused on
codes on graphs and iterative decoding. The second week, organized by
Brian Marcus and Joachim Rosenthal, turned to connections among coding
theory, system theory and symbolic dynamics.
A major goal of coding theory is to find a class of codes, together
with corresponding decoding algorithms, that realize in a practical way the
promise of Shannon's coding theorems. Nearly forty years ago, Gallager
introduced low density parity check codes (LDPC's), with iterative decod
ing algorithms, for this purpose. Recently, the subject of iterative decoding
has enjoyed a burst of intense activity with the discovery of turbo codes
and the realization that both turbo codes and LDPC's perform remark
ably well. Also, it has recently been recognized that the iterative decoding
algorithms used for these codes are instances of belief propagation, an algo
rithm originally developed by Pearl in artificial intelligence. This has led to
much work devoted to understanding and exploiting this connection. While
Pearl's algorithm is guaranteed to work well on graphs without cycles, it
has been demonstrated empirically to work well on graphs with cycles. A
central impetus of current work on iterative decoding is to understand why
this is so. The first week of the workshop focused on these issues.
Coding theory, system theory and symbolic dynamics have much in
common. Among the central themes in each of these subjects are the
construction of state space representations, understanding of fundamen
tal structural properties of sequence spaces, construction of input/output
systems, and understanding the special role played by algebraic structure.
The second week of the workshop was devoted to understanding how these
themes are treated in each of the subjects and how ideas from one subject
may be useful in the others. This continued the interactions initiated in an
IEEE workshop on codes, systems and symbolic dynamics in 1993.
This volume is a collection of papers on the subjects of the workshop.
Our intention was to collect in one place many pieces of work on these
subjects; some were presented at the workshop, while others were not.
vii
viii PREFACE
Part 1 contains papers with significant tutorial or overview content.
Parts 2 and 3 contain papers on the main themes of week 1, 'Codes on
Graphs' and 'Iterative Decoding'(although Part 3 contains some system
theoretic papers on non-iterative decoding). Parts 4 and 5 contain papers
on the main themes of week 2, 'Convolutional Codes' and 'Symbolic Dy
namics and Automata Theory'.
In addition to the material collected in this volume, the reader can
find copies of slides of many of the workshop talks as well as an exceptional
trove of related bibliographic material on the IMA Web site at:
http://www.ima.umn.edu/csg
For a collection of related papers, the reader may also wish to consult
the Special Issue on 'Codes on Graphs and Iterative Algorithms' of the
IEEE Transactions on Information Theory to be published in 200l.
Finally, we are happy to acknowledge the excellent organization and
facilities provided by the IMA as well as financial support from the IMA, the
IEEE Information Theory Society and the U. S. National Security Agency.
Brian Marcus (IBM Almaden Research Center)
Joachim Rosenthal (University of Notre Dame)
CONTENTS
Foreword ............................................................. v
Preface ............................................................. vii
PART 1. OVERVIEWS
An introduction to the analysis of iterative coding
systems ............................................................... 1
Tom Richardson and Rudiger Urbanke
Connections between linear systems and convolutional
codes ................................................................ 39
Joachim Rosenthal
Multi-dimensional symbolic dynamical systems. . . . . . . . . . . . . . . . . . . . . .. 67
Klaus Schmidt
PART 2. CODES ON GRAPHS
Linear-congruence constructions of low-density
parity-check codes. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. 83
J. Bond, S. Hui, and H. Schmidt
On the effective weights of pseudocodewords for
codes defined on graphs with cycles. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. 101
G. David Forney, Jr., Ralf Koetter,
Frank R. Kschischang, and Alex Reznik
Evaluation of Gallager codes for short block
length and high rate applications. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. 113
David J.C. MacKay and Matthew C. Davey
Two small Gallager codes ........................................... 131
David J. C. MacKay and Matthew C. Davey
Mildly non-linear codes. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. 135
Alan Parks
Capacity-achieving sequences. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. 153
M.A. Shokrollahi
ix
x CONTENTS
Hypertrellis: A generalization of trellis and factor
graph ............................................................... 167
Wai Ho Mow
PART 3: DECODING TECHNIQUES
BSC thresholds for code ensembles based on
"typical pairs" decoding. . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. 195
Srinivas Aji, Hui Jin, Aamod Khandekar,
David J.C. MacKay, and Robert J. McEliece
Properties of the tailbiting BCJR decoder ........................... 211
John B. Anderson and Kemal E. Tepe
Iterative decoding of tail-biting trellises and
connections with symbolic dynamics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. 239
G. David Forney, Jr., Frank R. Kschischang,
Brian Marcus, and Selim Tuncel
Algorithms for decoding and interpolation. . . . . . . . . . . . . . . . . . . . . . . . . .. 265
M argreet K uijper
An algebraic description of iterative decoding schemes. . . . . . . . . . . . . .. 283
Elke Offer and Emina Soljanin
Recursive construction of Gr6bner bases for the
solution of polynomial congruences. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. 299
Henry O'Keeffe and Patrick Fitzpatrick
On iterative decoding of cycle codes of graphs. . . . . . . . . . . . . . . . . . . . . .. 311
Gilles Zemor
PART 4. CONVOLUTIONAL CODES AND CODES OVER RINGS
Convolutional codes over finite Abelian groups:
Some basic results. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. 327
Fabio Fagnani and Sandro Zampieri
Symbolic dynamics and convolutional codes ......................... 347
Bruce Kitchens
Linear codes and their duals over artinian rings. . . . . . . . . . . . . . . . . . . .. 361
Thomas Mittelholzer
CONTENTS xi
Unit memory convolutional codes with maximum
distance . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. 381
Roxana Smarandaehe
Basic properties of multidimensional convolutional
codes ............................................................... 397
Paul Weiner
PART 5. SYMBOLIC DYNAMICS AND AUTOMATA THEORY
Length distributions and regular sequences. . . . . . . . . . . . . . . . . . . . . . . . .. 415
Frederique Bassino, Marie-Pierre Beal,
and Dominique Perrin
Handelman's theorem on polynomials with positive
multiples. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. 439
Valerio de Angelis and Selim Tuneel
Topological dynamics of cellular automata ........................... 447
Petr Kurka
A spanning tree invariant for Markov shifts .......................... 487
Douglas Lind and Selim Tuneel
List of workshop participants. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. 499