Probabilistic Analysis of Algorithms: On Computing Methodologies for Computer Algorithms Performance Evaluation

253 Pages·1987·7.26 MB·English

by  Micha Hofri

Texts and Monographs in Computer Science
Editor
David Gries
Advisory Board
F.L. Bauer
S.D. Brookes
C.E. Leiserson
M. Sipser
Texts and Monographs in Computer Science
Suad Alagic Relational Database Technology
Suad Alagic and Michael A. Arbib The Design of Well-Structured and Correct Programs
S. Thomas Alexander Adaptive Signal Processing: Theory and Applications
Michael A. Arbib, A.J. Kfoury, and Robert N. Moll A Basis for Theoretical Computer Science
Michael A. Arbib and Ernest G. Manes Algebraic Approaches to Program Semantics
F.L. Bauer and H. Wiissner Algorithmic Language and Program Development
Kaare Christian The Guide to Modula-2
Edsger W. Dijkstra Selected Writings on Computing: A Personal Perspective
Nissim Francez Fairness
Peter W. Frey. Ed. Chess Skill in Man and Machine, 2nd Edition
R. T. Gregory and E. V. Krishnamurthy Methods and Applications of Error-Free Computation
David Gries. Ed. Programming Methodology: A Collection of Articles by Members of IFIP WG2.3
David Gries The Science of Programming
Micha Hofri Probabilistic Analysis of Algorithms
A.J. Kfoury. Robert N. Moll. and Michael A. Arbib A Programming Approach to Computability
E. V. Krishnamurthy Error-Free Polynomial Matrix Computations
Franco P. Preparata and Michael Ian Shamos Computational Geometry: An Introduction
Brian Randell. Ed. The Origins of Digital Computers: Selected Papers
Arto Salomaa and Matti Soittola Automata-Theoretic Aspects of Formal Power Series
Jeffrey R. Sampson Adaptive Information Processing: An Introductory Survey
J.T. Schwartz, R.B. Dewar. E. Dubinsky, and E. Schonberg Programming with Sets: An Introduction to
SETL
William M. Waite and Gerhard Goos Compiler Construction
Niklaus Wirth Programming in Modula-2
Micha Hofri
Probabilistic Analysis
of Algorithms
On Computing Methodologies for Computer
Algorithms Performance Evaluation
With 14 Illustrations
Springer-Verlag New York Berlin Heidelberg
London Paris Tokyo
Micha Hofn
Departmenl of Computer Science
Technion- lIT
Haifa 32000
Israel
Series Editor
David Gries
Department of CompUler Science
Cornell University
Ithaca. New York 14853
USA
Library of Congress Cataloging·in·Publication.Data
Hofri. Micha.
Probabilistic analysis of algorithms.
(Te~ts and monographs in romputl'r science)
Bibliography: p.
IlICludcs inde~.
I. Ele<:tronic digital computers- Programming.
2. Algorithms. 3. Probabilities. I. Title. II. Series.
OA76.6.1159 1987 005.1'2 87·16581
C 1987 by Springer.verlag New York Inc.
Softcover reprint of the hardcover 1st edition 1987
All rights reserved. This work may 001 be translated or copied in whole or in pan without the: written
permi&sion of the publisher (Springer. Verlag. 115 Fifth Avenue. New Yorir:. New York 10010.
USA). e.\tepl for brief uccrpu in conection with reviews or scholarly analysis. Use in connection
with any form of information stonr.ge and retrieval. elcctronic adaptation. computer software. or by
similar or dissimilar methodology now known or hereafter developed is forbidden.
TlIc use of general descriptive names. trade names. trademans. etc. in this publkation. even if the
former are not especially identified. is not to be taken as a sign thai such names. as understood by
the Trade Mans and Merchandise Mans Act. may accordingly be used freely by anyone.
Tut prepared by the author in camera·ready form on Laser Writer Plus.
Printed and bound by R.R. Donnclley &. Sons. Harrisonburg. Virginia.
Printed in the United States of America.
98765432 1
ISBN·13: 978·146i2·916()..2 e·ISBN·13: 978·146124800-2
001: 10.1007/978·146124800-2
- 'llill
'19n illW
j?n~'l
PREFACE
This book originated from notes used for a course I gave in the Department of
Computer Science of the Technion during the Spring term of 1986. The course had
the same title as this volume. It was intended for both graduates and
undergraduates who are close to finishing their studies and are looking for a
suitable area in which to specialize. This was one of the factors that determined
the coverage of the course - and consequently of the text as well; the latter
contains much material that I had no time to discuss in class, as well as a good
deal more exercises than the students could be asked to tackle.
Until quite recently, "analysis of algorithms" was nearly synonymous with
determining the "complexity class" of an algorithm. This has the objective, most
often, of finding whether in all cases the running time (or storage requirements) of
the algorithm operation is or is not bounded by some specified function of the size
of a suitably devised representation of the problem. It usually boils down to the
consideration of some extreme, especially crafted problem instances. The
realization that there is more one could say to characterize the cost of using an
algorithm is probably due to the influence of Knuth's series on "The Art of
Computer Programming", which started out in 1968. There, clearly, the operation
of algorithms was shown to be associated with probabilistic concepts and
processes.
Random elements, and hence the call for stochastic analysis, may enter
algorithms in essentially two ways. On the one hand, we find the so called
"probabilistic algorithms", such that choose part of their actions on the basis of
random elements, explicitly introduced into the algorithm specification (pseudo
random numbers, simulated coin flipping and the like). Numerous algorithms of
this class were recently developed, some showing prowess well beyond anything
one has believed hitherto possible (primality testing algorithms provide a good
example). On the other hand, we find the operation of deterministic algorithms on
input data over which some probability measure can be stipulated. While the
viii Preface
sources of the randomness present a true dichotomy, the required analyses turn out
typically to be of the same nature in both cases. Among the algorithms for which
we provide detailed analyses, the reader will find examples of both varieties.
While the analyses proper are similar, we show in Chapter 1 that the second type
brings up methodological and conceptual problems that the first case need not
entail. The difficulty there may be phrased as lending substance to the notion of
two algorithms having the same complexity "on the average", or "in distribution".
The problem may also be seen to reside in the attribution of a priori probability
measures to the input instance space. At the time of writing there is no coherent
accepted theory or even taxonomy for these vexing issues, comparable to standard
complexity theory; we shall mostly skirt them, using reasonable - sometimes
seemingly facile -assumptions, invoking naturalness as our guideline.
The probabilistic analysis of algorithms, as a discipline, draws on a fair
number of branches of mathematics. Principally: probability theory (especially as
applied to stochastic processes), graph theory, combinatorics, real and complex
analysis, and occasionally algebra, number theory, computation theory,
operational calculus and more. It was unreasonable to expect the students to have
more than a cursory knowledge of most of the techniques we used, so much of the
time was given over to introducing and exploring these methods as we went along.
Arranging the text so it could be conveniently used both as a text and as a
reference posed a problem which was solved by departing in the book version
from the order of the class presentation to a large extent, collecting most of the
methodological material in Chapter 2.
The prerequisites that were assumed are basic courses in discrete
mathematics, calculus (including a smattering of differential equations), linear
algebra, probability theory, data structures and graph algorithms; all these being
required courses in our department. The last two are assumed to impart to the
students some algorithmic literacy.
The emphasis throughout is on the analytic and probabilistic aspects of our
activity, rather than the algorithmic ones. Good texts for the latter exist and are
multiplying satisfactorily; references are scattered throughout the book. For a
while I occupied myself with the question whether the text should have some
chapter on the needed tools from probability theory, or an introduction to the
subject, or at least an appendix - and decided against it. This subject has been
royally served in the last decade or two, with very good books, at every level of
depth and sophistication. Indeed, since I am not even aware of all the good sources
that exist now, the reference section mentions only those recently used in my
work, as well as for the preparation of this text. What I did include is a section -
2.6 - containing a few results from probability theory that are outside the normal
curricula and that I had occasion to witness their usefulness.
Computer algorithms deal with discrete quantities, and estimating their
operations is very often reducible to a counting problem, in one guise or another.
Hence the above mentioned need of combinatorics. Chapter 2 contains many
Preface ix
combinatorial tools and concepts, either in the text proper or in exercises. Further
sources for this important area are discussed in Chapter 1.
In addition to Chapter 1, which is used mainly to establish the context of the
book, and the methodological Chapter 2 mentioned above, a few sample analyses
are collected in the subsequent three chapters. The choice of the particular
algorithms to analyze in detail was another non-trivial issue. The quest was easier
for the first, simpler ones which are used mainly to introduce the nomenclature
and a point of view. Here I could capitalize on ideas found in the so-far rather
scanty literature on the subject; but beyond that I was mainly guided by personal
preferences and experience, and to some extent by requests of students in the
class, inasmuch as they agreed with the types of analysis I were interested in or
able to display. The last reservation hints at some of the difficulty in teaching this
subject: for some interesting analyses the sheer technical complexity (as distinct
from their mathematical profundity) is such that presenting them in class is an
unjustifiable harassment of the students (and the teacher too, no doubt). An
analysis in which I was involved could serve as an illustration of such an
unpleasantness (Fayolle et aI., 1986).
The exercises are mostly extensions of the text, and sometimes were used to
avoid giving much space to technicalities that tax the endurance of the reader and
that the student should anyway gain some proficiency in performing on his own.
Occasionally a large exercise presents a complete research problem, broken down
to steps, with hints provided on the way to proceed.
Given the course (or book) prerequisites, the text is quite self-contained.
There was no effort to keep the level of detail in the presentations uniform, and
earlier sections tend to be more detailed. In the later sections a larger part of the
derivation devolves to exercises, as mentioned above.
The material I used in class was (in order of presentation): Chapter 1,
portions of sections 2.2.1, 2.2.2, 2.6.1, Chapter 3, 2.2.3-4, 2.4 and Chapter 4
(barring section 4.2.3). The teaching assistant, beside doing some of the exercises
also covered Section 2.1, most of 2.3, and some of 2.5.
NOTATION
The references are all cited by author(s) and year, as in the above example,
(Fayolle et aI., 1986), and are collected at the back of the volume. I make no claim
there for either completeness or historical fairness; these are simply the sources I
actually used in preparing the text.
Equations are numbered separately in each major section (1.2, 3.3 etc.).
Equation (7) in section 3.2 will be denoted as (7) throughout 3.2, and by (3.2-7)
elsewhere; similar notation applies to exercises. The notation (A.1.7) and (C.3)
refers to equations number (1.7) and (3) in appendices A and C, respectively. The
mark 0 indicates the end of a proof, a theorem which is not followed by a proof,
or of a longer example.
x Preface
Symbols are defined as needed. I tried to follow a few ground rules for
uniformity: random variables and processes are denoted by capital italics, realized
values by lower-case ones; probability distribution-, density- and mass-functions
by F, f, and p, respectively, with suitable subscripts. Generating functions of
several varieties by forms of the letter g, or by a letter derived from the name of
the generated series or variable. Integer indices by i ,j, k, I, m, n (shades of
FORTRAN?). Kronecker's delta by 0ij' E and V are reserved for the expectation
and variance of random variables. There are a few exceptions. Most of the latter
resulted from my trying, when following an analysis developed elsewhere, to
adhere to the notation used there, so as to facilitate reference to the original paper.
Symbols that are frequently used are collected in the Notation Index, at the back of
the volume. Acronyms are expanded in the general index.
Starred exercises are those for which no satisfactory solution has yet been
worked out.
ACKNOWLEDGMENTS
The decision to write the book was taken following a conversation with Philippe
Flajolet. Alexander H.G. Rinnooy-Kan suggested the chapter on bin-packing. The
involvement of Ms Lynn Montz, the Computer Science Editor of Springer-Verlag,
New York, in the book production, was of a considerable help and is much
appreciated. Mrs. Raya Anavi set an example in typing part of the material. The
following colleagues and students read portions of the text and provided
comments and corrections that greatly improved the presentation: Eyal Bardavid,
John Bruno, Guy Fayolle, Aura Ganz, Zehava Koren, Shay Leshkovitz, Johann
Makowsky, Hadas Shachnai and Moshe Sidi. Their contributions are gratefully
acknowledged.
Still, their help does not entitle the above to any claim on errors and
misrepresentations, which remain wholly mine. Help in further reducing these is
avidly solicited, and will be very welcome.
The typesetting was done on equipment of the Department of Computer
Science of the Technion. The help of the principal system engineers, Aythan A vior
and Shlomo Goldberg, has been invaluable in getting around a myriad trivial
obstacles that were as exasperating as they were unexpected.
Haifa, Israel, March 1987. M. Hofri
ACKNOWLEDGMENTS
Figure 3.4, p. 130 is based on Fig. 11, p.87 of Donald E. Knuth, THE
ART OF COMPUTER PROGRAMMING, Vol 3, © 1973 by Addison
Wesley Publishing Company, Inc. Reading Massachusetts.
Reprinted with permission.
Figures 4.1 and 4.2, p.149, and portions of Tables 1 and 2,
p.158, are reproduced from Figures 1 and 2 p.276, and Tables 2 and
3, p.280 of J.e. Lagarias, A.M. Odlyzko and D.B. Zagier: "On the
Capacity of DisjointIy Shared Networks.", published in Computer
Networks, 10, 275-285, © 1985 by North-Holland Publishing
Company, Amsterdam. Reprinted with permission.
Figures 1.1 and 1.2, pp.5, 6 are reprinted from unnumbered
figures on pp.2, 3 in Rainer Kemp, FUNDAMENTALS OF THE
AVERAGE CASE ANALYSIS OF PARTICULAR ALGORITHMS, © 1984
by John Wiley & Sons Ltd. and B. G. Teubner, Stuttgart. Reprinted
with permission.