Table Of ContentComplexity of Algorithms
Lecture Notes, Spring 1999
Peter G´acs
Boston University
and
La´szl´o Lova´sz
Yale University
1
Contents
0 Introduction and Preliminaries 1
0.1 The subject of complexity theory . . . . . . . . . . . . . . . . . . . . . . . . . . . 1
0.2 Some notation and definitions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2
1 Models of Computation 3
1.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3
1.2 Finite automata . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5
1.3 The Turing machine . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8
1.4 The Random Access Machine . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18
1.5 Boolean functions and Boolean circuits . . . . . . . . . . . . . . . . . . . . . . . . 24
2 Algorithmic decidability 31
2.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31
2.2 Recursive and recursively enumerable languages . . . . . . . . . . . . . . . . . . . 33
2.3 Other undecidable problems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37
2.4 Computability in logic . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42
3 Computation with resource bounds 50
3.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 50
3.2 Time and space . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 50
3.3 Polynomial time I: Algorithms in arithmetic . . . . . . . . . . . . . . . . . . . . . 52
3.4 Polynomial time II: Graph algorithms . . . . . . . . . . . . . . . . . . . . . . . . 57
3.5 Polynomial space . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 62
4 General theorems on space and time complexity 65
4.1 Space versus time . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 71
5 Non-deterministic algorithms 72
5.1 Non-deterministic Turing machines . . . . . . . . . . . . . . . . . . . . . . . . . . 72
5.2 Witnesses and the complexity of non-deterministic algorithms . . . . . . . . . . . 74
5.3 General results on nondeterministic complexity classes . . . . . . . . . . . . . . . 76
5.4 Examples of languages in NP . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 79
5.5 NP-completeness . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 85
5.6 Further NP-complete problems . . . . . . . . . . . . . . . . . . . . . . . . . . . . 89
6 Randomized algorithms 99
6.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 99
6.2 Verifying a polynomial identity . . . . . . . . . . . . . . . . . . . . . . . . . . . . 99
6.3 Prime testing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 103
6.4 Randomized complexity classes . . . . . . . . . . . . . . . . . . . . . . . . . . . . 108
2
7 Information complexity: the complexity-theoretic notion of randomness 112
7.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 112
7.2 Information complexity . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 112
7.3 The notion of a random sequence . . . . . . . . . . . . . . . . . . . . . . . . . . . 117
7.4 Kolmogorov complexity and data compression . . . . . . . . . . . . . . . . . . . . 119
8 Pseudo-random numbers 124
8.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 124
8.2 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 124
8.3 Classical methods . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 125
8.4 The notion of a psuedorandom number generator . . . . . . . . . . . . . . . . . . 127
8.5 One-way functions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 130
8.6 Discrete square roots . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 133
9 Parallel algorithms 135
9.1 Parallel random access machines . . . . . . . . . . . . . . . . . . . . . . . . . . . 135
9.2 The class NC . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 138
10 Decision trees 143
10.1 Algorithms using decision trees . . . . . . . . . . . . . . . . . . . . . . . . . . . . 143
10.2 The notion of decision trees . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 146
10.3 Nondeterministic decision trees . . . . . . . . . . . . . . . . . . . . . . . . . . . . 147
10.4 Lower bounds on the depth of decision trees . . . . . . . . . . . . . . . . . . . . . 151
11 Communication complexity 155
11.1 Communication matrix and protocol-tree . . . . . . . . . . . . . . . . . . . . . . 155
11.2 Some protocols . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 159
11.3 Non-deterministic communication complexity . . . . . . . . . . . . . . . . . . . . 160
11.4 Randomized protocols . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 164
12 The complexity of algebraic computations 166
13 Circuit complexity 167
13.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 167
13.2 Lower bound for the Majority Function . . . . . . . . . . . . . . . . . . . . . . . 168
13.3 Monotone circuits . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 170
14 An application of complexity: cryptography 172
14.1 A classical problem . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 172
14.2 A simple complexity-theoretic model . . . . . . . . . . . . . . . . . . . . . . . . . 172
14.3 Public-key cryptography . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 173
14.4 The Rivest-Shamir-Adleman code . . . . . . . . . . . . . . . . . . . . . . . . . . . 175
0
0 Introduction and Preliminaries
0.1 The subject of complexity theory
The need to be able to measure the complexity of a problem, algorithm or structure, and
to obtain bounds and quantitive relations for complexity arises in more and more sciences:
besides computer science, the traditional branches of mathematics, statistical physics, biology,
medicine, socialsciencesandengineeringarealsoconfrontedmoreandmorefrequentlywiththis
problem. In the approach taken by computer science, complexity is measured by the quantity
of computational resources (time, storage, program, communication) used up by a particualr
task. These notes deal with the foundations of this theory.
Computation theory can basically be divided into three parts of different character. First,
the exact notions of algorithm, time, storage capacity, etc. must be introduced. For this, dif-
ferent mathematical machine models must be defined, and the time and storage needs of the
computations performed on these need to be clarified (this is generally measured as a function
of the size of input). By limiting the available resources, the range of solvable problems gets
narrower; this is how we arrive at different complexity classes. The most fundamental com-
plexity classes provide an important classification of problems arising in practice, but (perhaps
more surprisingly) even for those arising in classical areas of mathematics; this classification
reflects the practical and theoretical difficulty of problems quite well. The relationship between
different machine models also belongs to this first part of computation theory.
Second, one must determine the resource need of the most important algorithms in various
areas of mathematics, and give efficient algorithms to prove that certain important problems
belong to certain complexity classes. In these notes, we do not strive for completeness in
the investigation of concrete algorithms and problems; this is the task of the corresponding
fields of mathematics (combinatorics, operations research, numerical analysis, number theory).
Nevertheless, a large number of concrete algorithms will be described and analyzed to illustrate
certain notions and methods, and to establish the complexity of certain problems.
Third, one must find methods to prove “negative results”, i.e. for the proof that some
problems are actually unsolvable under certain resource restrictions. Often, these questions can
beformulatedbyaskingwhethercertaincomplexityclassesaredifferentorempty. Thisproblem
area includes the question whether a problem is algorithmically solvable at all; this question can
todaybeconsideredclassical, andtherearemanyimportantresultsconceriningit; inparticular,
the decidability or undecidablity of most concrete problems of interest is known.
Themajorityofalgorithmicproblemsoccurringinpracticeis,however,suchthatalgorithmic
solvability itself is not in question, the question is only what resources must be used for the
solution. Such investigations, addressed to lower bounds, are very difficult and are still in their
infancy. In these notes, we can only give a taste of this sort of results. In particular, we
discuss complexity notions like communication complexity or decision tree complexity, where
by focusing only on one type of rather special resource, we can give a more complete analysis
of basic complexity classes.
It is, finally, worth noting that if a problem turns out to be “difficult” to solve, this is not
necessarily a negative result. More and more areas (random number generation, communication
protocols, cryptography, data protection) need problems and structures that are guaranteed to
1
be complex. These are important areas for the application of complexity theory; from among
them, we will deal with random number generation and cryptography, the theory of secret
communication.
0.2 Some notation and definitions
Afinitesetofsymbolswillsometimesbecalledanalphabet. Afinitesequenceformedfromsome
elements of an alphabet Σ is called a word. The empty word will also be considered a word,
and will be denoted by ∅. The set of words of length n over Σ is denoted by Σn, the set of all
∗ ∗
words (including the empty word) over Σ is denoted by Σ . A subset of Σ , i.e. , an arbitrary
set of words, is called a language.
Note that the empty language is also denoted by ∅ but it is different, from the language {∅}
containing only the empty word.
Let us define some orderings of the set of words. Suppose that an ordering of the elements
∗
of Σ is given. In the lexicographic ordering of the elements of Σ , a word α precedes a word β if
eitherα isaprefix(beginningsegment)ofβ orthefirstletterwhichisdifferentinthetwowords
is smaller in α. (E.g., 35244 precedes 35344 which precedes 353447.) The lexicographic ordering
doesnotorderallwordsinasinglesequence: forexample, everywordbeginningwith0precedes
the word 1 over the alphabet {0,1}. The increasing order is therefore often preferred: here,
shorter words precede longer ones and words of the same length are ordered lexicographically.
This is the ordering of {0,1}∗ we get when we write up the natural numbers in the binary
number system.
The set of real numbers will be denoted by R, the set of integers by Z and the set of rational
numbers (fractions) by Q. The sign of the set of non-negative real (integer, rational) numbers
is R+ (Z+, Q+). When the base of a logarithm will not be indicated it will be understood to
be 2.
Let f and g be two real (or even complex) functions defined over the natural numbers. We
write
f = O(g)
if there is a constant c > 0 such that for all n large enough we have |f(n)| ≤ c|g(n)|. We write
f = o(g)
if f is 0 only at a finite number of places and f(n)/g(n) → 0 if n → ∞. We will also use
sometimes an inverse of the big O notation: we write
f = Ω(g)
if g = O(f). The notation
f = Θ(g)
means that both f = O(g) and g = O(f) hold, i.e. there are constants c1,c2 > 0 such that
for all n large enough we have c1g(n) ≤ f(n) ≤ c2g(n). We will also use this notation within
formulas. Thus,
2 2
(n+1) = n +O(n)
2
2 2 2
means that (n+1) can be written in the form n +R(n) where R(n) = O(n ). Keep in mind
that in this kind of formula, the equality sign is not symmetrical. Thus, O(n) = O(nn) but
O(n2) (cid:6)= O(n). When such formulas become too complex it is better to go back to some more
explicit notation.
0.1 Exercise Is it true that 1+2+···+n = O(n3)? Can you make this statement sharper?
♦
1 Models of Computation
1.1 Introduction
In this section, we will treat the concept of “computation” or algorithm. This concept is
fundamental for our subject, but we will not define it formally. Rather, we consider it an
intuitive notion, which is amenable to various kinds of formalization (and thus, investigation
from a mathematical point of view).
An algorithm means a mathematical procedure serving for a computation or construction
(the computation of some function), and which can be carried out mechanically, without think-
ing. This is not really a definition, but one of the purposes of this course is to demonstrate that
a general agreement can be achieved on these matters. (This agreement is often formulated
as Church’s thesis.) A program in the Pascal (or any other) programming language is a good
exampleofanalgorithmspecification. Sincethe“mechanical”natureofanalgorithmisitsmost
important feature, instead of the notion of algorithm, we will introduce various concepts of a
mathematical machine.
Mathematical machines compute some output from some input. The input and output can
be a word (finite sequence) over a fixed alphabet. Mathematical machines are very much like
the real computers the reader knows but somewhat idealized: we omit some inessential features
(e.g. hardware bugs), and add an infinitely expandable memory.
Here is a typical problem we often solve on the computer: Given a list of names, sort them
in alphabetical order. The input is a string consisting of names separated by commas: Bob,
Charlie, Alice. The output is also a string: Alice, Bob, Charlie. The problem is to compute a
function assigning to each string of names its alphabetically ordered copy.
In general, a typical algorithmic problem has infinitely many instances, whci then have
arbitrarily large size. Therefore we must consider either an infinite family of finite computers of
growing size, or some idealized infinite computer. The latter approach has the advantage that
it avoids the questions of what infinite families are allowed.
Historically, the first pure infinite model of computation was the Turing machine, intro-
duced by the English mathematician Turing in 1936, thus before the invention of programable
computers. The essence of this model is a central part that is bounded (with a structure inde-
pendentoftheinput)andaninfinitestorage(memory). (Moreexactly,thememoryisaninfinite
one-dimensional array of cells. The control is a finite automaton capable of making arbitrary
local changes to the scanned memory cell and of gradually changing the scanned position.) On
Turing machines, all computations can be carried out that could ever be carried out on any
other mathematical machine-models. This machine notion is used mainly in theoretical inves-
3
tigations. It is less appropriate for the definition of concrete algorithms since its description is
awkward, and mainly since it differs from existing computers in several important aspects.
ThemostimportantweaknessoftheTuringmachineincomparisonrealcomputersisthatits
memory is not accessible immediately: in order to read a distant memory cell, all intermediate
cellsmustalsoberead. ThisisremediedbytheRandomAccessMachine(RAM).TheRAMcan
reach an arbitrary memory cell in a single step. It can be considered a simplified model of real
world computers along with the abstraction that it has unbounded memory and the capability
to store arbitrarily large integers in each of its memory cells. The RAM can be programmed in
an arbitrary programming language. For the description of algorithms, it is practical to use the
RAM since this is closest to real program writing. But we will see that the Turing machine and
the RAM are equivalent from many points of view; what is most important, the same functions
are computable on Turing machines and the RAM.
Despite their seeming theoretical limitations, we will consider logic circuits as a model of
computation, too. Agivenlogiccircuitallowsonlyagivensizeofinput. Inthisway, itcansolve
only a finite number of problems; it will be, however, evident, that for a fixed input size, every
function is computable by a logical circuit. If we restrict the computation time, however, then
thedifferencebetweenproblemspertainingtologiccircuitsandtoTuring-machinesortheRAM
will not be that essential. Since the structure and work of logic circuits is the most transparent
and tractable, they play very important role in theoretical investigations (especially in the proof
of lower bounds on complexity).
If a clock and memory registers are added to a logic circuit we arrive at the interconnected
finite automata that form the typical hardware components of today’s computers.
Letusnotethatafixedfiniteautomaton,whenusedoninputsofarbitrarysize,cancompute
only very primitive functions, and is not an adequate computation model.
Oneofthesimplestmodelsforaninfinitemachineistoconnectaninfinitenumberofsimilar
automata into an array. This way we get a cellular automaton.
The key notion used in discussing machine models is simulation. This notion will not be
defined in full generality, since it refers also to machines or languages not even invented yet.
But its meaning will be clear. We will say that machine M simulates machine N if the internal
states and transitions of N can be traced by machine M in such a way that from the same
inputs, M computes the same outputs as N.
4
1.2 Finite automata
A finite automaton is a very simple and very general computing device. All we assume that if
it gets an input, then it changes its internal state and issues an output. More exactly, a finite
automaton has
— an input alphabet, which is a finite set Σ,
(cid:3)
— an output alphabet, which is another finite set Σ, and
— a set Γ of internal states, which is also finite.
To describe a finite automaton, we need to specify, for every input a ∈ Σ and state s ∈ Γ,
the output α(a,s) ∈ Σ(cid:3) and the new state ω(a,s) ∈ Γ. To make the behavior of the automata
well-defined, we specify a starting state START.
At the beginning of a computation, the automaton is in state s0 = START. The input to
the computation is given in the form of a string a1a2...an ∈ Σ∗. The first input letter a1 takes
the automaton to state s1 = ω(a1,s0); the next input letter takes it into state s2 = ω(a2,s1)
etc. The result of the computation is the string b1b2...bn, where bk = α(ak,sk−1) is the output
at the k-th step.
Thus a finite automaton can be described as a 6-tuple (cid:9)Σ,Σ(cid:3),Γ,α,ω,s0(cid:10), where Σ,Σ(cid:3),Γ are
finite sets, α : Σ×Γ → Σ(cid:3) and ω : Σ×Γ → Γ are arbitrary mappings, and START∈ Γ.
Remarks. 1. There are many possible variants of this notion, which are essentially equivalent.
Often the output alphabet and the output signal are omitted. In this case, the result of the
computation is read off from the state of the automaton at the end of the computation.
(cid:3)
In the case of automata with output, it is often convenient to assume that Σ contains the
blank symbol ∗; in other words, we allow that the automaton does not give an output at certain
steps.
2. Your favorite PC can be modelled by a finite automaton where the input alphabet
consists of all possible keystrokes, and the output alphabet consists of all texts that it can write
onthescreenfollowingakeystroke(weignorethemouse,ports,floppydrivesetc.) Notethatthe
numberofstatesismorethanastronomical(ifyouhave1GBofdiskspace,thanthisautomaton
1010
has something like 2 states). At the cost of allowing so many states, we could model almost
anything as a finite automaton. We’ll be interested in automata where the number of states is
much smaller - usually we assume it remains bounded while the size of the input is unbounded.
Every finite automaton can be described by a directed graph. The nodes of this graph are
(cid:3)
the elements of Γ, and there is an edge labelled (a,b) from state s to state s if α(a,s) = b
(cid:3)
and ω(a,s) = s. The computation performed by the automaton, given an input a1a2...an,
corresponds to a directed path in this graph starting at node START, where the first labels of
the edges on this path are a1,a2,...,an. The second labels of the edges give the result of the
computation (figure 1.1).
(1.1) Example Let us construct an automaton that corrects quotation marks in a text in the
following sense: it reads a text character-by-character, and whenever it sees a quotation like
”...”, it replaces it by “...”. All the automaton has to remember is whether it has seen an even
5
(b,x) (a,x)
(c,x) (c,y)
(a,x)
(a,y) (b,y) (b,y)
START
(c,x)
aabcabc yyxyxyx
Figure 1.1: A finite automaton
(’’,‘‘)
... ...
(z,z) (a,a) (a,a) (z,z)
START OPEN
(’’,’’)
Figure 1.2: An automaton correcting quotation marks
or an odd number of ” symbols. So it will have two states: START and OPEN (i.e., quotation
is open). The input alphabet consists of whatever characters the text uses, including ”. The
output alphabet is the same, except that instead of ” we have two symbols “ and ”. Reading
any character other than ”, the automaton outputs the same symbol and does not change its
state. Reading ”, it outputs “ if it is in state START and outputs ” if it is in state OPEN; and
it changes its state (figure 1.2). ♦
1.1 Exercise Construct a finite automaton with a bounded number of states that receives
two integers in binary and outputs their sum. The automaton gets alternatingly one bit of each
number,startingfromtheright. Ifwegetpastthefirstbitofoneoftheinputsnumbers,aspecial
symbol • is passed to the automaton instead of a bit; the input stops when two consecutive •
symbols are occur. ♦
1.2 Exercise Construct a finite automaton with as few states as possible that receives the
digits of an integer in decimal notation, starting from the left, and the last output is YES if the
number is divisible by 7 and NO if it is not. ♦
1.3 Exercise (a)Forafixedpositiveintegern, constructafiniteautomatonthatreadsaword
of length 2n, and its last output is YES if the first half of the word is the same as the second
half, and NO otherwise. (b) Prove that the automaton must have at least 2n states. ♦
6
1.4 Exercise Prove that there is no finite automaton that, for an input in {0,1}∗ starting
with a “1”, would decide if this binary number is a prime. ♦
7
