Complexity of Algorithms

242 Pages·2010·1.31 MB·English

1
Complexity of Algorithms
Lecture Notes, Spring 1999
Peter G´acs
Boston University
and
L´aszlo´ Lov´asz
Yale University
2
Contents
1 Introduction and Preliminaries 1
1.1 The subject of complexity theory . . . . . . . . . . . . . . . . . . . . . . . . . 1
1.2 Some notation and definitions . . . . . . . . . . . . . . . . . . . . . . . . . . . 2
2 Models of Computation 5
2.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5
2.2 Finite automata . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7
2.3 The Turing machine . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10
2.3.1 The notion of a Turing machine. . . . . . . . . . . . . . . . . . . . . . 10
2.3.2 Universal Turing machines. . . . . . . . . . . . . . . . . . . . . . . . . 12
2.3.3 More tapes versus one tape . . . . . . . . . . . . . . . . . . . . . . . . 17
2.4 The Random Access Machine . . . . . . . . . . . . . . . . . . . . . . . . . . . 21
2.5 Boolean functions and Boolean circuits . . . . . . . . . . . . . . . . . . . . . . 27
3 Algorithmic decidability 35
3.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 35
3.2 Recursive and recursively enumerable languages . . . . . . . . . . . . . . . . . 36
3.3 Other undecidable problems . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41
3.4 Computability in logic . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 47
3.4.1 G¨odel’s incompleteness theorem. . . . . . . . . . . . . . . . . . . . . . 47
3.4.2 First-order logic . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 49
3.4.3 A simple theory of arithmetic; Church’s Theorem . . . . . . . . . . . . 52
4 Computation with resource bounds 57
4.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 57
4.2 Time and space . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 58
4.3 Polynomial time I: Algorithms in arithmetic . . . . . . . . . . . . . . . . . . . 60
4.3.1 Arithmetic operations . . . . . . . . . . . . . . . . . . . . . . . . . . . 60
3
4 CONTENTS
4.3.2 Gaussian elimination . . . . . . . . . . . . . . . . . . . . . . . . . . . . 63
4.3.3 *Discrete square roots . . . . . . . . . . . . . . . . . . . . . . . . . . . 66
4.4 Polynomial time II: Graph algorithms . . . . . . . . . . . . . . . . . . . . . . 68
4.4.1 How is a graph given? . . . . . . . . . . . . . . . . . . . . . . . . . . . 68
4.4.2 Searching a graph . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 68
4.4.3 Maximum bipartite matching and alternating paths . . . . . . . . . . 70
4.5 Polynomial space . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 73
5 General theorems on space and time complexity 77
5.1 Time . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 77
5.2 Time-constructible and well-computable functions . . . . . . . . . . . . . . . 80
5.3 Space versus time . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 84
6 Non-deterministic algorithms 87
6.1 Non-deterministic Turing machines . . . . . . . . . . . . . . . . . . . . . . . . 87
6.2 Witnesses and the complexity of non-deterministic algorithms . . . . . . . . . 90
6.3 General results on nondeterministic complexity classes . . . . . . . . . . . . . 92
6.4 Examples of languages in NP . . . . . . . . . . . . . . . . . . . . . . . . . . . 96
6.4.1 Graphs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 96
6.4.2 Arithmetic and algebra . . . . . . . . . . . . . . . . . . . . . . . . . . 98
6.4.3 Systems of linear inequalities . . . . . . . . . . . . . . . . . . . . . . . 101
6.5 NP-completeness . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 104
6.6 Further NP-complete problems . . . . . . . . . . . . . . . . . . . . . . . . . . 109
6.6.1 NP-complete problems for graphs . . . . . . . . . . . . . . . . . . . . . 109
6.6.2 NP-complete problems in arithmetic and algebra . . . . . . . . . . . . 115
7 Randomized algorithms 121
7.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 121
7.2 Verifying a polynomial identity . . . . . . . . . . . . . . . . . . . . . . . . . . 121
7.3 Prime testing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 126
7.4 Randomized complexity classes . . . . . . . . . . . . . . . . . . . . . . . . . . 131
8 Information complexity and the notion of randomness 137
8.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 137
8.2 Measuring information complexity . . . . . . . . . . . . . . . . . . . . . . . . 138
8.3 *Self-delimiting information complexity . . . . . . . . . . . . . . . . . . . . . 143
8.4 The notion of a random sequence . . . . . . . . . . . . . . . . . . . . . . . . . 147
8.5 Kolmogorov complexity and data compression . . . . . . . . . . . . . . . . . . 149
CONTENTS 5
9 Pseudo-random numbers 155
9.1 Classical methods. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 155
9.2 The notion of a pseudorandom number generator . . . . . . . . . . . . . . . . 157
9.3 One-way functions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 160
9.4 Random number generators and one-way functions . . . . . . . . . . . . . . . 162
10 An application of complexity: cryptography 167
10.1 A classical problem . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 167
10.2 A simple complexity-theoretic model . . . . . . . . . . . . . . . . . . . . . . . 168
10.3 Public-key cryptography . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 169
10.4 The Rivest-Shamir-Adleman code . . . . . . . . . . . . . . . . . . . . . . . . . 171
11 Parallel algorithms 175
11.1 Parallel random access machines . . . . . . . . . . . . . . . . . . . . . . . . . 175
11.2 The class NC . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 180
12 Decision trees 185
12.1 Algorithms using decision trees . . . . . . . . . . . . . . . . . . . . . . . . . . 185
12.2 The notion of decision trees . . . . . . . . . . . . . . . . . . . . . . . . . . . . 189
12.3 Nondeterministic decision trees . . . . . . . . . . . . . . . . . . . . . . . . . . 191
12.4 Lower bounds on the depth of decision trees . . . . . . . . . . . . . . . . . . . 194
12.5 Algebraic decision trees . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 198
13 Communication complexity 203
13.1 Examples . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 204
13.2 Communication matrix and protocol-tree . . . . . . . . . . . . . . . . . . . . 206
13.3 Non-deterministic communication complexity . . . . . . . . . . . . . . . . . . 210
13.4 Randomized protocols . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 214
14 Interactive proofs 217
14.1 How to save the last move in chess? . . . . . . . . . . . . . . . . . . . . . . . 217
14.2 How to check a password — without knowing it? . . . . . . . . . . . . . . . . 219
14.3 How to use your password — without telling it?. . . . . . . . . . . . . . . . . 220
14.4 How to prove non-existence? . . . . . . . . . . . . . . . . . . . . . . . . . . . 221
14.5 How to handle submissions that keep the main result secret? . . . . . . . . . 224
14.6 How to referee exponentially long papers? . . . . . . . . . . . . . . . . . . . . 225
14.6.1 Two Provers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 225
14.6.2 Probabilistically Checkable Proofs . . . . . . . . . . . . . . . . . . . . 225
0 CONTENTS
14.6.3 Limits of approximability . . . . . . . . . . . . . . . . . . . . . . . . . 226
15 The complexity of algebraic computations 229
16 Circuit complexity 231
16.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 231
16.2 Lower bound for the Majority Function . . . . . . . . . . . . . . . . . . . . . 232
16.3 Monotone circuits . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 235
Chapter 1
Introduction and Preliminaries
1.1 The subject of complexity theory
The need to be able to measure the complexity of a problem, algorithm or structure, and to
obtain bounds and quantitive relations for complexity arises in more and more sciences: be-
sides computer science, the traditional branches of mathematics, statistical physics, biology,
medicine, social sciences and engineering are also confronted more and more frequently with
this problem. In the approach taken by computer science, complexity is measured by the
quantity of computational resources (time, storage, program, communication) used up by a
particualr task. These notes deal with the foundations of this theory.
Computationtheorycanbasicallybedividedintothreepartsofdifferentcharacter. First,
the exact notions of algorithm, time, storage capacity, etc. must be introduced. For this,
differentmathematicalmachinemodelsmustbedefined,andthetimeandstorageneedsofthe
computationsperformedontheseneedtobeclarified(thisisgenerallymeasuredasafunction
of the size of input). By limiting the available resources, the range of solvable problems
gets narrower; this is how we arrive at different complexity classes. The most fundamental
complexity classes provide an important classification of problems arising in practice, but
(perhaps more surprisingly) even for those arising in classical areas of mathematics; this
classification reflects the practical and theoretical difficulty of problems quite well. The
relationship between different machine models also belongs to this first part of computation
theory.
Second,onemustdeterminetheresourceneedofthemostimportantalgorithmsinvarious
areasofmathematics, andgiveefficientalgorithmstoprovethatcertainimportantproblems
belong to certain complexity classes. In these notes, we do not strive for completeness in
the investigation of concrete algorithms and problems; this is the task of the corresponding
fields of mathematics (combinatorics, operations research, numerical analysis, number the-
1
2 CHAPTER 1. INTRODUCTION AND PRELIMINARIES
ory). Nevertheless, a large number of concrete algorithms will be described and analyzed to
illustrate certain notions and methods, and to establish the complexity of certain problems.
Third, one must find methods to prove “negative results”, i.e. for the proof that some
problems are actually unsolvable under certain resource restrictions. Often, these questions
can be formulated by asking whether certain complexity classes are different or empty. This
problem area includes the question whether a problem is algorithmically solvable at all; this
questioncantodaybeconsideredclassical,andtherearemanyimportantresultsconcerining
it; in particular, the decidability or undecidablity of most concrete problems of interest is
known.
The majority of algorithmic problems occurring in practice is, however, such that algo-
rithmic solvability itself is not in question, the question is only what resources must be used
forthesolution. Suchinvestigations,addressedtolowerbounds,areverydifficultandarestill
in their infancy. In these notes, we can only give a taste of this sort of results. In particular,
we discuss complexity notions like communication complexity or decision tree complexity,
where by focusing only on one type of rather special resource, we can give a more complete
analysis of basic complexity classes.
It is, finally, worth noting that if a problem turns out to be “difficult” to solve, this
is not necessarily a negative result. More and more areas (random number generation,
communication protocols, cryptography, data protection) need problems and structures that
are guaranteed to be complex. These are important areas for the application of complexity
theory; from among them, we will deal with random number generation and cryptography,
the theory of secret communication.
1.2 Some notation and definitions
A finite set of symbols will sometimes be called an alphabet. A finite sequence formed from
some elements of an alphabet Σ is called a word. The empty word will also be considered a
word, and will be denoted by ∅. The set of words of length n over Σ is denoted by Σn, the
set of all words (including the empty word) over Σ is denoted by Σ∗. A subset of Σ∗, i.e. ,
an arbitrary set of words, is called a language.
Note that the empty language is also denoted by ∅ but it is different, from the language
{∅} containing only the empty word.
Letusdefinesomeorderingsofthesetofwords. Supposethatanorderingoftheelements
ofΣisgiven. Inthelexicographic orderingoftheelementsofΣ∗,awordαprecedesawordβ
if either α is a prefix (beginning segment) of β or the first letter which is different in the two
wordsissmallerinα. (E.g.,35244precedes35344whichprecedes353447.) Thelexicographic
ordering does not order all words in a single sequence: for example, every word beginning
1.2. SOME NOTATION AND DEFINITIONS 3
with 0 precedes the word 1 over the alphabet {0,1}. The increasing order is therefore often
preferred: here, shorter words precede longer ones and words of the same length are ordered
lexicographically. Thisistheorderingof{0,1}∗wegetwhenwewriteupthenaturalnumbers
in the binary number system.
The set of real numbers will be denoted by R, the set of integers by Z and the set of
rationalnumbers(fractions)byQ. Thesignofthesetofnon-negativereal(integer,rational)
numbers is R (Z , Q ). When the base of a logarithm will not be indicated it will be
+ + +
understood to be 2.
Let f and g be two real (or even complex) functions defined over the natural numbers.
We write
f =O(g)
if there is a constant c > 0 such that for all n large enough we have |f(n)| ≤ c|g(n)|. We
write
f =o(g)
if f is 0 only at a finite number of places and f(n)/g(n) → 0 if n → ∞. We will also use
sometimes an inverse of the big O notation: we write
f =Ω(g)
if g =O(f). The notation
f =Θ(g)
means that both f = O(g) and g = O(f) hold, i.e. there are constants c ,c > 0 such that
1 2
for all n large enough we have c g(n)≤f(n)≤c g(n). We will also use this notation within
1 2
formulas. Thus,
(n+1)2 =n2+O(n)
means that (n+1)2 can be written in the form n2 +R(n) where R(n) = O(n2). Keep in
mindthatinthiskindofformula, theequalitysignisnotsymmetrical. Thus, O(n)=O(nn)
but O(n2)(cid:54)=O(n). When such formulas become too complex it is better to go back to some
more explicit notation.
Exercise 1.2.1 Is it true that 1 + 2 + ··· + n = O(n3)? Can you make this statement
sharper?
4 CHAPTER 1. INTRODUCTION AND PRELIMINARIES