Probability and information: an integrated approach

236 Pages·1996·1.627 MB·English

by  David Applebaum

Probability and Information
an integrated approach
DAVID APPLEBAUM

Probability and information

Probability and information
An integrated approach
DAVID APPLEBAUM
Department of Probability and Statistics,
The University of Sheffield.
Published by the Press Syndicate of the University of Cambridge
The Pitt Building, Trumpington Street, Cambridge CB2 1RP
40 West 20th Street, New York, NY 10011-4211,USA
10 Stamford Road, Oakleigh, Melbourne 3166, Australia
c Cambridge University Press 1996
(cid:13)
First published 1996
A catalogue record for this book is available from the British Library
Library of Congress cataloguing in publication data
Applebaum, David, 1956–
Probability and information: an integrated approach/DavidApplebaum.
p. cm.
ISBN 0 521 55507 8 (hc). – ISBN 0 521 55528 0 (pb)
1. Probabilities. 2. Information theory. I. Title.
QA273.A66 1996
519.2–dc20 95-49423CIP
ISBN 0 521 55507 8 hardback
ISBN 0 521 55528 0 paperback
Transferred to digital printing 2003
To my parents, Sadie and Robert
To live effectively is to live with adequate information.
Norbert Wiener The Human Use of Human Beings
The study of probability teaches the student that clear logical thinking is also of use
in situations where one is confronted with uncertainty (which is in fact the case in
almost every practical situation).
A. Renyi Remarks on the Teaching of Probability
CONTENTS
Preface to the first edition page xiii
Preface to the second edition xvi
1 Introduction 1
1.1 Chance and information 1
1.2 Mathematical models of chance phenomena 2
1.3 Mathematical structure and mathematical proof 5
1.4 Plan of this book 6
2 Combinatorics 9
2.1 Counting 9
2.2 Arrangements 10
2.3 Combinations 12
2.4 Multinomial coefficients 15
2.5 The gamma function ( ) 16
∗
Further reading 19
3 Sets and measures 20
3.1 The concept of a set 20
3.2 Set operations 23
3.3 Boolean algebras 26
3.4 Measures on Boolean algebras 29
Further reading 36
4 Probability 37
4.1 The concept of probability 37
4.2 Probability in practice 38
4.3 Conditional probability 43
4.4 Independence 49
4.5 The interpretation of probability 51
4.6 The historical roots of probability 56
Further reading 61
5 Discrete random variables 63
5.1 The concept of a random variable 63
5.2 Properties of random variables 64
ix
5.3 Expectation and variance 70
5.4 Covariance and correlation 74
5.5 Independent random variables 78
5.6 I.I.D. random variables 80
5.7 Binomial and Poissonrandom variables 82
5.8 Geometric, negative binomial and hypergeometric random variables 86
Further reading 94
6 Information and entropy 95
6.1 What is information? 95
6.2 Entropy 97
6.3 Joint and conditional entropies; mutual information 101
6.4 The maximum entropy principle 105
6.5 Entropy, physics and life 106
6.6 The uniqueness of entropy 108
∗
Further reading 113
7 Communication 115
7.1 Transmission of information 115
7.2 The channel capacity 118
7.3 Codes 119
7.4 Noiseless coding 123
7.5 Coding and transmission with noise—Shannon’s theorem 129
∗
7.6 Brief remarks about the history of information theory 136
Further reading 138
8 Random variables with probability density functions 139
8.1 Random variables with continuous ranges 139
8.2 Probability density functions 141
8.3 Discretisation and integration 144
∗
8.4 Laws of large numbers 147
8.5 Normal random variables 150
8.6 The central limit theorem 155
8.7 Entropy in the continuous case 161
Further reading 168
9 Random vectors 169
9.1 Cartesian products 169
9.2 Boolean algebras and measures on products 171
9.3 Distributions of random vectors 173
9.4 Marginal distributions 179
9.5 Independence revisited 181
9.6 Conditional densities and conditional entropy 183
9.7 Mutual information and channel capacity 187
Further reading 194
Exploring further 195
Appendix 1 Proof By Mathematical Induction 197