Statistical Inference for Engineers and Data Scientists

421 Pages·2018·25.714 MB·English

by  Pierre Moulin, Venugopal V. Veeravalli

Statistical Inference for Engineers and Data Scientists
A mathematically accessible and up-to-date introduction to the tools needed to address
modern inference problems in engineering and data science, ideal for graduate students
taking courses on statistical inference and detection and estimation, and an invaluable
reference for researchers and professionals.
With a wealth of illustrations and examples to explain the key features of the the-
ory and to connect with real-world applications, additional material to explore more
advanced concepts, and numerous end-of-chapter problems to test the reader’s knowl-
edge, this textbook is the “go-to” guide for learning about the core principles of
statistical inference and its application in engineering and data science.
The password-protected Solutions Manual and the Image Gallery from the book are
available at www.cambridge.org/Moulin.
Pierre Moulin is a professor in the ECE Department at the University of Illinois at
Urbana-Champaign. His research interests include statistical inference, machine learn-
ing, detection and estimation theory, information theory, statistical signal, image, and
video processing, and information security. Moulin is a Fellow of the IEEE, and served
as a Distinguished Lecturer for the IEEE Signal Processing Society. He has received
two best paper awards from the IEEE Signal Processing Society and the US National
Science Foundation CAREER Award. He was founding Editor-in-Chief of the IEEE
Transactions on Information Security and Forensics.
Venugopal V. Veeravalli is the Henry Magnuski Professor in the ECE Department at the
University of Illinois at Urbana-Champaign. His research interests include statistical
inference and machine learning, detection and estimation theory, and information the-
ory, with applications to data science, wireless communications, and sensor networks.
Veeravalli is a Fellow of the IEEE, and served as a Distinguished Lecturer for the IEEE
Signal Processing Society. Among the awards he has received are the IEEE Browder J.
Thompson Best Paper Award, the National Science Foundation CAREER Award, the
Presidential Early Career Award for Scientists and Engineers (PECASE), and the Wald
Prize in Sequential Analysis.
Statistical Inference for

Engineers and Data Scientists

PIERRE MOULIN
University of Illinois, Urbana-Champaign
VENUGOPAL V. VEERAVALLI
University of Illinois, Urbana-Champaign
University Printing House, Cambridge CB2 8BS, United Kingdom
One Liberty Plaza, 20th Floor, New York, NY 10006, USA
477 Williamstown Road, Port Melbourne, VIC 3207, Australia
314–321, 3rd Floor, Plot 3, Splendor Forum, Jasola District Centre, New Delhi – 110025, India
79 Anson Road, #06–04/06, Singapore 079906
Cambridge University Press is part of the University of Cambridge.
It furthers the University’s mission by disseminating knowledge in the pursuit of
education, learning, and research at the highest international levels of excellence.
www.cambridge.org
Information on this title: www.cambridge.org/9781107185920
DOI: 10.1017/9781107185920
±c Cambridge University Press 2019
This publication is in copyright. Subject to statutory exception
and to the provisions of relevant collective licensing agreements,
no reproduction of any part may take place without the written
permission of Cambridge University Press.
First published 2019
Printed in the United Kingdom by TJ International Ltd. Padstow Cornwall
A catalogue record for this publication is available from the British Library.
ISBN 978-1-107-18592-0 Hardback
Additional resources for this publication at www.cambridge.org/Moulin.
Cambridge University Press has no responsibility for the persistence or accuracy of
URLs for external or third-party internet websites referred to in this publication
and does not guarantee that any content on such websites is, or will remain,
accurate or appropriate.
We balance probabilities and choose the most likely
– Sherlock Holmes

Brief Contents

Preface pagexvii
List of Acronyms xx
1 Introduction 1
Part I Hypothesis Testing 23
2 Binary Hypothesis Testing 25
3 Multiple Hypothesis Testing 54
4 Composite Hypothesis Testing 71
5 Signal Detection 105
6 Convex Statistical Distances 145
7 Performance Bounds for Hypothesis Testing 160
8 Large Deviations and Error Exponents for Hypothesis Testing 184
9 Sequential and Quickest Change Detection 208
10 Detection of Random Processes 231
Part II Estimation 257
11 Bayesian Parameter Estimation 259
12 Minimum Variance Unbiased Estimation 280
13 Information Inequality and Cramér–Rao Lower Bound 297
14 Maximum Likelihood Estimation 319
viii Brief Contents
15 Signal Estimation 358
Appendix A Matrix Analysis 384
Appendix B Random Vectors and Covariance Matrices 390
Appendix C Probability Distributions 391
Appendix D Convergence of Random Sequences 393
Index 395
Contents
Preface pagexvii
List of Acronyms xx
1 Introduction 1
1.1 Background 1
1.2 Notation 1
1.2.1 Probability Distributions 2
1.2.2 Conditional Probability Distributions 2
1.2.3 Expectations and Conditional Expectations 3
1.2.4 Unified Notation 3
1.2.5 General Random Variables 3
1.3 Statistical Inference 4
1.3.1 Statistical Model 5
1.3.2 Some Generic Estimation Problems 6
1.3.3 Some Generic Detection Problems 6
1.4 Performance Analysis 7
1.5 Statistical Decision Theory 7
1.5.1 Conditional Risk and Optimal Decision Rules 8
1.5.2 Bayesian Approach 9
1.5.3 Minimax Approach 10
1.5.4 Other Non-Bayesian Rules 11
1.6 Derivation of Bayes Rule 12
1.7 Link Between Minimax and Bayesian Decision Theory 14
1.7.1 Dual Concept 14
1.7.2 Game Theory 15
1.7.3 Saddlepoint 15
1.7.4 Randomized Decision Rules 16
Exercises 18
References 21
Part I Hypothesis Testing 23
2 Binary Hypothesis Testing 25
2.1 General Framework 25
2.2 Bayesian Binary Hypothesis Testing 26
x Contents
2.2.1 Likelihood Ratio Test 27
2.2.2 Uniform Costs 28
2.2.3 Examples 28
2.3 Binary Minimax Hypothesis Testing 32
2.3.1 Equalizer Rules 33
2.3.2 Bayes Risk Line and Minimum Risk Curve 34
2.3.3 Differentiable V(π ) 35
0
2.3.4 Nondifferentiable V(π ) 35
0
2.3.5 Randomized LRTs 37
2.3.6 Examples 38
2.4 Neyman–Pearson Hypothesis Testing 40
2.4.1 Solution to the NP Optimization Problem 41
2.4.2 NP Rule 42
2.4.3 Receiver Operating Characteristic 43
2.4.4 Examples 44
2.4.5 Convex Optimization 46
Exercises 47
3 Multiple Hypothesis Testing 54
3.1 General Framework 54
3.2 Bayesian Hypothesis Testing 55
3.2.1 Optimal Decision Regions 56
3.2.2 Gaussian Ternary Hypothesis Testing 58
3.3 Minimax Hypothesis Testing 58
3.4 Generalized Neyman–Pearson Detection 62
3.5 Multiple Binary Tests 62
3.5.1 Bonferroni Correction 63
3.5.2 False Discovery Rate 64
3.5.3 Benjamini–Hochberg Procedure 65
3.5.4 Connection to Bayesian Decision Theory 66
Exercises 67
References 70
4 Composite Hypothesis Testing 71
4.1 Introduction 71
4.2 Random Parameter± 72
4.2.1 Uniform Costs Over Each Hypothesis 73
4.2.2 Nonuniform Costs Over Hypotheses 76
4.3 Uniformly Most Powerful Test 77
4.3.1 Examples 77
4.3.2 Monotone Likelihood Ratio Theorem 79
4.3.3 Both Composite Hypotheses 80
4.4 Locally Most Powerful Test 82
4.5 Generalized Likelihood Ratio Test 84