Statistical Theory: A Concise Introduction

237 Pages·2022·6.636 MB·English

by  Felix Abramovich, Ya'acov Ritov

Statistical Theory
Designed for a one-semester advanced undergraduate or graduate statistical theory course, Statistical
Theory: A Concise Introduction, Second Edition clearly explains the underlying ideas, mathematics,
and principles of major statistical concepts, including parameter estimation, confidence intervals,
hypothesis testing, asymptotic analysis, Bayesian inference, linear models, nonparametric statistics,
and elements of decision theory. It introduces these topics on a clear intuitive level using illustrative
examples in addition to the formal definitions, theorems, and proofs.
Based on the authors’ lecture notes, the book is self-contained, which maintains a proper
balance between the clarity and rigor of exposition. In a few cases, the authors present a “sketched”
version of a proof, explaining its main ideas rather than giving detailed technical mathematical and
probabilistic arguments.
Features:
• Second edition has been updated with a new chapter on Nonparametric Estimation;
a significant update to the chapter on Statistical Decision Theory; and other updates
throughout
• No requirement for heavy calculus, and simple questions throughout the text help students
check their understanding of the material
• Each chapter also includes a set of exercises that range in level of difficulty
• Self-contained, and can be used by the students to understand the theory
• Chapters and sections marked by asterisks contain more advanced topics and may be
omitted
• Special chapters on linear models and nonparametric statistics show how the main
theoretical concepts can be applied to well-known and frequently used statistical tools
The primary audience for the book is students who want to understand the theoretical basis of
mathematical statistics—either advanced undergraduate or graduate students. It will also be an
excellent reference for researchers from statistics and other quantitative disciplines.
Felix Abramovich is a professor at the Department of Statistics and Operations Research at
Tel Aviv University.
Ya’acov Ritov is a professor in the Department of Statistics at the University of Michigan, Ann
Arbor. He is a professor emeritus of Statistics at the Hebrew University of Jerusalem.
CHAPMAN & HALL/CRC
Texts in Statistical Science Series
Joseph K. Blitzstein, Harvard University, USA
Julian J. Faraway, University of Bath, UK
Martin Tanner, Northwestern University, USA
Jim Zidek, University of British Columbia, Canada
Recently Published Titles
Fundamentals of Causal Inference
With R
Babette A. Brumback
Sampling
Design and Analysis, Third Edition
Sharon L. Lohr
Theory of Statistical Inference
Anthony Almudevar
Probability, Statistics, and Data
A Fresh Approach Using R
Darrin Speegle and Brain Claire
Bayesian Modeling and Computation in Python
Osvaldo A. Martin, Raviv Kumar and Junpeng Lao
Bayes Rules!
An Introduction to Applied Bayesian Modeling
Alicia Johnson, Miles Ott and Mine Dogucu
Stochastic Processes with R
An Introduction
Olga Korosteleva
Introduction to Design and Analysis of Scientific Studies
Nathan Taback
Practical Time Series Analysis for Data Science
Wayne A. Woodward, Bivin Philip Sadler and Stephen Robertson
Statistical Theory
A Concise Introduction, Second Edition
Felix Abramovich and Ya’acov Ritov
Applied Linear Regression for Longitudinal Data
With an Emphasis on Missing Observations
Frans E.S. Tan and Shahab Jolani
For more information about this series, please visit: https://www.routledge.com/Chapman--
HallCRC-Texts-in-Statistical-Science/book-series/CHTEXSTASCI
Statistical Theory
A Concise Introduction
Second Edition
Felix Abramovich
Tel Aviv University, Israel
Ya’acov Ritov
University of Michigan, USA
and
The Hebrew University of Jerusalem, Israel
Second edition published 2023
by CRC Press
6000 Broken Sound Parkway NW, Suite 300, Boca Raton, FL 33487-2742
and by CRC Press
4 Park Square, Milton Park, Abingdon, Oxon, OX14 4RN
© 2023 Felix Abramovich and Ya’acov Ritov
First edition published by CRC Press 2013
CRC Press is an imprint of Taylor & Francis Group, LLC
Reasonable efforts have been made to publish reliable data and information, but the author and publisher cannot
assume responsibility for the validity of all materials or the consequences of their use. The authors and publishers have
attempted to trace the copyright holders of all material reproduced in this publication and apologize to copyright holders
if permission to publish in this form has not been obtained. If any copyright material has not been acknowledged please
write and let us know so we may rectify in any future reprint.
Except as permitted under U.S. Copyright Law, no part of this book may be reprinted, reproduced, transmitted, or
utilized in any form by any electronic, mechanical, or other means, now known or hereafter invented, including pho-
tocopying, microfilming, and recording, or in any information storage or retrieval system, without written permission
from the publishers.
For permission to photocopy or use material electronically from this work, access www.copyright.com or contact the
Copyright Clearance Center, Inc. (CCC), 222 Rosewood Drive, Danvers, MA 01923, 978-750-8400. For works that are
not available on CCC please contact [e-mail]
Trademark notice: Product or corporate names may be trademarks or registered trademarks and are used only for iden-
tification and explanation without intent to infringe.
Library of Congress Cataloging-in-Publication Data
Names: Abramovich, Felix, author. | Ritov, Ya’acov, 1951- author.
Title: Statistical theory : a concise introduction / Felix Abramovich, Tel
Aviv University, Israel, Ya’acov Ritov, University of Michigan, USA and
The Hebrew University of Jerusalem, Israel.
Description: Second edition. | Boca Raton : Chapman & Hall, CRC Press,
2023. | Series: Chapman and Hall/CRC texts in statistical science |
Includes bibliographical references and index.
Identifiers: LCCN 2022024632 (print) | LCCN 2022024633 (ebook) | ISBN
9781032007458 (hardback) | ISBN 9781032007472 (paperback) | ISBN
9781003175407 (ebook)
Subjects: LCSH: Probabilities--Textbooks. | Statistics--Textbooks.
Classification: LCC QA276 .A23 2023 (print) | LCC QA276 (ebook) | DDC
519.5--dc23/eng20221007
LC record available at https://lccn.loc.gov/2022024632
LC ebook record available at https://lccn.loc.gov/2022024633
ISBN: 978-1-032-00745-8 (hbk)
ISBN: 978-1-032-00747-2 (pbk)
ISBN: 978-1-003-17540-7 (ebk)
DOI: 10.1201/ 9781003175407
Typeset in Nimbus
by KnowledgeWorks Global Ltd.
Publisher’s note: This book has been prepared from camera-ready copy provided by the authors.
Contents
ListofFigures ix
ListofTables xi
PrefacetotheSecondEdition xiii
1 Introduction 1
1.1 Preamble 1
1.2 Likelihood 3
1.3 Sufficiency 5
1.4 ∗Minimalsufficiency 7
1.5 ∗Completeness 9
1.6 Exponentialfamilyofdistributions 10
1.7 Exercises 12
2 PointEstimation 15
2.1 Introduction 15
2.2 Maximumlikelihoodestimation 15
2.3 Methodofmoments 20
2.4 Methodofleastsquares 21
2.5 ∗M-estimators 21
2.6 Goodness-of-estimation:meansquarederror 22
2.7 Unbiasedestimation 24
2.7.1 Definitionandmainproperties 24
2.7.2 Uniformlyminimumvarianceunbiasedestimators:TheCramer–Raolower
bound 25
2.7.3 ∗TheCramer–Raolowerboundformultiparametercase 29
2.7.4 Rao–Blackwelltheorem 31
2.7.5 ∗Lehmann–Scheffe´ theorem 33
2.8 Exercises 33
3 ConfidenceIntervals,Bounds,andRegions 37
3.1 Introduction 37
3.2 Quotingtheestimationerror 37
3.3 Confidenceintervals 39
3.4 Confidencebounds 43
3.5 ∗Confidenceregions 44
3.6 Exercises 46
v
vi CONTENTS
4 HypothesisTesting 49
4.1 Introduction 49
4.2 Simplehypotheses 50
4.2.1 TypeIandTypeIIerrors 50
4.2.2 Choiceofacriticalvalue 52
4.2.3 Thep-value 53
4.2.4 Maximalpowertests.Neyman–Pearsonlemma 55
4.3 Compositehypotheses 59
4.3.1 Powerfunction 59
4.3.2 Uniformlymostpowerfultests 61
4.3.3 Generalizedlikelihoodratiotests 65
4.4 Dualitybetweenhypothesistestingandconfidenceintervals(regions) 68
4.5 Sequentialtesting 69
4.6 ∗Multipletesting 72
4.6.1 Family-wiseerror 72
4.6.2 Falsediscoveryrate 74
4.7 Exercises 75
5 AsymptoticAnalysis 79
5.1 Introduction 79
5.2 ConvergenceandconsistencyinMSE 80
5.3 Convergenceandconsistencyinprobability 81
5.4 Convergenceindistribution 83
5.5 Thecentrallimittheorem 85
5.6 Asymptoticallynormalconsistency 87
5.7 Asymptoticconfidenceintervals 89
5.8 AsymptoticpropertiesofMLEs,Waldconfidenceintervals,andtests 92
5.9 ∗Multiparametercase 95
5.10 ∗AsymptoticpropertiesofM-estimators 97
5.11 Score(Rao)asymptotictestsandconfidenceregions 100
5.12 AsymptoticdistributionoftheGLRT,Wilks’theorem 101
5.13 Exercises 106
6 BayesianInference 109
6.1 Introduction 109
6.2 Choiceofpriors 111
6.2.1 Conjugatepriors 111
6.2.2 Noninformative(objective)priors 113
6.3 Pointestimation 116
6.4 Intervalestimation:Crediblesets 118
6.5 Hypothesistesting 119
6.5.1 Simplehypotheses 119
6.5.2 Compositehypotheses 120
6.5.3 Testingapointnullhypothesis 121
6.6 ∗Asymptoticpropertiesoftheposteriordistribution 122
6.7 Exercises 123
CONTENTS vii
7 ∗ElementsofStatisticalDecisionTheory 125
7.1 Introductionandnotations 125
7.2 Riskfunctionandadmissibility 126
7.3 Minimaxriskandminimaxrules 128
7.4 BayesriskandBayesrules 129
7.5 PosteriorexpectedlossandBayesactions 130
7.6 AdmissibilityofBayesrules 132
7.7 MinimaxityandBayesrules 134
7.8 Exercises 136
8 ∗LinearModels 139
8.1 Introduction 139
8.2 Definitionandexamples 139
8.3 Estimationofregressioncoefficients 141
8.4 Residuals.Estimationofthevariance 144
8.5 Examples 146
8.5.1 Estimationofanormalmean 146
8.5.2 Comparisonbetweenthemeansoftwoindependentnormalsampleswitha
commonvariance 146
8.5.3 Simplelinearregression 146
8.6 Goodness-of-fit:Multiplecorrelationcoefficient 147
8.7 Confidenceintervalsandregionsforthecoefficients 148
8.8 Hypothesistestinginlinearmodels 149
8.8.1 Testingsignificanceofasinglepredictor 149
8.8.2 Testingsignificanceofagroupofpredictors 150
8.8.3 Testingagenerallinearhypothesis 151
8.9 Predictions 152
8.10 Analysisofvariance 154
8.10.1 One-wayANOVA 154
8.10.2 Two-wayANOVAandbeyond 156
9 ∗NonparametricEstimation 157
9.1 Introduction 157
9.2 Theempiricaldistributionfunctionandthehistogram 157
9.3 Kerneldensityestimation 160
9.4 Theminimaxrate 164
9.5 Nonparametrickernelregression 166
9.6 Nonparametricestimationbyorthonormalseries 167
9.6.1 Orthonormalseries 167
9.6.2 Cosineseries 168
9.6.3 Nonparametricdensityestimationbycosineseries 170
9.6.4 Nonparametricregressionandorthonormalcosineseries 171
9.7 Splinesmoothing 173
9.8 Choiceofthesmoothingparameter 177
A ProbabilisticReview 185
A.1 Introduction 185
A.2 Basicprobabilisticlaws 185
A.3 Randomvariables 186
A.3.1 Expectedvalueandthevariance 187
A.3.2 Chebyshev’sandMarkov’sinequalities 188
A.3.3 ExpectationoffunctionsandtheJensen’sinequality 188
viii CONTENTS
A.3.4 Jointdistribution 189
A.3.5 Covariance,correlation,andtheCauchy–Schwarzinequality 189
A.3.6 Expectationandvarianceofasumofrandomvariables 190
A.3.7 ConditionaldistributionandBayesTheorem 190
A.3.8 Distributionsoffunctionsofrandomvariables 191
A.3.9 Randomvectors 191
A.4 Specialfamiliesofdistributions 193
A.4.1 Bernoulliandbinomialdistributions 193
A.4.2 Geometricandnegativebinomialdistributions 193
A.4.3 Hypergeometricdistribution 194
A.4.4 Poissondistribution 195
A.4.5 Uniformdistribution 196
A.4.6 Exponentialdistribution 196
A.4.7 Weibulldistribution 197
A.4.8 Gamma-distribution 198
A.4.9 Beta-distribution 198
A.4.10 Cauchydistribution 198
A.4.11 Normaldistribution 199
A.4.12 Log-normaldistribution 199
A.4.13 χ2distribution 200
A.4.14 t-distribution 200
A.4.15 F-distribution 200
A.4.16 Multinormaldistribution 201
A.4.16.1 Definitionandmainproperties 201
A.4.16.2 Projectionsofnormalvectors 202
B SolutionsofSelectedExercises 203
B.1 Chapter1 203
B.2 Chapter2 204
B.3 Chapter3 207
B.4 Chapter4 208
B.5 Chapter5 210
B.6 Chapter6 213
B.7 Chapter7 214
Bibliography 217
Index 219
List of Figures
1.1 Thelikelihoodfunctionforbabydata. 4
1.2 ThelikelihoodfunctionforMr.Skeptic’sdata. 5
2.1 LikelihoodfunctionforU(0,θ). 16
2.2 LikelihoodfunctionforU(θ,θ+1). 17
2.3 MSEforthreeestimatorsof pinB(10,p):MSE(pˆ ,p)(solidline),MSE(pˆ ,p)
1 2
(dashedline),andMSE(pˆ ,p)(dottedline). 24
3
3.1 90%confidenceintervalsforthenormalmeanµ computedfrom100independent
samplesofsize100fromthenormaldistributionN (µ,1).Thetruevalueofthe
parameter was µ =0. Intervals that do not include the true value are marked by
asterisks. 40
3.2 ThreeconfidenceregionsforaN (µ,σ2)sample,wheren=30,Y¯ =15,s2=4,
30
and α =0.05: the rectangle Bonferroni confidence regionCR from (3.5), the
B
trapezoidCR from(3.6)andtheunboundedstripCR from(3.7). 46
E χ2
4.1 ErrorprobabilitiesofbothtypesforExample4.1. 51
4.2 p-valueandα forExample4.5:α =0.05(leftfigure)andα =0.01(rightfigure). 55
4.3 Thelikelihoodratioλ(y)in(4.7)asafunctionofy¯. 58
4.4 Thepowerfunctionsofanidealtest(solid)anda“good”test(dashed). 60
4.5 The power functions for a one-sided test of Example 4.10 (left figure) and a
two-sidedtestofExample4.11(rightfigure). 61
4.6 Powerfunctionsfortwopairsoftestsatlevelα. 62
4.7 Minimizing(y¯−µ)2 subjecttoconstraint µ ≤µ :y¯≤µ (leftfigure)andy¯>µ
0 0 0
(rightfigure). 67
4.8 Thegeneralizedlikelihoodratioλ(y)asafunctionofT(y)inExample4.19. 68
4.9 Graphical illustration of the sequential Wald test for testing H : µ = 0 vs.
0
H :µ =1.Theboundarylinesare0.5t±2.9. 72
1
5.1 Tenrandompathsofasampleproportion pˆn=n−1∑ni=1Yi asafunctionofn,fora
seriesofBernoullitrialsB(1,0.5). 83
5.2 BarplotoftheprobabilityfunctionofPois(2),B(10,0.2),B(20,0.1).Notethat
allthreehaveameanof2. 85
5.3 Differentgammadensities,scaledtohavemean0andvariance1,andtheN (0,1)
density. 87
5.4 FourconfidenceregionsforaN (µ,σ2)sample,wheren=30,Y¯ =15,s2=4,
30
and α =0.05: the Bonferroni’sCR from (3.5) (rectangle), the exactCR from
B E
(3.6)(trapezoid),theasymptoticCR from(5.20),andtheasymptoticWaldCR
A1 A2
from(5.21)(grayellipsoid). 97
5.5 Asymptoticconfidenceinterval(a)andregioninamoregeneralcase(b)”,i.e.,(a)
correspondstoaninterval,while(b)toamoregeneralregion. 105
ix