Regression and Other Stories

552 Pages·2020·6.095 MB·English

by  Andrew Gelman, Jennifer Hill, Aki Vehtari

RegressionandOtherStories
Manytextbooksonregressionfocusontheoryandthesimplestofexamples.Realstatistical
problems,however,arecomplexandsubtle.Thisisnotabookaboutthetheoryof
regression.Itisabookabouthowtouseregressiontosolverealproblemsofcomparison,
estimation,prediction,andcausalinference.Itfocusesonpracticalissuessuchassample
sizeandmissingdataandawiderangeofgoalsandtechniques.Itjumpsrightinto
methodsandcomputercodeyoucanusefreshoutofthebox.
Keyfeatures:
● Realexamples,realstoriesfromtheauthors’real-worldexperiencedemonstratewhat
canbeachievedbyregressionandwhatthelimitationsare
● Usescomputationwiththepopularopen-sourceprogramsRandStaninsteadofderiving
formulas,withallcodeavailableonline
● Emphasisonusinggraphicsandpresentationtounderstandandcheckmodelsthathave
beenfittodata
● Practicaladviceforunderstandingassumptionsandimplementingmethodsfor
experimentsandobservationalstudies
● Smoothtransitiontologisticregressionandgeneralizedlinearmodels
● Clearpresentationofkeyideasindatacollection,sampling,generalization,andcausal
inference
Theauthorsareexperiencedresearcherswhohavepublishedarticlesinhundredsof
differentscientificjournalsinfieldsincludingstatistics,computerscience,policy,public
health,politicalscience,economics,sociology,andengineering.Theyhavealsopublished
articlesintheWashingtonPost,NewYorkTimes,Slate,andotherpublicvenues.Their
previousbooksincludeBayesianDataAnalysis,TeachingStatistics:ABagofTricks,and
DataAnalysisandRegressionUsingMultilevel/HierarchicalModels.
ANDREW GELMANisHigginsProfessorofStatisticsandProfessorofPoliticalScienceat
ColumbiaUniversity.
JENNIFER HILLisProfessorofAppliedStatisticsatNewYorkUniversity.
AKI VEHTARIisAssociateProfessorinComputationalProbabilisticModelingatAalto
University.
AnalyticalMethodsforSocialResearch
AnalyticalMethodsforSocialResearchpresentstextsonempiricalandformalmethodsfor
thesocialsciences.Volumesintheseriesaddressboththetheoreticalunderpinningsofana-
lytical techniques as well as their application in social research. Some series volumes are
broadinscope,cuttingacrossanumberofdisciplines.Othersfocusmainlyonmethodolog-
icalapplicationswithinspecificfieldssuchaspoliticalscience,sociology,demography,and
publichealth.Theseriesservesamixofstudentsandresearchersinthesocialsciencesand
statistics.
SeriesEditors
R.MichaelAlvarez,CaliforniaInstituteofTechnology
NathanielL.Beck,NewYorkUniversity
StephenL.Morgan,JohnsHopkinsUniversity
LawrenceL.Wu,NewYorkUniversity
OtherTitlesintheSeries
MaximumLikelihoodforSocialScience,byMichaelD.WardandJohnS.Ahlquist
ComputationalSocialScience,byR.MichaelAlvarez
SpatialAnalysisfortheSocialSciences,byDavidDarmofal
CounterfactualsandCausalInference,SecondEdition,byStephenL.MorganandChristo-
pherWinship
TimeSeriesAnalysisfortheSocialSciences,byJanetM.Box-Steffensmeier,JohnR.Freeman,
MatthewP.Hitt,andJonC.W.Pevehouse
StatisticalModelingandInferenceforSocialScience,bySeanGailmard
FormalModelsofDomesticPolitics,byScottGehlbach
Data Analysis Using Regression and Multilevel/Hierarchical Models, by Andrew Gelman
andJenniferHill
PoliticalGameTheory:AnIntroduction,byNolanMcCartyandAdamMeirowitz
EssentialMathematicsforPoliticalandSocialResearch,byJeffGill
SpatialModelsofParliamentaryVoting,byKeithT.Poole
EventHistoryModeling:AGuideforSocialScientists,byJanetM.Box-Steffensmeierand
BradfordS.Jones
EcologicalInference:NewMethodologicalStrategies,editedbyGaryKing,OriRosen,and
MartinA.Tanner
Regression and Other Stories
ANDREW GELMAN
ColumbiaUniversity,NewYork
JENNIFER HILL
NewYorkUniversity
AKI VEHTARI
AaltoUniversity,Finland
UniversityPrintingHouse,CambridgeCB28BS,UnitedKingdom
OneLibertyPlaza,20thFloor,NewYork,NY10006,USA
477WilliamstownRoad,PortMelbourne,VIC3207,Australia
314–321,3rdFloor,Plot3,SplendorForum,JasolaDistrictCentre,NewDelhi–110025,India
79AnsonRoad,#06–04/06,Singapore079906
CambridgeUniversityPressispartoftheUniversityofCambridge.
ItfurtherstheUniversity’smissionbydisseminatingknowledgeinthepursuitof
education,learning,andresearchatthehighestinternationallevelsofexcellence.
www.cambridge.org
Informationonthistitle:www.cambridge.org/9781107023987
DOI:10.1017/9781139161879
(cid:2)c AndrewGelman,JenniferHill,andAkiVehtari2021
Thispublicationisincopyright.Subjecttostatutoryexception
andtotheprovisionsofrelevantcollectivelicensingagreements,
noreproductionofanypartmaytakeplacewithoutthewritten
permissionofCambridgeUniversityPress.
Firstpublished2021
PrintedintheUnitedKingdombyTJInternationalLtd,PadstowCornwall
AcataloguerecordforthispublicationisavailablefromtheBritishLibrary.
ISBN978-1-107-02398-7Hardback
ISBN978-1-107-67651-0Paperback
CambridgeUniversityPresshasnoresponsibilityforthepersistenceoraccuracyof
URLsforexternalorthird-partyinternetwebsitesreferredtointhispublication
anddoesnotguaranteethatanycontentonsuchwebsitesis,orwillremain,
accurateorappropriate.
Contents
Preface xi
Whatyoushouldbeabletodoafterreadingandworkingthroughthisbook xi
Funchaptertitles xii
Additionalmaterialforteachingandlearning xiii
Part1: Fundamentals 1
1 Overview 3
1.1 Thethreechallengesofstatistics 3
1.2 Whylearnregression? 4
1.3 Someexamplesofregression 5
1.4 Challengesinbuilding,understanding,andinterpretingregressions 9
1.5 ClassicalandBayesianinference 13
1.6 ComputingleastsquaresandBayesianregression 16
1.7 Bibliographicnote 17
1.8 Exercises 17
2 Dataandmeasurement 21
2.1 Examiningwheredatacomefrom 21
2.2 Validityandreliability 23
2.3 Allgraphsarecomparisons 25
2.4 Dataandadjustment: trendsinmortalityrates 31
2.5 Bibliographicnote 33
2.6 Exercises 34
3 Somebasicmethodsinmathematicsandprobability 35
3.1 Weightedaverages 35
3.2 Vectorsandmatrices 36
3.3 Graphingaline 37
3.4 Exponentialandpower-lawgrowthanddecline;logarithmicandlog-logrelationships 38
3.5 Probabilitydistributions 40
3.6 Probabilitymodeling 45
3.7 Bibliographicnote 47
3.8 Exercises 47
4 Statisticalinference 49
4.1 Samplingdistributionsandgenerativemodels 49
4.2 Estimates,standarderrors,andconfidenceintervals 50
4.3 Biasandunmodeleduncertainty 55
4.4 Statisticalsignificance,hypothesistesting,andstatisticalerrors 57
4.5 Problemswiththeconceptofstatisticalsignificance 60
4.6 Exampleofhypothesistesting: 55,000residentsneedyourhelp! 63
4.7 Movingbeyondhypothesistesting 66
VI CONTENTS
4.8 Bibliographicnote 67
4.9 Exercises 67
5 Simulation 69
5.1 Simulationofdiscreteprobabilitymodels 69
5.2 Simulationofcontinuousandmixeddiscrete/continuousmodels 71
5.3 Summarizingasetofsimulationsusingmedianandmedianabsolutedeviation 73
5.4 Bootstrappingtosimulateasamplingdistribution 73
5.5 Fake-datasimulationasawayoflife 76
5.6 Bibliographicnote 76
5.7 Exercises 76
Part2: Linearregression 79
6 Backgroundonregressionmodeling 81
6.1 Regressionmodels 81
6.2 Fittingasimpleregressiontofakedata 82
6.3 Interpretcoefficientsascomparisons,noteffects 84
6.4 Historicaloriginsofregression 85
6.5 Theparadoxofregressiontothemean 87
6.6 Bibliographicnote 90
6.7 Exercises 91
7 Linearregressionwithasinglepredictor 93
7.1 Example: predictingpresidentialvotesharefromtheeconomy 93
7.2 Checkingthemodel-fittingprocedureusingfake-datasimulation 97
7.3 Formulatingcomparisonsasregressionmodels 99
7.4 Bibliographicnote 101
7.5 Exercises 101
8 Fittingregressionmodels 103
8.1 Leastsquares,maximumlikelihood,andBayesianinference 103
8.2 Influenceofindividualpointsinafittedregression 107
8.3 Leastsquaresslopeasaweightedaverageofslopesofpairs 108
8.4 Comparingtwofittingfunctions: lmandstan_glm 109
8.5 Bibliographicnote 111
8.6 Exercises 111
9 PredictionandBayesianinference 113
9.1 Propagatinguncertaintyininferenceusingposteriorsimulations 113
9.2 Predictionanduncertainty: predict,posterior_linpred,andposterior_predict 115
9.3 PriorinformationandBayesiansynthesis 119
9.4 ExampleofBayesianinference: beautyandsexratio 121
9.5 Uniform,weaklyinformative,andinformativepriorsinregression 123
9.6 Bibliographicnote 128
9.7 Exercises 128
10 Linearregressionwithmultiplepredictors 131
10.1 Addingpredictorstoamodel 131
10.2 Interpretingregressioncoefficients 133
10.3 Interactions 134
10.4 Indicatorvariables 136
10.5 Formulatingpairedorblockeddesignsasaregressionproblem 139
CONTENTS VII
10.6 Example: uncertaintyinpredictingcongressionalelections 140
10.7 Mathematicalnotationandstatisticalinference 144
10.8 Weightedregression 147
10.9 Fittingthesamemodeltomanydatasets 148
10.10Bibliographicnote 149
10.11Exercises 150
11 Assumptions,diagnostics,andmodelevaluation 153
11.1 Assumptionsofregressionanalysis 153
11.2 Plottingthedataandfittedmodel 156
11.3 Residualplots 161
11.4 Comparingdatatoreplicationsfromafittedmodel 163
11.5 Example: predictivesimulationtocheckthefitofatime-seriesmodel 166
11.6 Residualstandarddeviationσandexplainedvariance R2 168
11.7 Externalvalidation: checkingfittedmodelonnewdata 171
11.8 Crossvalidation 172
11.9 Bibliographicnote 180
11.10Exercises 180
12 Transformationsandregression 183
12.1 Lineartransformations 183
12.2 Centeringandstandardizingformodelswithinteractions 185
12.3 Correlationand“regressiontothemean” 187
12.4 Logarithmictransformations 189
12.5 Othertransformations 195
12.6 Buildingandcomparingregressionmodelsforprediction 199
12.7 Modelsforregressioncoefficients 206
12.8 Bibliographicnote 210
12.9 Exercises 211
Part3: Generalizedlinearmodels 215
13 Logisticregression 217
13.1 Logisticregressionwithasinglepredictor 217
13.2 Interpretinglogisticregressioncoefficientsandthedivide-by-4rule 220
13.3 Predictionsandcomparisons 222
13.4 Latent-dataformulation 226
13.5 MaximumlikelihoodandBayesianinferenceforlogisticregression 228
13.6 Crossvalidationandlogscoreforlogisticregression 230
13.7 Buildingalogisticregressionmodel: wellsinBangladesh 232
13.8 Bibliographicnote 237
13.9 Exercises 237
14 Workingwithlogisticregression 241
14.1 Graphinglogisticregressionandbinarydata 241
14.2 Logisticregressionwithinteractions 242
14.3 Predictivesimulation 247
14.4 Averagepredictivecomparisonsontheprobabilityscale 249
14.5 Residualsfordiscrete-dataregression 253
14.6 Identificationandseparation 256
14.7 Bibliographicnote 259
14.8 Exercises 259
VIII CONTENTS
15 Othergeneralizedlinearmodels 263
15.1 Definitionandnotation 263
15.2 Poissonandnegativebinomialregression 264
15.3 Logistic-binomialmodel 270
15.4 Probitregression: normallydistributedlatentdata 272
15.5 Orderedandunorderedcategoricalregression 273
15.6 Robustregressionusingthetmodel 278
15.7 Constructivechoicemodels 279
15.8 Goingbeyondgeneralizedlinearmodels 283
15.9 Bibliographicnote 286
15.10Exercises 286
Part4: Beforeandafterfittingaregression 289
16 Designandsamplesizedecisions 291
16.1 Theproblemwithstatisticalpower 291
16.2 Generalprinciplesofdesign,asillustratedbyestimatesofproportions 293
16.3 Samplesizeanddesigncalculationsforcontinuousoutcomes 297
16.4 Interactionsarehardertoestimatethanmaineffects 301
16.5 Designcalculationsafterthedatahavebeencollected 304
16.6 Designanalysisusingfake-datasimulation 306
16.7 Bibliographicnote 310
16.8 Exercises 310
17 Poststratificationandmissing-dataimputation 313
17.1 Poststratification: usingregressiontogeneralizetoanewpopulation 313
17.2 Fake-datasimulationforregressionandpoststratification 320
17.3 Modelsformissingness 322
17.4 Simpleapproachesforhandlingmissingdata 324
17.5 Understandingmultipleimputation 326
17.6 Nonignorablemissing-datamodels 332
17.7 Bibliographicnote 333
17.8 Exercises 333
Part5: Causalinference 337
18 Causalinferenceandrandomizedexperiments 339
18.1 Basicsofcausalinference 339
18.2 Averagecausaleffects 342
18.3 Randomizedexperiments 345
18.4 Samplingdistributions,randomizationdistributions,andbiasinestimation 346
18.5 Usingadditionalinformationinexperimentaldesign 347
18.6 Properties,assumptions,andlimitationsofrandomizedexperiments 350
18.7 Bibliographicnote 355
18.8 Exercises 356
19 Causalinferenceusingregressiononthetreatmentvariable 363
19.1 Pre-treatmentcovariates,treatments,andpotentialoutcomes 363
19.2 Example: theeffectofshowingchildrenaneducationaltelevisionshow 364
19.3 Includingpre-treatmentpredictors 367
19.4 Varyingtreatmenteffects,interactions,andpoststratification 370
19.5 Challengesofinterpretingregressioncoefficientsastreatmenteffects 373
19.6 Donotadjustforpost-treatmentvariables 374