Table Of ContentRegression for Categorical Data
Thisbookintroducesbasicandadvancedconceptsofmoderncategori-
calregressionwithafocusonthestructuringconstituentsofregression.
Meantforstatisticians,appliedresearchers,andstudents,itincludesmany
topicsnotnormallyincludedinbooksoncategoricaldataanalysis,includ-
ingrecentdevelopmentsinflexibleandhigh-dimensionalregression.
In addition to standard methods such as logit and probit models and
their extensions to multivariate settings, the book presents more recent
developmentsinregularizedregressionwithafocusontheselectionof
predictors;toolsforflexiblenonparametricregressionthatyieldfitsthat
areclosertothedata;advancedmodelsforcountdata;nonstandardtree-
based ensemble methods; and tools for the handling of both nominal
and ordered categorical predictors. Issues of prediction are explicitly
consideredinachapterthatintroducesstandardandnewerclassification
techniques.
SoftwareincludinganRpackagethatcontainsdatasetsandcodefor
most of the examples is available from http://www.stat.uni-muenchen
.de/∼tutz/catdata.
Dr.GerhardTutzisaProfessorofStatisticsintheDepartmentofStatistics
atLudwig-MaximiliansUniversity,Munich.HewasformerlyaProfessor
attheTechnicalUniversityBerlin.Heistheauthororco-authorofnine
booksandmorethan100papers.
CAMBRIDGE SERIES IN STATISTICAL AND PROBABILISTIC MATHEMATICS
EditorialBoard
Z.Ghahramani(DepartmentofEngineering,UniversityofCambridge)
R.Gill(MathematicalInstitute,LeidenUniversity)
F.P.Kelly(DepartmentofPureMathematicsandMathematicalStatistics,University
ofCambridge)
B.D.Ripley(DepartmentofStatistics,UniversityofOxford)
S.Ross(DepartmentofIndustrialandSystemsEngineering,UniversityofSouthern
California)
M.Stein(DepartmentofStatistics,UniversityofChicago)
This series of high-quality upper-division textbooks and expository monographs covers
all aspects of stochastic applicable mathematics. The topics range from pure and applied
statistics to probability theory, operations research, optimization, and mathematical pro-
gramming. The books contain clear presentations of new developments in the field and
alsoofthestateoftheartinclassicalmethods.Whileemphasizingrigoroustreatmentof
theoreticalmethods,thebooksalsocontainapplicationsanddiscussionsofnewtechniques
madepossiblebyadvancesincomputationalpractice.
A complete list of books in the series can be found at http://www.cambridge.org/uk/
series/sSeries.asp?code=CSPM.
Recenttitlesincludethefollowing:
8. AUser’sGuidetoMeasureTheoreticProbability,byDavidPollard
9. TheEstimationandTrackingofFrequency,byB.G.QuinnandE.J.Hannan
10. DataAnalysisandGraphicsUsingR,byJohnMaindonaldandJohnBraun
11. StatisticalModels,byA.C.Davison
12. SemiparametricRegression,byDavidRuppert,M.P.Wand,andR.J.Carroll
13. ExercisesinProbability,byLoicChaumontandMarcYor
14. StatisticalAnalysisofStochasticProcessesinTime,byJ.K.Lindsey
15. MeasureTheoryandFiltering,byLakhdarAggounandRobertElliott
16. EssentialsofStatisticalInference,byG.A.YoungandR.L.Smith
17. ElementsofDistributionTheory,byThomasA.Severini
18. StatisticalMechanicsofDisorderedSystems,byAntonBovier
19. TheCoordinate-FreeApproachtoLinearModels,byMichaelJ.Wichura
20. RandomGraphDynamics,byRickDurrett
21. Networks,byPeterWhittle
22. SaddlepointApproximationswithApplications,byRonaldW.Butler
23. AppliedAsymptotics,byA.R.Brazzale,A.C.Davison,andN.Reid
24. RandomNetworksforCommunication,byMassimoFranceschettiandRonaldMeester
25. DesignofComparativeExperiments,byR.A.Bailey
26. SymmetryStudies,byMarlosA.G.Viana
27. ModelSelectionandModelAveraging,byGerdaClaeskensandNilsLidHjort
28. BayesianNonparametrics,editedbyNilsLidHjortetal.
29. FromFiniteSampletoAsymptoticMethods inStatistics,byPranab K.Sen,JulioM.
Singer,andAntonioC.PedrosadeLima
30. BrownianMotion,byPeterMo¨rtersandYuvalPeres
31. Probability,byRickDurrett
Regression for Categorical Data
GERHARD TUTZ
Ludwig-MaximiliansUniversita¨t
cambridgeuniversitypress
Cambridge,NewYork,Melbourne,Madrid,CapeTown,
Singapore,Sa˜oPaulo,Delhi,Tokyo,MexicoCity
CambridgeUniversityPress
32AvenueoftheAmericas,NewYork,NY10013-2473,USA
www.cambridge.org
Informationonthistitle:www.cambridge.org/9781107009653
(cid:3)C GerhardTutz2012
Thispublicationisincopyright.Subjecttostatutoryexception
andtotheprovisionsofrelevantcollectivelicensingagreements,
noreproductionofanypartmaytakeplacewithoutthewritten
permissionofCambridgeUniversityPress.
Firstpublished2012
PrintedintheUnitedStatesofAmerica
AcatalogrecordforthispublicationisavailablefromtheBritishLibrary.
LibraryofCongressCataloginginPublicationdata
Tutz,Gerhard.
Regressionforcategoricaldata/GerhardTutz.
p. cm.–(Cambridgeseriesinstatisticalandprobabilisticmathematics)
ISBN978-1-107-00965-3(hardback)
1.Regressionanalysis. 2.Categories(Mathematics) I.Title. II.Series.
QA278.2.T882011
519.5(cid:4)36–dc22 2011000390
ISBN978-1-107-00965-3Hardback
Additionalresourcesforthispublicationathttp://www.stat.uni-muenchen.de/∼tutz/catdata.
CambridgeUniversityPresshasnoresponsibilityforthepersistenceoraccuracyofURLsfor
externalorthird-partyInternetWebsitesreferredtointhispublicationanddoesnot
guaranteethatanycontentonsuchWebsitesis,orwillremain,accurateorappropriate.
Contents
Preface ix
1 Introduction 1
1.1 CategoricalData: ExamplesandBasicConcepts. . . . . . . . . . . . . . . . . 1
1.2 OrganizationofThisBook . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5
1.3 BasicComponentsofStructuredRegression . . . . . . . . . . . . . . . . . . . 6
1.4 ClassicalLinearRegression. . . . . . . . . . . . . . . . . . . . . . . . . . . . 15
1.5 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27
2 BinaryRegression: TheLogitModel 29
2.1 DistributionModelsforBinaryResponsesandBasicConcepts . . . . . . . . . 29
2.2 LinkingResponseandExplanatoryVariables . . . . . . . . . . . . . . . . . . 33
2.3 TheLogitModel . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37
2.4 TheOriginsoftheLogisticFunctionandtheLogitModel . . . . . . . . . . . . 48
2.5 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 49
3 GeneralizedLinearModels 51
3.1 BasicStructure . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 51
3.2 GeneralizedLinearModelsforContinuousResponses. . . . . . . . . . . . . . 53
3.3 GLMsforDiscreteResponses . . . . . . . . . . . . . . . . . . . . . . . . . . 56
3.4 FurtherConcepts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 60
3.5 ModelingofGroupedData . . . . . . . . . . . . . . . . . . . . . . . . . . . . 62
3.6 MaximumLikelihoodEstimation . . . . . . . . . . . . . . . . . . . . . . . . . 63
3.7 Inference . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 67
3.8 Goodness-of-FitforGroupedObservations . . . . . . . . . . . . . . . . . . . 72
3.9 ComputationofMaximumLikelihoodEstimates . . . . . . . . . . . . . . . . 75
3.10 HatMatrixforGeneralizedLinearModels . . . . . . . . . . . . . . . . . . . . 76
3.11 Quasi-LikelihoodModeling. . . . . . . . . . . . . . . . . . . . . . . . . . . . 78
3.12 FurtherReading . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 79
3.13 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 79
4 ModelingofBinaryData 81
4.1 MaximumLikelihoodEstimation . . . . . . . . . . . . . . . . . . . . . . . . . 82
4.2 DiscrepancybetweenDataandFit . . . . . . . . . . . . . . . . . . . . . . . . 87
4.3 DiagnosticChecks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 93
4.4 StructuringtheLinearPredictor . . . . . . . . . . . . . . . . . . . . . . . . . 101
4.5 ComparingNon-NestedModels . . . . . . . . . . . . . . . . . . . . . . . . . 113
4.6 ExplanatoryValueofCovariates . . . . . . . . . . . . . . . . . . . . . . . . . 114
4.7 FurtherReading . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 119
4.8 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 120
v
vi CONTENTS
5 AlternativeBinaryRegressionModels 123
5.1 AlternativeLinksinBinaryRegression. . . . . . . . . . . . . . . . . . . . . . 123
5.2 TheMissingLink . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 130
5.3 Overdispersion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 132
5.4 ConditionalLikelihood . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 138
5.5 FurtherReading . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 140
5.6 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 140
6 RegularizationandVariableSelectionforParametricModels 143
6.1 ClassicalSubsetSelection . . . . . . . . . . . . . . . . . . . . . . . . . . . . 144
6.2 RegularizationbyPenalization . . . . . . . . . . . . . . . . . . . . . . . . . . 145
6.3 BoostingMethods . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 163
6.4 SimultaneousSelectionofLinkFunctionandPredictors. . . . . . . . . . . . . 170
6.5 CategoricalPredictors. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 173
6.6 BayesianApproach . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 178
6.7 FurtherReading . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 179
6.8 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 179
7 RegressionAnalysisofCountData 181
7.1 ThePoissonDistribution . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 182
7.2 PoissonRegressionModel . . . . . . . . . . . . . . . . . . . . . . . . . . . . 185
7.3 InferenceforthePoissonRegressionModel . . . . . . . . . . . . . . . . . . . 186
7.4 PoissonRegressionwithanOffset . . . . . . . . . . . . . . . . . . . . . . . . 190
7.5 PoissonRegressionwithOverdispersion . . . . . . . . . . . . . . . . . . . . . 192
7.6 NegativeBinomialModelandAlternatives. . . . . . . . . . . . . . . . . . . . 194
7.7 Zero-InflatedCounts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 198
7.8 HurdleModels . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 200
7.9 FurtherReading . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 203
7.10 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 204
8 MultinomialResponseModels 207
8.1 TheMultinomialDistribution . . . . . . . . . . . . . . . . . . . . . . . . . . . 209
8.2 TheMultinomialLogitModel . . . . . . . . . . . . . . . . . . . . . . . . . . 210
8.3 MultinomialModelasRandomUtilityModel . . . . . . . . . . . . . . . . . . 215
8.4 StructuringthePredictor . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 215
8.5 LogitModelasMultivariateGeneralizedLinearModel . . . . . . . . . . . . . 217
8.6 InferenceforMulticategoricalResponseModels . . . . . . . . . . . . . . . . . 218
8.7 MultinomialModelswithHierarchicallyStructuredResponse . . . . . . . . . 223
8.8 DiscreteChoiceModels. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 226
8.9 NestedLogitModel . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 231
8.10 RegularizationfortheMultinomialModel . . . . . . . . . . . . . . . . . . . . 233
8.11 FurtherReading . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 238
8.12 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 239
9 OrdinalResponseModels 241
9.1 CumulativeModels . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 243
9.2 SequentialModels . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 252
9.3 FurtherPropertiesandComparisonofModels . . . . . . . . . . . . . . . . . . 255
9.4 AlternativeModels . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 257
9.5 InferenceforOrdinalModels . . . . . . . . . . . . . . . . . . . . . . . . . . . 261
CONTENTS vii
9.6 FurtherReading . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 265
9.7 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 265
10 Semi-andNon-ParametricGeneralizedRegression 269
10.1 UnivariateGeneralizedNon-ParametricRegression . . . . . . . . . . . . . . . 269
10.2 Non-ParametricRegressionwithMultipleCovariates . . . . . . . . . . . . . . 285
10.3 StructuredAdditiveRegression . . . . . . . . . . . . . . . . . . . . . . . . . . 289
10.4 FunctionalDataandSignalRegression . . . . . . . . . . . . . . . . . . . . . . 307
10.5 FurtherReading . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 313
10.6 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 314
11 Tree-BasedMethods 317
11.1 RegressionandClassificationTrees . . . . . . . . . . . . . . . . . . . . . . . 317
11.2 MultivariateAdaptiveRegressionSplines . . . . . . . . . . . . . . . . . . . . 328
11.3 FurtherReading . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 329
11.4 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 329
12 TheAnalysisofContingencyTables: Log-LinearandGraphicalModels 331
12.1 TypesofContingencyTables . . . . . . . . . . . . . . . . . . . . . . . . . . . 332
12.2 Log-LinearModelsforTwo-WayTables . . . . . . . . . . . . . . . . . . . . . 335
12.3 Log-LinearModelsforThree-WayTables . . . . . . . . . . . . . . . . . . . . 338
12.4 SpecificLog-LinearModels . . . . . . . . . . . . . . . . . . . . . . . . . . . 341
12.5 Log-LinearandGraphicalModelsforHigherDimensions. . . . . . . . . . . . 345
12.6 Collapsibility . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 348
12.7 Log-LinearModelsandtheLogitModel . . . . . . . . . . . . . . . . . . . . . 349
12.8 InferenceforLog-LinearModels . . . . . . . . . . . . . . . . . . . . . . . . . 350
12.9 ModelSelectionandRegularization . . . . . . . . . . . . . . . . . . . . . . . 354
12.10MosaicPlots . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 357
12.11 FurtherReading . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 358
12.12Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 359
13 MultivariateResponseModels 363
13.1 ConditionalModeling . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 365
13.2 MarginalParametrizationandGeneralizedLog-LinearModels . . . . . . . . . 370
13.3 GeneralMarginalModels: AssociationasNuisanceandGEEs . . . . . . . . . 371
13.4 MarginalHomogeneity . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 385
13.5 FurtherReading . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 392
13.6 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 393
14 RandomEffectsModelsandFiniteMixtures 395
14.1 LinearRandomEffectsModelsforGaussianData . . . . . . . . . . . . . . . . 396
14.2 GeneralizedLinearMixedModels . . . . . . . . . . . . . . . . . . . . . . . . 402
14.3 EstimationMethodsforGeneralizedMixedModels . . . . . . . . . . . . . . . 407
14.4 MulticategoricalResponseModels . . . . . . . . . . . . . . . . . . . . . . . . 416
14.5 TheMarginalizedRandomEffectsModel . . . . . . . . . . . . . . . . . . . . 419
14.6 LatentTraitModelsandConditionalML . . . . . . . . . . . . . . . . . . . . . 420
14.7 SemiparametricMixedModels . . . . . . . . . . . . . . . . . . . . . . . . . . 420
14.8 FiniteMixtureModels . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 422
14.9 FurtherReading . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 426
14.10Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 427
viii CONTENTS
15 PredictionandClassification 429
15.1 BasicConceptsofPrediction . . . . . . . . . . . . . . . . . . . . . . . . . . . 430
15.2 MethodsforOptimalClassification . . . . . . . . . . . . . . . . . . . . . . . . 438
15.3 BasicsofEstimatedClassificationRules . . . . . . . . . . . . . . . . . . . . . 445
15.4 ParametricClassificationMethods . . . . . . . . . . . . . . . . . . . . . . . . 451
15.5 Non-ParametricMethods . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 457
15.6 NeuralNetworks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 468
15.7 Examples . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 471
15.8 VariableSelectioninClassification . . . . . . . . . . . . . . . . . . . . . . . . 473
15.9 PredictionofOrdinalOutcomes . . . . . . . . . . . . . . . . . . . . . . . . . 474
15.10Model-BasedPrediction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 480
15.11FurtherReading . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 481
15.12Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 482
A Distributions 485
A.1 DiscreteDistributions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 485
A.2 ContinuousDistributions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 487
B SomeBasicTools 490
B.1 LinearAlgebra . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 490
B.2 TaylorApproximation. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 491
B.3 ConditionalExpectation,Distribution . . . . . . . . . . . . . . . . . . . . . . 493
B.4 EMAlgorithm. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 494
C ConstrainedEstimation 496
C.1 SimplificationofPenalties . . . . . . . . . . . . . . . . . . . . . . . . . . . . 496
C.2 LinearConstraints. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 498
C.3 FisherScoringwithPenaltyTerm . . . . . . . . . . . . . . . . . . . . . . . . 499
D Kullback-LeiblerDistanceandInformation-BasedCriteriaofModelFit 500
D.1 Kullback-LeiblerDistance . . . . . . . . . . . . . . . . . . . . . . . . . . . . 500
E NumericalIntegrationandToolsforRandomEffectsModeling 504
E.1 LaplaceApproximation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 504
E.2 Gauss-HermiteIntegration . . . . . . . . . . . . . . . . . . . . . . . . . . . . 505
E.3 InversionofPseudo-FisherMatrix . . . . . . . . . . . . . . . . . . . . . . . . 507
ListofExamples 509
Bibliography 513
AuthorIndex 545
SubjectIndex 554
