Table Of ContentBayesian Non- and Semi-parametric
Methods and Applications
PeterE.Rossi
PrincetonUniversityPress
PrincetonandOxford
Unauthenticated
Download Date | 9/1/19 11:50 AM
THEECONOMETRICANDTINBERGEN
INSTITUTESLECTURES
SeriesEditors
HermanK.vanDijkandPhilipHansFranses
TheEconometricInstitute,
ErasmusUniversityRotterdam
TheEconometricInstituteLecturesseriesisajointprojectofPrinceton
University Press and the Econometric Institute at Erasmus University
Rotterdam.
Thisseriescollectsthelecturesofleadingresearcherswhichtheyhave
given at the Econometric Institute for an audience of academics and
students.
Thelecturesareatahighacademiclevelanddealwithtopicsthathave
importantpolicyimplications.Theseriescoversawiderangeoftopics
ineconometrics.Itisnotconfinedtoanyoneareaorsub-discipline.
TheEconometricInstituteistheleadingresearchcenterineconomet-
rics and management science in the Netherlands. The Institute was
foundedin1956byJanTinbergenandHenriTheil,withTheilbeing
itsfirstdirector.TheInstitutehasreceivedworldwiderecognitionwith
anadvancedtrainingprogramforvariousdegreesineconometrics.
Otherbooksinthisseriesinclude
AnticipatingCorrelations:ANewParadigm
forRiskManagementbyRobertEngle
CompleteandIncompleteEconometricModels
byJohnGeweke
SocialChoicewithPartialKnowledgeof
TreatmentResponsebyCharlesF.Manski
YieldCurveModelingandForecasting:
TheDynamicNelson-SiegelApproach
byFrancisX.DieboldandGlennD.Rudebusch
Unauthenticated
Download Date | 9/1/19 11:50 AM
Copyright©2014byPrincetonUniversityPress
PublishedbyPrincetonUniversityPress,41WilliamStreet,Princeton,
NewJersey08540
IntheUnitedKingdom:PrincetonUniversityPress,6OxfordStreet,
Woodstock,OxfordshireOX201TW
press.princeton.edu
AllRightsReserved
LibraryofCongressCataloging-in-PublicationData
Rossi,PeterE.(PeterEric),1955–
Bayesiannon-andsemi-parametricmethodsandapplications/PeterE.Rossi.
pagescm.—(TheEconometricandTinbergenInstituteslectures)
Includesbibliographicalreferencesandindex.
ISBN978-0-691-14532-7(hardcover:alk.paper)1.Econometrics.2.Bayesian
statisticaldecisiontheory.3.Mathematicaleconomics.I.Title.
HB139.R642014
330.01'519542—dc23
2013038609
BritishLibraryCataloging-in-PublicationDataisavailable
ThisbookhasbeencomposedinSabonNextLTPro
Printedonacid-freepaper.∞
TypesetbySRNovaPvtLtd,Bangalore,India
PrintedintheUnitedStatesofAmerica
10987654321
Unauthenticated
Download Date | 9/1/19 11:50 AM
Contents
Preface vii
1 MixturesofNormals 1
1.1 FiniteMixtureofNormalsLikelihood
Function 6
1.2 MaximumLikelihoodEstimation 9
1.3 BayesianInferencefortheMixtureof
NormalsModel 15
1.4 PriorsandtheBayesianModel 16
1.5 UnconstrainedGibbsSampler 25
1.6 Label-Switching 29
1.7 Examples 34
1.8 ClusteringObservations 46
1.9 MarginalizedSamplers 49
2 DirichletProcessPriorandDensity
Estimation 59
2.1 DirichletProcesses—AConstruction 60
2.2 FiniteandInfiniteMixtureModels 64
2.3 Stick-BreakingRepresentation 68
2.4 PolyaUrnRepresentationandAssociated
GibbsSampler 70
2.5 PriorsonDPParametersand
Hyper-parameters 72
2.6 GibbsSamplerforDPModelsandDensity
Estimation 78
2.7 ScalingtheData 80
2.8 DensityEstimationExamples 81
Unauthenticated
Download Date | 9/1/19 11:50 AM
vi Contents
3 Non-parametricRegression 90
3.1 Jointvs.ConditionalDensityApproaches 90
3.2 ImplementingtheJointApproachwith
MixturesofNormals 93
3.3 ExamplesofNon-parametricRegression
UsingJointApproach 96
3.4 DiscreteDependentVariables 104
3.5 AnExampleofExpenditureFunction
Estimation 108
4 Semi-parametricApproaches 115
4.1 Semi-parametricRegressionwithDPPriors 115
4.2 Semi-parametricIVModels 122
5 RandomCoefficientModels 152
5.1 Introduction 152
5.2 Semi-parametricRandomCoefficientLogit
Models 157
5.3 AnEmpiricalExampleofaSemi-parametric
RandomCoefficientLogitModel 161
6 ConclusionsandDirections
forFutureResearch 187
6.1 WhenAreNon-parametricand
Semi-parametricMethodsMostUseful? 187
6.2 Semi-parametricorNon-parametric
Methods? 189
6.3 Extensions 191
Bibliography 195
Index 201
Unauthenticated
Download Date | 9/1/19 11:50 AM
Preface
Much of the work in Bayesian econometrics has focused on
showing the value of Bayesian methods for parametric models
(see, for example, Geweke (2005), Koop (2003), Li and Tobias
(2011), and Rossi, Allenby, and McCulloch (2005)). This lit-
erature has documented the superior sampling properties of
Bayesian parametric methods, particularly in situations with
limitedsampleinformation.Forexample,Bayesianhierarchical
models have become standard in the marketing literature be-
cause of the prevalence of panel data structures and consumer
heterogeneity. While these models that have proved so useful
inmicro-econometricsandmarketingapplicationsinvolvevery
large numbers of parameters, both the panel-unit level model
(typicallyalinearregressionormultinomiallogitspecification)
and the distribution of heterogeneity (the random “effects”
distribution—commonly assumed to be multivariate normal)
are parametric. The purpose of this book is to show how
Bayesian non-parametric methods can be applied to economet-
ric and marketing applications. The Bayesian non-parametric
methods used here are finite and infinite mixture models and
the central theme is to express the non-parametric problem as
fundamentallyaproblemofdensityapproximation/estimation.
A Bayesian approach to non-parametric problems is funda-
mentally what I will call a “full-information” approach. That
is, all Bayesian approaches are likelihood-based and require an
approximation to the joint distribution of all observables. This
contrasts to the approach often favored in the non-Bayesian
econometrics literature which I will call “partial-information.”
Inthe“partial-information”approach(oftenthebasisforGMM
methods), only certain assumptions (such as independence or
orthogonalityrestrictions)areusedtoformtheestimatorofthe
model parameters and no attempt is made to model the entire
distribution of the data. In the “partial-information” approach,
Brought to you by | New York University
Authenticated
Download Date | 4/22/17 3:54 AM
viii Preface
the sampling distribution of the estimator is derived (typically
viaasymptoticmethods)withoutmakingexplicitdistributional
assumptions. The resulting inference is often claimed to be
“distribution-free”forthisreason.However,itiswellrecognized
that there can be considerable efficiency losses from this ap-
proach.Theargumentthatisusedtojustifytheapproachisthat
it does not require arbitrary and unsubstantiated distributional
assumptions.Bayesianparametricapproachesarethencriticized
onthebasisthattheyrequirearbitrarydistributionassumptions
whichoftenarenotexamined.Concernsformodelspecification
cause the “partial-information” advocate to be willing to trade-
offefficiencyforrobustnessofinference.
Inprinciple,afullynon-parametricBayesianapproachshould
removeconcernsregardingmodelmis-specificationwhileretain-
ingthedesirablesamplingpropertiesofa“full-information”or
likelihood-based procedure. As apractical matter,however, the
full non-parametric approach may require the approximation
of high dimensional distributions. Bayesian approaches will
essentiallyinvolveveryflexibleorhighlyparameterizedmodels
thatcanbegivenanon-parametricinterpretation.Non-Bayesian
procedures for highly parameterized models often suffer from
the over-fitting problem where small subsets of data drive the
model fit and the model produces very non-smooth estimates.
The advantage of a Bayesian method in a highly parameter-
ized model is that, with proper priors, the tendency to “over-
fit” is reduced substantially. This is because, in many models,
proper priors impose what amounts to a penalty for highly
parameterized models and implement a form of shrinkage.1
ForBayesianprocedures,theconcernsregardingover-fittingare
replacedwiththeproblemofassessmentofproperpriors.There
has been insufficient attention devoted to the assessment of
priorsinBayesiannon-parametricapplications.Theassumption
oftenmadeisthatqualityofBayesiandensityapproximationis
1Heretheterm“shrinkage”referstothepropertyofBayesestimatorswith
proper priors to be somewhere between the location of the prior and the
likelihood,therebyreducingthesensitivityoftheestimatorstosamplingerror.
Brought to you by | New York University
Authenticated
Download Date | 4/22/17 3:54 AM
Preface ix
notinfluencedagreatdealbythepriorsettings.Thereisasense
in which this statement is obviously false—there certainly are
priorsettingswhichexertagreatdealofinfluenceandlimitthe
flexibilityoftheBayesianprocedure.Moreimportantly,someof
the standard, proper but diffuse, settings used in practice can
be highly informative in ways that, perhaps, are unintended.
Whatisrequiredisaprocedureforpriorassessmentthatretains
flexibility and smoothing. I find that it is relatively straightfor-
ward toavoidoverlyand,perhaps,unintentionallyinformative
priorsandfindaregionofpriorsettingswherepriorsensitivity
is limited and yet the procedure retains desirable smoothing
properties.
Inthenon-Bayesianliteratureonnon-parametricapproaches,
thestyleofresearchistoproposea“non-parametric”procedure
and then prove that as the sample size approaches infinity this
procedure“consistently”(basedonsomenormwhichmeasures
the difference between the non-parametric approximation and
the“true”model)recoversthetruemodel.Insomecases,further
analysis is done to determine the optimal rate at which para-
meters must be added to the “non-parametric” approximation
and to provide some sort of asymptotic distribution theory. Of
course,anynumberofapproachesareconsistentinthesensethat
there are many different possible sets of basis functions which
can be used to approximate an arbitrary distribution. In some
cases, some sort of theory of “testing” or a penalty function is
used to expand the model as the sample size increases at a rate
sufficienttoassureconsistencybutnottoorapidlytoavoidover-
fitting. For the user of the procedure, there is little guidance
as to how to apply the approximation to one sample of a fixed
size.Cross-validationmethodsaresometimesusedto“tune”the
approximation for a given sample, but have unknown finite
sampleproperties.
In the approach taken here, I will use mixtures of normals
as the basis for approximations. It is obvious that mixtures of
normals have the desired “consistency” property required of a
non-parametricapproach.Whatismoreimportantistodemon-
stratethattheBayesianproceduresprovidereasonableestimates
Brought to you by | New York University
Authenticated
Download Date | 4/22/17 3:54 AM
x Preface
forthedemandingsortsofexamplesconsideredinappliedwork.
With proper procedures for prior assessment, the “flexibility”
vs “over-fitting” problem is largely avoided. Approximations
involving literally thousands of parameters can be used with-
out concern. The Bayes Factors2 implicit in Bayesian inference
impose such strong smoothing and shrinkage properties that I
donotobservetheover-fittingproblem.ThismeansthatIdon’t
have to couple my inference procedure with additional (often
ad hoc) methods for avoiding the over-fitting problem. This
providesarealadvantagetoathoughtfulBayesianapproach.
There is a sense, however, that all non-parametric methods,
no matter how powerful, are very demanding of the data.
Ultimately, inference regarding densities demands a uniform
distribution or lack of sparseness in the data. It is not so
much the number of observations that matters, but more that
the observations are spread evenly across the, possibly high
dimensional, space. For sparse data or data with “holes,” non-
parametricmethodswillnotworkaswell.Thegreatadvantage
ofBayesianproceduresisthattheinferenceprocedureproduces
reliable information regarding the precision of inference re-
garding the density and any functional thereof. This inference
comesessentiallyfreeaspartoftheBayescomputingprocedure
and without other delta method or bootstrap computations.
The inference will show that in sparse areas of the space, the
density is imprecisely estimated. Although it is reassuring to
know that the Bayesian procedure will properly reflect lack of
sample information, this does not remove the problem which
is particularly acute with high dimensional distributions. For
this reason, interest may focus on semi-parametric Bayesian
methods. In semi-parametric methods, part of the problem
is modeled with parametric assumptions and non-parametric
2The Bayes Factor is the quantity by which one transforms the prior
probabilityofamodeltoitsposte(cid:1)riorprobability.Inanyparametricmodel,the
BayesFactorcanbeexpressedas p(Data|θ)p(θ)dθ,where p(Data|θ)isthe
distributionoftheobserveddatagiventheparameters(the“model”)andp(θ)is
theprior.
Brought to you by | New York University
Authenticated
Download Date | 4/22/17 3:54 AM
Preface xi
methods are employed for the remaining smaller dimensional
part of the problem. Linear index models are a good exam-
ple of a semi-parametric approach. For example, consider the
problem of modeling the joint distribution of (y,x). If this is
high dimensional, the semi-parametric approach would be to
model the bivariate joint distribution of (y,x(cid:2)γ) using non-
parametric density approximations. In this semi-parametric ap-
proach,thelinearindexservesasadimensionreductiondevice
(seeChapter4forfurtherdiscussion).
Paneldataprovideagoodexampleofanaturalsettingwhere
semi-parametric approaches are desirable. Invariably, panels in-
volve a large number of cross-sectional units tracked over a
limiteddurationoftime.Herethetimedimensionisinsufficient
toallowfornon-parametricapproachestothemodelatthecross-
sectionallevel,buttherearesufficientcross-sectionalunitstoal-
lowforanon-parametricdeterminationofarandomcoefficient
distribution.Forexample,withpurchasedata,wemightspecify
a standard multinomial logit model for each unit indexed by a
parametervectorwhichhasanarbitrarydistributionacrossunits
(seeChapter5).
Ultimately, part of the output of a non-parametric or semi-
parametric approach will be a density estimate or some func-
tional of this object. For example, a non-parametric random
coefficient model will yield a joint distribution of preference
parameters across consumers. This joint distribution will be
represented by a high dimensional density surface. A challenge
will be to summarize this density in a meaningful way. The
ability to compute univariate and bivariate marginals from
this distribution will become important and this is another
advantageofthemixtureapproachinthattheimpliedmarginal
distributionsareeasytocalculate.Visualizationtechniqueswill
play an important role in summarizing these distributions as
moments lose much of their significance or interpretability for
non-ellipticaldistributions.
I develop all of the methods here from first principles so
that this work is accessible to anyone with a reasonable intro-
ductory knowledge of Bayesian statistics. Specifically, I assume
Brought to you by | New York University
Authenticated
Download Date | 4/22/17 3:54 AM
