Introduction to Maximum Likelihood Estimation and AIC model selection

41 Pages·2011·2.29 MB·English

Sampling, Regression, Experimental Design and
Analysis for Environmental Scientists,
Biologists, and Resource Managers
C. J. Schwarz
Department of Statistics and Actuarial Science, Simon Fraser University
[email protected]
January 16, 2011
Contents
1 Maximum Likelihood Estimation & AIC and model selection -
a primer 2
1.1 Maximum Likelihood Estimation (MLE) . . . . . . . . . . . . . . 2
1.1.1 The probability model . . . . . . . . . . . . . . . . . . . . 3
1.1.2 The likelihood . . . . . . . . . . . . . . . . . . . . . . . . 7
1.1.3 The MLE . . . . . . . . . . . . . . . . . . . . . . . . . . . 8
1.1.4 The precision of the MLE . . . . . . . . . . . . . . . . . . 10
1.1.5 Numerical optimization . . . . . . . . . . . . . . . . . . . 11
1.1.6 Example2: Different sizes of sampling units. . . . . . . . . 14
1.1.7 Example: What is the probability of becoming pregnant
in a month? . . . . . . . . . . . . . . . . . . . . . . . . . . 17
1.1.8 Example: Zero-truncated distribution . . . . . . . . . . . 21
1.1.9 Example: Capture-recapture with 2 parameters . . . . . . 21
1.1.10 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . 24
1.2 AIC and Model Selection . . . . . . . . . . . . . . . . . . . . . . 24
1.2.1 Technical Details . . . . . . . . . . . . . . . . . . . . . . . 26
1.2.2 AIC and regression . . . . . . . . . . . . . . . . . . . . . . 28
1.2.3 Example: Capture-recapture . . . . . . . . . . . . . . . . 29
1.2.4 Example: Survival of Juvenile Pygmy Rabbits . . . . . . 38
1.3 References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 40
1
Chapter 1
Maximum Likelihood
Estimation & AIC and model
selection - a primer
1.1 Maximum Likelihood Estimation (MLE)
These notes provide a (very) brief introduction to Maximum Likelihood Esti-
mation(MLE)atalowertechnicallevelthanwouldbefoundinaformalcourse
in Statistics. The important part of these notes are the ideas – don’t worry too
much about the technical aspects unless you wish to use maximum likelihood
for a non-standard problem. In the latter case, please don’t hesitate to contact
me to discuss your problem.
A review of MLE is found at http://en.wikipedia.org/wiki/Maximum_
likelihood.
Maximum Likelihood Estimation (MLE) was first developed by Sir R. A.
Fisher a famous geneticist and statistician. It is closely related to the con-
ceptofprobabilityandprovidesacomprehensivejustifiablemethodtoestimate
parameters from data. It is a standard tool used by statisticians for many
problems.
The key advantages of MLE are that:
• a unified framework to obtain estimators.
2
CHAPTER 1. MAXIMUM LIKELIHOOD ESTIMATION & AIC AND
MODEL SELECTION - A PRIMER
• in large samples is fully efficient, i.e. extracts all the information possible
from data.
• provides a standard way to obtain estimates of precision (se) for any data
collection scheme along with a way to estimate confidence intervals.
However, MLE is not a panacea and there are several “problems” with MLE
that may require other methods (such as Bayesian methods):
• The properties of MLE in small samples may not be optimal nor exact
• Thelikelihoodfunctionmaynotbecomputationallytractable. Thisoften
arises because the likelihood has to integrate over hidden variables and
the likelihood is a multi-dimensional integral.
1.1.1 The probability model
ThefirststepinusingMLEistodefineaprobabilitymodelforyourdata. Thisis
crucialbecauseifyourprobabilitymodeliswrong,thenallfurtherinferencewill
also be incorrect. Statisticians recognize that all probability models are wrong,
but hope that they are close enough to reality to be useful. Consequently an
important part of MLE is model assessment – how well does the model fit the
data. This won’t be covered in this primer – contact me for details.
Forexample,considertheproblemofestimatingthedensityofobjects(grey
dots) in a region as shown on the next page
3
(cid:13)c2010CarlJamesSchwarz
4
CHAPTER 1. MAXIMUM LIKELIHOOD ESTIMATION & AIC AND
MODEL SELECTION - A PRIMER
Of course in real life the locations of all the objects would be unknown
becauseifthelocations(andnumberofobjects)wereknown,thenitwouldbea
simplemattertocountallofthedotsanddividebythestudyareatoobtainthe
densityvalue! Consequently, threesmallareaswererandomlyselectedfromthe
area (how would this be done?) and the number of objects within the sampling
quadrats are counted as shown on the next page:
5
(cid:13)c2010CarlJamesSchwarz
6
CHAPTER 1. MAXIMUM LIKELIHOOD ESTIMATION & AIC AND
MODEL SELECTION - A PRIMER
This gives three counts, 4, 10, and 15. Note that the sampling protocol has
to be explicit on what to do with objects that intersect the boundary of the
sampling frame – this will not be discussed further in these notes.
Anon-parametricwaytoproceedwouldbetosimplytaketheaveragecount
(4+10+15 = 9.67) and report this as the density per sampling unit.1 However,
3
this simple approach will fail (as seen later) in more complex cases where the
data are not a straightforward.
A MLE approach would start with a model for the data. In many cases,
counts of objects follow a Poisson distribution. This would be true if objects
occurred “at random” on the landscape. It is not always true – for example,
objects could tend to fall in clusters (new plants growing from seeds from a
parent plant), or objects could be more spread out (competition for resources)
than random. In both cases, a more complex probability model could be fit. If
a simple Poisson model is used, an important part of the MLE process is model
assessment as noted earlier.
APoissondistributiondependsupontheparameterµwhich(inthiscase)is
themeannumberofdotspersamplingunitorthetruedensityinthepopulation.
Let Y represent the random variable for the number of objects found in each
sample. If the value of µ is known, then the probability of seeing Y =y objects
2 in a sample follows the Poisson probability law:
e−µµy
P(Y =y|µ)=
y!
For example, if µ=1, then P(Y =2|µ=1)= e−112 =.184.
2!
Of course, the value of µ is NOT known and this is the parameter that we
would like to estimate from our sample of size 3.
1.1.2 The likelihood
MLE starts by defining the likelihood of each data value.
e−µµy
L(µ|Y =y)=
y!
The likelihood looks familiar and is in fact the same probability function but
with the roles of Y and µ are reversed. A probability function gives the proba-
bility of observing Y given the parameter µ and sums to 1 over all the possible
1Thisestimatorreliesonthesamplingbeingasimplerandomsamplefromthepopulation
isadesignbasedestimatorascoveredinthechapteronSamplingTheory.
2The notation Y =y is read as the random variable Y taking the value y. For example,
Y =1wouldimplythatthenumberofobjectsinthesamplewas1.
7
(cid:13)c2010CarlJamesSchwarz
CHAPTER 1. MAXIMUM LIKELIHOOD ESTIMATION & AIC AND
MODEL SELECTION - A PRIMER
values of Y. The likelihood function is a function of µ given the data and does
NOTsumto1overthepossiblevaluesofµ. [Notethatwhilethepossiblevalues
of Y were 0, 1, 2, 3, ..., the possible values are µ is the non-negative real line,
i.e. 0≤µ.]
We will assume that each of three samples were selected independently of
each other. [If this were not true, then the likelihood function below will be
more complex.] It seems reasonable that the density is homogeneous over the
entireregion(whichagaincanberelaxedifneeded). LetY representthecount
i
in the ith sample with i = 1,2,3. Because of the independence assumption
and the assumption of a common parameter to all samples, the joint likelihood
function is the product of the individual likelihood functions:
e−µµy1 e−µµy2 e−µµy3
L(µ|Y =y ,Y =y ,Y =y )= × ×
1 1 2 2 3 3 y ! y ! y !
1 2 3
Thiscouldbewrittenmorecompactlyusingproductnotationbutisnotimpor-
tant for what follows.
So if our sample values were 4, 10 and 15, the likelihood function is:
e−µµ4 e−µµ10 e−µµ15
L(µ|Y =4,Y =10,Y =15)= × ×
1 2 3 4! 10! 15!
andis functiononly ofµ, i.e.the onlyunknown quantityis theparameter tobe
estimated.
1.1.3 The MLE
Fisher argued that a sensible estimate for µ would be the value that MAXI-
MIZEStheLIKELIHOOD(thatiswhythisprocedureiscalledMaximumLike-
lihood). This intuitively says that the best guess for the unknown parameter
maximizes the joint probability of the data and is the most consistent with the
data.
The MLE could be found using graphical methods, i.e. plot L as a function
of µ and find the value of µ that maximizes the curve as shown below:
8
(cid:13)c2010CarlJamesSchwarz
CHAPTER 1. MAXIMUM LIKELIHOOD ESTIMATION & AIC AND
MODEL SELECTION - A PRIMER
However, simple calculus gives us a way to do this without resorting to the
looking a graphs.
Recall from calculus that the maximum of a function often occurs when
then derivative (the slope of the line is zero). As the function increases to
the maximum the derivate is positive; as the function decreases away from the
maximum, the slope is negative. It switches from a positive slope to a negative
slope at the maximum.
It turns out that (for mathematical reasons), it is more convenient to look
atthelog(likelihood)wherelog()isthenaturallogarithmtothebasee. Inour
9
(cid:13)c2010CarlJamesSchwarz