Comments on Likelihood fits with variable resolution

0.08 MB·English

by  Giovanni Punzi

PHYSTAT2003, SLAC Sep. 8-11, 2003 1
Comments on Likelihood fits with variable resolution
GiovanniPunzi
ScuolaNormaleSuperioreandINFN,56100Pisa,Italy
UnbinnedlikelihoodfitsarefrequentinPhysics,andofteninvolvecomplexfunctionswithseveralcomponents.
Wediscussthepotentialpitfallsofsituationswherethetemplatesusedinthefitarenotfixedbutdependonthe
4 eventobservables,asithappenswhentheresolutionofthemeasurementisevent–dependent, andtheprocedure
0 toavoidthem.
0
2
Whenseveralcategoriesofeventsarepresentinthe lihood function:
n
a samedata sample,anunbinnedMaximumLikelihood
J fit is often used to determine the proportion and the L(f)=Y(fN(xi,0,σ)+(1−f)N(xi,1,σ)) (1)
0 properties of each class of events. This procedure i
1 makesuseof“templates”,representingtheprobability
distribution of the observablesused in the fit for each with respect to the required parameter f (here
]
n class of events. In the simplest cases the templates N(x,µ,σ) indicates the gaussianfunction in the vari-
a are completely determined by the values assigned to able x). This is very simple to perform with the help
a- the parameters of the fit, but frequently a more so- of a numerical maximization program.
t phisticated approach is chosen where templates vary Let’s make a specific numeric example, where f =
a
onaneventbyeventbasis,accordingtotheresolution
d 1/3 and σ = 1 (see illustration in Fig. 1), and the
of the measurement for that particular event. These
. size of the data sample is 150 events. By repeteadly
s variations are due to the dependence of resolution on
c generating MC samples of 150 events each, we obtain
extra variables, that change on an event-by-eventba-
i thedistributionoftheMaximumLikelihoodestimator
s sis . This may happen, for instance, when events are
y of f, which is shown in Fig. 2.
recorded by a detector that has different resolutions
h
p in different regions within its acceptance.
pHxL
[ A common example of this kind of fit in HEP is
0.35
givenbylifetimeand/ormassfits(see[1]forasample
1 0.3
v list of recent experimental papers), where variations 0.25
5 in resolution occur as a consequence of different con- 0.2
4 figuration of each individual decay. The same kind of 0.15
0 issue hovewer is likely to arise in other situations. 0.1
1 Thepurposeofthisshortpaperistopointoutsome 0.05
0 -2 0 2 4 x
potential pitfalls in this kind of fitting procedure. I
4
0 will illustrate the point with reference to a simple toy Figure 1: Probability distribution of x for thetoy
/ problem. problem described in thetext. Contribution of type–A
s
and type–Beventsare also shown.
c
i
s
y 1. A toy problem Its mean is 0.3368± 0.0041 and SD = 0.083, in
h
agreement with expectations of 0.3333 and 0.088 re-
p
Consider an experiment in which two types of spectively (the latter coming from Fisher information
:
v events, A and B, can occur. Let f be the fraction ).
i
X of type–A events, that is, the probability of a generic
eventto be of type A. We wantto extracta measure-
r
a mentoff fromagivensampleofdata. Inordertodo 80
this, we measure the value of an observable x, having
60
the following probability distributions:
40
p(x|A)=N(0,σ)
20
p(x|B)=N(1,σ)
0.2 0.4 0.6 0.8 1
Where σ is a known constant and N(µ,σ) is the nor-
Figure 2: Distribution of ML estimate of thefraction f
mal distribution
of type-Aevents(see text)
This problem is easily solved using an “unbinned
Likelihoodfit”. This consists ofmaximizing the Like-
WELT002
2 PHYSTAT2003, SLAC Sep. 8-11, 2003
2. A toy problem, with variable resolution call it “conditional Likelihood”) rather than the full
distributionp(xi,σi|f). Thedifferencemattersforfit-
Let’s now suppose that the resolution of x is not tingunlessithappensthatthedistributionofσi isthe
constant, but rather depends on the event: we are same for all types of events: p(σi|A) = p(σi|B). In
assuming that each event xi comes together with an that case, p(σi) can be factored out, and the incom-
individual value of σ (let it be σi). This situation plete Likelihoodofeq.(2)differsfromthe true Likeli-
is encountered in many real–life problems, and the hood just by a factor independent of f, that does not
commonapproachfound in the literatureis to simply affect the maximization.
modify the Likelihood function as follows: In the specific MC test reported above, we simu-
lated a resolution 1.5 times worse for events of type
L(f)=YfN(xi,0,σi)+(1−f)N(xi,1,σi)) (2) B than for type A, setting the σi distribution as flat
i between 1 and 2 for A-type events, and flat between
1.5and 3 for type-B events. We intentionally avoided
This looks like a pretty obvious generalization of ex-
sayingthisexplicitlybefore,inordertoputthereader
pression (1). To test it in our toy problem, we mod-
in the typical situation encountered in reality, where
ified our toy MC from previous example, by making
noattentionispayedtothedistributionofthosereso-
σ fluctuate at each eventwithin an arbitrarilychosen
lutionsforthe differentclassesofeventsconsideredin
range(1.0to3.0),andagainmaderepeatedsimulated
the fit. It turns out from our example that this may
experiments of 150 events each, maximizing the Like-
lead to very biased results.
lihoodexpression(2)toestimatef. Theresultofthis
Insummary,expression(2)simplydoesnotworkfor
test is shownin Fig. 3, and rather surprisingly,shows
fitting,andbyalargeamount: itcanbesaidtobelong
a very large bias with respect to the true value of f.
to that particular class of solutions nicely defined in
[2] as ‘SNW solutions’.
60
Conversely, if we use in fitting the correct expres-
50
sion of the Likelihood (eq. 4) we get the result shown
40
in fig. 4, showing a negligible bias. The resolution
30 of the fit is also much better, as the difference in the
20 distributionsoftheσthemselvesgetsexploitedinsep-
10 aratingthetwosamples;thishoweverisaminorpoint
in comparison with the bias issue.
0.2 0.4 0.6 0.8 1
Figure 3: Distribution of ML estimate of the fraction f
200
of type-Aevents, obtained from a ”conditional
Likelihood” 150
100
This may seem really odd, until one realizes that
this new problem is very different from the previous
50
one. Our problem now has actually two observables:
each observation consists of the pair of values (xi,σi) 0.2 0.4 0.6 0.8 1
rather than just xi, and its probability density de-
pends on both. This means that the Likelihood must Figure 4: Distribution of ML estimate of thefraction f
of type-Aevents,using the full Likelihood function
now be writtenbasedonthe probabilitydistributions
of the (xi,σi) pair:
L(f)=Yfp(xi,σi|A)+(1−f)p(xi,σi|B) (3)
i 3. Additional tests
Remembering that p(xi,σi|X) = p(xi|σi,X)p(σi|X) One may wonder at what features of the distribu-
we can write the correct expression of the Likelihood
tions make for a large bias. Table I shows results for
for our problem as:
a few variants of the original problem. Tests include:
L(f)=YfN(xi,0,σi)p(σi|A) • Equal-width ranges of σ.
i
• Disjoint σ ranges.
+(1−f)N(xi,1,σi)p(σi|B) (4)
• Constant, but different σ for A and B.
wherep(σi|X)isthepdfofσi foreventsoftypeX,an
elementthat wasabsentineq.(2); infact, comparing
• Constant, and close σ’s for A and B.
thetwoexpressionsshowsthat(2)isactuallythecon-
ditionalprobabilitydistributionp(xi|σi,f)(onemight • Same-meanσ distributionwithdifferentwidths.
WELT002
PHYSTAT2003, SLAC Sep. 8-11, 2003 3
meaningless number. This is a definite possibility in
Table I Results of MC fittingtests.
a real case, where the nature of type–A events may
Resolutions “conditional” L(2) TrueLikelihood be so different from type–B to produce meaningless
σA σB f˜A σ(f˜A) fˆA σ(fˆA) values for the resolution estimator σi, that was de-
1.0 1.0 0.336±0.003 0.08 signed to work for type–B events – remember that
[1.0,2.0] [1.5,3.0] 0.514±0.007 0.14 0.335±0.002 0.03 the distribution of A is given as fixed. Note that the
[1.0,2.0] [1.5,2.5] 0.474±0.007 0.14 0.335±0.002 0.03 expression used (5) does know that much, and cor-
[1.0,2.0] [2.0,3.0] 0.579±0.008 0.15 0.333±0.000 0.00 rectly disregards the value of σi in the A hypothesis.
1.0 2.0 0.645±0.006 0.12 0.333±0.000 0.00 For events of type B, the variable σi correctly repre-
1.0 1.1 0.374±0.004 0.08 0.333±0.000 0.00 sents the sigma, event by event, of the observable x,
[0.5,3.5] [1.5,2.5] 0.330±0.006 0.12 0.332±0.002 0.03 and the L function correctly accounts for this, too.
1.0 [1.0,2.0] 0.482±0.009 0.09 0.333±0.000 0.00 It may come as a surprise that the result is largely
(σA actually=1.) modifiedL(5) TrueLikelihood biased. The reason for this rather spectacular failure
1.0 [1.0,2.0] 0.374±0.004 0.08 0.333±0.000 0.00 isthatthesecondpieceofL,relatedtoB-typeevents,
[0.5,3.5] [1.0,2.0] 0.414±0.004 0.08 0.332±0.003 0.03 gets confused by the presence of the events of type–A
with meaningless values of sigma: they unavoidably
enter both terms of L during the calculation. The
• Only one type of events has variable sigma. conclusionis: whenever you include σi in your Likeli-
hoodexpression,evenforjustoneclassofevents,you
In almost every tried situation we found expres-
must also account for its distribution, and you must
sion(2)toreturnlargelybiasedresults. Theexception
do so for all event classes.
occurs when the averageσ is the same; the resolution
onf is howevermuchworsethanwith the correctex-
pression. It looks like the most important element is
the difference between the averagevalues of σ for the 4. Conclusions
different samples; the actual variability within each
sample seems less important.
Wheneverthetemplatesusedinamulti-component
Asimplersituationexists,thatisprettycommonin
fit depend on additional observables, one should al-
practice,whereonehasjustonesignalcomponentover
ways use the correct, complete Likelihood expression
a background, and the signal distribution contains a
(4), including the explicit distributions of all observ-
variable sigma, while the background is represented
ables for all classes of events. This is necessary even
just by a fixed template. In this case, expression (2)
if just one of the components is based on a variable
becomes:
σ. The simpler expressions that are commonly used
should be considered unreliable unless one can show
L(f)=YfN(xi,0,1)+(1−f)N(xi,1,σi)) (5)
that the distribution of the variable σ is the same for
i
all components.
This expression of L is of course still incorrect, but A more general consideration suggested by the ex-
it better describes reality at least for one of the two amples discussed above is that one should always be
event categories by incorporating explicitly the infor- waryof“intuitive”modificationsofaLikelihoodfunc-
mationthatithasafixedsigma. Hereavariabletem- tion. Foreverygivenproblemthereisonlyonecorrect
plate appears just in one component, and being this expression for the Likelihood (up to a multiplicative
the simplest configuration with a variable template, constant factor), and it is crucial to verify in every
it is interesting to ask whether it yields a reasonable case that the expression used is the right one, rather
approximation of the correct results. than rely on intuition.
If we apply this new Likelihood expression to the
last tested case, (σA = 1.0 and σB ∈ [1.0,2.0]), we
find that the result is still biased,althoughto a lesser
References
extent (Tab. I). This shows that the distribution
of σ must be kept into account even in the simplest
situation, where it appears in only one component of [1] Recentexamplesofvariable-resolutionfitsinHEP
the fit. can be found in: B. Aubert et al. [Babar Collab.],
Themechanismunderlyingthisproblemiseasierto Phys.Rev.Lett. 91 (2003) 121801. K.Abe et al.
see by looking at a variant of the previous case. Sup- [Belle Collab.], Phys.Rev.Lett. 88 (2002) 171801.
pose that resolutions are the same as above, but for P. Abreu et al. [Delphi Collab.] , Eur.Phys.J.
events of type-A the variable σi is distributed over a C16 (2000)555. ALEPH Collab., Phys.Lett. B492
wide range (0.5-3-5); this is not the actual value of (2000)275-287.M.Paulini,Int.J.Mod.Phys.A14
the resolution for those events, that is still fixed at 1, (1999) 2791-2886.
so for type-A events it represents just an additional [2] J. Heinrich, these proceedings.
WELT002