Table Of ContentPROBABILITY
For the Enthusiastic Beginner
David Morin
Harvard University
'DavidMorin2016d
ISBN-10: 1523318678
ISBN-13: 978-1523318674
PrintedbyCreateSpace
Additionalresourceslocatedat:
www.people.fas.harvard.edu/˜djmorin/book.html
Contents
Preface vii
1 Combinatorics 1
1.1 Factorials . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2
1.2 Permutations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3
1.3 Orderedsets,repetitionsallowed . . . . . . . . . . . . . . . . . . . 7
1.4 Orderedsets,repetitionsnotallowed . . . . . . . . . . . . . . . . . 12
1.5 Unorderedsets,repetitionsnotallowed. . . . . . . . . . . . . . . . 14
1.6 Whatweknowsofar . . . . . . . . . . . . . . . . . . . . . . . . . 20
1.7 Unorderedsets,repetitionsallowed . . . . . . . . . . . . . . . . . . 21
1.8 Binomialcoefficients . . . . . . . . . . . . . . . . . . . . . . . . . 29
1.8.1 CoinsandPascal’striangle . . . . . . . . . . . . . . . . . . 29
1.8.2 (a+b)n andPascal’striangle . . . . . . . . . . . . . . . . 31
1.8.3 PropertiesofPascal’striangle . . . . . . . . . . . . . . . . 33
1.9 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 35
1.10 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 35
1.11 Problems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 36
1.12 Solutions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41
2 Probability 57
2.1 Definitionofprobability . . . . . . . . . . . . . . . . . . . . . . . 57
2.2 Therulesofprobability . . . . . . . . . . . . . . . . . . . . . . . . 59
2.2.1 AND: The“intersection”probability,P(AandB) . . . . . 60
2.2.2 OR: The“union”probability,P(Aor B) . . . . . . . . . . 68
2.2.3 (In)dependenceand(non)exclusiveness . . . . . . . . . . . 71
2.2.4 Conditionalprobability . . . . . . . . . . . . . . . . . . . . 73
2.3 Examples . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 75
2.3.1 Theartof“not” . . . . . . . . . . . . . . . . . . . . . . . . 75
2.3.2 Pickingseats . . . . . . . . . . . . . . . . . . . . . . . . . 76
2.3.3 Socksinadrawer . . . . . . . . . . . . . . . . . . . . . . . 79
2.3.4 Coinsanddice . . . . . . . . . . . . . . . . . . . . . . . . 81
2.3.5 Cards . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 83
2.4 Fourclassicproblems . . . . . . . . . . . . . . . . . . . . . . . . . 85
2.4.1 TheBirthdayProblem . . . . . . . . . . . . . . . . . . . . 85
2.4.2 TheGame-ShowProblem . . . . . . . . . . . . . . . . . . 87
2.4.3 TheProsecutor’sFallacy . . . . . . . . . . . . . . . . . . . 90
2.4.4 TheBoy/GirlProblem . . . . . . . . . . . . . . . . . . . . 93
2.5 Bayes’theorem . . . . . . . . . . . . . . . . . . . . . . . . . . . . 97
2.6 Stirling’sformula . . . . . . . . . . . . . . . . . . . . . . . . . . . 106
2.7 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 108
2.8 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 109
2.9 Problems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 109
2.10 Solutions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 114
3 Expectationvalues 133
3.1 Expectationvalue . . . . . . . . . . . . . . . . . . . . . . . . . . . 133
3.2 Variance . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 141
3.3 Standarddeviation . . . . . . . . . . . . . . . . . . . . . . . . . . 146
3.4 Standarddeviationofthemean . . . . . . . . . . . . . . . . . . . . 150
3.5 Samplevariance . . . . . . . . . . . . . . . . . . . . . . . . . . . . 155
3.6 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 163
3.7 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 164
3.8 Problems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 165
3.9 Solutions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 168
4 Distributions 182
4.1 Discretedistributions . . . . . . . . . . . . . . . . . . . . . . . . . 182
4.2 Continuousdistributions . . . . . . . . . . . . . . . . . . . . . . . 184
4.2.1 Motivation . . . . . . . . . . . . . . . . . . . . . . . . . . 184
4.2.2 Probabilitydensity . . . . . . . . . . . . . . . . . . . . . . 186
4.2.3 Probabilityequalsarea . . . . . . . . . . . . . . . . . . . . 189
4.3 Uniformdistribution . . . . . . . . . . . . . . . . . . . . . . . . . 191
4.4 Bernoullidistribution . . . . . . . . . . . . . . . . . . . . . . . . . 192
4.5 Binomialdistribution . . . . . . . . . . . . . . . . . . . . . . . . . 193
4.6 Exponentialdistribution. . . . . . . . . . . . . . . . . . . . . . . . 196
4.6.1 Discretecase . . . . . . . . . . . . . . . . . . . . . . . . . 196
4.6.2 Rates,expectationvalues,andprobabilities . . . . . . . . . 199
4.6.3 Continuouscase . . . . . . . . . . . . . . . . . . . . . . . 202
4.7 Poissondistribution . . . . . . . . . . . . . . . . . . . . . . . . . . 207
4.7.1 Discretecase . . . . . . . . . . . . . . . . . . . . . . . . . 207
4.7.2 Continuouscase . . . . . . . . . . . . . . . . . . . . . . . 209
4.8 Gaussiandistribution . . . . . . . . . . . . . . . . . . . . . . . . . 215
4.9 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 221
4.10 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 222
4.11 Problems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 222
4.12 Solutions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 227
5 Gaussianapproximations 250
5.1 BinomialandGaussian . . . . . . . . . . . . . . . . . . . . . . . . 250
5.2 Thelawoflargenumbers . . . . . . . . . . . . . . . . . . . . . . . 256
5.3 PoissonandGaussian . . . . . . . . . . . . . . . . . . . . . . . . . 260
5.4 Binomial,Poisson,andGaussian . . . . . . . . . . . . . . . . . . . 263
5.5 Thecentrallimittheorem . . . . . . . . . . . . . . . . . . . . . . . 264
5.6 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 269
5.7 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 270
5.8 Problems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 270
5.9 Solutions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 272
6 Correlationandregression 277
6.1 Theconceptofcorrelation . . . . . . . . . . . . . . . . . . . . . . 277
6.2 Amodelforcorrelation . . . . . . . . . . . . . . . . . . . . . . . . 280
6.3 Thecorrelationcoefficient,r . . . . . . . . . . . . . . . . . . . . . 285
6.4 ImprovingthepredictionforY . . . . . . . . . . . . . . . . . . . . 291
6.5 Calculating ρ(x,y) . . . . . . . . . . . . . . . . . . . . . . . . . . 294
6.6 Thestandard-deviationbox . . . . . . . . . . . . . . . . . . . . . . 297
6.7 Theregressionlines . . . . . . . . . . . . . . . . . . . . . . . . . . 300
6.8 Tworegressionexamples . . . . . . . . . . . . . . . . . . . . . . . 305
6.8.1 Example1: Retakingatest . . . . . . . . . . . . . . . . . . 305
6.8.2 Example2: ComparingIQ’s . . . . . . . . . . . . . . . . . 310
6.9 Least-squaresfitting . . . . . . . . . . . . . . . . . . . . . . . . . . 313
6.10 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 317
6.11 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 318
6.12 Problems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 318
6.13 Solutions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 320
7 Appendices 335
7.1 AppendixA:Subtletiesaboutprobability . . . . . . . . . . . . . . 335
7.2 AppendixB:Euler’snumber,e . . . . . . . . . . . . . . . . . . . . 339
7.2.1 Definitionofe . . . . . . . . . . . . . . . . . . . . . . . . 339
7.2.2 Raisingetoapower . . . . . . . . . . . . . . . . . . . . . 341
7.2.3 Theinfiniteseriesforex . . . . . . . . . . . . . . . . . . . 344
7.2.4 Theslopeofex . . . . . . . . . . . . . . . . . . . . . . . . 346
7.3 AppendixC:Approximationsto(1+a)n . . . . . . . . . . . . . . 346
7.4 AppendixD:Theslopeofex . . . . . . . . . . . . . . . . . . . . . 350
7.4.1 Firstderivation . . . . . . . . . . . . . . . . . . . . . . . . 350
7.4.2 Secondderivation. . . . . . . . . . . . . . . . . . . . . . . 353
7.5 AppendixE:Importantresults . . . . . . . . . . . . . . . . . . . . 356
7.6 AppendixF:Glossaryofnotation. . . . . . . . . . . . . . . . . . . 359
Preface
Thisbookiswrittenforhighschoolandcollegestudentslearningaboutprobability
forthefirsttime.Mostofthebookisverypractical,withalargenumberofconcrete
examples and worked-out problems. However, there are also parts that are a bit
theoretical(atleastforanintroductorybook),withmanymathematicalderivations.
Allinall,ifyouarelookingforabookthatservesasaquickreference,thismaynot
betheoneforyou. Butifyouarelookingforabookthatstartsatthebeginningand
derives everything from scratch in a comprehensive manner, then you’ve come to
therightplace. Inshort,thisbookwillappealtothereaderwhohasahealthylevel
of enthusiasm for understanding how and why the standard results of probability
comeabout.
Probabilityisaveryaccessible(andextremelyfun!)subject,packedwithchal-
lenging problems that don’t require substantial background or serious math. The
examplesinChapter2areatestamenttothis. Ofcourse, thereareplentyofchal-
lenging topics in probability that do require a more formal background and some
heavy-dutymath. ThiswillbecomeevidentinChapters4and5(andthelatterpart
of Chapter 3). However, technically the only math prerequisite for this book is a
comfort with algebra. Calculus isn’t relied on, although there are a few problems
thatdoinvolvecalculus. Thesearemarkedclearly.
All of the problems posed at the ends of the chapters have solutions included.
Thedifficultyisindicatedbystars; mostproblemshavetwostars. Onestarmeans
plugandchug,whilethreestarsmeansomeseriousthinking. Besuretogiveasolid
effortwhensolvingaproblem,anddon’tlookatthesolutiontoosoon. Ifyoucan’t
solveaproblemrightaway,that’sperfectlyfine. Justsetitasideandcomebackto
itlater. It’sbettertosolveaproblemlaterthantoreadthesolutionnow. Ifyoudo
eventuallyneedtolookatasolution,coveritupwithapieceofpaperandreadone
lineatatime,togetahinttogetstarted. Thensetthebookasideandworkthings
outforreal. Thatway, youcanstill(mostly)solveitonyourown. Youwilllearn
a great deal this way. If you instead head right to the solution and read it straight
through,youwilllearnverylittle.
For instructors using this book as the assigned textbook for a course, a set of
homework exercises is posted at www.people.fas.harvard.edu/˜djmorin/book.html.
Asolutionsmanualisavailabletoinstructorsuponrequest.Whensendingarequest,
pleasepointtoasyllabusand/orwebpageforthecourse.
The outline of this book is as follows. Chapter 1 coverscombinatorics, which
is the study of how to count things. Counting is critical in probability, because
probabilitiesoftencomedowntocountingthenumberofwaysthatsomethingcan
happen. InChapter2wediveintoactualprobability. Thischapterincludesalarge
numberofexamples,rangingfromcoinstocardstofourclassicproblemspresented
in Section 2.4. Chapter 3 covers expectation values, including the variance and
standard deviation. A section on the “sample variance” is included; this is rather
mathematicalandcanbeskippedonafirstreading. InChapter4weintroducethe
concept of a continuous distribution and then discuss a number of the more com-
mon probability distributions. In Chapter 5 we see how the binomial and Poisson
distributions reduce to a Gaussian (or normal) distribution in certain limits. We
also discuss the law of large numbers and the central limit theorem. Chapter 6 is
somewhatofastand-alonechapter, coveringcorrelationandregression. Although
thesetopicsareusuallyfoundinbooksonstatistics,itmakessensetoincludethem
here,becausealloftheframeworkhasbeenset. Chapter7containssixappendices.
AppendixCdealswithapproximationsto (1+a)n whicharecriticalinthecalcu-
lationsinChapter5,AppendixElistsallofthemainresultswederiveinthebook,
andAppendixFcontainsaglossaryofnotation;youmaywanttorefertothiswhen
startingeachchapter.
A few informational odds and ends: This book contains many supplementary
remarksthatareseparatedofffromthemaintext;theseendwithashamrock,“♣.”
The letters N, n, and k generally denote integers, while x and t generally denote
continuous quantities. Upper-case letters like X denote a random variable, while
lower-case letters like x denote the value that the random variable takes. We re-
fer to the normal distribution by its other name, the “Gaussian” distribution. The
numericalplotsweregeneratedwithMathematica. Iwillsometimesuse“they”as
a gender-neutral singular pronoun, in protest of the present failing of the English
language. AndIwilloftenusean“’s”toindicatethepluralofone-letteritems(like
6’sondicerolls). Lastly,weofcoursetakethefrequentistapproachtoprobability
inthisintroductorybook.
IwouldparticularlyliketothankCareyWitkovformeticulouslyreadingthrough
theentirebookandofferingmanyvaluablesuggestions.JoeSwingleprovidedmany
helpfulcommentsandsanitychecksthroughoutthewritingprocess. Otherfriends
and colleagues whose input I am grateful for are Jacob Barandes, Sharon Bene-
dict,JoeBlitzstein,BrianHall,TheresaMorinHall,PaulHorowitz,DavePatterson,
AlexiaSchulz,andCorriTaylor.
Despite careful editing, there is essentially zero probability that this book is
errorfree(asyoucanshowinProblem4.16!).Ifanythinglooksamiss,pleasecheck
the webpage www.people.fas.harvard.edu/˜djmorin/book.html for a list of typos,
updates, additional material, etc. And please let me know if you discover some-
thingthatisn’talreadyposted. Suggestionsarealwayswelcome.
DavidMorin
Cambridge,MA
Chapter 1
Combinatorics
TO THE READER: This book is available as both a paperback and an eBook. I
have made a few chapters available on the web, but it is possible (based on past
experience) that a pirated version of the complete book will eventually appear on
file-sharingsites. Intheeventthatyouarereadingsuchaversion,Ihavearequest:
Ifyoudon’tfindthisbookuseful(inwhichcaseyouprobablywouldhavereturned
it, ifyouhadboughtit), orifyoudofinditusefulbutaren’tabletoaffordit, then
no worries; carry on. However, if you do find it useful and are able to afford the
Kindle eBook (priced below $10), then please consider purchasing it (available
onAmazon). Ifyoudon’talreadyhavetheKindlereadingappforyourcomputer,
you can download it free from Amazon. I chose to self-publish this book so that I
could keep the cost low. The resulting eBook price of around $10, which is very
inexpensivefora350-pagemathbook,islessthanamovieandabagofpopcorn,
with the added bonus that the book lasts for more than two hours and has zero
calories(ifusedproperly!).
–DavidMorin
Combinatoricsisthestudyofhowtocountthings.By“things”wemeanthevarious
combinations,permutations(differentorderings),subgroups,andsoon,thatcanbe
formed from a given set of objects/people/etc. For example, how many different
outcomesarepossibleifyouflipacoinfourtimes? Howmanydifferentfull-house
hands are there in poker? How many different committees of three people can be
chosenfromfivepeople? Whatifweadditionallydesignateonepersonasthecom-
mittee’s president? Knowing how to count these types of things is critical for an
understanding of probability, because when calculating the probability of a given
event,weoftenneedtocountthenumberofwaysthattheeventcanhappen.
The outline of this chapter is as follows. In Section 1.1 we introduce the con-
cept of factorials, which are ubiquitous in the study of probability. In Section 1.2
welearnhowtocountthenumberofpossiblepermutations(orderings)ofasetof
objects.Section1.3coversthenumberofpossiblecombinedoutcomesofarepeated
experiment,whereeachrepetitionhasanidenticalsetofpossibleresults. Examples
2 Chapter1. Combinatorics
include rolling dice and flipping coins. In Section 1.4 we learn how to count the
numberofsubgroupsthatcanbeformedfromagivensetofobjects,wheretheor-
der within the subgroup matters. An example is choosing a committee of people
inwhichallofthepositionsaredistinct. Section1.5coverstherelatedquestionof
thenumberofsubgroupsthatcanbeformedfromagivensetofobjects,wherethe
order within the subgroup doesn’t matter. An example is a poker hand; the order
of the cards in the hand is irrelevant. We find that the answer takes the form of a
binomialcoefficient.InSection1.6wesummarizethevariousresultswehavefound
sofar. Wediscoverthatoneresultismissingfromourcountingrepertoire,andwe
remedy this in Section 1.7. In Section 1.8 we look at the binomial coefficients in
moredetail.
Afterlearninginthischapterhowtocountallsortsofthings,we’llseeinChap-
ter 2 how the counting can be used to calculate probabilities. It’s usually a trivial
steptoobtainaprobabilityonceyou’vecountedtherelevantthings,sotheworkwe
doherewillprovewellworthit.
1.1 Factorials
Beforegettingintothediscussionofactualcombinatorics,wefirstneedtolookata
certainquantitythatcomesupagainandagain. Thisquantityiscalledthefactorial.
We’ll see throughout this chapter that when dealing with a situation that involves
an integer N, we often need to consider the product of the first N integers. This
productiscalled“N factorial,”anditisdenotedby“N!”.1 Forthefirstfewintegers,
wehave:
1!=1,
2!=1·2=2,
3!=1·2·3=6,
4!=1·2·3·4=24,
5!=1·2·3·4·5=120,
6!=1·2·3·4·5·6=720. (1.1)
As N increases, N!gets verylargeveryfast. Forexample, 10! = 3,628,800, and
20! ≈ 2.43·1018. In Chapter 2 we will introduce an approximation to N! called
Stirling’sformula.Thisformulamakesitclearwhatwemeanbythestatement,“N!
getsverylargeveryfast.”
Weshouldaddthat0!isdefinedtobe1.Ofcourse,0!doesn’tmakemuchsense,
becausewhenwetalkabouttheproductofthefirst N integers,itisunderstoodthat
we start with 1. Since 0 is below this starting point, it is unclear what 0! actually
means. However, there is no need to try too hard to make sense of it, because as
we’llseebelow,ifwesimplydefine0!tobe1,thenanumberofformulasturnout
tobeverynice.
1Idon’tknowwhysomeonelongagopickedtheexclamationmarkforthisnotation. Butjustre-
memberthatithasnothingtodowiththemorecommongrammaticaluseoftheexclamationmarkfor
emphasis.Sotrynottogettooexcitedwhenyousee“N!”!
1.2. Permutations 3
HavingdefinedN!,wecannowstartcountingthings. Withtheexceptionofthe
resultinSection1.3,allofthemainresultsinthischapterinvolvefactorials.
1.2 Permutations
Apermutationofasetofobjectsisawayoforderingthem.Forexample,ifwehave
threepeople–Alice,Bob,andCarol–thenonepermutationofthemisAlice,Bob,
Carol. AnotherpermutationisCarol,Alice,Bob. AnotherisBob,Alice,Carol. It
turnsoutthattherearesixpermutationsinall,aswewillseebelow.Thegoalofthis
sectionistolearnhowtocountthenumberofpossiblepermutations. We’lldothis
bystartingoffwiththeverysimplecasewherewehaveonlyoneobject. Thenwe’ll
consider two objects, then three, and so on, until we see a pattern. The route we
takeherewillbeacommononethroughoutthisbook:Althoughmanyoftheresults
can be derived in a few lines of reasoning, we’ll take the longer route where we
startwithafewsimpleexamplesandthengeneralizeuntilwearriveatthedesired
results. Concreteexamplesalwaysmakeiteasiertounderstandageneralresult.
Oneobject
Ifwehaveonlyoneobject,thenthereisclearlyonlyonewayto“order”it;thereis
noorderingtobedone. Alistofoneobjectsimplyconsistsofthatoneobject,and
that’sthat. Ifweusethenotationwhere P standsforthenumberofpermutations
N
of N objects,thenwehaveP =1.
1
Twoobjects
With two objects, things aren’t completely trivial like they are in the one-object
case, butthey’restillverysimple. Ifwelabelourtwoobjectsas1and2, thenwe
canorderthemintwoways:
1 2 or 2 1
SowehaveP =2. Atthispoint,youmightbethinkingthatthisresult,alongwith
2
the above P = 1 result, suggests that P = N for any positive integer N. This
1 N
wouldmeanthatthereshouldbethreedifferentwaystoorderthreeobjects. Well,
notsofast...
Threeobjects
Thingsgetmoreinterestingwiththreeobjects. Ifwecallthem1,2,and3,thenwe
canlistoutthepossibleorderings. ThepermutationsareshowninTable6.1.
123 213 312
132 231 321
Table1.1:Permutationsofthreeobjects.
