Principles of Probability Theory and its Applications

551 Pages·2022·8.923 MB·English

by  Majid Eyvazion, Mohammad Reza Mehroup

The science of probability theory was first noticed in the middle of the 18th century with the study of
games of chance. As time went by, the applications of this science expanded to the point where today, the theory
of probability plays a crucial role in recognizing, modeling, and improving many uncertain real-world
phenomena.
The book in front of you is the result of several years of studying, using, and teaching the concepts of
probability theory and its applications. Despite the many complexities of this science, the main goal and focus
of this book are to present its contents simply and fluently and to help students create a deep understanding of
the subjects of this science. To achieve this goal, each chapter of the book is divided into two main parts. In the
first part, the text of each chapter is explained in a simple way and has examples that increase learning and
understanding of the content. Since the authors believe solving different problems is the best way to get good at
probability theory and fully understand it, the second half of each chapter in the book is filled with many
classified problems. The order of the problems is also designed such that the readers feel themselves progressing
along the learning path step by step.
In the first chapter of this book, the main ideas of combinatorial analysis, which is the primary way to
figure out probabilities, are explained.
In Chapter 2, the definition of probability, the principles of probability theory, and the main methods of
probability calculation are stated, and in Chapter 3, conditional probability and its applications are presented.
In Chapter 4, random variables are addressed, and in Chapter 5, the expected value of random variables
and some of their properties are mentioned. Also, in Chapters 6 and 7, some widely used discrete and continuous
random variables are introduced.
In Chapter 8, joint random variables and conditional distributions are discussed, and in Chapters 9 and
10, the distribution and expected value of a function of several random variables are addressed.
Finally, in Chapter 11, expected value of multiple random variables, correlation, and covariance are
explained.
For those who encounter difficulties in solving the problems in this book, another book titled "Solution
Manual for Principles of Probability and its Applications" has been prepared by the authors, in which there are
explanatory answers to all questions and, in many cases, more than one solution to the problems is proposed.
The contents of this book are not without flaws, and we incentivize our dear readers to contact us via
[email protected] if they find any weak points or faults in the book or have any comments or
suggestions.
Ultimately, the authors hope that this collection can be effective in getting to know and learning the
science of probability better and be a step, however small, in developing this science.
CHAPTER 1: COMBINATORIAL ANALYSIS ............................................................................................................................................... 1
1.1. Introduction ....................................................................................................................................................................................................... 1
1.2. The Basic Principle of Counting ................................................................................................................................................................... 1
1.3. Permutations .................................................................................................................................................................................................... 5
1.3.1 Permutation of “n” Distinct elements .......................................................................................................................................... 5
1.3.2 Permutation of “r” Distinct elements from “n” Distinct elements ........................................................................................ 7
1.3.3 Permutation of “n” Elements, some of which have the Same value ...................................................................................... 8
1.3.4 Permutation of “n” Distinct elements at a Round table ......................................................................................................... 10
1.4. Combinations ................................................................................................................................................................................................. 12
1.5. Significant Identities of the Combinatorial Topic ................................................................................................................................ 16
1.6. The Ball and Urn (cell) Model ..................................................................................................................................................................... 23
1.7. Chapter Problems .......................................................................................................................................................................................... 48
CHAPTER 2: AXIOMS OF PROBABILITY ................................................................................................................................................ 58
2.1. Introduction .................................................................................................................................................................................................... 58
2.2. Random Trial, Sample Space, and Event ................................................................................................................................................ 58
2.3. An Introduction to the Algebra of Sets ..................................................................................................................................................... 59
2.4. Definition of Probability .............................................................................................................................................................................. 64
2.5. Some Probabilistic Propositions Resulting from Principles of the Probability Theory ............................................................... 79
2.6. Chapter Problems ..........................................................................................................................................................................................96
CHAPTER 3: CONDITIONAL PROBABILITY AND INDEPENDENCE ..................................................................................................... 106
3.1. Introduction .................................................................................................................................................................................................. 106
3.2. Conditional Probability Concept ............................................................................................................................................................. 106
3.3. The Law of Multiplication in Probability ............................................................................................................................................... 113
3.4. Independence of Events .............................................................................................................................................................................. 115
3.5. The Law of Total Probability .................................................................................................................................................................... 125
3.6. Bayes' Law ..................................................................................................................................................................................................... 133
3.7. The Law of Total Probability in Reduced Space .................................................................................................................................. 138
3.8. Chapter Problems ........................................................................................................................................................................................ 143
CHAPTER 4: RANDOM VARIABLES ...................................................................................................................................................... 162
4.1. Introduction .................................................................................................................................................................................................. 162
4.2. Types of Random Variables ....................................................................................................................................................................... 166
4.3. Discrete Random Variables ....................................................................................................................................................................... 167
4.4. Continuous Random Variables ................................................................................................................................................................ 169
4.5. Mixed Random Variables ........................................................................................................................................................................... 174
4.6. Cumulative Distribution Function ......................................................................................................................................................... 176
4.7. Some Important Values of Random Variables ...................................................................................................................................... 181
4.8. The Distribution of a Function of a Random Variable ....................................................................................................................... 184
4.9. Conditioning on Continuous Space ........................................................................................................................................................ 188
4.10. Chapter Problems ......................................................................................................................................................................................... 191
CHAPTER 5: EXPECTED VALUE .......................................................................................................................................................... 206
5.1. Introduction ................................................................................................................................................................................................. 206
5.2. The Expected Value of Discrete, Continuous, and Mixed Random Variables ............................................................................. 209
I
5.3. Some notes about the Expected Value of a Random Variable .......................................................................................................... 214
5.4. Expected Value of a Function of a Random Variable .........................................................................................................................223
5.5. Central Tendency Measures of a Random Variable ........................................................................................................................... 227
5.6. Dispersion Measures of a Random Variable........................................................................................................................................ 228
5.6.1. Variance ......................................................................................................................................................................................... 228
5.6.2. Standard Deviation ..................................................................................................................................................................... 230
5.6.3. Expected Distance from the Mean ............................................................................................................................................ 231
5.7. Other Measures of a Random Variable .................................................................................................................................................. 231
5.8. Approximate Expected Value of The Function of a Random Variable ...........................................................................................232
5.9. Moment Generating Function ................................................................................................................................................................ 233
5.10. Factorial Moment Generating Function .............................................................................................................................................. 238
5.11. Distribution and Expected Value Of 𝑿|𝒂≤𝑿≤𝒃 .............................................................................................................................. 242
5.12. Markov's and Chebyshev's Inequalities .................................................................................................................................................. 244
5.12.1. Markov's Inequality ...................................................................................................................................................................... 244
5.12.2. Chebyshev's Inequality ................................................................................................................................................................ 246
5.12.3. One-Sided Chebyshev's Inequality ........................................................................................................................................... 248
5.13. Chapter Problems ........................................................................................................................................................................................ 251
CHAPTER 6: SPECIAL DISCRETE RANDOM VARIABLES .................................................................................................................... 264
6.1. Introduction ................................................................................................................................................................................................. 264
6.2. The Bernoulli Random Variable.............................................................................................................................................................. 264
6.3. The Binomial Random Variable .............................................................................................................................................................. 269
6.3.1. Properties of The Binomial Random Variable........................................................................................................................ 272
6.4. The Geometric Random Variable ............................................................................................................................................................ 275
6.4.1. The Mean And Variance of The Geometric Distribution .................................................................................................... 277
6.4.2. The Memoryless Property of The Geometric Distribution .................................................................................................. 279
6.4.3. The Geometric Random Variable of Failure Type ................................................................................................................ 282
6.5. The Negative Binomial Random Variable (Pascal) ............................................................................................................................ 283
6.5.1. The Properties of Negative Binomial Random Variable ..................................................................................................... 286
6.5.2. The Negative Binomial Random Variable of Failure Type................................................................................................. 289
6.5.3. The Problem of Points ................................................................................................................................................................. 290
6.6. The Poisson Random Variable .................................................................................................................................................................. 291
6.6.1. The Properties of The Poisson Random Variable ................................................................................................................. 295
6.6.2. The Poisson Process ...................................................................................................................................................................... 297
6.6.3. Approximating The Probability Function of The Number Of Successes in n Dependent Trials .............................. 299
6.7. The Hypergeometric Random Variable................................................................................................................................................. 304
6.7.1. The Properties of Hypergeometric Random Variable ......................................................................................................... 305
6.8. The Discrete Uniform Random Variable .............................................................................................................................................. 309
6.9. Chapter Problems ......................................................................................................................................................................................... 311
CHAPTER 7: SPECIAL CONTINUOUS RANDOM VARIABLES .............................................................................................................. 333
7.1. Introduction ................................................................................................................................................................................................. 333
7.2. Continuous Uniform Random variable ................................................................................................................................................ 333
7.2.1. Some Properties of a Continuous Uniform Random Variable .......................................................................................... 338
7.3. Normal Random Variable ......................................................................................................................................................................... 340
7.3.1. Some Properties of a Normal Random Variable .................................................................................................................... 341
7.3.2. The Normal Distribution Approximation to the Binomial Distribution .........................................................................352
7.4. The Exponential Random Variable .........................................................................................................................................................355
7.4.1. Some Properties of the Exponential Distribution ................................................................................................................ 359
7.4.2. The Two-Parameter Exponential Distribution ..................................................................................................................... 361
7.5. The Gamma Random Variable ................................................................................................................................................................ 362
7.5.1. The Three-Parameter Gamma Random Variable ............................................................................................................................... 367
7.6. Other Continuous Distributions............................................................................................................................................................. 368
7.6.1. The Beta Random Variable ........................................................................................................................................................ 368
7.6.2. The Weibull Random Variable ...................................................................................................................................................370
II
7.6.3. The Cauchy Distribution ..............................................................................................................................................................371
7.6.4. The Pareto Random Variable ......................................................................................................................................................371
7.7. The Failure Rate Function ......................................................................................................................................................................... 372
7.8. Chapter Problems ........................................................................................................................................................................................ 376
CHAPTER 8: JOINT RANDOM VARIABLES AND CONDITIONAL DISTRIBUTION ............................................................................... 394
8.1. Introduction ................................................................................................................................................................................................. 394
8.2. Joint Random Variables ............................................................................................................................................................................ 394
8.2.1. Jointly Discrete Random Variables .......................................................................................................................................... 394
8.2.2. Jointly Continuous Random Variables .................................................................................................................................... 399
8.3. Some well-known Joint Distributions ................................................................................................................................................... 405
8.3.1. The Multinomial Distribution ................................................................................................................................................... 405
8.3.2. The Multivariate Hypergeometric Distribution ................................................................................................................... 406
8.3.3. The Bivariate Uniform Random Variable ............................................................................................................................... 407
8.3.4. The Bivariate Normal Distribution .......................................................................................................................................... 410
8.4. The Independence of Random Variables................................................................................................................................................ 410
8.5. Conditional Distributions ......................................................................................................................................................................... 418
8.5.1. Discrete Case .................................................................................................................................................................................. 418
8.5.2. Continuous Case ............................................................................................................................................................................ 421
8.6. Chapter Problems ........................................................................................................................................................................................ 427
CHAPTER 9: DISTRIBUTION OF A FUNCTION OF MULTIPLE RANDOM VARIABLES ........................................................................... 442
9.1. Introduction .................................................................................................................................................................................................. 442
9.2. Distribution of a Function of Multiple Random Variables................................................................................................................ 442
9.2.1. Discrete Case .................................................................................................................................................................................. 442
9.2.2. Continuous Case ............................................................................................................................................................................ 444
9.3. The Sum of Independent Random Variables ....................................................................................................................................... 448
9.4. The Central Limit Theorem ...................................................................................................................................................................... 457
9.5. Order Statistics ............................................................................................................................................................................................ 461
9.6. Chapter Problems ......................................................................................................................................................................................... 471
CHAPTER 10: EXPECTED VALUE OF MULTIPLE RANDOM VARIABLES, COVARIANCE, AND CORRELATION ................................. 482
10.1. Introduction ................................................................................................................................................................................................. 482
10.2. The Expected Value of a Function of Multiple Random Variables ................................................................................................. 482
10.3. The Expected Value of The Sum of Random Variables ...................................................................................................................... 487
10.4. Covariance Between Two Random Variables ..................................................................................................................................... 493
10.5. Correlation Coefficient Between Random Variables ......................................................................................................................... 506
10.6. Chapter Problems ........................................................................................................................................................................................ 510
CHAPTER 11: DETERMINING THE PROBABILITY FUNCTION AND EXPECTED VALUE BY CONDITIONING ..................................... 520
11.1. Introduction ................................................................................................................................................................................................. 520
11.2. Determining The Probability Function by Conditioning .................................................................................................................. 521
11.3. Determining The Expected Value by Conditioning ............................................................................................................................523
11.4. The Expected Value and Variance of The Sum of A Random Number of Random Variables ................................................. 529
11.5. Chapter Problems ....................................................................................................................................................................................... 533
REFERENCES ................................................................................................................................................................................................................. 541

III

In this chapter, we will introduce some methods to count the number of elements
in a discrete and finite set. It will be observed in subsequent chapters that using
enumeration theory or combinatorial analysis is one of the primary and fundamental
methods in probability computation. However, the reader should note that there are
not fixed and specific methods for determining the number of states of a set. The
purpose of this chapter is to learn the primary and fundamental principles of
enumeration and to achieve the ability to generalize them for problems that have not
been observed before.

A ll methods of counting rely on the Basic Principle of Counting or the
Principle of Multiplication, which is expressed as follows:

Suppose that two trials are to be done. If the first trial can obtain one
out of the 𝒏 possible results and each of those results correspond with
the 𝒎 possible results of the second trial, then altogether there are
𝒏×𝒎 possible results for performing the two trials.

1 | P ag e