Table Of ContentOther Titles of Interest
AKHIEZER and PELETMINSKII
Methods of Statistical Physics
BOWLER
Lectures on Statistical Mechanics
Journals
Automatica
Computers and Mathematics with Applications
Journal of Applied Mathematics and Mechanics
Reports on Mathematical Physics
USSR Computational Mathematics and Mathematical Physics
Probability Theory
and Mathematical Statistics
for Engineers
by
V. S. PUGACHEV
Institute of Control Sciences
Academy of Sciences of the USSR, Moscow, USSR
Translated by
I. V. SINITSYNA, Moscow, USSR
Translation Editor
P. EYKHOFF
Eindhoven University of Technology,
The Netherlands
PERGAMON PRESS
OXFORD · NEW YORK · TORONTO · SYDNEY · PARIS · FRANKFURT
U.K. Pergamon Press Ltd., Headington Hill Hall,
Oxford OX3 OBW, England
U.S.A. Pergamon Press Inc., Maxwell House, Fairview Park,
Elmsford, New York 10523, U.S.A.
CANADA Pergamon Press Canada Ltd., Suite 104,
150 Consumers Rd., Willowdale, Ontario M2J 1P9, Canada
AUSTRALIA Pergamon Press (Aust.) Pty. Ltd., P.O. Box 544,
Potts Point, N.S.W. 2011, Australia
FRANCE Pergamon Press SARL, 24 rue des Ecoles,
75240 Paris, Cedex 05, France
FEDERAL REPUBLIC Pergamon Press GmbH, Hammerweg 6,
OF GERMANY D - 6242 Kronberg-Taunus, Federal Republic of Germany
Copyright © 1984 Pergamon Press Ltd.
All Rights Reserved. No part of this publication may be
reproduced, stored in a retrieval system or transmitted in
any form or by any means: electronic, electrostatic,
magnetic tape, mechanical, photocopying, recording or
otherwise, without permission in writing from the
publishers.
First edition 1984
Library of Congress Cataloging in Publication Data
Pugachev, V. S. (Vladimir Semenovich)
Probability theory and mathematical statistics for engineers.
Translation of: Teoriia veroiatnostèt i matematicheskaia
statistika.
Includes bibliographical references.
1. Probabilities. 2. Mathematical statistics.
I. Title.
QA273.P8313 1984 519.1 82-13189
British Library Cataloguing in Publication Data
Pugachev, V. S.
Probability theory and mathematical statistics for engineers.
1. Probabilities 2. Mathematical statistics
I. Title
519.2 QA273
ISBN 0-08-029148-1
Printed in Hungary by Franklin Printing House
PREFACE
THE original Russian book is based on the lecture courses delivered by the
author to students of the Moscow Aviation Institute (Applied Mathematics
Faculty) during many years.
The book is designed for students and postgraduates of applied mathe
matics faculties of universities and other institutes of higher technical edu
cation. It may be useful, however, also for engineers and other specialists
who have to use statistical methods in applied research and for mathe
maticians who deal with probability theory and mathematical statistics.
The book is intended first of all for specialists of applied mathematics.
This fact determined the structure and the character of this book. A suffi
ciently rigorous exposition is given of the basic concepts of probability
theory and mathematical statistics for finite-dimensional random variables,
without using measure theory and functional analysis.
The construction of probability theory is based on A. N. Kolmogorov's
axioms. But the axioms are introduced only after studying properties of the
frequencies of events and the approach to probability as an abstract notion
which reflects an experimentally observable regularity in the behaviour of
frequencies of events, i.e. their stability. As the result of such an approach
the axioms of probability theory are introduced as a natural extension of
properties of the frequencies of events to probabilities.
Almost everywhere throughout the book, especially in studying mathe
matical statistics, vector random variables are considered without prelim
inary studying scalar ones. This intensifies the applied trend of the book
because in the majority of practical problems we deal with multi-dimensional
random vectors (finite sets of scalar random variables).
In order to employ the presented methods to direct practical problems
using computers, references are given throughout the book to standard
programs given in IBM Programmer's Manual cited in sequel as IBM PM.
Besides the foundations of probability theory an exposition is given of
all the basic parts of mathematical statistics for finite-dimensional random
variables.
Apart from routine problems of point and interval estimation and general
theory of estimates, the book contains also the stochastic approximation
method, multi-dimensional regression analysis, analysis of variance, factor
analysis, the theory of estimation of unknown parameters in stochastic
difference equations, the elements of recognition theory and testing hypothe-
vi Preface
ses, elements of statistical decision theory, the principles of statistical simula
tion (Monte Carlo) method.
While translating the book into English many improvements were made
in the exposition of the theory and numerous misprints were corrected.
In particular, new sections were added to Chapters 2, 3, 4, 5 devoted to the
fundamental notions of the entropy of random variables and the information
contained in them (Sections 2.5, 3.6.5, 4.5.10, 4.6 and 5.5).
In Chapter 1 the basic properties of frequencies of events are considered
and a frequency approach to the notion of probability is given. The cases
are considered where the probabilities of events may directly be calculated
from the equiprobability of different outcomes of a trial. After that the no
tion of an elementary event is given. The basic axioms of probability theory
are formulated, the notions of probability space, probability distribution,
conditional probability, dependence and independence of events are intro
duced and the basic formulae, directly following from the axioms including
the formulae determining binomial and polynomial distributions, are de
rived. Then the Poisson distribution is derived.
In Chapter 2 random variables and their distributions are considered.
The basic characteristics of the distributions of finite-dimensional random
variables, i.e. a density and a distribution function, are studied. It is shown
that a density as a generalized function, containing a linear combination of
δ-functions, exists for all three types of random variables encountered in
problems of practice, i.e. continuous, discrete and discrete-continuous. An
example is given of a random variable which has not a density of such a
type. The notions of dependence and independence of random variables are
introduced. Finally the notion of entropy is given and the main properties
of entropy are studied.
In Chapter 3 the numerical characteristics of random variables are studied.
First the definition of an expectation is given and basic properties of ex
pectations are studied. Then the definitions of the second-order moments
are given and their properties are studied. After this the moments of
any orders of real random variables are defined. Besides the moments
the notions of a median and quantiles for real scalar random variables are
given. The chapter concludes with the study of the one-dimensional normal
distribution.
Chapter 4 is devoted to the distributions and conditional distributions of
projections of a random vector. The expressions for the density of a projec
tion of a random vector and the conditional density of this projection, given
the value of the projection of the random vector on the complementary
subspace, are derived in terms of the density of the random vector. Some
examples of dependent and independent random variables are given and
the relation between the notions of correlation and dependence is discussed.
Conditional moments are defined. Characteristic functions of random
Preface vu
variables and the multi-dimensional normal distribution are discussed. The
notions of mean conditional entropy and of amount of information about
a random variable contained in another random variable are given.
In Chapter 5 the methods for finding the distributions of functions of
random variables, given the distributions of their arguments, are studied.
Here we consider a general method for determining the distribution functions
of functions of random variables, three methods for determining the densities,
i.e. the method of comparison of probabilities, the method of comparison
of probability elements and the δ-function method, as well as a method for
finding the characteristic functions and the moments method. The proof of
the limit theorem for the sums of independent, identically distributed, random
variables is given. The basic distributions encountered in mathematical
statistics are derived in the numerous examples showing the application of
the general methods outlined. The last section is devoted to studying the
effects of transformations of random variables on the amount of information
contained in them.
In Chapter 6 the statement of the basic problems of mathematical statis
tics, i.e. the problem of estimation of unknown probabilities of events,
distributions of random variables and their parameters is given at first. Then
the basic modes of convergence of sequences of random variables are
considered. The general definitions concerning estimates and confidence
regions are given; also the basic methods for finding confidence regions for
unknown parameters are studied. After this a frequency as the estimate of a
probability and estimates of moments determined by sample means are
studied. The chapter concludes with the studying of the basic methods of
testing hypotheses about distribution parameters.
The general theory of estimates of distribution parameters and basic
methods for finding the estimates, i.e. the maximum likelihood method and
moments method, are outlined in Chapter 7. Recursive estimation of the
root of a regression equation and the extremum point of a regression by
means of stochastic approximation method are studied.
Chapter 8 is devoted to the basic methods for estimation of densities
and distribution functions of random variables and the methods for approxi
mate analytical representation of distributions. The methods for testing
hypotheses about distributions by the criteria of K. Pearson, A. N. Kol-
mogorov and N. V. Smirnov are studied and the estimation of distribution
parameters by means of minimum χ2 method is considered. In the last
section of the chapter a summary of a statistical simulation method is given
as a technique for approximate calculations and a method for scientific
research.
In Chapter 9 statistical regression models are studied. The general method
for determining the mean square regression in a given class of functions, in
particular linear mean square regression, is studied at first. Then the methods
viii Preface
for estimation of linear regressions (regression analysis) and the methods
for testing hypotheses about regressions are given. Finally the bases of vari
ance analysis theory are derivedfromthegeneraltheory of designing linear
regression models.
Statistical models of other types are studied in Chapter 10. At first the
models described by difference equations, in particular autoregression
models, are considered. A method for estimation of sequences of random
variables determined by difference equations and unknown parameters in
difference equations is discussed as well as application of this method to
linear and non-linear autoregression models. Then some methods for de
signing factor models (elements of factor analysis) and recognition models
are studied. The similarity is demonstrated of some recognition problems
and problems of testing hypotheses about distribution parameters. In the
last section a short summary of elements of statistical decision theory (meth
ods for designing the models of decision-making processes) is given.
The Harvard system of references is used in the English translation of the
book. The author does not pretend in any way to provide a complete list
of literature references in the field concerned. In the list only those sources
are given which are cited in the text.
The formulations of all basic results and statements are given in italics.
The beginnings and the ends of the evaluations, proofs and discussions
which lead to certain results are indicated by black triangular indices
► and «.
Only a short summary is given of basic methods of modern mathematical
statistics of finite-dimensional random variables in Chapters 6-10.
For a deeper and more complete study of mathematical statistics one may
be recommended to read the books by H. Cramer (1946), M. G. Kendall
and A. Stuart (1976, 1977, 1979), S. Wilks (1962), C. R. Rao (1973),
T. W. Anderson (1958) and the books on various parts of mathematical
statistics, to which in Chapters 6-10 references are given.
In order to study the mathematical foundations of probability theory we
advise the books by M. Loève (1978), J. Neveu (1965) and P. L. Hennequin
and A. Tortrat (1965).
For information about the notions and theorems from various parts of
mathematics used in the book we advise the book by Korn and Korn (1968).
For recalling linear algebra the reader may use the books by Gantmacher
(1959), Lancaster (1969), Noble and Daniel (1977) and Wilkinson (1965).
For recalling mathematical analysis the book by Burkill and Burkill (1970)
may be used.*
t We recommend Russian readers also the Russian books by Fichtenholz (1964),
Nikol'skij (1977) and Smirnow (1979, vols. 1,2) for recalling mathematical analysis and the
books by Golovina (1974), Marcev (1978) and Smirnow (1979, vol. 3, Pt. 1) for recalling
linear algebra.
Preface ix
Sections 2.5, 3.6.5, 4.5.10, 4.6 and 5.5 devoted to the notions of entropy
and information contained in random variables, Section 8.4 devoted to the
statistical simulation method and Chapters 9 and 10 have been written with
the active assistance of I. N. Sinitsyn who has also helped me to edit the
whole Russian manuscript. Without his help the book probably would not
appear so soon. I consider it my pleasant duty to express my sincere gratitude
to I. N. Sinitsyn for his invaluable assistance.
I express also my gratitude to I. V. Sinitsyna for her excellent translation
of the book into English and for typing and retyping various versions of
Russian and English manuscripts.
I owe also many thanks to Professor P. Eykhoff for his kind collaboration
with me as a co-editor of the English translation of the book resulting in
considerable improvement of the English version of the book.
I wish to acknowledge my gratitude to N. I. Andreev and N. M. Sotsky
for their valuable remarks and discussions which promoted considerable
improvement of the book, to N. S. Belova, A. S. Piunikhin, I. D. Siluyanova
and O. V. Timokhina for their assistance in preparing the Russian manu
script for press, to M. T. Yaroslavtseva for the assistance in preparing for
press the Russian manuscript and the last three chapters of the English
manuscript, to S. Ya. Vilenkin for the consultations on computational
aspects of the methods outlined in the book and for the organization of
computer calculations for a number of examples.
I owe also my gratitude to I. V. Brûza, Eindhoven University of Technol
ogy, who carefully checked the list of references at the end of the book,
corrected it, and converted it into a form suitable for English-speaking
readers.
V. S. PUGACHEV
Moscow■, December, 1980
CHAPTER 1
PROBABILITIES OF EVENTS
1.1. Random phenomena
1.1.1. Examples of random phenomena. In his daily life man meets with
random phenomena at every step. There is no process without them. The
simplest example of random phenomena are the measurement errors. We
know that absolutely accurate measurements do not exist. While measuring
repeatedly the same object, for instance when we weigh it many times with
an analytical balance, we always receive similar but different results. It may
be explained by the fact that the result of every measurement contains a
random error and the results of different measurements contain different
errors. It is impossible to predict what will be the error of a given specific
measurement or to determine it after measuring. If we make an experimental
study of some phenomenon and represent the results by a graph it is seen
that the points, if sufficient in number, never lie on a single curve but are
subject to random scatter. This scatter is explained both by measurement
errors and the action of other random causes.
A second example of random phenomena is missile scatter. The missiles
never get in the same point even when you aim at the same point. One would
think the conditions for all the shots are the same. But the missiles follow
different trajectories and arrive at different points. It is impossible to predict
in what point a given missile arrives. One of the reasons for this is the funda
mental impossibility to measure exactly the parameters of atmospheric
conditions at all the points of the missile trajectory exactly at the time in
stants at which the missile will pass these points. The aerodynamic forces
and their moments acting on the missile depend on these parameters and
cause the uncertainty of the hit point of the missile.
As a third example of random phenomena we can point to the failures
of various technical equipment. In spite of the high quality of modern
engineering there are sometimes failures of some devices. The failure of a
device is a random phenomenon. It is impossible to predict if it fails or not
and, if it fails, to predict the instant of a failure.
The noises in radio-receivers also belong to the random phenomena. The
so-called "ether" is always satiated with various electromagnetic radiations.
2*
2 Probability Theory and Mathematical Statistics for Engineers
The electric discharges in the atmosphere, the movement of atmospheric
electricity, working equipment created by man and so on act as sources of
such radiations. Therefore tuning cannot prevent outside radiations to
make noises in a receiver. The more remote the transmitter is the more noises
distort the received signals. It becomes apparent that at the same time we
hear in the radio-set the received signals and crackles as well. This well-
known phenomenon also represents a random phenomenon because it is
impossible to predict when and what outside electromagnetic radiation will
get into the radio-set. It is fundamentally impossible to avoid the outside
radiations in the radio-sets since they are destined for receiving faint electro
magnetic radiations.
The irregular oscillations (vibrations) of an aircraft flying in a turbulent
atmosphere also represent a random phenomenon. These oscillations of a
plane are due to random gusts of wind in a turbulent atmosphere.
1.1.2. Nature of random phenomena. Like any phenomena random phe
nomena are caused by quite definite reasons. All phenomena of the external
world are interrelated and influence each other (the law of phenomenological
interdependence). Therefore each observable phenomenon is causally related
with innumerable other phenomena and its pattern of development depends
on the multiplicity of the factors. It is therefore impossible to trace all these
innumerable relations and to investigate their actions. When some phenom
enon is studied only a limited number of basic factors affecting the pattern
of the phenomenon can be established and traced. A number of secondary
factors is neglected. This gives an opportunity to study the essence of a
phenomenon deeply and to determine its regularity. At the same time acting
in such a way man impoverishes the phenomenon, makes it schematic. In
other words, an observable phenomenon is substituted by its suitable sim
plified model. In consequence of this any law of science reflects the essence
of an observable phenomenon, but it is always considerably poorer than the
true phenomenon. No law can characterize a phenomenon comprehensively,
in plentitude and variety. The deviations from regularity caused by joint
action of an innumerable variety of neglected factors we call random phe
nomena.
If we make an experimental study of any phenomenon with the purpose
of obtaining its regularities we have to observe it many times under equal
conditions. By equal conditions we mean equal values of all numerical
characteristics of controlled factors. All uncontrolled factors may be differ
ent. Consequently the action of the controlled factors will be practically the
same under different observations of the same phenomenon. This fact
reflects the regularities of a given phenomenon. Random deviations from
the regularities caused by the action of uncontrolled factors are different
