Table Of ContentRajan Srinivasan
Importance Sampling
Springer-Verlag Berlin Heidelberg GmbH
q,
. ·
E UO NLINE LIBRARY
ngmeermg
http://www.springer.delengine/
Rajan Srinivasan
Importance
Sampling
Appl ications
in Communications
and Detection
With 114 Figures
, Springer
Dr. Rajan Srinivasan
University of Twente
Room EL/TN 9160
P0217
7500 AE Enschede
Netherlands
e-mail: [email protected]
Library of Congress Cataloging-in-Publication Data
Srinivasan, Rajan:
Importance Sampling: Applications in Communications and Detection / Rajan Srinivasan.
(Engineering online library)
ISBN 978-3-642-07781-4 ISBN 978-3-662-05052-1 (eBook)
DOI 10.1007/978-3-662-05052-1
This work is subject to copyright. AII rights are reserved, whether the whole or part of the material is
concerned, specificalIy the rights of translation, reprinting, reuse of illustrations, recitation,
broadcasting, reproduction on microfilm or in other ways, and storage in data banks. Duplication of
this publication or parts thereof is permitted only under the provisions of the German Copyright Law
of September 9, 1965, in its current version, and permission for use must always be obtained from
Springer-Verlag Berlin Heide1berg GmbH.
Violations are liable for prosecution act under German Copyright Law.
http://www.springer.de
© Springer-Verlag Berlin Heidelberg New York 2002
Originally published by Springer-Verlag Berlin Heidelberg N ew York in 2002
The use of general descriptive names, registered names, traaemarKS, elc. m 11llS pUDllcaIion does not
imply, even in the absence of a specific statement, that such names are exempt from the relevant
protective laws and regulations and therefore free for general use.
Typesetting: Camera ready by author
Cover-Design: de'blik, Berlin
Printed on acid free paper SPIN: 10874744 62/3020/kk -5432 1 O
In memory of my father
Mysore Srinivasan
(1916-1989)
Preface
This research monograph deals with fast stochastic simulation based on im
portance sampling (IS) principles and some of its applications. It is in large
part devoted to an adaptive form of IS that has proved to be effective in appli
cations that involve the estimation of probabilities of rare events. Rare events
are often encountered in scientific and engineering processes. Their charac
terization is especially important as their occurrence can have catastrophic
consequences of varying proportions. Examples range from fracture due to
material fatigue in engineering structures to exceedance of dangerous levels
during river water floods to false target declarations in radar systems.
Fast simulation using IS is essentially a forced Monte Carlo procedure
designed to hasten the occurrence of rare events. Development of this simu
lation method of analysis of scientific phenomena is usually attributed to the
mathematician von Neumann, and others. Since its inception, MC simula
tion has found a wide range of employment, from statistical thermodynamics
in disordered systems to the analysis and design of engineering structures
characterized by high complexity. Indeed, whenever an engineering problem
is analytically intractable (which is often the case) and a solution by nu
merical techniques prohibitively expensive computationally, a last resort to
determine the input-output characteristics of, or states within, a system is to
carry out a simulation. Simulation is concerned with replicating or mimicking
a system and its operation by mechanizing the exact mathematical equations
that describe it and all its inputs using a computer. The reliability of a simu
lation is governed primarily by the authenticity of the analytical model, that
is, by how closely the mathematical descriptions used fit the actual physical
system and its environs. The accuracy is determined by the precision of the
computations.
In several applications, systems are driven or perturbed by stochastic
inputs that may arise from natural sources or are derived from the outputs
of other systems. It is often of interest to determine the average behaviour of
a system in terms of its response. The MC method then uses a (discretized)
model of these stochastic processes to generate random numbers, and runs
them through the simulated system to give rise to responses of interest. If
this is done a sufficiently large number of times, the law of large numbers
guarantees that the averaged results will approach the mean or expected
VIII Preface
behaviour of the system. Hence, analysis by simulation can playa very useful
role in the design process of complex systems. The Me method however is
not limited to studying systems with stochastic inputs. An early and classical
use has been in the evaluation of integrals of functions over complicated
multidimensional regions. Random points are generated over a simpler or
more convenient region which contains the desired region of integration. The
points which fall in the latter region are used to evaluate the integrands and
the results are weighted and summed to provide an estimate of the integral.
There are important application areas wherein system performance is
closely linked with the occurrence of certain rare phenomena or events. In
digital communications, for example, bit error probabilities over satellite links
using error correction coding can be required to be as low as 10-10. In packet
switching over telecommunication networks, an important parameter is the
probability of packet loss at a switch. These probabilities are required to be
of the order of 10-9• False alarm probabilities of radar and sonar receivers are
usually constrained to not exceed values close to 10-6• Development of these
sophisticated systems is primarily due to complex signal processing opera
tions that underlie them. In such situations, analysis by mathematical or nu
merical techniques becomes very difficult owing to memories, nonlinearities,
and couplings present in the system, and high dimensionality of the processes
involved. Conventional Me simulation also becomes ineffective because of ex
cessively long run times required to generate rare events in sufficiently large
numbers for obtaining statistically significant results.
It is in situations such as those described above that IS has a powerful
role to play. It was first researched by physicists and statisticians. Its use
subsequently spread to the area of reliability in the domains of civil and
mechanical engineering. In more recent times, IS has found several applica
tions in queuing theory, performance estimation of highly reliable computing
systems, and digital communications. Since the mid 1990's, it has made ap
preciable inroads into the analysis and design of detection algorithms that
have applications in radar (and sonar) systems. Research in IS methods and
new applications still goes on, especially as engineering systems become more
complex and increasingly reliable.
In simulations based on IS, probability distributions of underlying processes
that give rise to rare events in a system are changed or biased, causing these
events to occur more frequently. This renders them quickly countable. Each
event is then weighted appropriately and summed, to provide unbiased esti
mates of the rare event probabilities. It turns out that if the biasing distri
bution is carefully chosen, the resulting estimate has markedly lower (error)
variance than the conventional Me estimate. Apart from use of IS in specific
applications, an important aspect of its research has therefore been concerned
with the search for good biasing distributions. Several theoretical results on
the latter subject are available in the literature, especially those concerning
use of tilted (or twisted) densities. These densities have been known for a long
Preface IX
time and have played a major role in development of the theory of large devi
ations. In fast simulation they have been shown to have attractive optimality
properties in asymptotic situations, when dealing with large numbers of ran
dom variables. The approach taken in this monograph is somewhat different
insofar as applications are concerned. Choice of good biasing distributions in
a specific situation is largely left to the ingenuity of the analyst. This need
cause no alarm to an intending user of IS. In many applications, choice of
a family of (parameterized) biasing distributions can usually be made with
a little thought. Once this is done, the rest of the procedure is concerned
with determining parameters of the family that provide estimates that have
low variances. It is in fact the most direct approach to obtaining accurate
estimators based on IS, whenever these can be mechanized without too much
difficulty. The chief aim of this monograph, therefore, is to introduce inter
ested researchers, analysts, designers, and advanced students to the elements
of fast simulation based on adaptive IS methods, with several expository nu
merical examples of elementary and applied nature being provided herein
that hopefully render the techniques readily usable.
The concept of IS is introduced and described in Chapter 1 with emphasis
on estimation of rare event probabilities. Different biasing methods are de
scribed in Chapter 2 and evaluated in terms of the variances of the estimates
that they provide. The concept of adaptive biasing is introduced in this chap
ter and optimization algorithms are developed. The IS simulation problem is
posed as one of variance minimization using stochastic algorithms. The third
chapter is devoted to sums of random variables. Tail probability estimation
is discussed and a method based on conditional probability is developed. A
simulation methodology for determining a number that is exceeded by sums
of random variables with a given small probability is formulated. It is referred
to as the inverse IS problem and forms the basis for parameter optimization
in systems to achieve specified performance levels. This has several practical
applications, as demonstrated by numerical examples. In this same chapter,
a new approximation for tail probability is derived. The derivation of the
Srinivasan density, an approximation for densities of sums of independent
and identically distributed random variables, is given. Several simulation ex
amples are given in these chapters to illustrate the nature of results that
can be expected from well designed IS algorithms. The next chapter, Chap
ter 4, is a short one containing derivations of approximations for detection
and false alarm probabilities of likelihood ratio tests. They complement some
well known classical results in this topic.
The remaining chapters, 5 to 8, are on applications of the IS techniques
discussed previously. Chapter 5 presents an effective solution for applying IS
to constant false alarm rate detection algorithms that are used in radar and
sonar receivers. It is shown how adaptive techniques can be applied to their
analysis and design. Several detection situations are described and numerical
results provided. Results are provided in Chapter 6 on ensemble detection,
x Preface
a technique that combines the outputs of several processors to achieve ro
bustness properties. In Chapter 7 is described blind simulation, a procedure
for handling situations in which the statistical distributions of underlying
processes may be unknown or partially known. It is applied to a detection
algorithm to demonstrate its capabilities. The second area of application
studied in this monograph is in Chapter 8. It deals with performance evalu
ation of digital communication systems that cannot be handled analytically
or even using standard numerical techniques. Parameter optimization is also
addressed. Several examples are given that serve to illustrate how adaptive
IS can be used for such systems. Some of the application examples in these
last chapters are treated briefly in regard to their setting and mathemat
ical background. In particular, the topic of space-time adaptive processing
(STAP) detection algorithms in Chapter 5 mainly consists of indications of
how IS could be used in their simulation. This was necessary since the ma
terial is still nascent and can be the subject of further research.
Much of the material in this book consists of my research results developed
since 1996, when I first became interested in the subject of fast simulation.
Being strictly in the nature of a monograph, I have not dealt with several
topics and results of IS that are important in their own right. From those sci
entists and authors whose works have not been included or mentioned, I beg
indulgence. Undeniably, several of the results reported here were obtained in
discussions and collaboration with students and colleagues. They have been
given adequate due by way of referenced published literature. Nevertheless,
it is my pleasure to recall and recognize here the encouragement, sugges
tions, and help that I received from friends and colleagues while penning this
work. In particular I wish to thank Mansij Chaudhuri, S Ravi Babu, Edison
Thomas, and A. Vengadarajan, who were my colleagues in India, and David
Remondo Bueno, Wim van Etten, and Hans Roelofs, from Europe.
January 2002 Rajan Srinivasan
University of Twente
The Netherlands
Table of Contents
Preface . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. VII
1. Elements of Importance Sampling ........................ 1
1.1 Rare events and simulation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1
1.2 Fast simulation. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2
1.2.1 Random functions. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4
1.3 Optimal biasing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4
Example: An optimal biasing density. . . . . . . . . . . . . . . . 6
1.3.1 Conditions on f* ........................ . . . . . . . . . 7
1.4 The simulation gain .................................... 8
2. Methods of Importance Sampling. . . . . . . . . . . . . . . . . . . . . . . . . 9
2.1 Conventional biasing methods. . . . . . . . . . . . . . . . . . . . . . . . . . .. 10
2.1.1 Scaling.......................................... 10
Example: Scaling the Wei bull density ............... 11
Example: Estimating a probability close to unity ..... 13
Transformations. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. 14
2.1.2 Translation...................................... 16
Example: Optimal translated density. . . . . . . . . . . . . . .. 17
Example: Translating the Gaussian density .......... 18
Example: Translating the Wei bull density. . . . . . . . . . .. 19
An equivalence. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. 20
2.1.3 Exponential twisting ............................ " 21
Example: Twisting the Gaussian density. . . . . . . . . . . .. 24
Example: Twisting the Gamma density ............ " 24
2.2 Adaptive IS - optimized biasing. . . . . . . . . . . . . . . . . . . . . . . . .. 25
2.2.1 Variance estimation and minimization ............. " 25
Adaptive optimization ....................... . . . .. 27
Example: Optimum translation for Rayleigh density. .. 27
2.2.2 Estimation by IS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. 29
Example: Estimating variance using IS . . . . . . . . . . . . .. 33
2.3 Combined scaling and translation. . . . . . . . . . . . . . . . . . . . . . . .. 34
Example: Two-dimensional biasing for a Weibull density 35
2.4 Other biasing methods. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. 36
