Discrete Stochastic Processes and Optimal Filtering, Second Edition

293 Pages·2009·3.2 MB·English

by  Jean?Claude Bertein, Roger Ceschi(auth.)

Discrete Stochastic Processes and Optimal Filtering
To our families.
We wish to thank Mme Florence François for having typed the manuscript, and
Mr Stephen Hazlewood who assured the translation of the work.
Discrete Stochastic
Processes and
Optimal Filtering

Second Edition

Jean-Claude Bertein
Roger Ceschi
First published 2005 in France by Hermes Science/Lavoisier entitled Processus stochastiques discrets et
filtrages optimaux © Hermes Science/Lavoisier
First edition published 2007 by ISTE Ltd
Second edition published 2010 in Great Britain and the United States by ISTE Ltd and John Wiley &
Sons, Inc.
Apart from any fair dealing for the purposes of research or private study, or criticism or review, as
permitted under the Copyright, Designs and Patents Act 1988, this publication may only be reproduced,
stored or transmitted, in any form or by any means, with the prior permission in writing of the publishers,
or in the case of reprographic reproduction in accordance with the terms and licenses issued by the CLA.
Enquiries concerning reproduction outside these terms should be sent to the publishers at the
undermentioned address:
ISTE Ltd John Wiley & Sons, Inc.
27-37 St George’s Road 111 River Street
London SW19 4EU Hoboken, NJ 07030
UK USA
www.iste.co.uk www.wiley.com
© ISTE Ltd 2007, 2010

The rights of Jean-Claude Bertein and Roger Ceschi to be identified as the authors of this work have been
asserted by them in accordance with the Copyright, Designs and Patents Act 1988.
Library of Congress Cataloging-in-Publication Data

Bertein, Jean-Claude.
Discrete stochastic processes and optimal filtering / Jean-Claude Bertein, Roger Ceschi. -- 2nd ed.
p. cm.
Includes bibliographical references and index.
ISBN 978-1-84821-181-0
1. Signal processing--Mathematics. 2. Digital filters (Mathematics) 3. Stochastic processes. I. Ceschi, Roger.
II. Title.
TK5102.9.B465 2009
621.382'2--dc22
2009038813
British Library Cataloguing-in-Publication Data
A CIP record for this book is available from the British Library
ISBN 978-1-84821-181-0
Printed and bound in Great Britain by CPI Antony Rowe, Chippenham and Eastbourne
Table of Contents
Preface . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ix
Introduction. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xi
Chapter 1. Random Vectors . . . . . . . . . . . . . . . . . . . . . . . . . . 1
1.1. Definitions and general properties. . . . . . . . . . . . . . . . . . . 1
1.2. Spaces L1(dP) and L2(dP). . . . . . . . . . . . . . . . . . . . . . . 20
1.2.1. Definitions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20
1.2.2. Properties. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22
1.3. Mathematical expectation and applications. . . . . . . . . . . . . 23
1.3.1. Definitions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23
1.3.2. Characteristic functions of a random vector . . . . . . . . . . 34
1.4. Second order random variables and vectors. . . . . . . . . . . . . 39
1.5. Linear independence of vectors of L2(dP) . . . . . . . . . . . . . 46
1.6. Conditional expectation (concerning random vectors
with density function) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 51
1.7. Exercises for Chapter 1 . . . . . . . . . . . . . . . . . . . . . . . . . 56
Chapter 2. Gaussian Vectors. . . . . . . . . . . . . . . . . . . . . . . . . . 63
2.1. Some reminders regarding random Gaussian vectors. . . . . . . 63
2.2. Definition and characterization of Gaussian vectors . . . . . . . 66
2.3. Results relative to independence. . . . . . . . . . . . . . . . . . . . 68
2.4. Affine transformation of a Gaussian vector. . . . . . . . . . . . . 72
2.5. The existence of Gaussian vectors. . . . . . . . . . . . . . . . . . . 74
2.6. Exercises for Chapter 2 . . . . . . . . . . . . . . . . . . . . . . . . . 84
vi Discrete Stochastic Processes and Optimal Filtering
Chapter 3. Introduction to Discrete Time Processes . . . . . . . . . . 93
3.1. Definition . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 93
3.2. WSS processes and spectral measure. . . . . . . . . . . . . . . . . 105
3.2.1. Spectral density. . . . . . . . . . . . . . . . . . . . . . . . . . . . 105
3.3. Spectral representation of a WSS process. . . . . . . . . . . . . . 109
3.3.1. Problem. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 109
3.3.2. Results. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 111
3.4. Introduction to digital filtering. . . . . . . . . . . . . . . . . . . . . 114
3.5. Important example: autoregressive process . . . . . . . . . . . . . 127
3.6. Exercises for Chapter 3 . . . . . . . . . . . . . . . . . . . . . . . . . 132
Chapter 4. Estimation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 139
4.1. Position of the problem . . . . . . . . . . . . . . . . . . . . . . . . . 139
4.2. Linear estimation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 142
4.3. Best estimate – conditional expectation . . . . . . . . . . . . . . . 154
4.4. Example: prediction of an autoregressive process AR (1). . . . 162
4.5. Multivariate processes . . . . . . . . . . . . . . . . . . . . . . . . . . 163
4.6. Exercises for Chapter 4 . . . . . . . . . . . . . . . . . . . . . . . . . 172
Chapter 5. The Wiener Filter . . . . . . . . . . . . . . . . . . . . . . . . . 177
5.1. Introduction. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 177
5.1.1. Problem position . . . . . . . . . . . . . . . . . . . . . . . . . . . 178
5.2. Resolution and calculation of the FIR filter. . . . . . . . . . . . . 179
5.3. Evaluation of the least error. . . . . . . . . . . . . . . . . . . . . . . 181
5.4. Resolution and calculation of the IIR filter . . . . . . . . . . . . . 183
5.5. Evaluation of least mean square error . . . . . . . . . . . . . . . . 187
5.6. Exercises for Chapter 5 . . . . . . . . . . . . . . . . . . . . . . . . . 188
Chapter 6. Adaptive Filtering: Algorithm of the Gradient
and the LMS. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 195
6.1. Introduction. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 195
6.2. Position of problem. . . . . . . . . . . . . . . . . . . . . . . . . . . . 198
6.3. Data representation . . . . . . . . . . . . . . . . . . . . . . . . . . . . 200
6.4. Minimization of the cost function. . . . . . . . . . . . . . . . . . . 202
6.4.1. Calculation of the cost function. . . . . . . . . . . . . . . . . . 205
6.5. Gradient algorithm . . . . . . . . . . . . . . . . . . . . . . . . . . . . 209
6.6. Geometric interpretation. . . . . . . . . . . . . . . . . . . . . . . . . 212
6.7. Stability and convergence. . . . . . . . . . . . . . . . . . . . . . . . 216
Table of Contents vii
6.8. Estimation of gradient and LMS algorithm . . . . . . . . . . . . . 221
6.8.1. Convergence of the algorithm of the LMS . . . . . . . . . . . 223
6.9. Example of the application of the LMS algorithm. . . . . . . . . 224
6.10. Exercises for Chapter 6. . . . . . . . . . . . . . . . . . . . . . . . . 233
Chapter 7. The Kalman Filter. . . . . . . . . . . . . . . . . . . . . . . . . 235
7.1. Position of problem. . . . . . . . . . . . . . . . . . . . . . . . . . . . 235
7.2. Approach to estimation. . . . . . . . . . . . . . . . . . . . . . . . . . 239
7.2.1. Scalar case . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 239
7.2.2. Multivariate case . . . . . . . . . . . . . . . . . . . . . . . . . . . 242
7.3. Kalman filtering . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 243
7.3.1. State equation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 243
7.3.2. Observation equation . . . . . . . . . . . . . . . . . . . . . . . . 244
7.3.3. Innovation process . . . . . . . . . . . . . . . . . . . . . . . . . . 246
7.3.4. Covariance matrix of the innovation process . . . . . . . . . 246
7.3.5. Estimation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 248
7.3.6. Riccati’s equation. . . . . . . . . . . . . . . . . . . . . . . . . . . 257
7.3.7. Algorithm and summary . . . . . . . . . . . . . . . . . . . . . . 259
7.4. Exercises for Chapter 7 . . . . . . . . . . . . . . . . . . . . . . . . . 261
7.5. Appendices . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 267
7.6. Examples treated using Matlab software. . . . . . . . . . . . . . . 273
Table of Symbols and Notations. . . . . . . . . . . . . . . . . . . . . . . . 281
Bibliography. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 283
Index. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 285
Preface
Discrete optimal filtering applied to stationary and non-stationary signals
allows us to process, according to chosen criteria, all of the problems that we
might encounter in situations of noisy signal extraction, in the most efficient
manner possible.
It makes up a necessary stage in the most diverse domains: calculation of
the orbits or guidance of aircraft in the aerospace or aeronautic domain,
calculation of filters in the telecommunications or command systems
domain, or even in the processing of seismic signals domain – the list is
endless.
Furthermore, the study and the results obtained from discrete signals lend
themselves easily to a calculator.
In their work, the authors have taken pains to stress the educational
aspects, preferring this to displays of erudition; all of the preliminary
mathematics and probability theories necessary for a sound understanding of
optimal filtering have been treated in the most rigorous fashion. It should not
be necessary to turn to other works to acquire a sound knowledge of the
subjects studied.
Thanks to this work, the reader will not only be able to understand
discrete optimal filtering but will also be able to easily delve deeper into the
different aspects of this wide field of study.
Introduction
The objective of this book is the progressive and rigorous presentation of
the bases of discrete optimal filtering. The optimal character can be
understood in the sense that we always choose that criterion at the minimum
of the norm −L2 of error.
Chapter 1 tackles random vectors, their principal definitions and
properties.
Chapter 2 discusses the subject of Gaussian vectors. Given the practical
importance of this notion, the definitions and results are accompanied by
numerous commentaries and explanatory diagrams.
Chapter 3, “Introduction to Discrete Time Processes”, is by its very
nature more “physics-based” than the preceding chapters and can be
considered as an introduction to numerical filtering. Results that are essential
for what follows will be given in this chapter.
Chapter 4, “Estimation”, brings us the pre-requisites essential for the
construction of optimal filters. The results obtained on projections in Hilbert
spaces make up the cornerstone of future demonstrations.
Chapter 5 discusses the Wiener filter, an electronic device well adapted to
processing second-order stationary signals. Practical calculations of such
filters, as an answer to finite or infinite pulses, will be developed.
Adaptive filtering, which is the subject of Chapter 6, can be considered as
a relatively direct application of the determinist or stochastic gradient
xii Discrete Stochastic Processes and Optimal Filtering
method. At the end of the adaptation or convergence process, we again
encounter the Wiener filter.
This book is completed with a study of Kalman filtering which enables
stationary or non-stationary signal processing; from this point of view we
can say that it generalizes Wiener’s optimal filter.
Each chapter is enhanced by a series of exercises with answers, and
solutions to examples are also supplied using Matlab software, which is well
adapted to signal processing problems.