Table Of ContentLecture Notes
in Control and Information Sciences 612
Editor: M. Thoma
O
O
fD io ~ m 0 I N rD
I%
~ --. nq. ~ io
0 3 IO ~, ~-0 0
Series Advisory Board
A. Bensoussan M.J. • Grimble • P. Kokotovic • H. Kwakernaak
J.L. Massey Y.Z. • Tsypkin
Author
Dr RudolfKulhav~
Institute of Information Theory and Automation
Academy of Sciences of the Czech Republic
P.O. Box 18, 182 08 Prague, Czech Republic
ISBN 3-540-76063-6 Springer-Ver|ag Berlin Heidelberg New York
British Library Cataloguing in Publication Data
Kulhav~, Rudolf
Recursive nonIinear estimation : a geometric approach. -
(Lecture notes in control and information sciences ; 216)
l.Parameter estimation 2.Estimation theory
l.Title
519.5'4
3540760636 ISBN
Library of Congress Cataloging-in-Publication Data
A catalog record for this book is available from the Library of Congress
Apart from any fair dealing for the purposes of research or private study, or criticism or review, as
permitted under the Copyright, Designs and Patents Act 1988, this publication may only be reproduced,
stored or transmitted, in any form or by any means, with the prior permission in writing of the
publishers, or in the case of reprographic reproduction in accordance with the terms of licences issued
by the Copyright Licensing Agency. Enquiries concerning reproduction outside those terms should be
sent to the publishers.
© Springer-Vedag London Limited 1996
Printed in Great Britain
The publisher makes no representation, express or implied, with regard to the accuracy of the
informationc ontained in this book and cannot accept any legal responsibility or liability for any errors
or omissions that may be made.
Typesetting: Camera ready by author
Printed and bound at the AthenPareeusms Ltd, Gateshead
6913830-543210 Printed on acid-free paper
oT Lenka, Lukdg and Matin
Preface
Motivation
The theory of parameter estimation is known to be a well-developed, mature field of
applied mathematics. Some of the probabilistic tools commonly utilized now in statis-
tics originated in the late 17th and early 18th century. Yet, solutions optimal from the
theoretical point of view may be of little practical value if they involve computations
that are too difficult to be implemented on any real computer. This is typically the case
when the models considered are non-linear, non-normal, or high-dimensional. There
are essentially two ways of arriving at a feasible algorithm--either to look for a sim-
pler model, or to approximate the theoretical solution. In both cases, the ideal solution
is abandoned.
We clearly face a dilemma. Should we stick to theory--and compute little? Or
should we rather insist on feasibility--and sacrifice theory? This book suggests a
compromising solution that views approximation as an inherent part of inference. The
bottom line is that no substantial difference is made between the statistical and com-
putational uncertainty.
Level
The book is intended for scientists and graduate students interested in parameter esti-
mation and system identification for complex, 'nonstandard' models. The mathemat-
ical prerequisites are rather low although some preliminary knowledge of statistical
methods of parameter estimation will be helpful.
Every attempt was made to avoid unnecessary technicalities. This is consistent
with the author's firm belief that an escape from the dilemma 'statistical theory ver-
sus feasible algorithms' will eventually be found in a thoughtful modification of the
current paradigms rather than in higher mathematics itself.
Throughout the book iti s assumed that data can be modelled as either continuous
or discrete random variables. The density functions are considered strictly positive.
The unknown parameters are supposed to come from a subset of the real space. These
restrictions allow us to avoid the systematic use of measure theory and introduction
of differential-geometric notions. The loss of generality is, hopefully, outweighed by
a more lucid and widely acceptable presentation, with focus on ideas rather than tech-
nical points.
VIII
Organization
The book is organized as a step-by-step introduction into a different way of thinking
about statistical methods of parameter estimation, characterized by the systematic use
of information "distances" rather than the likelihood function or posterior density of
the unknown parameter.
Chapter 1 introduces the reader into the problem, explains why the theoretical
paradigms need to be approximated and gives a brief survey of typical approaches to
design of approximation.
In Chap. 2, the problem of probability-based parameter estimation is transposed
into the language of information measures. The likelihood function and the poste-
rior density are shown to be directly related to the classical information measures--
inaccuracy, Kullback-Leibler distance and Shannon's entropy. A number of examples
illustrate the view of parameter estimation as 'probability matching'.
Chapter 3 introduces the key tool which is decomposition of the information "dis-
tance" between the empirical and theoretical densities into two components--one in-
dependent of data given the value of a properly chosen statistic, the other independent
of the unknown parameter. It gives rise to a specific Pythagorean-like geometry of
probability density functions. The connection between the Pythagorean-like decom-
position and the statistical notion of a sufficient statistic is elucidated.
Chapter 4 applies the Pythagorean geometry for approximation of the likelihood
function and posterior density in the case that the statistic used is not sufficient for the
model considered. A proper choice of the statistic is discussed in detail. The construc-
tion proposed is shown to be equivalent to the choice of a necessary statistic.
Chapter 5 discusses some of possible ways of numerical implementation of thea p-
proximate estimation. A simple modification of the basic scheme is shown that makes
the basic algorithm capable of tracking slowly varying parameters. The effect of the
modification on the posterior density is essentially equivalent with the effect of expo-
nential forgetting.
Chapter 6 gives a summary of key points of the proposed solution, with emphasis
on the features that distinguish it from other approaches. Open problems and chal-
lenges are indicated.
Throughout the book the cases of independent observations and controlled dy-
namic systems are considered in parallel. The former, simpler and more transparent
case serves as introduction and motivation to the latter which is the real challenge.
Geometric Insight
I build as much as possible on geometric intuition, making maximum use of orthog-
onal and minimum distance projections. I like to envisage key ideas through pictures
and geometric concepts and I do hope that most readers will enjoy it too. The Eu-
clidean case, simple and transparent, serves repetitively as motivation for analogous
solutions in the spaces of probability density functions. A number of examples and
solvedp roblems is used to illustrate and enlighten the major steps of exposition.
IX
stnemgdelwonkcA
The book has been greatly influenced, both in contents and form, by numerous dis-
cussions with colleagues and friends working in fields as different as system iden-
tification, statistical parameter estimation, differential geometry, probability theory,
information theory, mathematical system theory. The list of their names would be too
long to put here, but my thanks go to all of them for making me see the problem in a
broader perspective.
My personal enthusiasm for the geometric way of thinking about statistical prob-
lems started in 1985 when I discovered in one of Moscow's second-hand book-
shops the monograph Statistical Decision Rules and Optimal Inference by Nikola~
N. (~encov. I take this opportunity to pay tribute to the book that has become a source
of unceasing inspiration for me.
Results achieved since 1985 have been to a large extent the product of a stimu-
lating environment of the Institute of Information Theory and Automation in Prague.
My colleagues and students contributed a good deal to the production of this book,
both by raising tough questions and by helping me to see the answers. I am particu-
larly indebtedt o Vficlav Peterka who introduced me into the vast gardens of Bayesian
identification and to Miroslav Kgrn~ who roused my interest in recursive nonlinear
estimation and who constantly encouraged and supported me on my way through.
The biggest debt I owe is to my family--for all those hours that ought to have
been spent with them.
Prague, March 1996 Rudolf ~2vahluK
Table of Contents
Inference Under Constraints
. . . . . . . . . . . . . . . . . . . . . . .
1.1 The Bayesian Paradigm ....................... 1
1.2 Recursive Estimation ........................ 2
1.3 Admissible Complexity ....................... 3
1.4 Common Approaches to Approximation .............. 5
1.5 Why the Geometric Approach? ................... 10
2 From Matching Data to Matching Probabilities .............. 13
2.1 Data Matching in Euclidean Space ................. 13
2.2 Probability-based Estimation .................... 17
2.3 Estimation via Inaccuracy ..................... 25
2.4 Maximum Likelihood versus Bayesian Estimation ........ 34
2.5 Inaccuracy for Discrete Data .................... 38
2.6 Examples: Probability Matching .................. 47
2.7 Historical Notes ........................... 61
3 Optimal Estimation with Compressed Data ................ 63
3.1 Data Compression in Euclidean Space ............... 63
3.2 Kullback-Leibler Distance ..................... 70
3.3 Pythagorean View of Statistical Estimation ............ 77
3.4 Examples: Normal and LinearM odels ............... 104
3.5 Data Compression ......................... 110
3.6 Historical Notes ........................... 126
4 Approximate Estimation with Compressed Data ............. 129
4.1 Further Compressing Data: Euclidean Case ............ 129
4.2 Idea of Approximate Estimation .................. 132
4.3 Key Properties of Approximation ................. 141
4.4 Relation to Large Deviation Theory ................ 147
4.5 Choice of Statistic ......................... 151
4.6 Examples: Models and Statistics .................. 159
4.7 Approximatioonf Empirical Expectation ............. 171
4.8 Historical Notes ........................... 176
Xtl Table cf Contents
5 Numerical Implementation . . . . . . . . . . . . . . . . . . . . . . . . 179
5.1 Direct Optimization ......................... 179
5.2 Prior Approximation of Critical Maps ............... 183
5.3 Parameter Tracking via Forgetting ................. 186
5.4 Historical Notes ........................... 190
6 Cc nduding Remarks . . . . . . . . . . . . . . . . . . . . . . . . . . . 191
6.1 Summary of Key Points ...................... 191
6.2 Open Problems ........................... 195
6.3 Outlock ............................... 197
A Selected Topics from Probability Theory ................. 199
A. 1 Probability Spaces ......................... 199
A.2 Discrete Random Variables ..................... 202
A.3 Con~'nuous Random Variables ................... 204
A.4 Ncrmal Distribution ........................ 207
A.5 H61der Inequality .......................... 207
B Selected Topics from Convex Optimization ............... 209
B. 1 Convex Sets and Functions ..................... 209
B.2 Minimization of Convex Functions ................. 210
B.3 Jensen's Inequality ......................... 210
B.4 L egendre- Fenchel Transform .................... 210
References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 213
Index . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 223
Basic Notations and Conventions
Special Symbols
/x
defined as
proportionality, i.e., equality up to a normalizing
factor
[] end of example
Sets
X = {x~,...,xn} C3 is a set having elements Xl,... nx,
aE~ a is an element of
AC% A is a subset of %
[XI
number of elements of the set C~
empty set
R set of real numbers
Cartesian Products
~xJ~ Cartesian product of the sets ~ and Z
(y, z) E tj x Z (y, z) is an element of ~ x Z
Vectors and Sequences
T v vector transposed to the vector v
llvlI Euclidean norm of the vector v, I[vll 2 2 = v Y,7=t
yN = (Yl,... ,YN) a finite sequence (row vector) of length N starting
at 1
