Table Of ContentAsymptotic Theory of Nonlinear Regression
Mathematics and Its Applications
Managing Editor:
M. HAZEWINKEL
Centrefor Mathematics and Computer Science, Amsterdam, The Netherlands
Volume 389
Asymptotic Theory of
Nonlinear Regression
by
Alexander V. Ivanov
Glushkov Institutefor Cybernetics.
Kiev. Ukraine
Springer-Science+Business Media, B.V.
A C.I.P. Catalogue record for this book is available from the Library of Congress.
ISBN 978-90-481-4775-5 ISBN 978-94-015-8877-5 (eBook)
DOI 10.1007/978-94-015-8877-5
Printed an acid-free paper
An
Rights Reserved
© 1997 Springer Science+Business Media Dordrecht
Originally published by Kluwer Academic Publishers in 1997
Softcover reprint ofthe hardcover Ist edition 1997
No part of the material protected by this copyright notice may be reproduced or
utilized in any fonn or by any means, electronic or mechanical,
including photocopying, recording or by any infonnation storage and
retrieval system, without written pennission from the copyright owner.
Contents
INTRODUCTION 1
1 CONSISTENCY 5
1 Introductory Remarks . . . . . . . . . . . . . . . . . . . . . . . .. 5
2 Large Deviations of the Least Squares Estimator in the Case of
Errors Having an Exponential Moment . . . . . . . . . . . . . . .. 9
3 Large Deviations of the Least Squares Estimator in the Case of
Errors with a Moment of Finite Order . . . . . . . . . . . . . . .. 25
4 The Differentiability of Regression Functions and the Consistency
of the the Least Squares Estimator . . . . . . 45
5 Strong Consistency . . . . . . . . . . . . . . . 58
6 Taking the Logarithm of Non-Linear Models. 73
2 ApPROXIMATION BY A NORMAL DISTRIBUTION 79
7 Stochastic Asymptotic Expansion of Least Squares Estimators 79
8 Asymptotic Normality of Least Squares Estimators: First Results. 92
9 Asymptotic Normality of Least Moduli Estimators . . . . . 108
10 Asymptotic Expansion of the Distribution of Least Squares
Estimators. . . . . . . . . . . . . . . . . . . . . . . . . . . . 119
11 Calculation of First Polynomials of an Asymptotic Expansion of the
Distribution of a Least Squares Estimator . . . . . . . . . . 144
3 ASYMPTOTIC EXPANSIONS RELATED TO THE LEAST SQUARES
ESTIMATOR 155
12 Asymptotic Expansion of Least Squares Estimator Moments. 155
13 Asymptotic Expansions Related to the Estimator of the Variance
of Errors of Observation . . . . . . . . . . . . . . . . . . . . . . .. 168
14 Asymptotic Expansion of the Distribution of the Variance
Estimator of Observational Error in Gaussian Regression 188
15 Jack Knife and Cross-Validation Methods of Estimation of the
Variance of Errors of Observation. . . . . . . . . . . . . . . . . 196
16 Asymptotic Expansions of Distributions of Quadratic Functionals
of the Least Squares Estimator . . . . . . . . . . . . . . . . . . . . 207
v
vi CONTENTS
17 Comparison of Powers of a Class of Tests of Hypotheses on a
Non-Linear Regression Parameter. . . . . . . . . . . . . . . . 229
4 GEOMETRIC PROPERTIES OF ASYMPTOTIC EXPANSIONS 251
18 Certain Aspects of the Differential Geometry of Models of
Non-Linear Regression . . . . . . . . . . . . . . . . . . . 251
18.1 Embedded Riemannian Manifolds and Statistical
Connectedness . . . . . . . . . . . . . . . . . . . 251
18.2 Statistical Curvature . . . . . . . . . . . . . . . . 254
18.3 Measures of the Non-Linearity of Regression Models 256
18.4 Statistical Invariants . . . . . . . . . . . . . . . . . 267
18.5 Invariants of Non-Linear Regression with a Scalar
Parameter. . . . . . . . . . . . . . . . . . . . . . . 268
18.6 Invariants of Non-Linear Regression with a Vector
Parameter. . . . . . . . . . . . . . . . . . . . . . . 272
19 The Geometric Interpretation of Asymptotic Expansions. 276
19.1 Geometry of the AE of the LSE Moments. . . . . 276
19.2 The Geometry of AEs Associated with the Estimator of the
Variance (J'2 •••••••••••••••••••••••••• 279
19.3 Geometry of AE of Distributions of Quadratic Functionals
of the LSE ........................... 281
19.4 Geometries of the Statistical Criteria for Testing Hypotheses
about Non-Linear Regression Parameters .. . . . . . . .. 284
ApPENDIX 289
I Subsidiary Facts ..... . 289
II List of Principal Notations. 298
COMMENTARY 303
Chapter 1 303
Chapter 2 304
Chapter 3 305
Chapter 4 306
BIBLIOGRAPHY 309
INDEX 325
Introduction
Let us assume that an observation Xi is a random variable (r.v.) with values in
(1R1 , 81) and distribution Pi (1R1 is the real line, and 81 is the cr-algebra of its
Borel subsets). Let us also assume that the unknown distribution Pi belongs to a
certain parametric family {Pi() , () E e}. We call the triple £i = {1R1 , 81, Pi(), () E e}
a statistical experiment generated by the observation Xi.
We shall say that a statistical experiment £n = {lRn, 8n, P; ,() E e} is the
= =
product of the statistical experiments £i, i 1, ... ,n if PO' P1() X ... X Pn() (IRn
is the n-dimensional Euclidean space, and 8n is the cr-algebra of its Borel subsets).
In this manner the experiment £n is generated by n independent observations
=
X (X1, ... ,Xn).
In this book we study the statistical experiments £n generated by observations
of the form
Xj = g(j, (}) + cj, j = 1, .. . ,n. (0.1)
e e
In (0.1) g(j, (}) is a non-random function defined on c, where c is the closure in
e
IRq of the open set ~ IRq, and C j are independent r. v .-s with common distribution
function (dJ.) P not depending on ().
Let Ell (E) be the expectation with respect to the measure PO' (P). Let us
assume that Ell Xj = g(j, (}) (Ecj = 0). We shall call the representation (0.1)
a regression model, and g(j, (}), understood as a function over N x (}C (N is the
set of natural numbers) a regression function. If there exists a parametrisation of
a regression function for which g(j, (}) is a linear form of the coordinates of the
=
vector () ((}1, ... ,(}q), then the observational model (0.1) is then called a linear
regression model. Otherwise the model (0.1) is called non-linear.
=
The totality of statistical inference methods for unknown parameters ()
((}1, ... ,(}q) of the function g(j, ()) and probability characteristics of the r.v. Cj
(for example, the variance cr2) by observations Xj , j = 1, ... , n, traditionally be
longs to regression analysis. According to this, such regression models (0.1), either
linear or non-linear, as are admitted by experimentalists, are referred to as linear
or non-linear regression models.
The prevalence in regression analysis of linear models (see, for example, the
books of Seber [200] and Sen and Srivastava [202]) reflects the fact that for solv
ing concrete problems - using the terminology accepted in applications - the un
known response of the system studied is replaced by its polynomial approximation
1
A. V. Ivanov, Asymptotic Theory of Nonlinear Regression
© Springer Science+Business Media Dordrecht 1997
2 INTRODUCTION
and the coefficients of the approximating polynomials play the role of the estimated
parameters. The non-linear regression models have important advantages over the
linear ones. The main one consists in the greater adequacy of the non-linear
models that have essentially fewer unknown parameters. Often the parameters of
non-linear models have the meaning of physical variables at the same time as the
linear parameters are usually devoid of physical significance.
The last quarter of a century's study of non-linear regression models has
steadily attracted the interest of specialists; we refer only to the books of Bard [9],
Ratkowsky [192,193]' Gallant [84], Bates and Watts [27], Demidenko [70,71], Seber
and Wild [201], and Ross [196]. However, on introducing into statistics the use of
non-linear regression analysis it is necessary to overcome a series of mathematical
difficulties which do not have analogues in the linear theory. For example, the least
squares estimator (l.s.e.) non-linearly entering in a regression function parameters
can not be found in an explicit form, and this complicates the description of its
mathematical properties. Another principal peculiarity consists in this: character
istics of the l.s.e. (bias, correlation matrix, and such like) are local, i. e., depend on
unknown values of the parameters. These two cases approximate the model (0.1)
with general models of parametric statistical inference by independent non-equally
distributed observations, although it does not follow that we should neglect that
=
all inhomogeneity of the sample (0.1) X (Xl' ... ' Xn) is concentrated only in
=
the shifts g(j, 8), j 1, ... , n.
The maximum likelihood method plays an important role in the theory of
parametric estimation, for the application of which it is necessary to know to
within an unknown parameter the distribution (of the density) ofthe observations.
Concurrent with it, the Bayes' method of estimation and the method of maximal
a posteriori density also have such a peculiarity.
Not forgetting about the success, fallen to the the theory of normal regression
(the ej are Gaussian (0,0'2) r.v.-s), let us emphasise that the meaning of regres
sion analysis consists in the performance of statistical inference using minimal
information about the distribution of the r.v.-s ej. Consequently the methods of
estimation used in regression analysis must be oriented towards the use of mea
gre information about the errors of observation ej. Such methods are those of
M -estimators, the minimisation of empirical risk, minimal contrast (if the con
trast function does not depend upon the density of observations), the method of
recurrent estimation, and others (see the book by Birkes and Dodge [36]).
In the stream of publications devoted to the problems of regression analysis
the central place is occupied by the least squares method of estimates of para
meters, which has a protracted history. The basic statement of the linear theory
was worked out in the researches of Gauss [85] and Markov [153,154], and then
developed by Aitken [7], Neyman and David [166], Linnik [148], Rao [189], and
many others.
DEFINITION: The l.s.e. of an unknown parameter 8 E e, obtained from the ob
servations X = (Xl, ... ,Xn) in the form (0.1) is any vector (O~, ... ,O~) = On =
INTRODUCTION 3
9 e
n(X) E c having the following property
L:
=
L(r) [Xj - g(j,r)]2, (0.2)
'E '
(the symbol stands in the whole book for the summation over the index j from
1 to n).
If follows that we should recognise that the asymptotic (i. e., n -+ 00) properties
of the l.s.e. with parameter () of the non-linear model (0.1) were in fact not studied
until the works of Jennrich [138] and Malinvaud [152], and the basic results in this
area of mathematical statistics were obtained in the flow of the last two decades.
A large part of the proposed investi~ation devoted to the study of the asymp
totic statistical properties of the l.s.e. (}n of the parameter () of the model (0.1),
for example such as the probabilities of large deviations l.s.e., is weak and strong
consistency, stochastic asymptotic expansions (s.a.e.) and asymptotic expansions
(a.e.), l.s.e., 9 of dJ., and various functionals of them (in particular, the variance
n
estimator Ec = 0'2 > a of the errors of observation Cj, and others). A series of
questions is also examined that is connected with the differential geometry of the
a.e. obtained in this book. Less attention is given in this book to the least moduli
estimator (l.m.e.).
e =
DEFINITION: The I. m. e. of the parameter () E for the observations X (Xl, ... ,
= = e
Xn) of the form (0.1) is any random vector (O~, ... ,O~) On On (X) E c having
the property
L:
R(r) = IXj - g(j,r)l· (0.3)
The least moduli method as a method of smoothing out observations first
appeared in the work of Boscovitch [79] earlier than Gauss' method of least squares;
however, it did not obtain such a wide prevalence as the latter because of the non
differentiability of the function (0.3). This book contains some statements about
the consistency and asymptotic normality of the l.m.e. On.
In the text the sections and theorems are provided with continuous numbering
(in the Appendix a theorem's number contains the letter 'A'). Lemmas, remarks
and corollaries have two numbers: the first refers to the number of the theorem
to which they are related; the second is its own number. In the body of the work
a double numbering is adopted for formulae: the first refers to the number of the
section and the second to the number of the formula. In conditions and proofs
we often write 'for n > no' in place of the expression 'for sufficiently large n', or
perhaps nothing if we seek to avoid needless repetition.
Chapter 1
Consistency
1 INTRODUCTORY REMARKS
Let En = {Rn, En, PO' ,8 E e} be a statistical experiment generated by the inde
e
pendent observations (0.1), E Eq. Let us write
L(7) = L(7,X) = ~)Xj - g(j,7W,
X = (Xl, ... ,Xn) E lRn, 7 = (71, ... ,7q) E ec.
We consider briefly the question of the existence of the l.s.e. On, i.e., the question
e
of the existence of the r.v. On = On(X) with values in c satisfying the equation
e
Here the simplest fact, but one very important in applications, is this: if c is
compact and g(j, 8) E C(eC), j 2: 1, then the l.s.e. On exists [138].
e
Clearly, if c is compact, and g(j,8) E C(eC), j 2: 1, then there may hold
an analogous assertion for the l.m.e. On and any other estimates of the parameter
8, which are defined as point optima of a functional continuous in the arguments
(8,X).
e
It is clear that there is also interest in the case when is an unbounded set.
Let us mention one assertion covering this case for the continuous functions g(j, 8),
j 2: l.
Let us assume that the functions g(j, 8) E C(eC), j 2: 1, in which inf L(8, X)
e
is attained in c for each X E lRn. Then the l.s.e. On exists [174].
Even if only for one of the functions g(j, 8)
lim \g(j,8)\=oo,
1111-+00
then for each X E lRn and number m > 0 it is possible to determine a compact
e
set Tx,m C c such that
inf L(8, X) > m
IIE9c\Tx,m
e
and consequently inf L(8, X) is attained in c.
5
A. V. Ivanov, Asymptotic Theory of Nonlinear Regression
© Springer Science+Business Media Dordrecht 1997
