Approximation of Generalized Ridge Functions in High Dimensions

3.8 MB·

by  Sandra Keiper

Approximation of Generalized Ridge Functions in High
Dimensions
Sandra Keiper
Department of Mathematics, Technische Universita¨t Berlin
10623 Berlin, Germany
E-mails: [email protected]
7
1
January 26, 2017
0
2

n
Abstract
a
J
This paper studies the approximation of generalized ridge functions, namely of functions which are

4 constantalongsomesubmanifoldsofRN. Weintroducethenotionoflinear-sleevefunctions,whosefunction
2 valuesonlydependonthedistancetosomeunknownlinearsubspaceL. Weproposetwoeffectivealgorithms
toapproximatelinear-sleevefunctionsf(x)=g(dist(x,L)2),whenboththelinearsubspaceL⊂RN andthe
]
A functiong∈Cs[0,1]areunknown. Wewillproveerrorboundsforbothalgorithmsandprovideanextensive
numericalcomparisonofboth. Wefurtherproposeanapproachofhowtoapplythesealgorithmstocapture
N
general sleeve functions, which are constant along some lower dimensional submanifolds.
.
h
t Keywords. RidgeFunctions,FunctionApproximation,BigData,HighDimensions,ActiveVariables,Active
a
m Subspaces, Optimization over Grassmannian Manifolds
[

1 1 Introduction
v
8
1 Nowadays we are living in a world where the acquisition, analysis and storage of big data play a major role.
0 Digital communication, medical imaging, seismology and cosmology are only a few examples, which show the
7 necessity to handle massive data sets. Usually data is modeled as functions f :X →Y, where X can be RN or
0
a general curved surface. In particular the approximation of such functions from point queries, when N is very
.
1 large, is an important field. Such problems arise, for example, in learning theory [24], in modeling physical and
0 biological systems [17], as well as neural networks [3] and in parametric and stochastic PDEs [5].
7
Because of the so-called curse of dimensionality, a notion introduced in 1961 by Richard Bellman [1], the
1
handling of functions in many variables is an ambitious task. Namely, functions on RN with smoothness of
:
v order s can in general be recovered with an accuracy of at most n−s/N, applying n-dimensional spaces of
i
X linear or nonlinear approximation. Thus, the learning of functions depending on a large number of variables is
particularlydifficultevenwithsmoothnessassumptionsonf [8,9,20]. Certainly, weneedtoimposeadditional
r
a structure on f to achieve efficient learning [4, 14, 16, 22, 23].
1.1 Ridge Functions
One popular approach to break the curse of dimensionality is to consider ridge functions of the form
RN ⊇Ω(cid:51)x(cid:55)→f(x)=g(Ax), (1)
whereA∈Rm×N,withmconsiderablysmallerthanN,isusuallycalledridgematrix andg ∈Cs(Rm),1≤s≤2,
is called the ridge profile. The requirement for the function to have at least one derivative is essential. In fact,
it was shown in [18] that ridge functions need to have a first derivative uniformly bounded away from zero in
the origin in order to reduce the complexity of the approximation task.
1
For particular choices of A different approaches to successfully learn ridge functions have been investigated.
For example, if A is of the form AT = [e ,...,e ], for e ∈ RN being the canonical unit vectors and
i1 im ik
i ∈ {1,...,N}, f can be rewritten as a function which depends only on a few variables, i.e., f(x ,...,x ) =
k 1 N
g(x ,...,x ). An approach to recover the active variables and approximating the ridge profile g has been
i1 im
givenin[10]. Itwasshownthatbyadaptivesamplingwecanobtainsimilarestimatesasiftheactivecoordinates
i ,...,i are known to us.
1 m
Another special case of (1) is to assume that m = 1 and that the matrix A is, therefore, a vector, usually
called ridge vector and denoted by AT =:a. In this case, f is of the form
f(x)=g((cid:104)x,a(cid:105)). (2)
The recovery of such ridge functions from point queries was first considered by Cohen, Daubechies, DeVore,
Kerkyacharian,andPicardin[4]forridgefunctionswithapositiveridgevector. Itwasshownthattheaccuracy
of their method is close to the approximation rate of one-dimensional continuous functions.
However, the algorithm from [4] does not apply to arbitrary ridge vectors. In [14, 16, 23] new algorithms
were introduced to waive the assumption of a positive ridge vector. The main idea of the algorithm in [16] is to
approximate the gradient of f by divided differences, exploiting the fact that the gradient of f is some scalar
multiple of the ridge vector. The accuracy of the approximation of the gradient is determined by the choice of
the step size in the computation of the divided differences, whereas the number of sampling points is fixed.
The approach by Fornasier, Schnass and Vybiral [14] is rather based on compressed sensing and applies to
(1) very generally. Thus, not the gradient but the directional derivatives of f were approximated at a certain
number of random points in random directions. However, especially for the methods in [4, 14], the authors
need a restrictive assumption to use compressed sensing techniques. That is, the ridge vector a can be well-
approximated by a sparse subset of its coefficients. In [23] this assumption could be removed by leveraging the
Dantzig selector [2] to recover an approximation of a.
However, the structure assumption on f to be a ridge function can be very restrictive. If we, for example,
consider a sensor network, where we have a certain number of sensors, say N, which measure the moisture,
temperatureandpressuretoforecastforestfire,theaimistocomputetheriskoffirebyafunctionf :R3N →R
depending on the measurements of the sensors. It is then very unlikely that the combination of measurements
which yield the same risk of fire lie on a 3N−1-dimensional hyperplane, since also parameters like topography
and vegetation influence the prediction. Much more likely is the assumption that these combinations lie on a
lower dimensional manifold.
1.2 Sleeve Functions
To allow for the recovery of more general functions, which are constant along some lower-dimensional subman-
ifolds, we will introduce the notion of sleeve functions and as a special case of linear-sleeve functions. Within
thispaperwewilltheninvestigateandanalyzealgorithmstocapturelinear-sleevefunctionsandwewillpropose
a technique to apply these methods to general sleeve functions.
Definition 1.1 Let g ∈ Cs[−r,r], r ∈ R , M ⊂ RN a d-dimensional smooth submanifold of RN, and
+
tub (M) := {x ∈ RN : dist(x,M) < r}, then we call f : tub (M) → R a sleeve function if we can rewrite f in
r r
terms of g by
f(x)=g(dist(x,M)2), (3)
for x ∈ tub (M). In the case where M is a linear subspace, we call f a linear-sleeve function and denote
r
L:=M, to emphasize the special case, i.e., we write:
f(x)=g(dist(x,L)2). (4)
The need to restrict g to a bounded domain is twofold; on the one hand, if we wish to recover g from
a finite number of sampling points, we do need this restriction and on the other hand, it is useful for the
approximation task to have a unique mapping x (cid:55)→ x with dist(x,M) = dist(x,x ). Thus, in the case of M
0 0
being a linear subspace, r can be chosen arbitrarily, where in the general case r is chosen to be the radius of a
non-self-intersecting tube around M. For an illustration of linear-sleeve functions we refer to Figure 2.
2
Note that the notion of sleeve functions is indeed a generalization of ridge function, thus, if L is an N −1-
dimensional subspace, we can rewrite dist(x,L)2 = (cid:104)x,a(cid:105)2, where a is the normal vector of L. Also note that
thisformulationisverydifferentfromtheoneintroducedin[14]. Indeed, iff isoftheformf =g(A·), thelevel
sets are linear subspaces, whereas this is not true for linear-sleeve functions (cf. Figure 2).
Furthermore, observe that even by separating the approximation task in approximating g and M, we cannot
simply use manifold learning algorithms to approximate M, since manifold learning algorithms (cf. e.g. [7, 25])
usually assume that we can sample from the manifold. However, we need to reconstruct the level sets (or at
least one, namely M) without knowing in advance to which level set the sampling points belong; actually it is
very likely that all sampling points belong to different level sets.
1.3 Our Contribution
Our work studies the approximation of linear-sleeve function of the form f(x)=g(dist(x,L)) for x∈tub (L),
1
where g ∈ Cs[0,1] and L ⊂ RN is a d-dimensional subspace of RN. We will provide and analyze two different
algorithms to capture linear sleeve functions from point queries. Our main contributions can be summarized as
follows.
• Adaptive Algorithm. The first algorithm, to which we refer to as ATPE, is based on the fact that the
gradientoff insomex∈RN isperpendiculartothelevelsetoff inx. Wewillshowthattherestrictionof
f totheplane,whichisperpendiculartothegradientandwhichisthetangentplaneofthecorresponding
level set, is again a linear-sleeve function. We will then argue that applying the same fact iteratively to
the restrictions of f, the tangent plane computed in the N −d-th step gives a reconstruction of L. In
ATPE will then substitute the gradient by divided differences, because we cannot compute the gradient
by point queries of f.
• Optimization Algorithm The second algorithm, to which we refer to as OGM, is based on a minimization
over the Grassmannian manifold. Namely, it will define an objective function, whose minimizer is L.
However, we will see that we cannot define this objective functions using only point samples of f. In
OGM we therefore approximate this objective function by an objective function whose minimizer L˜ will
be proven to be close to L.
• Error Bounds. Those two algorithms are of a rather different nature. Whereas the approximation success
of the first algorithm depends only on the error of the gradient approximation by divided differences, the
success of the second algorithms depends on the error of the approximation of the objective function.
The first main theorem states that the error of the approximation of the d-dimensional subspace L using
ATPE can be bounded by
√
(cid:107)L−L˜(cid:107) ≤C(1+K)N−d N −dhs,
HS
where L˜ is the approximation of L, h can be chosen arbitrarily small but fixed, C,K are some positive
constantsandthenumberoffunctionevaluationsisgivenby(N+1)(N−d). Forthesecondmaintheorem
we prove that using OGM the approximation error is given by
√
(cid:107)L−L˜(cid:107) ≤C˜ N −dM−1,
HS
whereM isthenumberoffunctionsamplesandC˜ aconstantonlydependingpolynomialonthedimension
ofthespace. NotethatOGM,differentlytoATPE,yieldsareconstructionerrorwhichdecreaseswiththe
number of sampling points and is not constrained by a fixed number of sampling points and is, therefore,
advantageous. However, the first algorithm is more promising to apply also to the manifold case.
• Impact on the Approximation of General Sleeve Functions. The next step would be to find algorithms to
recover general sleeve functions of the type (3). Due to the fact, that we would need to optimize over all
possibled-dimensionalsubmanifolds,toapproximategeneralsleevefunctionsinasimilarwayasproposed
by OGM, we anticipate that an adaptation of ATPE is more promising.
We believe that one can also use gradient approximations to capture general sleeve functions of the type
(3). Roughlysaid,weproposetousethegradientstocomputesamplesfromthemanifold. Moreprecisely,
knowingthegradientoff atsomepointx, againwouldenableustoapproximatethesleeveprofileg and,
3
Figure 1: Generalized ridge function of the form f(x):=g(dist(x,M))
under additional assumption, we could use the direction given by the gradient and the value of f in x
to translate x to the manifold. A careful estimation of the distribution of the translated sample points
should then enable us to apply manifold learning algorithms (e.g [25]) to estimate the manifold M.
The paper is organized as follows: After introducing some preliminaries, we will present and analyze ATPE
in Section 3. In Section 4 we will introduce and analyze OGM. The consideration will be completed in Section
5 by some promising numerical results.
2 Preliminaries
To put our results in a precise setting, we introduce the class LR(s) of all linear-sleeve functions f(x) :=
g(dist(x,L)2), where g ∈Cs[0,1] and L⊂RN a subspace. We use the following norm, subsequently referred to
as H¨older norm, on Cs. For k <s≤k+1, with k ∈N, we define
k
(cid:88)
(cid:107)g(cid:107) :=(cid:107)g(cid:107) :=|g(k)| + (cid:107)g(j)(cid:107) ,
Cs Cs[0,1] Lip(s−k) C[0,1]
j=0
where g(j) denotes the j-th derivative of g, and, for 0<β ≤1, we set
|g(x)−g(y)|
|g| :=sup .
Lip(β) x(cid:54)=y |x−y|β
Note that we call g Lipschitz continuous if |g| is bounded. We then call |g| Lipschitz constant or
Lip(1) Lip(1)
Lipschitznorm ofg. Ifwewanttohighlightthedimensionofthevectorspace,wesometimeswrite(cid:107)·(cid:107) :=(cid:107)·(cid:107)
(cid:96)N p
p
forthe(cid:96) normofavector, for p=1,2. Theweak(cid:96) normofavectorx∈RN isthesmallestconstantM, such
p p
that
#{i:x ≥ε}≤Mε−1/p,ε>0.
i
We further recall the following useful property of any norm on RN.
Lemma 2.1 ([16]) Let (cid:107)·(cid:107) be any norm on RN and x∈RN with (cid:107)x(cid:107)=1, x˜∈RN \{0} and λ∈R. Then
x˜ 2(cid:107)x˜−λx(cid:107)
(cid:107)sign(λ) −x(cid:107)≤ .
(cid:107)x˜(cid:107) (cid:107)x˜(cid:107)
We will denote the i-th canonical unit vector, with a one in the i-th coordinate and zero elsewhere, by e .
i
The Grassmannian manifold of all d-dimensional subspaces of RN is denoted by G(d,N) and for an orthogonal
projection P :RN →RN, the operator norm is given by the Hilbert-Schmidt norm
(cid:118)
(cid:117) N
(cid:117)(cid:88)
(cid:107)P(cid:107) =(cid:116) (cid:107)Pe (cid:107)2.
HS i 2
i=1
4
(a) (b)
Figure 2: Generalized ridge function of the form f(x) := g(dist(x,L)2), where L is some one-dimensional affine
subspace of RN. The figures show three level sets for the function f(x ,x ,x ) = x2 +x2 = dist(x,L)2, where
1 2 3 2 3
L := span(1,0,0) (blue line). (a) The level sets are illustrated in the ambient space R3. (b) Projection of the
level sets onto the subspace orthogonal to L, which is here the xy-plane.
The orthogonal complement of a subspace P is denoted by P⊥ and the distance of a vector x ∈ RN to a
subspace, respective subset, P ⊂RN is defined by
dist(x,P):=min(cid:107)x−y(cid:107) .
2
y∈P
In the sequel, for two quantities A,B ∈ R, which may depend on several parameters, we shall write A (cid:46) B, if
there exists a constant C >0 such that A≤CB, uniformly in the parameters. If the converse inequality holds
true, we write A(cid:38)B and if both inequalities hold, we shall write A(cid:16)B.
Finally, wewanttorecallsomeapproximationpropertiesoffunctionsinC[0,1]. FortwogivenintegersS >1
andM ≥2,weconsiderthespaceS ,h:=1/M,ofpiecewisepolynomialsofdegreeS−1withequallyspaced
h,S
knots at the points ih, i=1,...,M −1, and having continuous derivatives of order S−2. It is well-known (cf.
e.g. [8]) that there is a class of linear operators Q which maps C[0,1] into S . These operators are usually
h h,S
called quasi-interpolants. For a function g ∈C[0,1], the application of a quasi-interpolant only depends on the
values of g at the points ih, i = 0,...,M. Furthermore, we can choose the operator Q to fulfill the following
h
property: For all g ∈Cs([0,1])
(cid:107)g−Q g(cid:107) ≤C|g| hs, (5)
h C[0,1] Cs[0,1]
with C, a constant depending only on S [8].
3 An Adaptive Algorithm Estimating Tangent Planes
Anobviousapproachtoapproximatesleevefunctionsoftheform(3)istoapplythemethodstorecoverclassical
ridge functions of the form (2), i.e., to approximate a linear sleeve function by a classical ridge function. Of
course, this method can only provide good approximation results if the sleeve function is close to a classical
ridge function in a certain sense, cf. [15]. However, it would be more convenient to approximate a function of
the form (3) by an estimator of the same form.
We first observe that we can rewrite a linear-sleeve function as
f(x)=g(dist(x,L)2)=g((cid:107)P x(cid:107)2),
P 2
where P is the orthogonal projection to the (N −d)-dimensional subspace P orthogonal to L. For simplicity
P
we will denote the orthogonal projection of a vector x ∈ RN to a subspace H = span{u ,...,u } ∈ G(d,N)
1 d
by Hx := P (x). This notation relates to the matrix representation of an orthogonal projection given by
H
H =(cid:80)d u uT.
i=1 i i
5
The algorithm, we will introduce in this section, will, similarly as in [4, 16], exploit the fact that we can
estimate the tangent plane in some x ∈RN of the (N −1)-dimensional submanifold
0
(cid:8)x∈RN :dist(x,L)2 =dist(x ,L)2(cid:9)
0
as the unique hyperplane which is perpendicular to the gradient of f in x . We will then show that the
0
function f restricted to this tangent plane is again of the form (4). Of course, we cannot compute the gradient
by sampling the function; in the subsequent proposed algorithm we, therefore, approximate the gradient by
divided differences.
Figure 3: The restriction of f to the affine subspace which is orthogonal to the gradient of f in some x is again a
1
linear sleeve function. In the illustrated case we can recover L after two steps, because the gradient of the restric-
tion of f in some x is orthogonal to L.
2
3.1 The Algorithm
As mentioned before the idea of the first algorithm is to use the fact that the gradient of the linear-sleeve func-
tion f in some x is perpendicular to the level sets and that the restriction of f to the corresponding tangent
0
plane is again a linear sleeve function (cf. Figure 3). We will further show that applying this fact iteratively
to restrictions of f finds after N −d steps the wanted subspace L. To prove this statement we introduce an
adaptive algorithm to exactly recover the subspace L by computing gradients of f (see Algorithm 1) and show
in Theorem 3.1 that this algorithm can recover the subspace L exactly. Thus, our first aim is to show that the
system, which is formed by the gradients of the restrictions of f, forms a basis for P = L⊥. Observe that an
essential step in this algorithm is to compute gradients of f and is therefore not useful to approximate f from
point samples. It only serves as an auxiliary tool to introduce the first main Algorithm 2.
Algorithm 1: ATPC - Approximation by Adaptive Tangent Plane Computation
Data: f(·)=g(dist(·,L)2).
Result: Td.
begin
fN ←−f.
TN ←−RN.
for i=N,...,d+1 do
1. For some arbitrarily chosen normalized x ∈Ti compute ∇fi(x ).
i i
2. u ←−∇fi(x )/(cid:107)∇fi(x )(cid:107) .
i i i 2
3. Ti−1 ←−(span{u ,...,u })⊥.
i N
4. Let fi−1 be the restriction of f to Ti−1.
6
As mentioned before this algorithm iteratively computes restrictions of f, such that after N −d steps the
restrictionoff willbeexactlydefinedonL. AssumewehavecomputedthetangentplaneTi andtherestriction
fi of f the subspace Ti ⊂ RN, for i = N,...,d+1. The algorithm then chooses uniformly at random a point
x ∈ Ti and computes the gradient of fi in x (1.). It then normalizes this gradient (2.) and determines the
i i
subspace Ti−1 which is orthogonal to this gradient in Ti (3.). Finally we restrict fi to Ti−1 and repeat this
procedure until i=d+1. The following theorem states that ATPC indeed succeeds to recover L.
Theorem 3.1 Let f(x) = g(dist(x,L)2) = g((cid:107)Px(cid:107)2), g ∈ Cs[0,1], 1 < s ≤ 2, for x ∈ tub (L) and the linear
2 1
subspace P⊥ = L ⊂ RN. Compute Td as proposed in Algorithm 1. Then L coincides with Td almost surely
(e.g. due to the Lebesque measure on RN).
Proof. We write L as L = span{u ,...,u } where {u ,...,u ,u ,...,u } is an orthonormal basis of RN,
1 d 1 d d+1 N
and let V =[u ...u u ...u ] be the corresponding matrix. We begin by computing the gradient of f and
1 d d+1 N
obtain
∇f(x )=2g(cid:48)((cid:107)Px (cid:107)2)Px ,
N N 2 N
which is obviously perpendicular to L if Px (cid:54)=0, which is almost surely true.
N
Due to the orthogonality of ∇f(x ) to L, we can assume that u = ∇f(x )/(cid:107)∇f(x )(cid:107) and then by
N N N N 2
definitionTN−1 =(span{u })⊥ =span{u ,...,u }. WenowdefinefN−1 tobetherestrictionoff toTN−1
N 1 N−1
and hN−1 :RN−1 →R by
(cid:20) (cid:21) (cid:20) (cid:21)
xˆ xˆ
hN−1(xˆ):=f(V )=fN−1(V ),
0 0
for xˆ ∈ RN−1. Then ∇hN−1T(xˆ) = ∇f(x)TVˆ , where Vˆ := [u ...u ], i = 1,...,N, and x =: V(xˆ,0)T ∈
N−1 i 1 i
TN−1. Thus, the gradient of hN−1, considered as a vector in RN is given by
(cid:20)∇hN−1(xˆ)(cid:21)
=VT ∇f(x),
0 N−1
where V :=[u ...u 0...0]. We conclude that for some x ∈TN−1 chosen uniformly at random, we have
i 1 i N−1
∇fN−1(x )=VVT ∇f(x )=2g(cid:48)((cid:107)Px (cid:107)2)VVT Px. (6)
N−1 N−1 N−1 N−1 2 N−1
A straightforward computation shows that VVT Px is the projection of Px to TN−1. Thus, it is obvious
N−1
thatforsomex ∈TN−1,chosenasinAlgorithm1,∇fN−1(x )isperpendiculartoL,i.e.,ifPx (cid:54)=0,
N−1 N−1 N−1
which is almost surely the case.
Further, ∇fN−1(x ) is also orthogonal to u , since it lies in TN−1. Therefore, we set
N−1 N
u =∇fN−1(x )/(cid:107)∇fN−1(x )(cid:107)
N−1 N−1 N−1 2
for some x ∈ TN−1 and TN−2 := span{u ,...,u }. Note that again Px (cid:54)= 0 holds almost surely.
N−1 1 N−2 N−1
We repeat this procedure until we get a basis {u ,...,u } of P, which yields the desired space L=P⊥.
d+1 N
The previous theorem shows that, if we could compute the gradient in (N −d) points, we would be able
to recover the space L, respectively its orthogonal complement P, exactly. However, ATPC is not based on
sampling the function f, since gradients cannot be computed exactly using point queries. Thus, we can only
approximate the gradients by computing the divided differences
(cid:20)f(x+he )−f(x)(cid:21)N
∇ f(x)= i .
h h
i=1
We adapt the first step in ATPC by substituting the computation of the gradients by computing divided dif-
ference and propose Algorithm 2, to which we will refer as ATPE, for the approximation task.
7
Algorithm 2: ATPE - Approximation by Adaptive Tangent Plane Estimation
Data: f(·)=g(dist(·,L)2).
Result: L˜.
begin
f˜N ←−f.
T˜N ←−RN.
for i=N,...,d+1 do
1. For some arbitrarily chosen normalized x˜ ∈T˜i compute
i
(cid:34) (cid:35)N
f˜i(x˜ +he )−f˜i(x˜ )
∇ f˜i(x˜ )= i k i .
h i h
k=1
2. u˜ ←−∇ f˜i(x˜ )/(cid:107)∇ f˜i(x˜ )(cid:107) .
i h i h i 2
3. T˜i−1 ←−(span{u˜ ,...,u˜ })⊥.
i N
4. Let f˜i−1 be the restriction of f to T˜i−1.
L˜ ←−T˜d.
The described procedure, of course, cannot find the correct plane P. However, it is able to compute a good
approximationofP,wheretheapproximationerrordependsonthechoiceofh. Forreasonsofclarity,theproof
of the next theorem is moved to the next subsection.
Theorem 3.2 Let f be a linear-sleeve function of the form (4), i.e., f ∈ LR(s) for some s ∈ (1,2]. Assume
that the derivative of g ∈ Cs([0,1]) is bounded by some positive constants c ,c . By sampling the function f
2 3
at (N −d)(N +1) appropriate points, ATPE constructs almost surely an approximation of L by a subspace
L˜ ⊂RN, such that the error is bounded by
√
(cid:107)L−L˜(cid:107) ≤CCˆ(1+CˆK)N−d N −dhs/2(s−1)i,
HS
for some arbitrarily small h>0, where
K =4|g(cid:48)| +2c +max{(cid:107)∇f(x )(cid:107):i=N,...,d+1}
Cs 3 i
Cˆ =2/min{(cid:107)∇fN−i(x )(cid:107) :i=N,...,d+1}
i 2
C2 =h2−sc +4|g(cid:48)|shs−1(2+h)2(s−1)+c |g(cid:48)| (2+h)s−1,
3 C 3 Cs
which are constants depending only on the H¨older norm and bounds of g(cid:48).
In particular if g(cid:48) is Lipschitz continuous, i.e. s=2, it holds
√
(cid:107)L−L˜(cid:107) (cid:46)(1+C˜)N−d N −dh,
HS
with C˜ =CˆK.
It then only remains to recover the ridge profile g. The estimation of g is rather straightforward. As the
gradient gives the direction in which f changes, f becomes a one-dimensional function in the direction of the
gradient. Hence, we can estimate g with well-known numerical methods. Indeed, we have already seen that the
gradientoff insomepointxisgivenby∇f(x)=g(cid:48)((cid:107)Px(cid:107)2)Px,i.e.,thenormalizeddirectionisa:=Px/(cid:107)Px(cid:107) .
2 2
Setting x :=ta yields
t
t2
f(x )=g((cid:107)Px (cid:107)2)=g( (cid:107)Px(cid:107)2)=g(t2).
t t 2 (cid:107)Px(cid:107)2 2
2
However, similar to the algorithm in [16], this algorithm uses a fixed number of samples and the estimation
cannot be improved by taking more samples. We, therefore, also aim for an algorithm which yields a recon-
struction whose error decreases with the number of sampling points (cf. Section 4). Also note that due to the
adaptive character of ATPE the reconstruction error increases for smaller values of the Lipschitz continuity s.
8
To complete this subsection, we want to remark that we can perform a similar, but slightly worse, error
analysis for the case that f is of the form f(x) = g((cid:107)Px(cid:107) ), whereas before we considered sleeve functions of
2
the form f(x)=g((cid:107)Px(cid:107)2). See Remark 1 in the next subsection for a short explanation.
2
3.2 Proof of Theorem 3.2
As mentioned above, the idea is to approximate the gradients of f =fN and fi for i=d+1,...,N −1. Since
weneedN+1samplesforeachgradientapproximation,weneed(N−d)(N+1)samplesaltogether. Wealready
know from Theorem 3.1 that the subspace P can be written in terms of the gradients of the restrictions of f
and f itself. Hence, we assume P = L⊥ = span{u ...,u }, where the u ’s are given as stated in Theorem
d+1 N i
3.1. We split the proof by establishing several lemmata.
Lemma 3.3 Under the assumptions of Theorem 3.2 and with the choice of the u˜ ’s, i = d+1,...,N, as
i
proposed in Algorithm 2, we have
√
2C N −dhs/2
(cid:107)u˜ −u (cid:107) ≤ =:S , (7)
N N 2 ∇f(x˜ ) 0
N
where x˜ ∈RN is chosen uniformly at random u = ∇f(x˜N) .
N N (cid:107)∇f(x˜N)(cid:107)2
Proof. First, let us estimate the error between ∇f(x) and ∇ f(x). We can compute
h
f(x+he )−f(x) g((cid:107)P(x+he )(cid:107)2)−g((cid:107)Px(cid:107)2)
∇ f(x) = i = i 2 2
h i h h
(cid:107)P(x+he )(cid:107)2−(cid:107)Px(cid:107)2
=g(cid:48)(ξ ) i 2 2
i,h h
(cid:80)N (cid:104)x+he ,u (cid:105)2−(cid:104)x,u (cid:105)2
=g(cid:48)(ξ ) j=d+1 i j j
i,h h
N
(cid:88)
=g(cid:48)(ξ ) 2u (cid:104)x,u (cid:105)+hu2
i,h ji j ji
j=d+1
 
N
(cid:88)
=g(cid:48)(ξi,h)2[Px]i+h u2ji,
j=d+1
for some ξ between (cid:107)Px(cid:107)2 and (cid:107)P(x+he )(cid:107)2, where u denotes the i-th entry of the vector u . We then
i,h 2 i 2 ji j
estimate
N N
(cid:12)(cid:12)ξi,h−(cid:107)Px(cid:107)22(cid:12)(cid:12)≤|(cid:107)P(x+hei)(cid:107)22−(cid:107)Px(cid:107)22|≤ (cid:88) (cid:12)(cid:12)(cid:0)2h(cid:104)x,uj(cid:105)uji+h2u2ji(cid:1)(cid:12)(cid:12)=2h|[Px]i|+h2 (cid:88) u2ji ≤2h+h2,
j=d+1 j=d+1
where we used the fact that u is a unit vector and that, therefore, all its entries (in absolute value) and the
entries of its projection are smaller than or equal to one. Thus, the error which we obtain by approximating
the gradient can be estimated as
 2
N N
(cid:88) (cid:88)
(cid:107)∇f(x)−∇hf(x)(cid:107)22 = g(cid:48)(ξi,h)2h2 u2i,j
i=1 j=d+1
N
+4(cid:88)(g(cid:48)(ξ )−g(cid:48)((cid:107)Px(cid:107) ))2(Px)2
i,h 2 i
i=1
N N
+2h(cid:88)|g(cid:48)(ξi,h)|(cid:12)(cid:12)g(cid:48)(ξi,h)−g(cid:48)((cid:107)Px(cid:107)22)(cid:12)(cid:12)|(Px)i| (cid:88) u2ji
i=1 j=d+1
=:T +T +T .
1 2 3
9
To estimate those terms we take the following inequality into account:
N  N 2 N  N  N  N  N  N (cid:32) N (cid:33)
(cid:88) (cid:88) (cid:88) (cid:88) (cid:88) (cid:88) (cid:88) (cid:88) (cid:88)
 u2ji =  u2ji u2ji≤  u2ji= u2ji =N −d,
i=1 j=d+1 i=1 j=d+1 j=d+1 i=1 j=d+1 j=d+1 i=1
where we used in the second as well as in the last step that {u }d forms an orthonormal system, which spans
j j=1
H,sothat(cid:80)d u2 ≤1foreachi=1,...,N and(cid:80)N u2 =1foreachj =1,...,d. Nowthedesiredestimates
j=1 ji i=1 ji
follow immediately:
 2
N d
(cid:88) (cid:88)
T1 ≤h2(cid:107)g(cid:48)(cid:107)2∞  u2ji ≤(N −d)(cid:107)g(cid:48)(cid:107)2∞h2,
i=1 j=1
N
(cid:88)
T ≤4|g(cid:48)|2 (2h+h2)2(s−1) (Px)2 ≤|g(cid:48)|2 (2h+h2)2(s−1),
2 Cs i Cs
i=1
T ≤(cid:107)g(cid:48)(cid:107) (cid:107)g(cid:107) (N −d)h(2h+h2)s−1.
3 ∞ Cs
Thus, we can find a constant C >0, independent of the dimensions d and N, such that
√
(cid:107)∇f(x)−∇ f(x)(cid:107) ≤C N −dhs/2. (8)
h 2
For the exact choice of C we refer to Theorem 3.2. Hence, applying Lemma 2.1, with λ=1/(cid:107)∇ f(x)(cid:107) , proves
h 2
the claim.
Next, we use the approximation of the gradient to approximate the tangent plane TN−1 at x with T˜N−1 =
span{∇ f(x)⊥}. The approximation error is then, of course, given by (7). Further, we let fN−1 and f˜N−1 be
h
the restriction of f to TN−1 and T˜N−1, respectively.
Again we want to compute the column vectors u of V, i = N,...,d+1, step by step as the normalized
i
gradients of f, fi. But instead of computing the gradient of fj we can only approximate it through an
approximation of the gradient of f˜j. Thus, we iteratively set the columns u˜ , i = N,...,d+1 of V˜ as the
i
normalized approximated gradients of f˜. The error of the approximation in each step can then be estimated
i
by means of the following lemmata, in particular, by means of Lemma 3.4 for the first step and Lemma 3.7 for
the i-th step.
Before stating these lemmata, let us recall the definition of the matrices V,V˜,V and V˜, i=N−1,...,d+1.
i i
Let
u˜ =∇ f˜i(x˜ )/(cid:107)∇ f˜i(x˜ )(cid:107) and u =∇ fi(x )/(cid:107)∇ fi(x )(cid:107) ,
i h i h i 2 i h i h i 2
for i = N,...,d+1, where x = VVTx˜ and x˜ as well as fi and f˜i are chosen as proposed in the Algorithms
i i i i
ATPC and ATPE. Then {u ,...,u } as well as {u˜ ,...,u˜ } form orthonormal systems and u ,u˜ , i =
d+1 N d+1 N i i
1,...,d, are chosen that the whole systems {u ,...,u } and {u˜ ,...,u˜ } form an orthonormal basis for RN.
1 N 1 N
We can now define
(cid:2) (cid:3) (cid:2) (cid:3)
V = u u ... u 0 ... 0 , V = u u ... u ,
i 1 2 i 1 2 N
V˜ =(cid:2)u˜ u˜ ... u˜ 0 ... 0(cid:3), V˜ =(cid:2)u˜ u˜ ... u˜ (cid:3).
i 1 2 i 1 2 N
Lemma 3.4 With the same assumptions and choices as in Theorem 3.2 and Lemma 3.3, we have
(cid:107)u˜ −u (cid:107) ≤Cˆ(1+K)Ss−1.
N−1 N−1 2 0
We first have to prove the following lemma:
Lemma 3.5 With the same assumptions and choices as in Theorem 3.2 and Lemma 3.3, let x := VVT x˜,
N−1
for some x˜∈T˜N−1. We then have
(cid:107)Px−Px˜(cid:107) ≤(cid:107)x˜(cid:107) (cid:107)u −u˜ (cid:107) .
2 2 N N 2
10