Flexible sparse regularization

0.17 MB·

by  Dirk A. Lorenz

Flexible sparse regularization
Dirk A. Lorenz Elena Resmerita
6
1 January 19, 2016
0
2

n
a Abstract
J

8 The seminal paper of Daubechies, Defrise, DeMol made clear that
1
ℓp spaces with p [1,2) and p-powers of the corresponding norms are

∈
] appropriatesettingsfordealingwithreconstructionofsparsesolutions
A
of ill-posed problems by regularization. It seems that the case p = 1
N
provides the best results in most of the situations compared to the
.
h cases p (1,2). Anextensive literaturegives great creditalsotousing
at ℓp space∈s with p (0,1) together with the corresponding quasinorms,
m although one has∈to tackle challenging numerical problems raised by
[ the non-convexity of the quasi-norms. In any of these settings, either

1 super, linear or sublinear, the question of how to choose the exponent
v p has been not only a numerical issue, but also a philosophical one.
9
In this work we introduce a more flexible way of sparse regularization
2
4 by varying exponents. We introduce the corresponding functional an-
4 alytic framework, that leaves the setting of normed spaces but works
0
with so-called F-norms. One curious result is that there are F-norms
.
1 which generate the ℓ1 space, but they are strictly convex, while the
0
6 ℓ1-norm is just convex.
1
:
v
1 Introduction
i
X
r
a We study variational regularization of inverse problems where the solution
is sought in a space of sequences. This case appears, for example, when the
solution is some function that is represented in some basis or frame, and
hence, the result can be expressed by means of the sequence of coefficients
with respect to that basis or frame. In this context it is often the case that
the sought after solution has the feature of sparsity, i.e. the basis or frame
is chosen in such a way that meaningful solutions of the inverse problem can
be written as sparse linear combinations of basis or frame vectors. In the
context of inverse problems, this approach was made popular by the work
1
[14] while penalties of ℓp-type have been used before for regularization, e.g.
in [11,12,17,22,25] and also in statistics under the name LASSO [26].
Recent results on sparse regularization deal with convergence and con-
vergence rates of solutions for ℓp regularization with 1 p < 2 [6,16,18,21],
≤
with extensions to 0 < p < 1 [10,15,28] and even more general penalties in
sequence spaces [4].
Most of these studies deal with norms (in the case of (weighted) ℓp penal-
ties with 1 p < 2) and quasi-norms in the case of penalties with 0 < p < 1.
≤
In this work we propose to look at the slightly more general framework of
so-called F-norms, where penalties of type x pk are considered.
k
| |
We collect some known results about these F-norms, provide some new
P
results that are of interest in the context of sparse regularization and investi-
gate regularization properties of these penalties. Note that an application of
this theoretical framework has been dealt with for complex valued signals in
a finite dimensional context, more precisely for parallel Magnetic Resonance
Imaging, showing a promising numerical behaviour - see [9].
2 F-norms and the sequence spaces ℓpk
For a (real or complex) linear space X an F-norm is a functional : X
k·k →
[0, [ such that (cf. [20]):
∞
1. x = 0 if and only if x = 0;
k k
2. λx = x for all scalars λ, λ = 1;
k k k k | |
3. x+y x + y for any x,y X;
k k ≤ k k k k ∈
4. for scalars λ 0 if follows that λ x 0;
n n
→ k k →
5. if x 0 we have for all scalars λ that λx 0;
n n
k k → k k →
6. If x 0 and λ 0, then λ x 0.
n n n n
k k → → k k →
Such F-norms induce translation invariant metrics by d(x,y) = x y and
k − k
conversely, if d is a translation invariant metric, then x = d(x,0) is an
k k
F-norm.
The examples of F-norms that we are going to use in this paper are the
so-called ℓpk F-norms, defined as follows: For a sequence p of positive real
k
{ }
numbers and a (real or complex) sequence x = x let
k
{ }
x = x pk
k kpk | k|
X
2
and denote
ℓpk = x = x : x pk < .
k k
{ { } | | ∞}
Although F-spaces have been studiXed in some generality and detail al-
ready, e.g. in [20], and the sequence spaces ℓpk have also been treated by [19,
20,24], they have not been considered for regularization, to the best of our
knowledge.
On ℓpk we consider d : ℓpk ℓpk [0,+ ) defined by
× → ∞
d(x,y) = x y pk. (1)
k k
| − |
X
Note that d is a metric on ℓpk and (ℓpk,d) is a complete metric space.
Severalinteresting propertiesofthesespacesareinorder. Thedualspaces
of ℓpk can be characterized as follows.
Proposition 2.1 ( [24]). (i) If p 0 as k , then the topological
k
→ → ∞
dual of ℓpk provided with the topology induced by the corresponding F-
norm is the space
ℓ∞(p ) := x = x : sup x pk <
k k k
{ { } k∈N| | ∞}
(ii) The topological dual of ℓpk with inf p > 0 is ℓ∞.
k k
The spaces are naturally ordered:
Proposition 2.2. If 0 < p q for all k larger than some K N, then
k k
≤ ∈
ℓpk ℓqk and the inclusion is dense.
⊂
Proof. The case q 1 and p q for all k has been treated in [24, Lemma
k k k
≤ ≤
2]. The proof here, however, is essentially the same.
Since all p are positive, it holds that x 0 for x ℓpk. Hence x 1
k k k
→ ∈ | | ≤
for k larger than some N N. Hence, for k max(N,K) we have x qk
k
∈ ≥ | | ≤
x pk , which shows the assertion.
k
| |
The density of the inclusion ℓpk ℓqk follows from the fact that the
⊂
sequences with finite support are dense in both spaces.
Also, the important Schur property of ℓ1 holds for these spaces as soon
as the exponents tend to one:
Proposition 2.3 ( [19, Theorem 2]). If p 1, then it holds that weak con-
k
→
vergence in ℓpk (i.e. component-wise convergence) implies strong convergence
in ℓpk. In other words, convergence xn x∗ (n ) for every k, implies
k → k → ∞
xn x∗ 0 as n .
k − kpk → → ∞
3
A Kadec-Klee (oder Radon-Riesz) property can be verified in the ℓpk
framework as well, similarly to Lemma 2 in [23]:
Proposition 2.4. Let p (0,2] and let x ℓpk converge componen-
k n
{ } ⊂ { } ⊂
twise to x ℓpk and converge also in the sense x x as n .
∈ k nkpk → k kpk → ∞
Then the following convergence holds: x x 0 as n .
k n − kpk → → ∞
We provide further properties of the spaces ℓpk is the next sections where
we treat the cases of p 1 and p 1 separately.
k k
≥ ≤
3 Averaging functionals - the superlinear pow-
ers case
Now we turn to the case of superlinear powers, i.e the case
1 q 2, q 1, as k . (2)
k k
≤ ≤ → → ∞
Proposition 3.1. If (2) holds with q > 1, then the following statements
k
hold true:
(i) ℓ1 ℓqk ℓp while the latter inclusion is strict.
p>1
⊂ ⊂ ∩
(ii) If x ℓ1, then
∈
x sup x qk.
k kqk ≤ k k1
k
In particular, if x 1, then x x .
k k1 ≤ k kqk ≤ k k1
(iii) The F-norm is strictly convex and continuous;
k·kqk
(iv) The F-norm is Gˆateaux differentiable on ℓ1 and its derivative at
k·kqk
x is given by q x qk−1sign(x ) .
k k k
{ | | }
(v) If additionally x M for some M > 0, then
k kqk ≤
x max M,1 .
k k2 ≤ { }
Proof. The inclusion ℓ1 ℓqk follows from Proposition 2.2 and (iii)-(iv) are
⊂
shown in [8, Section 1.3] and [27, Exercise 22] for the special case q = 1+ 1.
k k
The general case with q 1 and q 1 works similarly.
k k
≥ →
For (ii) note that since x / x 1 we have
| k| k k1 ≤
x x qk 1
1 = | k| | k| x qk .
x ≥ x qk ≥ sup x qk | k|
k k1 k k1 kk k1
X X X
4
Now we prove ℓqk ℓp. First note that x < implies x 0.
⊂ ∩p>1 k kqk ∞ k →
Now let p > 1 arbitrarily and notice that p 1 q 1 for any k larger than
k
− ≥ −
a number k N. Without restricting generality, we can further assume that
0
∈
x 1 for any k larger than k N, which yields x p x qk for all
k 0 k k
| | ≤ ∈ | | ≤ | |
k k . Thus, if x < , then x ℓp as well.
≥ 0 k kqk ∞ ∈
− 1
In order to illustrate the strict inclusion, let xk = k qk. Then it holds
that x qk = k−1 which diverges, i.e. x / ℓqk. Let p > 1 arbitrarily
k k
| | { } ∈
and s such that 1 < s < p. As above, one hasq 1 < s 1 and consequently,
k
1 P1 P − −
kqk > ks, for k sufficiently large. This yields
1 p 1 p
x p = <
| k| kq1k k1s
(cid:16) (cid:17) (cid:16) (cid:17)
for k large enough, which implies xk ℓp for any p > 1, as k−ps con-
{ } ∈
verges.
P
(v) can be proven by distinguishing two cases. If M 1, then x M
≤ k kqk ≤
implies x 1, k N. Consequently, one has x 2 x qk, k N which
k k k
| | ≤ ∀ ∈ | | ≤ | | ∀ ∈
yields the result. In case M > 1, let y := xk, k N and y := y , where
k ck ∀ ∈ { k}
1
ck = Mqk. Due to kxkqk ≤ M, one has kykqk ≤ 1 and, according to the first
case, y 1. Hence, the inequality
k k2 ≤
x 2 x 2
| k| | k| = y 2 1,
M2 ≤ 2 | k| ≤
k k Mqk
X X X
completes the proof.
It is natural to ask, whether the inclusion ℓ1 ℓqk from (i) in Proposi-
⊂
tion 3.1 is also strict when q > 1. It may be surprising that this indeed need
k
not be the case: If the exponents q decay to one fast enough, the sets ℓqk
k
and ℓ1 do coincide, as shown below. To this end, we employ the following
result which can be found in the paper by Simons [24, Theorem 3] (where
it is attributed to Croft and Conway) but can be traced back to the earlier
work by Nakano [19, Theorem 1].
Theorem 3.2. For 0 < p 1 and π defined by 1 + 1 = 1 it holds that
k ≤ k pk πk
ℓpk = ℓ1 if and only if
There exists a natural number N > 1 such that Nπk < . (*)
∞
k
X
5
Theorem 3.3. Let 1 q < . Then ℓqk = ℓ1 holds if and only if
k
≤ ∞
1
There exists a natural number N > 1 such that N−qk−1 < (*′)
∞
k
X
(with the convention that N−∞ = 0).
Proof. Since q > 1, we always have ℓ1 ℓqk.
k
⊂
1
−
Assume that there exists N such that N qk−1 < holds and that
k ∞
x ℓqk, i.e. x qk < is satisfied. Set p = 1/q < 1 and define y such
th∈at y = x kq|k k(|i.e. y∞pk = x ). It folPlokws thakt y < . Lketting
| k| |Pk| | k| | k| k| k| ∞
π = 1 we see that 1 = 1 q = 1 1 , and hence 1 + 1 = 1. Thus, (*)
k 1−qk πk − k −pk Ppk πk
is fulfilled and we conclude by Theorem 3.2, that y pk < which shows
k| k| ∞
x < , i.e. x ℓ1.
k| k| ∞ ∈ P
The other direction is argued similarly.
P
Example 3.4. We examine a few possibilities for the sequence q 1 to
k
→
illustrate when one can expect ℓ1 = ℓqk:
1. For q = 1+1 it holds that 1 = k and hence condition (*′) is fulfilled
k k qk−1
(the series N−k converges for every N > 1). This shows that
P 1
ℓ1+k = ℓ1.
2. Forq = 1+ 1 onegets 1 = log(k)andsince N−log(k) converges
k log(k) qk−1
if and only if N > e (since N−log(k) = k−log(N)), we see that
P
1
ℓ1+log(k) = ℓ1.
3. For q = 1 + 1 with 0 < α < 1, we get 1 = log(k)α and by
k log(k)α qk−1
Cauchy condensation we see that N−log(k)α does not converge for
any N > 1. Hence
1 P
ℓ1+log(k)α = ℓ1.
6
Now we turn towards variational regularization of inverse problems. For
simplicity we focus on the case of linear inverse problems.
Consider g : ℓ2 R defined by
→ ∪{∞}
x if x ℓqk
g(x) = k kqk ∈
( otherwise
∞
6
and consider ill-posed operator equations Ax = y, with A chosen as below.
Choosing g with q > 1 as a regularization term has two major advantages:
k
it yields unique minimizers for the regularization problem and it is a differen-
tiable function, thus making things easier from a computational viewpoint.
Proposition 3.5. Let A : ℓ2 Y be a linear continuous operator, Y be a
→
Hilbert space and g defined as above, where the sequence (q ) satisfies asser-
k
tion (*′) of Theorem 3.3. Then for any α > 0, the Tikhonov functional
1
min Ax y 2 +αg(x) (3)
x∈ℓ1 2k − kY
(cid:26) (cid:27)
has a unique minimizer x ℓ1; it even holds that x ℓ2(qk−1).
α α
∈ ∈
Proof. Note that domg = ℓ , due to Theorem 3.3 and thus, the optimization
1
problem (3) over ℓ1 is correctly formulated. If we show that g is sequentially
weakly lower semicontinuous and coercive, then the arguments for existence
of minimizers follow standard techniques (see, e.g., [23]), while uniqueness
is ensured by the strict convexity of g. Especially there holds the necessary
and sufficient optimality condition
[ A∗(Ax y)] = αq sign(x ) x qk−1, k N.
α k k k k
− − | | ∀ ∈
Since the Hilbert space adjoint A∗ maps Y into ℓ2 and q is bounded and
k
{ }
bounded away from zero (it even holds q 1), it follows that x (qk−1) ℓ2
k k
≥ | | ∈
which is equivalent to x ℓ2(qk−1).
k
{ } ∈
Note that coercivity of g follows from Proposition 3.1 (v), while weak
sequential lower semicontinuity is a consequence of convexity of g, lower
semicontinuity ofcomponentfunctionst t qk andFatou’slemma forseries.
7→ | |
Remark 3.6. In the case q 1 one gets that every minimizer (note that
k
≡
uniqueness need not to hold then) is in ℓ0, i.e. in the space of sequences
with finite support, i.e., every minimizer is sparse. In cases where the ℓ1-
penalty is not used to promote sparse solutions, but only to enforce that the
minimizer lies in ℓ1 (as, e.g., for regularization with Besov B1,1 penalty), this
1
“over-regularization” is not desirable. By moving from an ℓ1-penalty to an
ℓqk-penalty with q 1 fast enough, we do not only get unique minimizers,
k
→
but also potentially “non-sparse minimizers”.
In the case 1. of Example 3.4, i.e. for q = 1+ 1 we get that minimizers
k k
2
have to lie in ℓ2(qk−1) = ℓk and this space contains non-sparse sequences such
as x = 2(−k2) or x = k−k.
k k
7
Stabilityandconvergence oftheregularizationmethodinthesenseg(x
α
−
x¯) 0 can be shown as in [16], where x¯ is a solution of the equation which
→
minimizes g.
As regards convergence rates, they are often formulated with respect to
the Bregman distance of the regularization functional.
Proposition 3.7. Let D : ℓqk ℓqk [0, ) be the Bregman distance with
× → ∞
respect to the ℓqk F-norm, where 1 < q 2, k N. Then the following in-
k
≤ ∀ ∈
equality holds, whenever x,y ℓqk satisfy x , y L for some positive
∈ k kqk k kqk ≤
number L:
1
D(x,y) (q 1)(x y )2.
k k k
≥ 6L − −
k
X
Proof. The Bregman distance D is given by
D(x,y) = x qk y qk q sign(q ) y qk−1(x y ).
k k k k k k k
| | −| | − | | −
k
X
For the estimate we use the inequality
p(p 1)(C +K)p−2
t p s p psign(p) s p(t s) − 1 (t s)2 (4)
| | −| | − | | − ≥ 2 −
which holds for s,t R, s C , t s K and 1 < p 2 (see [2, Lemma
1
∈ | | ≤ | − | ≤ ≤
12]).
Since x , y L we have x , y L1/qk and x y 2L1/qk.
k kqk k kqk ≤ | k| | k| ≤ | k − k| ≤
Inequality (4) gives
D(x,y) = x qk y qk q sign(q ) y qk−1(x y )
k k k k k k k
| | −| | − | | −
k
X
q (q 1)(3L1/qk)qk−2
k k − (x y )2
k k
≥ 2 −
k
X
1
(q 1)(x y )2
k k k
≥ 6L − −
k
X
where in the last inequality we used q > 1 which implies 3qk−2 1/3 and
k
≥
L(qk−2)/qk 1/L.
≥
In other words, the Bregman distace w.r.t. on a bounded ball is
k·kqk
bounded from below by the squared norm in ℓ2 with weight w = q 1.
w k k −
Weobtainthefollowingerrorestimateforregularizationwith penalty.
k·kqk
8
Theorem 3.8. Let A : ℓ2 Y be linear and bounded, x† ℓ2 and yδ Y
→ ∈ ∈
such that Ax† yδ δ and let q > 1, q 1. Further let
− y ≤ k k →
(cid:13) (cid:13)
(cid:13) (cid:13)xδ = argmin 1 Ax yδ 2 +α x
α 2 − Y k kqk
x
(cid:13) (cid:13)
and assume that x† , xδ (cid:13)L for som(cid:13)e L 0. If x† further fulfills the
qk α qk ≤ ≥
source condition, i.e. there existsw Y suchthat A∗w = (q x qk−1sign(x )) ,
(cid:13) (cid:13) (cid:13) (cid:13) k k k k
(cid:13) (cid:13) (cid:13) (cid:13) ∈ | |
then it holds that
6L δ
xδ x† +√α w .
α − ℓ2(qk−1) ≤ √2 √α k kY
(cid:16) (cid:17)
(cid:13) (cid:13)
Proof. Since by P(cid:13)ropositio(cid:13)n 3.1 (iv) one has that q x qk−1sign(x ) is the
k k k
| |
Gaˆteaux derivative of x , it is also a subgradient, and hence the condi-
k kqk
tion A∗w = (q x qk−1sign(x )) is the standard source condition (cf. [7]),
k k k k
| |
implying
1 δ 2
D(xδ,x†) +√α w .
α ≤ 2 √α k kY
(cid:16) (cid:17)
Together with Proposition 3.7, the result follows.
Note that the additional boundedness assumption xδ L is not a
α qk ≤
severe restriction, since by minimality of xδ we conclude that
α (cid:13) (cid:13)
(cid:13) (cid:13)
δ2 δ2
xδ + x† +L.
α qk ≤ 2α qk ≤ 2α
(cid:13) (cid:13) (cid:13) (cid:13)
Another point here i(cid:13)s, th(cid:13)at the sour(cid:13)ce (cid:13)condition does not need that x† is
sparse. The source condition is fulfilled, independently of the unknown true
solution, as soon as the range of A∗ contains ℓqk−1. However, the previous
results on convergence rates from [6] for ℓ1-regularization without sparsity of
x† do not apply here and it appears that the techniques employed there are
not directly applicable in this case.
4 Averaging functionals - the sublinear pow-
ers case
Now we turn to the case of sublinear powers, i.e. we consider
∞
ℓpk = x = x : x := x pk < , (5)
{ { k} k kpk | k| ∞}
k=1
X
9
with
0 < p 1.
k
≤
We outline below several properties of the F-norm defined at (5).
Proposition 4.1. The following statement holds true:
ℓpk ℓp
p>0
⊂ ∩
and the inclusion is strict.
Proof. Let x ℓpk. From x pk < + and p > 0, it follows that
k k
∈ | | ∞
x pk 0 as k and, as a consequence, x 0. Therefore, there exists
k k
k| | N→such tha→t x∞ 1 forPall k k . →
0 k 0
∈ | | ≤ ≥
Consider an arbitrary p > 0. Without loss of generality, we can say that
p p, for all k k . This gives x pk x p, k k , which implies that
k 0 k k 0
≤ ≥ | | ≥ | | ∀ ≥
x ℓp.
∈
The strict inclusion can be proven similarly as in Proposition 3.1.
One can show, by using a technique similar to the one in [28], that
Tikhonov regularization with the penalty is well-defined and conver-
k·kpk
gent. Denote by h : ℓ2 [0, ] the function defined by
→ ∞
x if x ℓpk
h(x) = k kpk ∈ (6)
( otherwise.
∞
Notethataninequality similar totheoneinProposition3.1(v) canbeshown
also for in case p = inf p > 0:
k·kpk k k
1
x M x max 1,Mp . (7)
k kpk ≤ ⇒ k k1 ≤ { }
This yields coercivity of the function h, which will be needed for proving the
following well-posedness statement.
Proposition 4.2. Let A : ℓ2 Y be a linear continuous operator which
→
is also weak∗-weak sequentially continuous, Y a Hilbert space and h : ℓ2
→
[0, ] defined by formula (6), where the sequence p satisfies assertion (*)
k
∞ { }
of Theorem 3.2 and inf p > 0. Then for any α > 0, the Tikhonov functional
k k
1
min Ax y 2 +αh(x) (8)
x∈ℓ1 2k − k
(cid:26) (cid:27)
has at least one minimizer x .
α
10