Continuous Relaxation of Discontinuous Shrinkage Operator: Proximal Inclusion and Conversion

Let ${\rm Id}:{\mathcal{H}}\rightarrow{\mathcal{H}}:x\mapsto x$ denote the identity operator on ${\mathcal{H}}$. An operator $T:{\mathcal{H}}\rightarrow{\mathcal{H}}$ is Lipschitz continuous with constant $\kappa>0$ (or $\kappa$-Lipschitz continuous for short) if

Let $2^{{\mathcal{H}}}$ denote the power set (the family of all subsets) of ${\mathcal{H}}$. An operator $\mathsf{T}:{\mathcal{H}}\rightarrow 2^{{\mathcal{H}}}$ is called a set-valued operator, where $\mathsf{T}(x)\subset{\mathcal{H}}$ for every $x\in{\mathcal{H}}$. Given a set-valued operator $\mathsf{T}$, a mapping $U:{\mathcal{H}}\rightarrow{\mathcal{H}}$ such that $U(x)\in\mathsf{T}(x)$ for every $x\in{\mathcal{H}}$ is called a selection of $\mathsf{T}$. The inverse of a set-valued operator $\mathsf{T}$ is defined by $\mathsf{T}^{-1}:{\mathcal{H}}\rightarrow 2^{{\mathcal{H}}}:y\mapsto\{x\in{\mathcal{H}}\mid y\in\mathsf{T}(x)\}$, which is again a set-valued operator in general.

An operator $\mathsf{T}:{\mathcal{H}}\rightarrow 2^{{\mathcal{H}}}$ is monotone if

where gra $\mathsf{T}:=\{(x,u)\in{\mathcal{H}}^{2}\mid u\in\mathsf{T}(x)\}$ is the graph of $\mathsf{T}$. The following fact is known [21, Proposition 20.10].

A monotone operator $\mathsf{T}$ is maximally monotone if no other monotone operator has its graph containing gra $\mathsf{T}$ properly.

A function $f:{\mathcal{H}}\rightarrow(-\infty,+\infty]:={\mathbb{R}}\cup\{+\infty\}$ is proper if the domain is nonempty; i.e., ${\rm dom}~{}f:=\{x\in{\mathcal{H}}\mid f(x)<+\infty\}\neq\varnothing$. A function $f$ is lower semicontinuous (l.s.c.) on ${\mathcal{H}}$ if the level set ${\rm lev}_{\leq a}f:=\left\{x\in{\mathcal{H}}:f(x)\leq a\right\}$ is closed for every $a\in{\mathbb{R}}$. A function $f:{\mathcal{H}}\rightarrow(-\infty,+\infty]$ is convex on ${\mathcal{H}}$ if $f(\alpha x+(1-\alpha)y)\leq\alpha f(x)+(1-\alpha)f(y)$ for every $(x,y,\alpha)\in{\rm dom}~{}f\times{\rm dom}~{}f\times[0,1]$. For $\eta\in(-\infty,+\infty]$, we say that $f$ is $\eta$-weakly convex if $f+(\eta/2)\left\|\cdot\right\|^{2}$ is convex. Here, $\eta<0$ (i.e., $f-(\left|\eta\right|/2)\left\|\cdot\right\|^{2}$ is convex) means that $f$ is $\left|\eta\right|$-strongly convex. Clearly, $\eta$-weak convexity of $f$ implies $\tilde{\eta}$-weak convexity of $f$ for an arbitrary $\tilde{\eta}\geq\eta$. When the “minimal” weak-convexity parameter is $\eta=+\infty$, $f+\alpha\left\|\cdot\right\|^{2}$ is nonconvex for every $\alpha\in{\mathbb{R}}$, i.e., $f$ is not weakly convex (for any parameter in ${\mathbb{R}}$). The set of all proper l.s.c. convex functions $f:{\mathcal{H}}\rightarrow(-\infty,+\infty]$ is denoted by $\Gamma_{0}({\mathcal{H}})$.

Given a proper function $f:{\mathcal{H}}\rightarrow(-\infty,+\infty]$, the Fenchel conjugate (a.k.a. the Legendre transform) of $f$ is $f^{*}:{\mathcal{H}}\rightarrow(-\infty,\infty]:u\mapsto\sup_{x\in{\mathcal{H}}}(\left\langle{x},{u}\right\rangle-f(x))$. The conjugate $f^{**}:=(f^{*})^{*}$ of $f^{*}$ is called the biconjugate of $f$. In general, $f^{*}$ is l.s.c. and convex [21, Proposition 13.13].²²2 It may happen that ${\rm dom}~{}f^{*}=\varnothing$. For instance, the conjugate of $f:{\mathbb{R}}\rightarrow{\mathbb{R}}:x\mapsto-x^{2}$ is $f^{*}(u)=+\infty$ for every $u\in{\mathbb{R}}$. If ${\rm dom}~{}f^{*}\neq\varnothing$, or equivalently if $f$ possesses a continuous affine minorant³³3 Function $f$ has a continuous affine minorant if there exist some $(a,b)\in{\mathcal{H}}\times{\mathbb{R}}$ such that $f(x)\geq\left\langle{a},{x}\right\rangle+b$ for every $x\in{\mathcal{H}}$., then $f^{*}$ is proper and $f^{**}=\breve{f}$; otherwise $f^{**}(x)=-\infty$ for every $x\in{\mathcal{H}}$ [21, Proposition 13.45]. Here, $\breve{f}$ is the l.s.c. convex envelope of $f$.⁴⁴4 The l.s.c. convex envelope $\breve{f}$ is the largest l.s.c. convex function $g$ such that $f(x)\geq g(x)$, $\forall x\in{\mathcal{H}}$.

Let $f:{\mathcal{H}}\rightarrow(-\infty,+\infty]$ be proper. Then, the set-valued operator $\partial f:{\mathcal{H}}\rightarrow 2^{\mathcal{H}}$ such that

is the subdifferential of $f$ [21].

We present a slight extension of the previous result [13, Lemma 1] below.

As a particular instance, if $f+(\eta/2)\left\|\cdot\right\|^{2}\in{\Gamma_{0}(\mathcal{H})}$ for some constant $\eta\in(-\infty,\gamma^{-1})$, existence and uniqueness of minimizer is automatically ensured. In the convex case of $f\in{\Gamma_{0}(\mathcal{H})}$, ${{\rm s\mbox{-}Prox}}_{\gamma f}$ reduces to the classical Moreau’s proximity operator [22].

Fact 5 presented above gives an interplay between $\eta$-weakly convex functions for $\eta\in[0,1)$ and the $(1-\eta)^{-1}$-Lipschitz continuous gradients of smooth convex functions, stemming essentially from the duality between strongly convex functions and smooth convex functions. Now, the question is the following: is there any such relation in the case of $\eta\in[1,+\infty)$?

While the proximity operator ${\rm s\mbox{-}Prox}_{\varphi}$ is (Lipschitz) continuous in the case of $\eta<1$, the case of $\eta\geq 1$ (more specifically, the case in which $\varphi+(\eta/2)\left\|\cdot\right\|^{2}\not\in\Gamma_{0}({\mathcal{H}})$ for any $\eta<1$) includes those functions of which the “set-valued” proximity operator contains a discontinuous shrinkage operator as its selection. The following proposition concerns the case of $\eta=1$, which will be linked to the case of $\eta>1$ later in Section 3-3.2.

Proposition 1 bridges the subdifferential of convex function and the proximity operator of 1-weakly convex function, indicating an interplay between maximality of monotone operators and 1-weak convexity of functions.

We start with the definition of the l.s.c. 1-weakly-convex envelope.

The first key result is presented below.

Theorem 1 implies that, if a discontinuous operator is a selection of $\mathbf{Prox}_{f}$ for a nonconvex function $f$, it can also be expressed as a selection of $\mathbf{Prox}_{\widetilde{f}}$ for a 1-weakly-convex function $\widetilde{f}$, which actually coincides with the l.s.c. 1-weakly-convex envelope of $f$. This fact indicates that, when seeking for a shrinkage operator as (a selection of) the proximity operator, one may restrict attention to the class of weakly convex functions.

We remark that Theorem 1 is trivial if $f+(1/2)\left\|\cdot\right\|^{2}\in\Gamma_{0}({\mathcal{H}})$, because $\widetilde{f}=f$ in that case by Fact 2. Hence, our primary focus in Theorem 1 is on $\eta$-weakly convex functions for $\eta>1$, although Theorem 1 itself has no such a restriction. The following corollary is a direct consequence of Fact 5 ($\eta<1$), Proposition 1 ($\eta=1$), and Theorem 1 ($\eta>1$).

The second key result presented below gives a principled way of converting the set-valued proximity operator of 1-weakly convex function to a MoL-Grad denoiser.

An interplay between surjectivity of an operator $T$ and continuity (under weak convexity) of a function can be seen through a ‘lens’ of proximity operator.⁷⁷7 It is known that convexity of $f:{\mathcal{H}}\rightarrow{\mathbb{R}}$ implies continuity of $f$ when ${\mathcal{H}}$ is finite dimensional [21, Corollary 8.40].

The hard shrinkage operator with the threshold $\tau\in{\mathbb{R}}_{++}$ is defined by [4]

for which it holds that

where

The l.s.c. 1-weakly convex envelope $\widetilde{g}_{\tau}:=(g_{\tau}+(1/2)(\cdot)^{2})^{**}-(1/2)(\cdot)^{2}$ of $g_{\tau}$ is given by (see Fig. 1)

Here, $\varphi_{\tau_{2}}^{\rm MC}(x):=\begin{cases}\left|x\right|-\frac{1}{2\tau_{2}}x^{2},&\mbox{if }\left|x\right|\leq\tau_{2},\\ \frac{1}{2}\tau_{2},&\mbox{if }\left|x\right|>\tau_{2},\end{cases}$ is the MC penalty of the parameter $\tau_{2}\in{\mathbb{R}}_{++}$ [10, 11].

The proximity operator of $\widetilde{g}_{\tau}$ is given by

where $\overline{\rm conv}$ denotes the closed convex hull ($\overline{\rm conv}\{0,\tau\}=[0,\tau]$ and $\overline{\rm conv}\{0,-\tau\}=[-\tau,0]$). Comparing (12) and (15), it can be seen that

as consistent with Theorem 1. This implies that ${\rm hard}_{\tau}$ is also a selection of the proximity operator $\mathbf{Prox}_{\widetilde{g}_{\tau}}$ of the 1-weakly convex function $\widetilde{g}_{\tau}$, which is maximally monotone. Indeed, $\mathbf{Prox}_{\widetilde{g}_{\tau}}$ in (15) is the (unique) maximally monotone extension of ${\rm hard}_{\tau}$ [20] (cf. Remark 2).

We now invoke Theorem 2 to obtain the continuous operator

where the firm shrinkage operator [9] for the thresholds $\tau_{1}\in{\mathbb{R}}_{++}$, $\tau_{2}\in(\tau_{1},+\infty)$, is defined by ${\rm firm}_{\tau_{1},\tau_{2}}:={\rm s\mbox{-}Prox}_{\tau_{1}\varphi_{\tau_{2}}^{\rm MC}}:{\mathbb{R}}\rightarrow{\mathbb{R}}:x\mapsto 0~{}\mbox{if }\left|x\right|\leq\tau_{1};x\mapsto{\rm sign}(x)\frac{\tau_{2}(\left|x\right|-\tau_{1})}{\tau_{2}-\tau_{1}}~{}\mbox{if }\tau_{1}<\left|x\right|\leq\tau_{2};x\mapsto x~{}\mbox{if }\left|x\right|>\tau_{2}$. We remark that $\tau_{1}\varphi_{\tau_{2}}^{\rm MC}$ is $\tau_{1}/\tau_{2}$-weakly convex ($\widetilde{g}_{\tau}/(\delta+1)$ in (17) is $1/(\delta+1)$-weakly convex) and its “single-valued” proximity operator gives firm shrinkage, while $\widetilde{g}_{\tau}$ in (16) is 1-weakly convex and a selection of its “set-valued” proximity operator gives hard shrinkage. We also mention that the limit of the continuous operator ${\rm firm}_{\tau/(\delta+1),\tau}$ with respect to the relaxation parameter $\delta$ coincides with the discontinuous operator ${\rm hard}_{\tau}$; i.e., $\lim_{\delta\downarrow 0}{\rm firm}_{\tau/(\delta+1),\tau}(x)=\widetilde{\rm hard}_{\tau}(x)=x1(\left|x\right|\geq\tau)\in\mathbf{Prox}_{g_{\tau}}(x)$ for every $x\in{\mathbb{R}}$ (see Fig. 2).

Fig. 3(a)–(g) illustrates the process of obtaining a continuous relaxation ${\rm s\mbox{-}Prox}_{\frac{2}{3}\widetilde{g}_{1}}$ of the discontinuous operator hard₁, corresponding to the case of $\tau:=1$ and $\delta:=1/2$. Comparing the graphs of the discontinuous operator hard₁, $\mathbf{Prox}_{g_{1}}$, and $\mathbf{Prox}_{\widetilde{g}_{1}}(=:\mathsf{H})$, one can observe that (see Theorem 1 for the second inclusion)

The maximally monotone operator $\mathbf{Prox}_{\widetilde{g}_{1}}$ (see Remark 1) is then converted to ${\rm s\mbox{-}Prox}_{(2/3)\widetilde{g}_{1}}={\rm firm}_{2\tau/3,\tau}$ in a step-by-step manner using (17) (which is based on Theorem 2). Inspecting the figure under Corollary 1, Figs. 3(b) and 3(h) correspond to Corollary 1.2 (not maximally monotone) for $\eta:=+\infty$ and $\eta:=4$, respectively, and Figs. 3(c) and 3(g) correspond to Corollary 1.1 (maximally monotone) for $\eta:=1$ and $\eta:=2/3$, respectively.

Letting $\eta:=1/(\delta+1)$, (17) for $\delta\in{\mathbb{R}}_{++}$ concerns the case of $\eta\in(0,1)$. In the case of $\eta\in[1,+\infty)$, on the other hand, the proximity operator of the $\eta$-weakly convex function $\eta\widetilde{g}_{\tau}$ is set-valued. Actually, it is not difficult to verify that $\mathbf{Prox}_{\eta\widetilde{g}_{\tau}}=\mathbf{Prox}_{\eta g_{\tau}}$ for every $\eta\in(1,+\infty)$. Note here that this is not true for $\eta:=1$, i.e., $\mathbf{Prox}_{\widetilde{g}_{\tau}}\neq\mathbf{Prox}_{g_{\tau}}$, as can be seen from Figs. 3(b) and 3(c). See Fig. 3(h) for the case of $\eta:=4$.

We have seen that the discontinuous hard shrinkage operator is converted to the continuous firm shrinkage operator via the transformation from (i) to (vi) in Fig. 3(a). We mimic this procedure for another discontinuous operator.

We consider the Euclidean case ${\mathcal{H}}:={\mathbb{R}}^{N}$ for $N\geq 2$. Let ${\boldsymbol{w}}\in{\mathbb{R}}_{+}^{N}$ be the weight vector such that $0\leq w_{1}\leq w_{2}\leq\cdots\leq w_{N}$. Given ${\boldsymbol{x}}\in{\mathbb{R}}^{N}$, we define $\left|{\boldsymbol{x}}\right|\in{\mathbb{R}}_{+}^{N}$ of which the $i$th component is given by $\left|x_{i}\right|$. Let $\left|{\boldsymbol{x}}\right|_{\downarrow}\in{\mathbb{R}}_{+\downarrow}^{N}=\{{\boldsymbol{x}}\in{\mathbb{R}}_{+}^{N}\mid x_{1}\geq x_{2}\geq\cdots\geq x_{N}\}$ denote a sorted version of $\left|{\boldsymbol{x}}\right|$ in the nonincreasing order; i.e., $\big{[}\left|{\boldsymbol{x}}\right|_{\downarrow}\big{]}_{1}\geq\big{[}\left|{\boldsymbol{x}}\right|_{\downarrow}\big{]}_{2}\geq\cdots\geq\big{[}\left|{\boldsymbol{x}}\right|_{\downarrow}\big{]}_{N}$. The reversely ordered weighted $\ell_{1}$ (ROWL) penalty [14] is defined by $\Omega_{{\boldsymbol{w}}}({\boldsymbol{x}}):={\boldsymbol{w}}^{\top}\left|{\boldsymbol{x}}\right|_{\downarrow}$. The penalty $\Omega_{{\boldsymbol{w}}}$ is nonconvex and thus not a norm.⁸⁸8If $w_{1}\geq w_{2}\geq\cdots\geq w_{N}$, the function ${\boldsymbol{x}}\mapsto{\boldsymbol{w}}^{\top}\left|{\boldsymbol{x}}\right|_{\downarrow}$ is convex, and it is called the ordered weighted $\ell_{1}$ (OWL) norm [23].

In this case, the associated proximity operator will be discontinuous. This implies in light of Fact 5 that $\Omega_{{\boldsymbol{w}}}$ is not even weakly convex. To see this, let us consider the case of $N=2$, and let $(0\leq)w_{1}<w_{2}$. In this case, by symmetry, the proximity operator is given by $\mathbf{Prox}_{\Omega_{{\boldsymbol{w}}}}({\boldsymbol{x}})={\rm sgn}({\boldsymbol{x}})\odot\mathbf{Prox}_{\Omega_{{\boldsymbol{w}}}}(\left|{\boldsymbol{x}}\right|),~{}{\boldsymbol{x}}\in{\mathbb{R}}^{2}$, where ${\rm sgn}(\cdot)$ is the componentwise signum function, $\odot$ denotes the Hadamard (componentwise) product, and for ${\boldsymbol{x}}\in{\mathbb{R}}_{+}^{2}$

Here, ${\boldsymbol{w}}_{\downarrow}:=[w_{2},w_{1}]^{\top}$, and $(\cdot)_{+}:{\mathbb{R}}^{2}\rightarrow{\mathbb{R}}^{2}:[y_{1},y_{2}]^{\top}\mapsto[\max\{y_{1},0\},\max\{y_{2},0\}]^{\top}$ is the ‘ramp’ function. A selection of $\mathbf{Prox}_{\Omega_{{\boldsymbol{w}}}}$ will be referred to as ROWL shrinkage.