Convergence Rate of Inertial Forward-Backward Splitting Algorithms Based on the Local Error Bound Condition

Let $H$ be a real Hilbert space. $f:H\to\mathbb{R}$ be a smooth convex function and continuously differentiable with $L_{f}$-Lipschitz continuous gradient, and $g:H\to\mathbb{R}\cup\left\{{+\infty}\right\}$ be a proper lower semi-continuous convex function. We also assume that the proximal operator of $\lambda g,$ i.e.,

can be easliy computed for all $\lambda>0.$ In this paper, we consider the following problem:

We assume that problem ($P$) is solvable, i.e., ${X}:=\arg\min F\neq\emptyset,$ and for ${x_{*}}\in{X}$ we set ${F_{*}}:=F\left({{x_{*}}}\right).$

In order to solve the problem ($P$), several algorithms have been proposed based on the use of the proximal operator due to the non differentiable part. One can consult Johnstone & Moulin (2017), Moudafi & Oliny (2003) and Villa & Salzo (2013) for a recent account on the proximal-based algorithms that play a central role in nonsmooth optimization. A typical optimization strategy for solving problem ($P$) is the Inertial Forward-Backward algorithm (IFB), which consists in applying iteratively at every point the non-expansive operator ${T_{\lambda}}:H\to H,$ defined as

In view of the composition of IFB, we can easily find that the inertial term ${\gamma_{k}}$ plays an important role for improving the speed of convergence of IFB. Based on Nesterov’s extrapolation technique (see, Nesterov, 2019), Beck and Teboulle proposed a “fast iterative shrinkage-thresholding algorithm” (FISTA) with $t_{1}=1$ and ${t_{k+1}}{\rm{=}}\frac{{{\rm{1+}}\sqrt{{\rm{1+4}}t_{k}^{2}}}}{2}$ for solving ($P$) (see, Beck & Teboulle, 2009). The remarkable properties of this algorithm are the computational simplicity and the significantly better global rate of convergence of the function values, that is $F\left({{x_{k}}}\right)-F\left({{x_{*}}}\right)=O\left({\frac{1}{{{k^{2}}}}}\right).$ Several variants of FISTA considered in works such as Apidopoulos & Aujol (2020), Chambolle & Dossal (2015), Calatroni & Chambolle (2019), Donghwan & Jeffrey (2018), Mridula & Shukla (2020), Su & Boyd (2016) and Tao & Boley (2016), the properties such as convergence of the iterates and rate of convergence of function values have also been studied.

Chambolle and Dossal (see, Chambolle & Dossal, 2015) pointed out that FISTA satisfies a better worst-case estimate, however, the convergence of the iterates is not known. They proposed a new ${t_{k}}=\frac{{k-1+a}}{a}\left({a>2}\right)$ to show that the iterates generated by the corresponding IFB, named “FISTA_CD”, converges weakly to the minimizer of $F$. Attouch and Peypouquet (see, Attouch & Peypouquet, 2016) further proved that the sequence of function values generated by FISTA_CD approximates the optimal value of the problem with a rate that is strictly faster than $O\left({\frac{1}{{{k^{2}}}}}\right),$ namely $F\left({{x_{k}}}\right)-F\left({{x_{*}}}\right)=o\left({\frac{1}{{{k^{2}}}}}\right).$ Apidopoulos et al. (see, Apidopoulos & Aujol, 2020) noticed that the basic idea of the choices of $t_{k}$ in Attouch & Cabot (2018), Beck & Teboulle (2009) and Chambolle & Dossal (2015) is the Nesterov’s rule: $t_{k}^{2}-t_{k+1}^{2}+{t_{k+1}}\geq 0,$ and they focused on the case that the Nesterov’s rule is not satisfied. They studied the ${\gamma_{k}}=\frac{n}{{n+b}}$ with ${0<b<3}$ and found that the exact estimate bound is: $F\left({{x_{k}}}\right)-F\left({{x_{*}}}\right)=O\left({\frac{1}{{{k^{\frac{{2b}}{3}}}}}}\right)$. Attouch and Peypouquet (see, Attouch & Cabot, 2018) considered various options of ${\gamma_{k}}$ to analyze the convergence rate of the function values and weak convergence of the iterates under the given assumptions. Further, they showed that the strong convergence of iterates can be satisfied for the special options of $f$. Wen, Chen and Pong (see, Wen & Chen, 2017) showed that for the nonsmooth convex minimization problem ($P$), under the local error bound condition (see, Tseng & Yun, 2009), the $R$-linear convergence of both the sequence $\left\{{{x_{k}}}\right\}$ and the corresponding sequence of objective values $\left\{{F\left({{x_{k}}}\right)}\right\}$ can be satisfied if ${\sup_{k}}{\gamma_{k}}<1;$ and they pointed out that the sequences $\left\{{{x_{k}}}\right\}$ and $\left\{{F\left({{x_{k}}}\right)}\right\}$ generated by FISTA with fixed restart or both fixed and adaptive restart schemes (see, O’Donoghue & Candès, 2015) are $R$-linearly convergent under the error bound condition. However, the local convergence rate of the iterates generated by FISTA for solving ($P$) is still unknown, even under the local error bound condition.

The local error bound condition, which estimates the distance from $x$ to $X^{*}$ by the norm of the proximal residual at $x,$ has been proved to be extremely useful in analyzing the convergence rates of a host of iterative methods for solving optimization problems (see, Zhou & So, 2017). Major contributions on developing and using error bound condition to derive convergence results of iterative algorithms have been developed in a series of papers (see, e.g. Hai, 2020; Luo & Tseng, 1992; Necoara & Nesterov, 2019; Tseng & Yun, 2009, 2010; Tseng, 2010; Zhou & So, 2017). Zhou and So (see, Zhou & So, 2017) established error bounds for minimizing the sum of a smooth convex function and a general closed proper convex function. Such a problem contains general constrained minimization problems and various regularized loss minimization formulations in machine learning, signal processing, and statistics. There are many choices of $f$ and $g$ satisfy the local error bound condition, including:

• •

  (Pang, 1987, Theorem 3.1) $f$ is strong convex, and $g$ is arbitrary.

• •

  (Luo & Tseng, 1992a, Theorem 2.3) $f$ is a quadratic function, and $g$ is a polyhedral function.

• •

  (Luo & Tseng, 1992, Theorem 2.1) $g$ is a polyhedral function and $f=h(Ax){\rm{+}}\left\langle{c,x}\right\rangle,$ where $A\in{R^{m\times n}},c\in{R^{n}}$ and $h$ is a continuous differentiable function with gradient Lipschitz continuous and strongly convex on any compact convex set. This covers the well-known LASSO.

• •

  (Luo & Tseng, 1993, Theorem 4.1) $g$ is a polyhedral function and $f\left(x\right)=\mathop{\max}\limits_{y\in Y}\left\{{{{\left({Ax}\right)}^{T}}y-h\left(y\right)}\right\}+{q^{T}}x,$ where $Y$ is a polyhedral set, $h$ is a strongly convex differentiable function with gradient Lipschitz continuous.

• •

  (Tseng, 2010, Theorem 2) $f$ takes the form $f\left(x\right)=h\left({Ax}\right),$ where $h$ is same as the above second item and $g$ is the grouped LASSO regularizer.

More examples satisfying the error bound condition can be referred to Tseng (2010), Zhou & So (2017), Tseng & Yun (2009), Pang (1987) and Luo & Tseng (1992).

It has been observed numerically that first-order methods for solving those specific structured instances of problem ($P$) converge at a much faster rate than that suggested by the theory in Tao & Boley (2016), Xiao & Zhang (2013) and Zhou & So (2017). A very powerful approach to analyze this phenomenon is the local error bound condition. Hence, the first point this work focuses is the improved convergence rate of IFBs with some special $t_{k}$ under the local error bound condition.

We also pay attention to the Nesterov’s rule: $t_{k}^{2}-t_{k+1}^{2}+{t_{k+1}}\geq 0.$ For the $t_{k}$ satisfies it, we can derive that ${t_{k+1}}-{t_{k}}<1$ and $\sum\limits_{k=1}^{+\infty}{\frac{1}{{{t_{k}}}}}$ is divergent, which will greatly limit the choice of $t_{k}.$ What we expect is whether we can find the more suitable $t_{k}$ and obtain the improved theoretical results if we replace the Nesterov’s rule by some new we proposed.

Contributions.

In this paper, based on the local error bound condition, we exploit an assumption condition for the important parameter $t_{k}$ in IFB, and prove the convergence results including convergence rate of function values and strong convergence of iterates generated by the corresponding IFB. The above mentioned assumption condition imposed on $t_{k}$ provides a theoretical basis for choosing a new $t_{k}$ in IFB to solving those problems satisfying the local error bound condition, like LASSO. We use a “comparison methods” to discuss six choices of $t_{k}$, which include the ones in original FISTA (see, Beck & Teboulle, 2009) and FISTA_CD (see, Chambolle & Dossal, 2015) and satisfy our assumption condition, and separately show the improved convergence rates of the function values and establish the sublinear convergence of the iterates generated by corresponding IFBs. We also establish the same convergence results for IFB with an adaptive modification (IFB_AdapM), which performs well in numerical experiments. It is remarkable that, under the local error bound condition, the strong convergence of the iterates generated by the original FISTA is established, the convergence rate of function values for FISTA_CD is improved to $o\left({\frac{1}{{{k^{2(a+1)}}}}}\right)$, and the IFB algorithms with some $t_{k}$ can achieve sublinear convergence rate $o\left({\frac{1}{{{k^{p}}}}}\right)$ for any positive integer $p>1$.

In this section, we derive a new assumption condition for the $t_{k}$ in IFB, and analyze the convergence results of the corresponding IFB under the local error bound condition.

We start by recalling a key result, which plays an important role in our theoretical analysis.

Next, we give a very weak assumption to show that the sequence $\left\{{F\left({{x_{k}}}\right)}\right\},$ which is generated by Algorithm 1 with $0\leq\gamma_{k}\leq 1$ for $k$ is large sufficiently, converges to $F\left({{x_{*}}}\right)$ independent on $t_{k}.$

Assumption $A_{0}:$ For any $\xi_{0}\geq{F^{*}},$ there exist $\epsilon_{0}>0$ and ${\tau_{0}}>0$ such that

whenever $\left\|{{T_{\frac{1}{{{L_{f}}}}}}\left(x\right)-x}\right\|<{\varepsilon_{0}}$ and $F\left(x\right)\leq\xi_{0}.$

Remark 2. Note that Assumption $A_{0}$ can be derived by the assumption that $F$ is boundedness of level sets.

The rest of this paper is based on the following assumption.

Assumption $A_{1}:$ (“Local error bound condition”, Tseng & Yun (2009)) For any $\xi\geq{F_{*}},$ there exist $\varepsilon>0$ and ${\bar{\tau}}>0$ such that

whenever $\left\|{{T_{\frac{1}{{{L_{f}}}}}}\left(x\right)-x}\right\|<\varepsilon$ and $F\left(x\right)\leq\xi.$

As mentioned in Section 1, the $t_{k}$ in FISTA accelerates convergence rate from $O\left({\frac{1}{k}}\right)$ to $O\left({\frac{1}{{{k^{2}}}}}\right)$ for the function values and $t_{k}$ in FISTA_CD improves the convergence rate to $o\left({\frac{1}{{{k^{2}}}}}\right).$ Other options for $t_{k}$ are considered in Attouch & Cabot (2018) and Apidopoulos & Aujol (2020). Hence, we see that $t_{k}$ is the crucial factor to guarantee the convergence of the iterates or to improve the rate of convergence for the function values. Apidopoulos et al. in Apidopoulos & Aujol (2020) points that if $t_{k}$ satisfies the Nesterov’s rule, then one can obtain a better convergence rate. However, we notice that the Nesterov’s rule will limit the choice of $t_{k}$ greatly. In the following, we present a new Assumption $A_{2}$ for $t_{k},$ which helps us to obtain some new options of $t_{k},$ and analyze the convergence of iterates and convergence rate of the function values for the Algorithm 1 with a class of abstract $t_{k}$ satisfied Assumption $A_{2}$ under the local error bound condition.

Assumption $A_{2}:$ There exists a positive constant $0<\sigma\leq 1$ such that $\mathop{\lim}\limits_{k\to\infty}k^{\sigma}\left({\frac{{{t_{k+1}}}}{{{t_{k}}}}-1}\right)=c,$ where $c>0.$

Remark 3. It follows that ${\gamma_{k}}\in\left]{0,1}\right[,$ for any $k$ is sufficiently large, ${\mathop{\lim}\limits_{k\to\infty}{t_{k}}=+\infty}$ and ${\mathop{\lim}\limits_{k\to\infty}\frac{{{t_{k+1}}}}{{{t_{k}}}}=1}$ from Assumptions $A_{2}$. (It is easy to verify that $t_{k}$ in FISTA and $t_{k}$ in FISTA_CD both satisfy the Assumption $A_{2}$ by choosing $\sigma=1$, also, we can see that there exist some $t_{k}$, which satisfy or do not satisfy Nesterov’s rule, satisfy Assumption $A_{2}$ (See Section 3))

Here, we introduce a new way, which we called “comparison method”, that considers a sequence $\left\{{{\alpha_{k}}}\right\}$ such that ${\alpha_{k}}=\frac{{{s_{k}}-1}}{{{s_{k+1}}}}\geq{\gamma_{k}},$ where $\left\{{{s_{k}}}\right\}$ is a nonnegative sequence, to estimate the bounds of objective function and the local variation of the iterates.

In the following, we show the improved convergence rates for the IFBs with six special $t_{k}$ satisfying the Assumption 2.

Case 1. ${t_{k}}={e^{{{\left({k-1}\right)}^{\alpha}}}},0<\alpha<1.$

Remark 4. Notice that $\forall p>1,\;{\left({k-1}\right)^{\alpha\left\lceil{{\textstyle{1\over\alpha}}-1}\right\rceil}}{e^{-{{\left({k-1}\right)}^{\alpha}}}}=o\left({\frac{1}{{{k^{p}}}}}\right),$ which means that the sublinear convergence rate of the IFB with the $t_{k}$ in Case 1 is faster than any order.

Case 2. ${t_{k}}=\frac{{{k^{r}}-1+a}}{a}\;\left({r>1},a>0\right).$

Remark 5. For the $t_{k}$ in Case 2, we show that the convergence rate of function values and iterates related to the value of $r$. The larger $r,$ the better convergence rate Algorithm 1 achieves.

Case 3. ${t_{k}}=\frac{{{k^{r}}-1+a}}{a}\;\left({r<1,a>0}\right).$