General inertial smoothing proximal gradient algorithm for the relaxation of matrix rank minimization problem

In the recent years, much work has been dedicated to the matrix rank minimization problem which emerge in many applications, especially in computer vision Zheng_Y and matrix completion Recht . Lots of models, methods and its variants have been studied, one can refer to the literaturesMa_S ; Cai_J ; Mesbahi ; Lai_M ; Fornasier_M ; Ma_T ; Ji_S ; Lu_Z ; He_Y ; Zhao_Q , in detail, the matrix rank minimization problem can be represented as

where $\mathrm{rank}(X)$ denotes the number of nonzero singular values.

In this paper, we concentrate on the exact continuous relaxation of matrix rank minimization problem proposed in Yu_Q_Zhang_X , that is the following nonconvex relaxation problem,

where $m\geq n$ and $f:\mathbb{R}^{m\times n}\to[0,\infty)$ is convex and not necessairly smooth, $\lambda$ is a positive parameter. $\Phi(X)=\sum_{i=1}^{n}\phi(\sigma_{i}(X))$ is an exact continuous relaxation of the matrix rank minimization problem. The capped-$\ell_{1}$ function $\phi$ with given $v>0$ is

It is observed that $\phi$ in (3) can be seen as a DC function, that is

with $\theta_{1}(t)=0,\,\theta_{2}(t)=t/v-1(t\geq 0)$ and $\mathcal{D}(t)=\{i\in\{1,2\}:\theta_{i}(t)=\max\{\theta_{1}(t),\theta_{2}(t)\}\}$. When the matrix $X$ is to be diagonal, the problem (2) is reduced to the exact continuous relaxation model of $\ell_{0}$ regularization problem which was given in Bian_W_Chen_X , i.e.,

where $\Phi(x)=\sum_{i=1}^{n}\phi(x_{i})$ and $\phi$ is defined as $\phi(t)=\mathop{\min\left\{1,|t|/v\right\}}$ for $t\in\mathbb{R}$. In Bian_W_Chen_X , one proposed the efficient smoothing proximal gradient method to solve this kind of problem with box constraint.

In Zhang_J , the authors established the smoothing proximal gradient method with extrapolation (SPGE) algorithm for solving (5). The extrapolation coefficient can be obtained $\sup\beta_{k}=1$. Further, under more strict condition of the extrapolation coefficient which includes the extrapolation coefficient in sFISTA with the fixed restart ODonoghue , it is proved that any accumulation point of the sequence generated by SPGE is a lifted stationary point of (5). Besides, the convergence rate based on the proximal residual is developed. Since the penalty item of problem (2) is nonconvex for a fixed $d$ in (12), the framework of SPGE algorithm in Zhang_J can not be directly applied to the matrix case.

As we know, incorporating inertial item which is also called extrapolation item to the proximal gradient method is a popular technique to improve the efficiency of proximal gradient algorithm for solving the following composition optimization problem,

where $f\colon\mathbb{R}^{n}\to\mathbb{R}$ is a smooth with Lipschitz continuous gradient and possibly nonconvex function, $g\colon\mathbb{R}^{n}\to(-\infty,+\infty]$ is a proper closed, and convex function. The composition structure that one item is smooth convex and the other is convex makes the proximal gradient method Parikh_N_Boyd_S widely used. Specifically, the updating scheme can be read as:

where $t_{k}$ is the stepsize. Throughout this paper, the proximal mapping Parikh_N_Boyd_S of $\lambda g$ is defined as

where $\lambda>0$ and $u\in\mathbb{R}^{n}$. The accelerated proximal gradient method for convex optimization problems have been well studied Nesterov_Y0 ; Nesterov_Y1 . In particular, Beck and Teboulle Beck_A_Teboulle_M established the remarkable Fast Iterative Shrinkage-Thresholding Algorithm (FISTA) for the convex case which was based on the Nesterov’s method Nesterov_Y1 ; Nesterov_Y . Some variants of FISTA have been developed by choosing appropriate extrapolation parameters, we refer the readers to Chambolle_A_Dossal_C ; Liang_J_and_C_B and their references therein. In Wu_Z_Li_M , under the proper parameter constraints, Wu and Li established the proximal gradient algorithm with different extrapolation (PGe) coeifficients on the proximal step and gradient step for solving the problem (6) when the smooth component is nonconvex and the nomsooth component is convex. The general framework is given as follows:

where $\alpha_{k},\beta_{k}$ are the extrapolation coefficients, and $\lambda_{k}$ is the stepsize. In Wu_zhongming , the authors further developed the inertial Bregman proximal gradient method for minimizing the sum of two possible nonconvex functions. In their method, the generalized Bregman distance replaces the Euclidean distance and two different inertial items on the proximal step and gradient step are taken respectively.

For matrix case, Toh and Yun Toh_K_C_Yun_S proposed the accelerated proximal gradient algorithm for the following convex problem:

where $f$ is a convex and smooth function with Lipschitz continuous gradient and $g$ is a proper closed convex function. The problem arises in applications such as in matrix completion problems Toh_K_C_Yun_S , multi-task learning A_Argyriou and principal component analysis (PCA) Mesbahi ; Xu_H . When the matrix is diagonal, the problem is reduced to the vector optimization problem (6).

Since optimization problem (2) is a matrix generalization of the vector optimization problem (5), it is natural for us to explore the possibility of extending some of the algorithms developed for problem (5) to solve problem (2). We observe that the loss function in above require a restrictive assumption that the function $f$ is smooth, it is difficult to apply the proximal gradient method directly for problem (2). Fortunately, the smoothing approxitation methods proposed in Chen_X can overcome the dilemma. Smoothing approximation method is an efficient method which is widely used in other problems. One can refer to Zhang_C_Chen_X ; Wu_F_Bian_W_Xue_X and their references therein. Especially, in Wei_B , the author proposed the smoothing fast iterative shrinkage thresholding algorithm (sFISTA) for the convex problem with the following extrapolation coefficient under the vector case, i.e.,

where $t_{0}=1$ and $\mu_{0}\in(0,1)$. They gave the global convergence rate $O(lnk/k)$ on the objective function.

Motivated by the smoothing method and the general accelerated techinique in (7), we propose a general inertial smoothing proximal gradient method (GIMSPG) for matrix case. Thanks to the special structure of penalty item of $(\ref{1.2})$, though the subproblem is nonconvex in the GIMSPG algorithm, it has a closed form solution. Most recently, Li and Bian li_wenjing studied a class of sparse group $\ell_{0}$ regularized optimization problem and gave an exact continuous relaxation model of it. They proposed the difference of convex(DC) algorithms for solving the relaxation model which has the DC structure. Moreover, they discussed the zero elements of the accumulation point has finite iterations. Inspired by this, under certain conditions of the parameters, we prove that the zero singular values of any accumulation point of the iterates generated by the GIMSPG algorithm have the same locations and the nonzero singular values are always not less than a fixed value. Besides, the GIMSPG algorithm has the ability identifying the zero singular values of any accumulation point within finite iterations. Furthermore, we show that any accumulation point of the sequence generated by GIMSPG algorithm is a lifted stationary point of the continuous relaxation problem (2).

The outline of this paper is as follows. Some preliminaries are presented in Section 2. Then we establish the GIMSPG algorithm framework and discuss its convergence in Section 3. In Section 4, we conduct numerical experiments to illustrate the efficiency of the proposed GIMSPG algorithm. Finally, we draw some conclusions in Section 5.

Notations. Let $\mathbb{R}^{m\times n}$ be the matrix space of $m\times n$ with the standard inner product, i.e., for any $X,Y\in\mathbb{R}^{m\times n}(m\geq n)$, $\langle X,Y\rangle={\rm{tr}}(X^{T}Y)$, where ${\rm{tr}}(X^{T}Y)$ is the trace of the matrix $X^{T}Y$. For any matrix $X\in\mathbb{R}^{m\times n}$, the Frobenius norm and nuclear norm are respectively denoted by $\|X\|{\|X\|}_{F}=\sqrt{{\rm{tr}}(X^{T}X)}$, $\|X\|_{*}=\sum_{i=1}^{n}\sigma_{i}(X)$ where $\sigma(X):=(\sigma_{1}(X),\ldots,\sigma_{n}(X))^{T}$ is the singular values of $X$ with $\sigma_{1}(X)\geq\sigma_{2}(X)\geq,\ldots,\geq\sigma_{n}(X)\geq 0$. Denote $\mathcal{A}(\sigma(X))=\left\{i\in\left\{1,2,\ldots,n\right\}:\sigma_{i}(X)\neq 0\right\}$, $\|X\|_{1}=\|vec(X)\|_{1}$, where $vec(X)$ is the vectorization operation of a matrix $X$. Denote $\partial f(X)$ be a Clarke subgradient of $f$ at $X\in\mathbb{R}^{m\times n}$ where $f(X):\mathbb{R}^{m\times n}\to\mathbb{R}$ is a locally Lipschitz continuous function. $\mathscr{D}(x)$ is a diagonal matrix generated by vector $x$. $E_{i}=\mathscr{D}(e_{i})$ where $e_{i}$ is the unit vector and the $i$-th element is 1. Denote $\mathbb{Q}^{n}$ be the set of $n\times n$ dimension unitary orthogonal matrix and $\mathbb{S}^{n}$ be the real and symmertric $n\times n$ matrices. For vector $x\in\mathbb{R}^{n}$, $\|x\|_{1}=\sum_{i=1}^{n}|x_{i}|$. Denote $\mathbb{N}=\left\{0,1,2,\ldots\right\}$, $\mathbb{D}^{n}=\left\{d\in\mathbb{R}^{n}:d_{i}\in\left\{1,2\right\},i=1,2,\ldots,n\right\}$.

In this section, we first recall some definitions and primary results which are the basis for the rest of the discussion. First, we give the form of the Clarke subdifferential of $\Phi$ at $X\in\mathbb{R}^{m\times n}$ followed by A_S_Lewis , that is

where $\mathcal{M}(X)=\{(U,V)\in\mathbb{Q}^{m\times m}\times\mathbb{Q}^{n\times n}:U^{T}U\!=\!V^{T}V\!=\!I,X\!=\!U\mathscr{D}(\sigma(X))V^{T}\}$.

In order to overcome the nondifferentiability of the convex loss function $f$ in (2), we use a smoothing function defined as in Yu_Q_Zhang_X to approximate convex function $f$.

Denote

It can be easily got that

and $\Phi(\bar{X})=\Phi^{d^{\bar{X}}}(\bar{X})$ with fixed $\bar{X}$ where $d^{\bar{X}}$ is defined in (11).

For the discussion convenience, we give the notations in the following way

where $\tilde{f}$ is a smoothing function of $f$, $\mu>0$, and $d\in\mathbb{D}^{n}$. By the formulation of (13), we have

In the following, we focus on the following nonconvex optimization problem with the given smoothing parameter $\mu>0$, and $d\in\mathbb{D}^{n}$:

When the matrix $X$ is the diagnoal matrix, since the penalty item of (15) is piecewise linearity function, the proximal operator of $\tau\Phi^{d}$ on $\mathbb{R}^{n}$ has a closed form solution with the given vectors $d\in\mathbb{D}^{n},w\in\mathbb{R}^{n}$, and a positive number $\tau>0$, in detail, the following proximal operator

can be calculated by $\hat{x}_{i}=\mathop{\max}\left\{\hat{w}_{i}-\frac{\tau}{v},0\right\}$, $\forall i=1,2,\ldots,n,$ where

Motivated by the smoothing approximation technique, the proximal gradient algorithm and the efficiency of inertial method, we consider the following general inetrial smoothing proxiaml gradient method of $\mathcal{\widetilde{F}}^{d^{k}}(\cdot,\mu_{k})$, i.e.,

where $\alpha_{k},\beta_{k}$ are different extrapolation parameters and certain conditions are required for them, $h_{k}$ is the parameter depending on $\tilde{L}$. Especially, when $\alpha_{k}=\beta_{k}=0$, the GIMSPG algorithm is reduced to the MSPG algorithm. Based on (16) and Lemma 3, we get that the problem $\mathop{\min Q_{d^{k}}(X,Y^{k},Z^{k},\mu_{k})}$ has a minimizer $\hat{X}=U\mathscr{D}(\hat{x})V^{T}$ with $\hat{x}={\rm{prox}}_{\tau\Phi^{d}}(w^{k})$, $\tau=\lambda{h^{-1}_{k}}\mu_{k}$ and $W^{k}=U\mathscr{D}({w^{k}})V^{T},$ $W^{k}=Y^{k}-{h^{-1}_{k}}\mu_{k}\nabla\tilde{f}(Z^{k},\mu_{k})$.

Next, we are ready to dicuss the convergence by constructing the auxiliary sequence. For any $k\in\mathbb{N}$, define

where $\{X^{k}\}$, $\{\mu_{k}\}$ are the sequences generated by GIMSPG algorithm and $\delta_{k+1}=\frac{h_{k+1}\alpha_{k+1}}{2}$. It needs to mention that the sequence is essential to the convergence analysis of the GIMSPG algorithm. Now, we start by showing that GIMSPG algorithm is well defined.

Then we are intend to prove that the auxiliary sequence $H_{\delta_{k+1}}(X^{k+1},X^{k},\mu_{k+1},\mu_{k})$ is monotonically decreasing.

Letting $d^{k}=d^{X^{k}}$ and according to $\mathcal{\widetilde{F}}^{d^{k}}(X^{k+1},\mu_{k})\geq\mathcal{\widetilde{F}}(X^{k+1},\mu_{k})$, we have

By the Definition 2(iv), we easily have that

Combining (3), (30) with the nonincreasing of $h_{k}$, $\alpha_{k}$, we get

According to the parameters constraint in Assumption 4, we easily have that $(\tilde{L}-h_{k}+h_{k}\alpha_{k}-\beta_{k}\tilde{L})\mu^{-1}_{k}+\alpha_{k}h_{k}\mu^{-1}_{k+1}<-h_{k}\mu^{-1}_{k+1}\varepsilon$, $\beta^{2}_{k}-\beta_{k}\leq 0$. Hence, the inequality (23) holds. So the desired results are obtained.