Nonlocal Adaptive Direction-Guided Structure Tensor Total Variation For Image Recovery ^††thanks: This work was supported by The Scientific and Technological Research Council of Turkey (TUBITAK) under 115R285.

The general inverse imaging problems seek the recovery of the underlying image $\textbf{f}\in\mathbb{R}^{NC}$ (assuming that each $N$-pixel channel is vectorized and all these $C$ channels are stacked together into a single vector) from an observation model of the form $\textbf{g}=\textbf{H}\textbf{f}+\bm{\eta}$, where $\bm{\eta}$ is the noise and H is a known linear operator. The type of the recovery problem is determined by the almost always ill-conditioned H, which mainly corresponds to the impulse response of the imaging device. If $\textbf{H}=\textbf{I}$, then the problem is denoising. Besides, it may be a matrix that functions as a mask (i.e., inpainting), or a circulant matrix with all shifts of a blurring kernel (i.e., deblurring). It may also be a composite operator, i.e., $\textbf{H}=\textbf{SF}$, where F blurs and S down-samples (i.e., single-image super-resolution problem), or F applies the Fourier transform and S works as a mask that retains only a subset of the Fourier coefficients (i.e., reconstruction from sparse Fourier measurements).

The variational approach to such recovery tasks aims at minimizing an energy function of the form:

where the former term is known as data fidelity and the latter one is the regularization term that encodes a ‘prior’. The parameter $\tau$ is the regularization parameter that balances the contribution of each term to the cost. The regularization criteria has a critical role in model-driven recovery, thus its proper selection has been immensely studied. One old (but still functional with its extended versions) regularizer is total variation (TV) by Rudin et al (1992), which favors piecewise-constant solutions. For years, TV has been extended in many aspects. Some researchers brought anisotropicity to better handle angled boundaries Grasmair and Lenzen (2010); Lou et al (2015); Bayram and Kamasak (2012); Lefkimmiatis et al (2015), some designed higher-order functionals to overcome staircase artifacts by favoring piecewise-smoothness Chan et al (2000); Bredies et al (2010); Papafitsoros and Schönlieb (2014), and some others employed nonlocal priors to benefit inherent nonlocal self similarity (NSS) as well Gilboa and Osher (2008); Lefkimmiatis and Osher (2015). Structure tensor total variation (STV) Lefkimmiatis et al (2015) is one such TV variant, which redesigns TV in semi-local fashion. Rather than penalizing the gradient, it penalizes a summary of the gradients within a local neighborhood. This neighborhood-awareness provides a better representation of the local image variations, and significantly boosts the performance. Recently in Demircan-Tureyen and Kamasak (2019, 2020), we have shown that penalizing the summary of the directional gradients within a patch instead of the gradients increases the performance as long as the local directions are accurately estimated. In this paper, we design a nonlocal counterpart to our prior direction-guided regularizer, under the influence of the nonlocal STV (NLSTV Lefkimmiatis and Osher (2015)), so that the long-distance dependencies across the image can also be modelled. We also design an algorithm that employs anisotropic Gaussian kernels and nonlocal structure tensors to estimate the directional parameters that are needed by the proposed regularizer.

In Lefkimmiatis and Osher (2015), the nonlocal counterpart of the STV was proposed. The extension has leveraged the STV term, which had already robustly encoded the local structural variation in semi-local fashion, by widening its scope to involve the nonlocal variations as well. This idea takes its source from the fact that a local patch often has similar nonlocal patches accross the image. This NSS prior was first revealed in the form of nonlocal means (NLM) filtering Buades et al (2011), which approximates the underlying intensity value at a spatial coordinate $i$ by taking the weighted average of the patches similar to the one around $i$th pixel. In Gilboa and Osher (2008) and Elmoataz et al (2008), the idea behind NLM was adopted to variational approach to enrich TV based models. They treated images as a graphs, which allowed them to define nonlocal gradients and the interaction of two distant image points became possible. The nonlocal TV (NLTV) term involves the following nonlocal gradient operator:

$\forall{j}\in\{1,2,\cdots,N\}$, where $w(i,j)$ is a function that assigns weights between points by considering the pairwise similarities of the intensities and the relative distances. Therefore, the NLTV prior is defined as follows:

The rational behind NLTV is to penalize the weighted averages of the gradient magnitudes gathered from the similar patches under the assumption that the gradients of the similar patches are also similar. The NLTV regularization term is designed only for the scalar-valued images, i.e., $C=1$. If we turn back to NLSTV Lefkimmiatis and Osher (2015), pursuing the same idea with NLTV, it involves a nonlocal structure tensor operator primarily defined as

where $\mathcal{N}[i]=\{j:w(i,j)>0\}$ and $J$ is the Jacobian operator. It was later reformulated as $(S_{w}\textbf{f})[i]=(J_{w}\textbf{f})[i](J_{w}\textbf{f})[i]^{T}$ to be able to decompose the NLSTV into linear functionals (See Lefkimmiatis et al (2015); Lefkimmiatis and Osher (2015)). Here, the operator $J_{w}:\mathbb{R}^{NC}\mapsto\mathbb{R}^{N\times 2\times NC}$ is nonlocal Jacobian of the form:

where each superscript referring to a certain channel and $D_{w}:\mathbb{R}^{N}\mapsto\mathbb{R}^{N\times 2\times N}$ is nonlocal gradient with the definition:

This definition differs from Eq. (2) in the sense that it can also exploit the underlying structure as opposed to the graph-based approach. Having defined the nonlocal Jacobian, the NLSTV is formulated as

where $\mathcal{S}_{p}$ is Schatten norm of order $p$, i.e., $\|\textbf{X}\|_{\mathcal{S}_{p}}=\|\sigma(\textbf{X})\|_{p}$ for a generic matrix X and $\sigma(\cdot)$ returning the singular values Bhatia (2013).

Our recovery algorithm involves two stages: (1) the stage that estimates the directional parameters required by the subsequent stage, (2) the stage that solves the inverse problem at hand by employing the proposed regularizer. The following subsection starts with the explanation of the proposed regularizer. Since we used the term adaptive direction-guided STV (ADSTV) in Demircan-Tureyen and Kamasak (2020) to refer the extended version of DSTV Demircan-Tureyen and Kamasak (2019), which gained the ability of handling multi-directional images, the regularization term that we propose here will be called as nonlocal ADSTV (NLADSTV).

We consider the problems of image denoising and image deblurring by confronting four related methods: STV, ADSTV, NLSTV, and NLADSTV. The experiments are conducted on the vector-valued images shown in Fig. 2. The images are taken from the popular Berkeley Segmentation Dataset (BSD) Arbelaez et al (2010). For the quantitative evaluation, we use the peak signal-to-noise ratio (PSNR). The source codes of STV¹¹1https://github.com/cig-skoltech/Structure_Tensor_Total_Variation and NLSTV²²2https://github.com/cig-skoltech/NLSTV that we use were made publicly available by the authors on GitHub. Our NLADSTV is implemented on top of NLSTV.

Since the algorithms under consideration are all aiming to minimize Eq. (1), they need regularization parameter $\tau$ to be tuned. We fine-tuned $\tau$ such that it leads to the best PSNR in all experiments. For both STV and ADSTV, the Gaussian kernel of support $3\times 3$ pixel with the standard deviation $\sigma=0.5$ is used. For NLSTV and NLADSTV, on the other hand, the sizes of the neighborhood (used for the nonlocal weights) and the search window are set to $7\times 7$ and $11\times 11$, respectively. Also, the number of the similar patches $L=9$, as suggested in Lefkimmiatis and Osher (2015). For the sake of fairness, we fixed the free parameter $\alpha^{+}$ of ADSTV and NLADSTV as 4, but it is possible to increase the restoration quality of these methods by fine-tuning $\alpha^{+}$. When it comes to the setting of the DPE procedure, the Gaussian kernel used for the computation of the local structure tensor has $9\times 9$ pixel support and the standard deviation of $1.5$, while the size of the neighborhood is $5\times 5$, the size of the search window is $11\times 11$, and $L=13$. In addition to them, note that the order of the Schatten norm $p=1$ for all regularizers under consideration.

According to the experiments conducted, our approach seems to produce promising results. However, the effectiveness of the NLADSTV based regularization highly depends on the performance of the directional parameter estimation (DPE) procedure. In the case of Gaussian blur for instance, since our DPE procedure was able to estimate less reliable parameters, the PSNR gain was not as satisfying as it was in the other experiments. Fortunately, because the structure tensor is a positive semi-definite matrix, the linearity factor $\textbf{c}[i]$ also behaves like a coherence factor, which measures the reliability of the estimated orientations. Thus for the unreliable orientation estimations, $\textbf{c}[i]$ becomes small and the NLADSTV starts working like the NLSTV. Therefore, the restoration quality never falls below the one that we obtain by using NLSTV.

The NLADSTV comes up with additional computational demands. The DPE procedure takes around 10 sec. on average, on a computer equipped with Intel Core Processor i7-7500U (2.70-GHz) with 16 GB of memory. Besides, the convergence of the NLADSTV based problem requires more iterations than the NLSTV, which makes the entire framework run 2 times slower than the NLSTV based image recovery.

This paper proposes a competitive framework for variational image recovery. We establish a two-stage algorithm to advance the nonlocal structure tensor total variation (NLSTV) regularizer, which already exploits local structural regularity and nonlocal self-similarity properties. We further enrich these priors by encoding some spatially varying directional characteristics of the underlying image. In order to understand these characteristics, we design an algorithm that estimates the directional parameters required by our regularizer. The experiments prove that the proposed method achieves significant performance improvements over its local counterpart and the NLSTV.