Image Restoration under Semantic Communications ^††thanks: This research is funded by Hanoi University of Science and Technology (HUST) under project number T2021-SAHEP-002.

Many researches in the literature have been carried out to find ways to apply semantic communication to enhance the transmission performance of these multimedia signals. In [8], the authors discussed the semantic communication principles, architecture, its application areas, and the approach to it with machine learning (ML) and information of things (IoTs). P. Jiang et al.[9] have proposed semantic-based video conferencing network, where the authors used the incremental redundancy hybrid automatic repeat request (IR-HARQ) feedback framework to guarantee the quality of the transmission according to signal-to-noise ratio (SNR) and channel state information (CSI). A deep learning enabled semantic communication systems (DeepSC-S) for speech signals transmission was proposed in [10], which learns and extracts speech signals at the transmitter and then recovering them at the receiver from the received features directly. Then, the authors compared its performance with the traditional approaches to verify the effectiveness of the system, especially in deeply faded environments. To the best of our knowledge, there is no related work considering using semantic information at the receiver to enhance the reconstruction quality and evaluating the system performance by the different evaluation metrics.

This paper studies an uplink massive multiple-input multiple-output (MIMO) semantic transmission system where a user transmits image data to the base station (BS). The sematic image data includes the spatial correlation between the pixels instead of the randomly non-structure signals. The data is then modulated and transmitted over the wireless environments. At the BS, the alternate direction method of multipliers (ADMM) is first used to reconstruct the transmitted image signals by minimizing the transmission error over the random fluctuations of fading channels. After that, we consider using a post-filter to improve the reconstruction quality from the output of the ADMM via mitigating residual errors based on image correlation structure [11]. This paper considers the traditional filters comprising a median filter, a Gaussian filter, and a block-matching and 3D filtering. Finally, the performance of image data recovery is assessed by three parameters commonly used in wireless communication (the physical layer) and multimedia (the application layer), namely, the bit error rate (BER), the peak signal-to-noise ratio (PSNR), and the structural similarity (SSIM) over different signal-to-noise ratio (SNR) values. Numerical results demonstrate the significant importance in exploiting the prior image information to enhance the reconstruction quality. Besides, the performance metric usually used in the physical layer may be contradicted to one in the application layer.

We consider an uplink of a Massive MIMO system where the BS is equipped with $M$ antennas communicates with a user having $K$ antennas. Let us suppose that this user selects the semantic content of the image data $\mathbf{I}$ and sends it to the BS. The image content will be simply stacked into $Z$ column vectors as follows $\mathbf{I}\rightarrow\{\mathbf{i}_{1},\cdots,\mathbf{i}_{Z}\}$. These column vectors are further modulated so that each vector is mapped to the constellation points of the alphabet set $\mathcal{O}$. Each value in the vector $\mathbf{i}_{z},\forall z=1,\ldots,Z,$ corresponds to one or several constellation points. This mapping process will be represented as $\mathbf{I}\overset{\mathcal{M}}{\mapsto}\mathbf{X}\Leftrightarrow\{i_{1},...,i_{N}\}\mapsto\{x_{1},...,x_{n}\}$, where $x_{z}\in\mathbb{C},\forall z,$ are the modulated symbols that will be transmitted over the MIMO channel and received at the BS; $\mathcal{M}$ indicates the mapping or the modulation. In particular, let us define $\mathbf{y}_{z}$ the received signal at the BS after propagating over the environment as

where $\textbf{H}\in\mathbb{C}^{M\times K}$ is the channel matrix between the user and the BS; and $\mathbf{n}_{k}\sim\mathcal{CN}(\mathbf{0},\sigma^{2}\mathbf{I}_{M})$ represents the additive white Gaussian noise (AWGN), where each element follows a circularly symmetric complex Gaussian variable with zero mean and variance $\sigma^{2}$. Here, $\mathcal{CN}(\cdot,\cdot)$ denotes the circular symmetric complex Gaussian noise and $\rho$ is the transmitted power allocated to each modulated data symbol. The signal-to-noise ratio (SNR) is be defined by $\mathrm{SNR}=\rho/\sigma^{2}$. Once the BS has received all the noisy signal $\{\mathbf{y}_{z}\}$, it will decode and exploit the inverse projection to reconstruct the image, denoted by $\tilde{\mathbf{I}}$ as $\{\mathbf{y}_{z}\}\overset{\mathcal{M}^{-1}}{\longmapsto}\mathbf{\tilde{I}},$ where $\mathcal{M}^{-1}$ represents the demodulation or the inverse mapping. Let us simply denote $\mathbf{\tilde{I}}=\mathcal{M}^{-1}(\{\mathbf{y}_{z}\})$. We emphasize that, from the signal received in (1) and the given instantaneous channels, the BS decodes the image data information by the recovery model presented hereafter.

The received measurement data at the BS contains noise, artifacts, and loss from the environment. The contamination may be extremely severe, especially at a low SNR regime. Our goal is to reconstruct the image $\tilde{\mathbf{I}}$ with the highest amount of desired information retained as the original semantic image data $\mathbf{I}$. Consequently, the optimization problem, which we would like to solve, is formulated as follows

where $\|\cdot\|_{F}$ is the Frobenius norm. The objective function of problem (2) is to recover the semantic image data that is close to the original one. The mathematical operation $\mathcal{R}(\cdot)$ in the first constraint reconstructs the entire image from the partial image information. This optimization problem is non-convex from the mathematical points of view. Besides, the received measurements are contaminated by the fading channels during transmission and noise at the receiver, so the useful information may be lost. In addition, problem (2) is only based on the features of the physical layer. In other words, it does not consider the image structure as a constraint, and therefore there is room to acquire a better the reconstruction quality than its locally optimal solution.

By utilizing the ADMM algorithm in [12, 13], the first stage can obtain a local solution to problem (2). This algorithm actually approximates the inherent nonconvexity of (2) to a convex optimization problem. From the received measurement data $\{\mathbf{y}_{z}\}$ and the channel matrix $\mathbf{H}$, the ADMM algorithm will iteratively decode $\tilde{x}_{z}$ which is the constellation point contained in the received data measurement. As wished, $\tilde{x}_{z}$ should be coincided with the modulated symbol $x_{z}$ sent by the user. From the set $\{\tilde{x}_{z}\}$, one can attain the decoded image information, which is reshaped and restored in $\tilde{\mathbf{I}}$. As presented in the previous section, the decoded image may still not good in terms of, for example the BER, PSNR, and SSIM due to the locality of the solution. Therefore, after the first stage, the decoded image will be further processed to improve the image quality by expecting that texture and fine details of the image will be better captured. It is motivated by exploiting the image structure from the fact that natural images are strongly correlated. The correlation is indeed deployed in the lowpass filters and we can borrow them in our applications.

In the second stage, the noise and residual errors are removed by a filtering process with an effective kernel, which may be classified into two types: pixel-based kernel, e.g., Gaussian filter and median filter, and patch-based kernel, e.g., block-matching and 3D (BM3D) filter. This section will look at the basis of these techniques.

When it comes to transmitting data from one point to another, the key metric to evaluate is how many errors will appear in the data that appears at the remote end, thus make the BER metric an important metric in measuring the performance of a communication system or a data channel. As the name implies, the BER is defined as the rate at which errors occur in a transmission system. The definition can be translated to the following formula

where $E$ and $B$ is the total number of error bits and the total number of bits that are transmitted, respectively. Based on BER, we can evaluate the quality of the transmission process as well as the effectiveness of using digital filters in restoring the original image. In a communication system, the receiver side BER may be affected by different transmission factors, for instance, channel noise, interference, attenuation, multipath fading, etc.

BER is an metric that are mainly used at the physical layer, so it only considers the accuracy of the transmitted data to the original data without considering the actual quality of the image and the effect of noise. Therefore, we need to look at other metrics used to evaluate image quality.

The PSNR is an expression for the ratio of the maximum possible value (power) of a signal to the power of distortion noise that affects the quality of the signal. Because many signals have a very wide dynamic range, (ratio between the largest and smallest possible values of a changeable quantity) the PSNR is usually expressed in terms of the logarithmic decibel scale.

Assume that the data we are considering is a 2D digital image signal. Let $f$ be the original image data matrix of size size $m\times n$, $g$ is the image data matrix to be compared and must have the same size $m\times n$ as the original. Mathematically, PSNR is defined as follows:

where MAX is the maximum pixel value, depending on the original image data. For example, if the input image has a double-precision floating-point data representation, then MAX is 1. If it has an 8-bit representation, MAX is 255. The remaining parameter is MSE, which means the Mean Squared Error of the two image and is calculated as follows:

MSE and the PSNR are both utilized to measure the quality of two images. The MSE represents the cumulative squared error between the compressed and the original image, whereas PSNR represents a measure of the peak error. The lower the value of MSE, the lower the error, and the higher the PSNR, the more similar restored image to the original image, thus means the better result acquired. The main limitation of this metric is that two degraded images with the same PSNR value could contain very different types of errors, some of which are easier to detect than others, resulting in different image quality.

Unlike the PSNR, the SSIM metric is often utilized to evaluate the similarity or similarity between the noisy signal and the original data. It was first proposed in [15] by Zhou Wang, based on the assumption that human visual perception is highly adaptive to copying structural information from an image. This assessment is designed by modeling any image data using three factors: luminance, contrast and structure [16].
To calculate the SSIM value of two image data f and g, first we must calculate the illuminance value of each image:

where $N$ is the number of pixels in the image $\mathbf{x}$, i.e., $\mathbf{x}=[x_{1},\cdots,x_{i},\cdots,x_{N}]^{T}$, thus making illuminance the average of all the pixels. After measuring the illuminance, we calculate the contrast which is the standard deviation of all pixels as follows

The SSIM formula is based on three comparison measurements between the input image: luminance, contrast and structure. However, from these two values, we can calculate the SSIM index based on the reduced formula as follows

in which $C_{1}=(K_{1}L)^{2}$ , $C_{2}=(K_{2}L)^{2}$ are the constants with L being the dynamic range of pixel values and the constant $K1,K2\ll 1$. The covariance between two images $f$ and $g$ (reshaped into vectors of the $N$ elements), denoted by $\sigma_{fg}$, is defined as

The resultant SSIM index is a decimal value between 0 and 1, and value 1 is only reachable in the case of two identical sets of data and therefore indicates perfect structural similarity. A value of 0 indicates no structural similarity. The difference with other techniques such as MSE or PSNR is that structural information considers pixels to have strong inter-dependencies with each other, especially when they are spatially close. These dependencies carry important information about the structure of the objects in the visual scene.

We observe the representation of reconstructed images with the SNR value of 10 [dB] in Fig.  2. The images in the first column are the original images or called the ground truth. The second column includes the decoded images after applying the ADMM algorithm. Meanwhile, the refined images result from deploying the Gaussian, median, and BM3D filters, respectively. We can observe that the decoded image by the ADMM algorithm still contains lots of noise and artifacts. This is because the ADMM algorithm considered in this paper decodes the image data based on the channel information only. By applying a low-pass filter to smooth out the decoded image, we observe much the better restoration quality. Noise and artifacts have been greatly reduced by using the Gaussian and median filters, but besides that, the edges and details are also lost. More advance, the BM3D filter offer the ability to preserve the edges and details of the image, but cannot completely remove the noise from the image, especially the noise in the Monarch image is still clearly visible. Median filters are very good at handling salt and pepper noise type due to their characteristics. Details are retained relatively well, comparable to the results of the BM3D filter, like the tripod in the Cameraman image. In the following, we investigate the system performances by different metrics as mentioned.

In Fig. 4, we visualize the simulation results of the average BER value of 12 natural images in the Set12 dataset [17]. It shows that the blue line corresponding to the case where the image is restored without a filter gives the lowest bit error rate and gradually decreases inversely with the SNR of the channel, means that when the channel’s state gets better, the BER would decrease. However, as the SNR increases, the filtered image’s BER stays very high and does not seem to improve, which leads to the conclusion, that the filters are not able to restore the recovered image to its original version. Instead, it will sacrifice image details to increase the quality, as can be seen in Fig. 2. This seems to be contradictory because in the previous section, we noticed that utilizing filters improve the image quality significantly compared to contaminated image after transmission and restoration using the ADMM algorithm. To clarify this disagreement, we need to understand that the purpose of ADMM algorithm is to solve the optimization problem and minimize the transmission error. Therefore, any subsequent action by the filter would increase the difference of data bits, resulting in a higher BER. This is the reason why the average BER value in the case of using the filter is not as good as the BER value in the case of not using the filter. It leads to the conclusion: For structured data such as digital image signals, BER is no longer a decisive criterion for evaluating the quality of the recovered data.

As mentioned above, the BER index in this case is not suitable for assessing the quality of the recovered image. Therefore, we need to use metrics commonly used in image processing. Figure 4 depicts the average PSNR value results of the different cases mentioned earlier. The results show that with the PSNR index, the image quality after going through the filter is significantly higher than the image at the output of the ADMM algorithm. The Median filter and the BM3D filter give approximately the same results, but the weak point of the Median filter is that it does not have a control parameter, leading to a dropped image quality at higher channel’s SNR.

Another parameter commonly used in image processing to evaluate quality is the SSIM index. Fig. 5 is the average SSIM result of the images in the Set12 dataset. In general, the image after passing through the filter gives significantly better SSIM results than the original image, especially in the low and medium SNR regions. The Median filter performs best in this area but as the channel quality gets better, similar to PSNR, the output image no longer improve. The BM3D filter here due to being optimized for the highest PSNR result did not give good performance at SSIM metric.