Deep Learning for Predictive Analytics in Reversible Steganography

Secrecy is at the heart of covert communications. The ability to pass secret information under surveillance is essential to espionage and military operations. It has been reported that deep learning can be used to identify locations in a cover image at which message embedding would not arouse suspicion [22, 23, 24]. More technically, a deep learning algorithm assigns a cost to every pixel to quantify the effect of making modifications. In this way, a high degree of perceptual and statistical undetectability can be reached by minimising the cost. Other approaches include using generative models to convert a given message into a cover image which is less vulnerable to steganalysis [25], to transform a cover image along with a secret message into a stego image [26], and to encode a message directly into a realistic stego image in the absence of an explicit cover image [27].

Robustness is a prioritised requirement for copyright protection. In commercial applications, entrepreneurs can prevent copyright infringement by embedding a registered watermark into digital assets. Deep learning has been used to embed invisible watermarks into images and videos in a durable way in order to identify copyright ownership and deter illegitimate copying [28]. Unauthorised screen recording is a new scenario in which an adversary attempts to capture an electronically displayed still photograph or video footage with a digital camera, causing optical interference in watermark extraction. It has been shown that neural networks can be trained to simulate optical interference and then learn to encode and decode watermarks in a robust manner [29]. Decoding accuracy can also be enhanced by using neural networks to remove light artefacts such as moiré fringes, a type of textile with a rippled or wavy appearance, prior to watermark extraction [30]. To facilitate augmented reality, deep learning has been used to decode hyperlinks embedded in physical photographs rather than digital media, subject to real-world variations in print quality, illumination, occlusion and viewing distance [31].

Reversibility is a desirable characteristic for applications in which accuracy and consistency of data are important. It is a requisite for preventing an accumulation of steganographic distortion over each data transmission. A reversible steganographic method often relies on intricate logical operations to regulate imperceptibility and guarantee reversibility. Such operations are hard to achieve using existing neural network models. One of the earliest approaches uses a U-shaped network (U-Net) to automatically encode a cover image and a secret message into a stego image and a separate neural network model for decoding [32]. Another approach encodes a message bitstream into a cover image via a generative model, trains a cycle-consistent generative adversarial network (CycleGAN) to learn the back-and-forth transformation between the cover and stego images, and extracts the message bitstream from the stego image by using another neural network model [33]. Furthermore, an intriguing study shows that both encoding (forward mapping) and decoding (inverse mapping) can be performed by using a single invertible neural network (INN) [34]. While a monolithic end-to-end system generally offers a large steganographic capacity, a common limitation within this type of framework is imperfect reversibility due to the presence of an information bottleneck (i.e. a form of lossy compression) in neural networks. The lack of transparency, interpretability and explainability in neural networks adds an extra dimension to the problem.

Although reversibility is not an all-or-nothing proposition, the unreliability of end-to-end learning may pose uncontrollable risks in certain circumstances. Problem decomposition or modularisation is essential to addressing complex problems. Existing reversible steganographic schemes often consist of coding and analytics modules. In general, the coding module handles the encoding and decoding mechanisms, which are designed to achieve perfect reversibility subject to an imperceptibility constraint. The analytics module models the data distribution and exploits data redundancy in order to optimise steganographic rate-distortion performance. For instance, the regular–singular (RS) scheme introduced by Fridrich et al. constructs a discrimination function to categorise image blocks into regular, singular and unusable groups on the basis of the smoothness prior, and uses invertible flipping of the least significant bits to achieve invisible and erasable embedding [35]. We can define the former part as the analytics module and the latter part as the coding module. The problem of limited reversibility in previous deep-learning approaches lies in the difficulty of lossless coding in an end-to-end learning fashion. A recent study achieves perfect reversibility with a deep-learning-based RS scheme following this established modular framework [36]. A conditional generative adversarial network (GAN), referred to as pix2pix [37], is applied to improve the performance of the discrimination function. It has been shown that by partitioning a scheme into coding and analytics modules and deploying neural networks in the latter, reversibility can be reliably guaranteed.

A combination of prediction-error modulation with deep learning has been described in two studies [38] and [39]. Neural networks are incorporated into the schemes based upon modularity. The most notable feature in common is that both studies divide pixels into query and context sets by using a chequered pattern and develop neural network models for predicting the query from the context. The main difference is neural network architectures. The former constructs a multi-scale convolutional neural network (MS-CNN) to address the issue of restricted receptive fields in traditional predictors. The latter applies a persistent memory network (MemNet) originally developed for image denoising and image super-resolution [40]. From a certain perspective, context-aware pixel intensity prediction is closely related to low-level computer vision tasks such as image denoising and image super-resolution because they all rely largely on low-level (or pixel-level) features such as edges, contours and textures, rather than high-level semantics [41]. Therefore, it is expected that advanced neural networks from the low-level computer vision domain can be applied directly with minor modifications. Another difference is the setting of the initial intensities of the query pixels. The former simply sets the query pixels to zero and learns to predict from scratch. By contrast, the latter initialises them to the mean of neighbouring pixels and performs prediction in a coarse-to-fine manner. Consequently, a problem of pixel-intensity prediction is, in a certain sense, transformed into a problem of image denoising or image super-resolution. Last but not least, the chequered pattern for context/query splitting features a dual-layer prediction: pixels assigned as the query in the first round, after partial embedding, can be assigned as the context in the second round. Such a practice gives rise to the problem of distributional shift—the data distribution in the target (online) domain for deployment shifts from that in the training (offline) domain. In consequence of the distortion introduced in the preceding steganographic process, assigning those distorted pixels as the context could introduce discrepancy and bias into the contextual information, thereby undermining predictive performance. The expected performance relies on the extrapolation (generalisation) capability of neural networks. To summarise, this work aims to seek answers to the following questions:

• •

  Can different intensity initialisation strategies affect predictive accuracy of neural networks substantially?

• •

  Can the problem of distributional shift be mitigated, thereby enabling efficient dual-layer prediction?

• •

  Can off-the-shelf neural networks from the low-level computer vision domain be transferred to perform pixel intensity prediction?

A schematic workflow of the encoding and decoding procedures is provided in Figure 1 with the definitions of variables and operations listed in Table I. The scheme based on prediction-error modulation can be broken down into an analytics module and a coding module. The former models the distribution of natural images and predicts the intensity of query pixels on the basis of the available context pixels. The latter encodes/decodes messages into/from prediction errors. Consider the following scenario: A sender (encoder) embeds a message $\bm{m}$ into a cover image $\bm{x}$ to produce a stego image $\bm{x^{\prime}}$ and then transmits it to a receiver (decoder) who extracts the message and restores the cover image on receipt of the stego image.

At the encoder side, we pre-process the cover image to prevent pixel intensity overflow during message embedding at the cost of generating extra auxiliary information. We treat the auxiliary information as a part of the payload and concatenate it with the intended message. Then, a chequered pattern is applied to split the image into a black set and a white set of pixels, denoted respectively by $\bm{x}_{\mathbb{B}}$ and $\bm{x}_{\mathbb{W}}$. In the first-layer prediction, we designate the black set as the context set and the white set as the query set. The roles are reversed in the second-layer prediction. We make use of a neural network or a predictive algorithm to estimate the intensities of the query pixels based on the observed context pixels. The prediction errors or residuals, denoted by $\bm{\varepsilon}$, are computed by subtracting the predicted values from the actual values. A portion of the payload is embedded in the residual domain through arithmetic operations. The modulated prediction errors, denoted by $\bm{\varepsilon}^{\prime}$, are then added to the predicted values, resulting in a slight distortion to the query pixels. The process can be repeated once more to embed another part of the payload by swapping the roles between the context and the query.

At the decoder side, we carry out message extraction and image restoration in a last-in-first-out manner. Similar to the encoding process, the decoding process begins by splitting pixels of the stego image into the context and the query, predicting the query from the context, and calculating the prediction errors. A partial payload is extracted and the prediction errors are de-modulated back to the original state through inverse operations. Steganographic distortion is removed by adding the restored prediction errors to the predicted values. The decoding process can likewise be repeated once more by swapping roles between the context and query. Finally, the restored image is post-processed with the extracted auxiliary information to undo the minute modifications caused by the overflow prevention measure.

To summarise, the algorithmic steps are enumerated as follows. For the encoding phase:

1. 1.

   $\{\check{\bm{x}},\bm{v}\}=\operatorname{scale}_{\Downarrow}(\bm{x})$;

2. 2.

   $\bm{p}=\operatorname{concat}(\bm{v},\bm{m})$;

3. 3.

   $\{\bm{x}_{\mathbb{B}},\bm{x}_{\mathbb{W}}\}=\operatorname{split}(\check{\bm{x}})$;

4. 4.

   $\bm{y}_{\mathbb{W}}=\operatorname{net}_{1}(\bm{x}_{\mathbb{B}},\bm{x}_{\mathcal{Q}}|\mathcal{Q}=\mathbb{W})$;

5. 5.

   $\bm{x}^{\prime}_{\mathbb{W}}=\operatorname{encode}(\bm{x}_{\mathbb{W}},\bm{y}_{\mathbb{W}},\bm{p}_{1})$;

6. 6.

   $\bm{y}_{\mathbb{B}}=\operatorname{net}_{2}(\bm{x}^{\prime}_{\mathbb{W}},\bm{x}_{\mathcal{Q}}|\mathcal{Q}=\mathbb{B})$;

7. 7.

   $\bm{x}^{\prime}_{\mathbb{B}}=\operatorname{encode}(\bm{x}_{\mathbb{B}},\bm{y}_{\mathbb{B}},\bm{p}_{2})$;

8. 8.

   $\bm{x}^{\prime}=\operatorname{merge}(\bm{x}^{\prime}_{\mathbb{B}},\bm{x}^{\prime}_{\mathbb{W}})$.

For the decoding phase:

1. 1.

   $(\bm{x}^{\prime}_{\mathbb{B}},\bm{x}^{\prime}_{\mathbb{W}})=\operatorname{split}(\bm{x}^{\prime})$;

2. 2.

   $\bm{y}_{\mathbb{B}}=\operatorname{net}_{2}(\bm{x}^{\prime}_{\mathbb{W}},\bm{x}_{\mathcal{Q}}|\mathcal{Q}=\mathbb{B})$;

3. 3.

   $\{\bm{x}_{\mathbb{B}},\bm{p}_{2}\}=\operatorname{decode}(\bm{x}^{\prime}_{\mathbb{B}},\bm{y}_{\mathbb{B}})$;

4. 4.

   $\bm{y}_{\mathbb{W}}=\operatorname{net}_{1}(\bm{x}_{\mathbb{B}},\bm{x}_{\mathcal{Q}}|\mathcal{Q}=\mathbb{W})$;

5. 5.

   $\{\bm{x}_{\mathbb{W}},\bm{p}_{1}\}=\operatorname{decode}(\bm{x}^{\prime}_{\mathbb{W}},\bm{y}_{\mathbb{W}})$;

6. 6.

   $\check{\bm{x}}=\operatorname{merge}(\bm{x}_{\mathbb{B}},\bm{x}_{\mathbb{W}})$;

7. 7.

   $\{\bm{v},\bm{m}\}=\operatorname{deconcat}(\bm{p})$;

8. 8.

   $\bm{x}=\operatorname{scale}_{\Uparrow}(\check{\bm{x}},\bm{v})$.

All images are assumed to be 8-bit greyscale throughout this study. Let us denote by $\vartheta$ a threshold for the stego channel such that

where $\operatorname{abs}$ denotes the absolute value. In other words, we use the parameter $\vartheta$ to determine for which prediction error values the payload embedding process can take place. The following presents the algorithmic details surrounding the notion of the stego channel.

Both encoding and decoding are carried out in the residual domain. Addition and subtraction on residual values are equivalent to the same arithmetic operations on pixel values (i.e. a pro rata increment or decrement in pixel intensity). Given that computations are defined in a Galois finite field, such arithmetic operations may cause pixel intensity overflow: intensity values that are unexpectedly small or large wrap around the minimum or maximum after manipulations. We pre-process the image to prevent pixels from moving off boundary (or becoming saturated) and mark the locations where off-boundary pixels may occur. To distinguish between the processed pixels $\check{x}$ and unprocessed pixels $x$ with the same value, we flag $1$ (true) for the former and $0$ (false) for the latter in an overflow-status register, that is,

and

where

The parameter $\vartheta$ is associated with the modulation step in the message embedding process. A greater value of $\vartheta$ spans a wider range of pixel values that may be modified beyond the boundary and thus having a higher probability of increasing the size of the register. The overflow-status register is an overhead of reversibility. It has to be concatenated with the message and embedded as a part of the payload. As a consequence, its size has to be deducted from the overall payload when assessing steganographic capacity. See Algorithms 1 and 2 for the pseudo-codes. The overflow handling process is codified in Table IIc.

Let $y_{i,j}$ denote a prediction of the pixel at location $(i,j)$. For each query pixel, we compute its prediction error by

For the prediction errors determined as the stego channel, we embed a bit of payload information by

where $\operatorname{sgn}$ extracts the sign of the prediction error (either positive or negative) and $p_{t}$ is the current payload bit. For the rest of the errors, we shift them by $\vartheta$ to prevent error values from overlapping (i.e. indistinguishable from the errors determined as stego channel); that is,

A modulated error is then added to the predicted value, resulting in a stego pixel

Let us take the case in which $\vartheta=2$ for example. When $\varepsilon=0$, we can map it to one of the three states, $\varepsilon^{\prime}=0$ or $\pm{1}$, to encode a ternary digit (a trit of information). When $\varepsilon=\pm{1}$, we shift it to $\pm{2}$ to avoid ambiguity and map it to one of the two states, $\varepsilon^{\prime}=\pm{2}$ or $\pm{3}$ (disregarding the sign), to represent one bit of information. For all other $\varepsilon$ such that $\varepsilon\notin(-\vartheta,+\vartheta)$, we shift each of them by $\vartheta$ in either a positive or negative direction, depending on the sign. However, converting the payload between binary (base $2$) and ternary (base $3$) numeral systems on the fly can be problematic. To circumvent this issue, we can map the errors with value $0$ to the digit $0$ if the observed payload bit $p_{t}$ is $0_{2}$; otherwise, we read the next payload bit $p_{t+1}$ and map the errors to the digit $-1$ if $[p_{t},p_{t+1}]=[1_{2},0_{2}]$ and to the digit $+1$ if $[p_{t},p_{t+1}]=[1_{2},1_{2}]$; that is,

Theoretically, the mapping permits one trit or $\log_{2}3\approx 1.585$ bits of information to be embedded. In practice, the compromised solution embeds one or two bits with a probability of $0.5$ each, thereby being capable of embedding $1.5$ bits on average for $\varepsilon=0$. Decoding is simply an inverse mapping. It begins by predicting the query pixel intensities and computing the prediction errors. Each error is de-modulated according to its magnitude. For errors between $\pm 1$, we map them back to $0$ and extract the corresponding payload bits. For errors whose magnitude is in $(1,2\vartheta)$ regardless of the sign, we de-modulate them by the floor division of the magnitude by $2$ and extract the payload bits by the remainder of the division. For the rest errors, we shift them back by $\vartheta$. The original pixel intensity is recovered by adding the de-modulated error to the predicted value.

According to the Law of Error [42], we can hypothesise that the frequency of an error approximates an exponential function of its magnitude and therefore follows a zero-mean Laplacian distribution. This symmetrical modulation pattern is derived from the fact that prediction errors are expected to centre around $0$. Therefore, it is advisable to designate errors with a small absolute magnitude (of high frequency) as the stego channel. A rise in the width of the stego channel represents an increased steganographic capacity. See Algorithms 3 and 4 for the pseudo-codes. The prediction-error modulation process is codified in Table IIIc. Figure 2 displays stego images, their tonal distributions, and locations of carrier pixels w.r.t. different settings of stego-channel parameter $\vartheta$. Results are generated by spreading a pseudo-random binary message over the whole image at the maximal steganographic capacity. It can be observed that the quality degradation is virtually imperceptible, the disturbance to the tonal distribution is subtle, and the pixels selected for carrying the payload are mostly clustered in smooth areas. We would like to note that this coding method is simple, readily scalable and computationally efficient but not necessarily optimal. The main theme of this study is the analytics module, which is independent of the coding module. Any coding method that reflects more accurately the statistical distribution of residuals can be applied without conflicting with the findings of this study. For the subject of mathematical optimisation of reversible steganographic coding, the interested reader is referred to the research study [43].

A convenient way to define pixel connectivity is to consider the von Neumann neighbourhood, that is, four adjacent pixels connected horizontally and vertically. If we sample the context and query pixels uniformly in such a way that each query pixel has $4$ connected context pixels, we end up forming a pattern analogous to a chequerboard, as illustrated in Figure 3. We can define the black set $\mathbb{B}$ as the context and the white set $\mathbb{W}$ as the query (or the other way round):

A naïve way to estimate the intensity of the central pixel is to calculate the mean of locally connected pixels. This heuristic predictive model, referred to as local-mean interpolation (LMI) [44], is based on the smoothness prior, a generic contextual assumption on real-world photographs. The recent studies have shown that neural network models (e.g. MS-CNN [38] and MemNet [39]) can be applied to improve predictive accuracy. Because predictive accuracy is instrumental in steganographic rate-distortion performance, the factors that could have the impacts upon predictive accuracy are primary concerns of this study.

The first step of the encoding phase is to split a cover image into $\bm{x}_{\mathbb{B}}$ and $\bm{x}_{\mathbb{W}}$. While the values of the query pixels are supposed to be predicted, the initial values of query pixels $\bm{x}_{\mathcal{Q}}$ have to be determined and cannot be set as null since the input to the applied neural networks must be a complete image. To this end, we consider two simple initialisation strategies: zero initialisation and local-mean initialisation.

Dual-layer prediction entails the problem of distributional shift because steganographic distortion appears after the first-layer embedding. In the first layer, the intensities of query pixels are predicted by a neural network model and then modified with the prediction-error modulation algorithm. In the second layer, the roles of context and query are switched. As a result, the context set consists of distorted pixels. This can also be viewed as uncertainty propagation in the sense that errors in the previous layer propagate to the next layer, impairing the predictive performance. Formally, we aim to minimise the loss of two test sets:

To this end, we explore three training strategies: universal training, independent training, and causal training. The training strategies differ by the configurations of the training set, as illustrated in Figure 4.

An image whose query pixels are initialised to either zero or local mean can be viewed as a noisy and low-resolution version of the original image. The goal of context-aware pixel-intensity prediction can therefore be considered as refining the observed image into a clean and high-resolution one. Based on this perception, we can adopt advanced deep-learning models originally devised for image denoising and image super-resolution. A neural network is essentially a non-linear function that learns to minimise a pre-defined loss function by transforming the input into useful features or representations in a latent space. For the task of pixel-intensity prediction, we suggest training the models to minimise the $\ell_{1}$ norm or mean absolute error (MAE). The reason for choosing the $\ell_{1}$ norm over the $\ell_{2}$ norm is that the latter tends to produce overly smoothed outputs in image enhancement tasks [45, 46, 47]. The $\ell_{2}$ norm, or mean squared error (MSE), encourages producing an average of plausible solutions and is prone to being affected by outliers. There are other preferable loss functions in the field of low-level computer vision. An example is the Euclidean distance between the high-level feature maps extracted by a $19$-layer VGG neural network [48] pre-trained on the ImageNet (a database for large-scale visual recognition) [49]. Another example is the adversarial loss which uses a discriminator to estimate the likelihood that a generated instance is real [50]. Nevertheless, we do not anticipate these ad hoc loss functions improving steganographic performance because prediction-error modulation relies mainly on pixel-wise distance. Apart from the choice of loss functions, when applying off-the-shelf models, upsampling and transposed convolutional layers should be replaced with regular convolutional layers because the sizes of the input and output images are the same in pixel-intensity prediction. While the neural network architectures vary from one to another and it is difficult to foresee their performance in different tasks (due to the ‘black box’ nature of neural networks), we attach relative importance to shortcut connection because it mitigates the performance degradation problem when increasing network depth [51] and allows models to learn explicitly the minute differences between the input and output images [52]. To ensure reproducibility, the architectural details of the MS-CNN, MemNet and RDN used for empirical evaluation are illustrated in Figure 5 and specified as follows.

The main theme of this study is predictive analytics. For a fair comparison, we evaluate the performance of different predictive models based on the same steganographic algorithm (i.e. prediction-error modulation). We do not compare steganographic performance with the end-to-end framework because it approaches the problem from a different standpoint. In general, the end-to-end framework cannot reliably satisfy the requirement for perfect reversibility given the limit of current deep-learning algorithms. For the predictive analytics module, we select LMI as a benchmark model [44] based on heuristics and both MS-CNN [38] and MemNet [39] as state-of-the-art models based on deep learning. All the selected models are specifically designed and applied for the prediction task with the same context–query arrangement (i.e. the chequered pattern). The mentioned deep learning models have also been shown to outperform other traditional methods for either same or different context–query arrangement, including nearest-neighbour interpolation [53], median-edge detector [54], gradient-adjusted predictor [55] and bilinear interpolation [56]. In summary, the coding algorithm and the context–query splitting pattern are held constant (as control variables) throughout the course of the investigation to prevent the interference in the experimental results. We select LMI, MS-CNN and MemNet as representative benchmarks and carry out a comparative study with the advanced RDN model.

The neural network models are trained and tested on the BOSSbase dataset [57], which originated from an academic competition for digital steganography. It comprises a collection of $10,000$ greyscale photographs covering a wide variety of subjects and scenes. To fit the models over a broad range of hardware options with reasonable training time, all the images are resized to a resolution of $256\times 256$ pixels by using the Lanczos resampling algorithm [58]. The training and test sets are randomly sampled at a ratio of $80$/$20$. All the neural network models are trained to minimise the $\ell_{1}$ loss unless specified otherwise. Further inferences and evaluations are made on a set of standard test images from the volume ‘miscellaneous’ in the USC-SIPI dataset [59]. To investigate predictive performance on challenging samples, we also take Brodatz texture images from the volume ‘textures’ in the USC-SIPI dataset [60]. We use peak signal-to-noise ratio (PSNR) (expressed in decibels; dB) [61] and structural similarity (SSIM) [62] for perceptual quality assessment. When evaluating the visual quality of predicted images, we take the whole image into account because the bias in the context pixels is almost negligible. The context pixels in each predicted image are almost always the same as those in the original image. We use embedding rate (expressed in bits per pixel; bpp) for steganographic capacity evaluation.

We compare the zero and local-mean initialisation strategies by the learning curves and the perceptual qualities. The learning curves are plotted in Figure 6. The curves represent the training loss over $100$ epochs for the input/target pairs ($\bm{x}_{\mathbb{B}}$, $\bm{x}$). It can be observed that while the convergence rate varies from model to model, zero initialisation reaches a slightly lower loss than the local-mean initialisation in the last epoch. A possible explanation is that although setting the pixel intensity to zero seems to be abrupt and oversimplified, this approach involves minimal human intervention in the machine learning pipeline. By contrast, local-mean initialisation can be viewed as introducing subjective prior knowledge (i.e. the smoothness prior) in the training process. The perceptual qualities of predicted images are measured in Figure 7. Through observing visual quality measurements on different models, we conclude that there is virtually no difference between the impacts of two initialisation strategies upon predictive accuracy. While local-mean initialisation provides a rough estimation in advance, its contribution to predictive accuracy is negligible. Since zero initialisation has a virtue of low computational complexity, we adopt it for the remaining experiments.

Predictive accuracies of the universal, independent, and causal training strategies for dual-layer prediction are depicted in Figure 8. The bars show the average PSNR and SSIM scores between the ground-truth and predicted images w.r.t. different training strategies. Steganographic distortion introduced in the first layer would propagate and decrease the predictive accuracy of the second layer. As a result, the second-layer scores are generally lower than the first-layer scores due to the distributional shift. The causal training achieves a comparatively high accuracy for the second-layer prediction, whereas the performance gap between the other two strategies is not distinct. This suggests that the causal training using a training set with a distribution similar to that of the test set can indeed alleviate the distributional shift to some extent. The narrow performance gap between the universal and independent training can be attributed to the translation-invariance property of CNNs. It suggests that training a single model can be as good as training two separate models. Although causal training requires additional computations when constructing the second training set, it is proved to be the most effective among the three training strategies. Hence, we opt for the causal training strategy for the remaining experiments.

Visual comparisons between a heurstic model and a neural network model are shown in Figures 9 and 10. We evaluate the PSNR scores for images predicted from the LMI and RDN. The reported scores are related to the first-layer prediction. A close inspection of zoomed-in views reveals that the RDN is better able to retrieve textural areas with reasonable accuracy in comparison with the naïve LMI, owing to the ability of neural networks to learn rich patterns. Figure 11 compares predictive accuracy of the RDN, MemNet, MS-CNN and LMI on texture images. The results reflect a significant improvement yielded by the RDN model over other predictive models, confirming the capability of the former to predict textural patterns.

The distribution of prediction errors plays a pivotal role in steganographic performance. Normally, errors would follow a zero-centred Laplacian distribution and a more accurate model results in a more concentrated distribution. According to the coding algorithm, a more concentrated distribution can lead to a better steganographic rate-distortion trade-off. To analyse the error distribution, we examine the probability distribution function (PDF), cumulative distribution function (CDF), and Lorenz curve of errors, as shown in Figures 14, 14, and 14, respectively. We use the entropy, variance, percentile statistics, and Gini coefficient to measure the degree of error concentration [63]. Shannon’s entropy is maximised for a uniform distribution and the converse is equally true: a smaller entropy means a more concentrated distribution [13]. The variance is a measure of the spread of the data around the mean. The 95^th percentile indicates the maximum error magnitude below which $95\%$ of errors fall. The Gini coefficient is a measure of statistical dispersion intended to represent the inequality of the error magnitude within an image. A coefficient of $0$ expresses perfect equality (dispersion), whereas a coefficient of $1$ corresponds to maximal inequality (concentration). For the overall degree of concentration, the RDN ranked in the first tier, followed by the MemNet and the MS-CNN in the second tier, with the LMI ranked last.

The steganographic rate-distortion curves are plotted in Figure 15. The models with a higher predictive accuracy indeed achieve a better rate-distortion trade-off. The perceptual quality of stego images is inversely proportional to the embedding rate since distortions accumulate along the embedding process. The maximum capacity depends on the image content. Images with highly textured details would have lower capacity. A rise of $\vartheta$ increases the maximum embedding rate at the cost of compromising the rate-distortion trade-off. Amongst all models, the RDN stands out, achieving state-of-the-art steganographic performance. Apart from the $\ell_{1}$ loss, we investigate the applicability of other common loss functions in the field of image restoration. We implement the $\ell_{2}$ loss and the multi-loss on the RDN model. The latter is comprised of $\ell_{1}$ loss, feature-space loss and adversarial loss. While an extra computational cost is devoted to the implmentation of the multi-loss training, the results suggest that there is marginal, if any, gain in steganographic performance by applying such loss function. This reinforces the argument that the prediction-error modulation algorithm relies primarily on pixel-wise distance.