Texture-guided Coding for Deep Features

In recent years, there has been an increasing amount of literature on deep feature compression. Sufficiently compressing features without significantly degrading the performance of CV tasks is a common goal. In this subsection, we roughly divide the compression of depth features into lossless and lossy compression.

Lossless compression treats depth features as generic data and compresses them using GZIP[16], ZLIB[17], BZIP2[18] or LZMA[19]. Although lossless compression achieves the best performance for the task, it has a limited compression ratio and does not adequately compress the deep feature data.

In lossy compression, deep features are compressed by classical video codec, such as high-efficiency video coding (HEVC)[1]. In [2], the HEVC range extensions[3] were applied to compress the deep features which were extracted from YOLO [4], where all the channels were combined to form a single image [2]. Different from [2], the feature of each channel was processed as the video frame in [5] where the HEVC intra[20] coding was used to compress all the feature channels extracted from the shallow layer of VGG [6], ResNet [7], etc. Deep features have similarities not only within each channel but also between channels. In Paper [8], the authors explored techniques for eliminating redundancy among feature channels. They tested three methods, including two inter-frame[21] approaches and one intra-frame[20] technique that tiled all channels into a single frame. The results showed that the tiling method performs best. Suzuki et al.[9] also investigated methods for removing redundancy among feature channels. They rearrange all channels of a feature into two or three large feature maps and then compress them using HEVC inter-mode.

The video-based solution reduces the redundancy of features. However, it usually assigns the same quantization parameter (QP) to different feature channels, which ignores the importance of the feature and cannot achieve a high representation ability for the feature. To solve this problem, the sensitivity-aware bit allocation was proposed[10], where the sensitivity of each feature channel was defined by its influence on the fidelity to the CV task. Moreover, different QP is assigned to the feature channel with different sensitivity, which achieves the adaptive bit allocation for features. Specifically, the feature channel with higher sensitivity would be allocated more bits to achieve high fidelity. In comparison, the channel with lower sensitivity would be allocated with fewer bits to guarantee that the overall bits do not exceed the given bit cost. In addition, Wang et al.[22] investigated the quantization of deep features. Unlike [2, 23, 5, 9, 24], which use uniform or logarithmic quantizers to reduce the data volume, it assigns different quantization intervals to each feature channel to achieve minimal network output errors.

These codec-based approaches achieve a good balance between task efficiency and computational cost when performing machine tasks. However, these methods only consider machine vision and do not consider human vision; in other words, they do not provide a texture image that humans can view.

In [25], they propose an end-to-end feature compression system that incorporates a task-specific loss function into the training process, resulting in better analytics performance with low bit rates. In addition to providing intra-frame patterns, the system introduces inter-frame coding schemes.

Since deep features can be larger than the original image size, [26] and [27] use an auto-encoder(AE) to reduce the feature dimension and size. Similarly, [28] introduces a Variational Auto-Encoder (VAE) model to reduce feature size while preserving task performance.

Like the AE,[29] and [30] achieve feature size reduction and recovery using a front-end encoding module and a back-end decoding module, respectively. Choi et al.[13] also use feature reduction to decrease the amount of data, focusing on reducing feature channels rather than directly manipulating the 2-D feature size. For analysis tasks, they can use a subset of the necessary features to complete.

End-to-end compression and feature size reduction methods reduce the required bit rate for feature transmission. However, both methods primarily focus on the analysis task and do not fully consider the reconstruction task. A hierarchical coding approach has been proposed to process both tasks jointly. Wang et al.[12] introduced a coding framework that integrates feature and texture, consisting of a base layer and an enhancement layer. In this framework, the base layer transmits the features server for the analysis task, and the enhancement layer transmits the residuals of the image reconstructed by the base layer from the original image. In [11], they proposed a similar hierarchical framework using an end-to-end model that includes a viewable analysis layer and a high-quality reconstruction layer. The analysis layer is used for the analysis task, while the reconstruction layer transmits the residuals of the features used for image reconstruction.

Hierarchical coding effectively utilizes texture and material information, enabling image reconstruction and analysis tasks to be performed simultaneously. However, as mentioned in previous methods, the residuals transmitted by the enhancement layer depend on the quality of the reconstructed image in the base layer. In features with low texture information, such as convolutional layers near the classifier, the quality of the reconstructed image is often inadequate, and in some cases, reconstruction may be impossible. Therefore, a more general layer coding framework is necessary to address these issues.

The overall pipeline of our proposed scheme is illustrated in Fig. 1. Our coding scheme is composed of two distinct branches. The first is the texture compression (TC) branch, and the second is the Feature Compression (FC) branch. The TC branch primarily serves humans and is dedicated to providing high-quality images to people. The FC branch serves machines and focuses on enhancing machine performance for task completion.

The TC branch utilizes the Neural Network-based(NN) codec for image compression. Features contain highly abstract semantic information that is challenging for humans to comprehend. In conventional feature compression frameworks, it is challenging to satisfy human and machine perception requirements simultaneously. To address human visual perception, we introduce a TC branch. This branch provides preview images for humans and assists the FC branch in feature reconstruction. The workflow is as follows: at the encoding end, the source image $I$ is downsampled and fed into a NN encoder to generate the texture stream $\mathcal{S}_{\mathcal{T}}$, where downsampling is performed to reduce the cost of image transmission. At the decoding end, the received $\mathcal{S}_{\mathcal{T}}$ is decoded to obtain the down-sampled image $\hat{I}_{\downarrow}$. Then $\hat{I}_{\downarrow}$ and the decoded features of the FC branch are jointly sent to the Image Reconstruction Network to obtain the reconstructed image $\tilde{I}$ for human.

The FC branch is primarily used for feature compression and reconstruction. It consists of an encoder and a decoder of the Feature Compression Neural Network (FCNN). At the encoding stage, the source image $I$ is fed into a feature extractor to obtain features $\mathcal{F}_{\mathcal{H}\mathcal{Q}}$, which consist of $C$ channels. However, the importance of these $C$ channels varies, and not every channel needs to be compressed and transmitted. To reduce the data volume of the features, we use a channel selection module in the FCNN to determine which channels to keep. This module outputs the channel selection information consisting of $C$ values, where 1 indicates channel retention, and 0 indicates channel removal. The retained channels are then used to construct a feature subset $\hat{\mathcal{F}}{m}$ with $M$($M\leq C$) channels. Finally, the quantized $\hat{\mathcal{F}}_{m}$ is flattened and passed through the HEVC encoder for compression and transmission. The quantization parameters and channel selection information are also transmitted. At the decoding stage, the flattened features are first decoded from the bitstream $\mathcal{S}_{\mathcal{F}}$ and then rearranged to reconstruct the feature subset $\hat{\mathcal{F}}{m}$. The missing channels are filled with zeroes. However, $\hat{\mathcal{F}}{m}$ contains only partial channel information and cannot be used directly for the task at hand. To reconstruct the full feature, the downsampled image reconstruction $I{\downarrow}$ from the TC branch is upsampled and passed through the feature extractor to obtain the features $\mathcal{F}_{LQ}$. By incorporating the channel selection information, the missing channel information is extracted from $\mathcal{F}_{LQ}$ and combined with $\hat{\mathcal{F}}{m}$ to reconstruct the feature $\hat{\mathcal{F}}_{HQ}$. The process of feature recombination is explained in the next section. Finally, this feature is fed into the Feature Reconstruction Module(FRM) of FCNN to recover the features for machine vision tasks.

Neural network design often involves gradually reducing feature dimensions while rapidly increasing convolutional kernel numbers to capture high-level semantic information for visual tasks. For instance, in conventional neural networks, the initial convolutional layer transforms a 3-channel RGB input image into 32 or 64 channels, halving the feature size. Consequently, the number of channels in intermediate convolutional feature maps increases more rapidly than the size reduction, leading to a substantial amount of feature data. Some researchers opt to reduce feature data by discarding channels, which may degrade task performance. To maintain task performance, researchers directly utilize the network to repair lost channels, mitigating the impact of dropped channels. However, regenerating channels filled with zeros poses a significant challenge for the network. To address this issue, we employ features extracted from texture images to substitute for the lost channels. Nevertheless, the disparity between features extracted from low-quality texture images and those from uncompressed images is considerable, leading to unsatisfactory performance improvements for visual tasks. Thus, applying an enhancement network to the reorganized features becomes necessary to enhance channel quality.

Based on this observation, we propose the Feature Compression Neural Network, which consists of a Channel Selection Module and a Feature Reconstruction Module. In our approach, we use the channel selection module to transmit the channels that contribute significantly to the task while filtering out the channels with low contribution. This allows us to achieve compression. To ensure task performance, we incorporate both the features of low-quality images and the high-contribution features into the Feature Reconstruction Module to recover high-quality features.

Low-quality images lose semantic information due to downsampling and image compression operations. To enhance the quality of the reconstructed image, we fuse feature maps containing semantic information with the low-quality image. However, the size of the depth features and the low-quality map differ and cannot be fused directly. Moreover, the size of depth features extracted from different locations varies, posing texture and feature fusion challenges.

Through observation, we find that the size difference between different feature layers is always an integer multiple of 2. Thus, it is possible to simultaneously transform the features and texture to the same size by either up-sampling the features or down-sampling the low-quality image. Building on this observation, we propose a generalized feature fusion network named Image Reconstruction Neural Network(IRNN) using the UNet[35] framework. The IRNN maximizes the utilization of information from both the feature and texture domains to accomplish image reconstruction. Guided by texture information, reconstructing the image from the feature domain becomes more efficient. The IRNN takes two inputs, features and texture, and outputs an enhanced image for human viewing. As Fig. 3 shows, the IRNN is constructed by the double conv unit (DCU), the down-sampling unit (DSU), the up-sampling unit (USU), and the up-sampling block (USB), as shown in Fig. 3, where the kernel size of the convolution layer of each unit is $3\times 3$ and stride = 1.

More specifically, the IRNN consists of two processes: top-down and bottom-up. The top-down process serves feature extraction and fusion and is composed of DCU and DSU modules. DCU is used to extract features from textured images and consists of two $3\times 3$ convolutions and two Leaky ReLU activations. Additionally, DCU is embedded in DSU, USU, and UPU modules to adjust feature channels. DSU, in the top-down process, adjusts feature channels and sizes and is composed of a max-pooling layer and DCU. The number of DSU modules in IRNN is determined by the input feature dimensions. The bottom-up process is used to reconstruct images from features and comprises USU, USB, and a $3\times 3$ convolution. USU consists of upsampling and DCU. Upsampling employs bilinear interpolation to reduce network parameters. The features of USU processes are residual-connected to the convolution results of the same layer and serve as input to the next USU. USB consists of a $3\times 3$ convolution and PixelShuffer for feature upsampling. The convolution result from USB, after adjusting the channels with a final $3\times 3$ convolution, is added to the upsampled low-resolution image $\hat{I}_{\downarrow}$ to output the reconstructed image $\tilde{I}$. The L2 loss is adopted as the loss function in our proposed IRNN:

where $I$ represents the target image, $\tilde{I}$ denotes the image reconstructed by the network, and $H$ and $W$ correspond to the image’s width and height, respectively. The $x$ and $y$ represent the positions of pixels.

In computer vision, VGGNet[6] is widely used for feature extraction in the back-end of various visual tasks. Our work chooses the general VGG16 as the back-end for classification tasks. VGG16 consists of 13 convolutional layers, five max-pooling layers, and three fully connected layers. We disconnect the VGG16 network at the 4-th pooling layer and split it into two subnetworks, one for feature extraction and the other for classification tasks. It is worth noting that the VGG16 network used is pre-trained, and its weights are provided by PyTorch[36]. To maintain consistency with the pre-trained weights, the weights of the feature extraction network and the classification network are frozen.

To verify the visual task performance of our method, we evaluate the classification performance on the ImageNet 2012 dataset. The dataset contains 1000 categories. For fairness, we randomly select one image from each category for testing. For comparison, we compress the original image using the classic video compression algorithm HEVC, denoted as Image Anchor. In addition, we extract features from the original image and then tile all feature channels into a two-dimensional grayscale image, which is then compressed using HEVC, denoted as Feature Anchor. In addition, three more popular end-to-end image compression models bmshj2018-hyperprior[39], mbt2018[40] and cheng-2020-anchor[34] are selected for comparison.

During compression, we use the intra-configuration of HEVC to compress features and images. We selected some test points with close bitrates for model comparison. It is worth noting that features need to be quantized before HEVC can compress them. Therefore, we adopt the quantization method of Chen et al.[5] to perform 8-bit quantization on the features, and the quantization and dequantization formulas are as follows:

Where $\hat{F}_{i}$ represents the quantized feature of the $i$-th channel, and $\tilde{F}_{i}$ denotes the dequantized $i$-th channel.

We use rate-accuracy curves to evaluate the performance of our proposed method. The rate is measured by bits per pixel (bpp), and the accuracy is measured by top-1 accuracy(ACC). The calculation formula of bpp is as follows:

where $W$ and $H$ represent the width and height of the original image, and $total\_bits$ denotes the total consumed bits. For our method, $total\_bits$ includes not only the bits of the feature stream and the texture stream, but also the bits consumed by the quantization parameters.

The experimental results are shown in Fig. 4. We set the weight parameters for our method to $\alpha=0.5$ and $\lambda=3$. From the figures, it can be observed that our method achieves better task performance at the same bpp under low bit rates. Compared to Image Anchor, our algorithm’s BD-Accuracy can reach $-69.46\%$, saving $-69.85\%$ bits at the same ACC. Fig. 4 also indicates that using texture images alone leads to poor task performance. However, task performance significantly improves when combined with the transmission of partial channels from the feature layer. This suggests the effectiveness of our collaborative strategy between texture and feature layers. Additionally, it is noteworthy that the performance improvement of features at high bit rates is limited, possibly due to quantization errors.

In addition to assessing the Rate-Accuracy performance, we evaluated the compression performance of our algorithm on various depth feature layers. The experimental results are summarized in Table I. The feature layers in the table represent features extracted from different positions of VGG16, where ”Pool1” denotes a shallow feature layer with the largest data volume, and ”Pool4” represents a deep feature layer with the smallest data volume. The compression rate in the table is computed by dividing the compressed data volume by the uncompressed data volume. A lower value indicates better compression performance. The calculation formula is given by:

The table reveals that, under the same accuracy, our feature compression algorithm achieves a lower compression rate compared to directly using HEVC to compress features, with an average compression performance improvement of approximately 3 times. This demonstrates that our algorithm not only achieves excellent compression performance on deep feature layers but also maintains strong performance on shallow feature layers, highlighting the generality of our method.

Guided by the texture image, our method achieves better machine vision task performance at the same bitrate. It provides humans with preview images at the decoding end to assist decision-making, meeting the ”dual vision” requirements of human eye visual perception and machine vision cognition. Moreover, the features can also be applied to texture images to facilitate image reconstruction.

We conducted experiments with three strategies to validate the performance of IRNN and the auxiliary role of features in image reconstruction. The first method applies Bicubic upsampling to the compressed low-resolution texture image $\hat{I}_{\downarrow}$ to restore the original resolution with a $2\times$ factor, denoted bicubic. The second method feeds $\hat{I}_{\downarrow}$ solely into the IRNN network for image reconstruction, denoted Texture-SR. The third method takes features and $\hat{I}_{\downarrow}$ as inputs for joint reconstruction by IRNN, denoted Texture-Feature-SR. The experimental setup remains the same as the previous section, with 1000 images used for evaluation. The feature’s QP and texture quality are set to 11 and 4, respectively.

The experimental results are as follows: The average PSNR for the first method is 23.254dB, for the second method is 23.949dB, and for the third method is 24.009dB. Comparing the results of the first and second methods, our IRNN reconstruction network improves performance by 0.695dB. A comparison between Bicubic and Texture-SR demonstrates that our IRNN reconstruction network can improve performance by 0.695 dB. Furthermore, a comparison between Methods Texture-SR and Texture-Feature-SR reveals that features can guide image reconstruction. As shown in Fig. 5, the target of the preview image reconstructed by the joint features and texture is slightly clearer. However, the improvement is not significant compared to Texture-SR. This could be attributed to our feature selection strategy, as depicted in Fig. 6. Since the feature layer close to the classifier inherently contains less texture information, and the feature selection module tends to prioritize features relevant to machine vision tasks, the improvement is not substantial.

In order to assess the impact of two strategies on detection performance, we conducted ablation experiments. The strategies in question are direct channel discarding and using low-quality features to fill the missing channels. The experimental setups are detailed in Table II. The ”Texture” parameter indicates whether low-quality features extracted from the texture are used to replace the lost channels. When ”Texture” is enabled, it means that features $\hat{\mathcal{F}}_{HQ}$ from Fig. 2 are utilized for subsequent training; otherwise, the network is trained using $\mathcal{F}^{m}_{HQ}$. The ”FRM” parameter indicates whether the FRM is employed to reconstruct the features $\hat{\mathcal{F}}_{HQ}$ or $\mathcal{F}^{m}_{HQ}$ for enhancement.

The experimental results, presented in Fig. 7, reveal comparisons between Method 1 and Method 2. It is observed that the classification performance when directly discarding feature channels and when utilizing low-quality features to fill in the lost channels is comparable, leading to a degradation in classification performance. Moreover, upon closer examination, filling the lost channels in Method 1 directly with low-quality feature channels results in a slight decrease in classification performance, while replacing the filled low-quality graphs in Method 2 with zeros maintains their classification performance. These findings suggest that the gap between low-quality and high-quality features is too significant to directly fill in the missing feature channels, thus limiting the potential improvement in visual task performance.

Additionally, a comparison between Method 1 and Method 3 demonstrates that, with the same QP compression and the assistance of the FRM module, Method 1 can further reduce the amount of data for the features—i.e., it can discard more channels—but at the cost of decreased classification performance.

Furthermore, contrasting Method 2 with Method 4 reveals that, with the assistance of the FRM module, it can further reduce the amount of data for the features, and the performance of the visual task remains unchanged. This underscores the efficient ability of FRM to reconstruct and enhance features.