Scene Graph Lossless Compression with Adaptive Prediction for Objects and Relations

General purpose lossless codecs usually convert any forms of data into a sequence of symbols, and compress the sequence into a compressed stream.

A popular tool for general-purpose lossless compression is zlib zlib , which combines LZ77 DBLP:journals/tit/ZivL77 and Huffman coding 4051119Huffman . Many later compression codecs improve such scheme by optimizing the string matching algorithm in LZ77 (such as LZMALZMA and BrotliDBLP:journals/tois/AlakuijalaFFKOS19 ) or switching to better entropy coder such as Asymmetric Numeral SystemDBLP:journals/corr/Duda13 (such as ZstdZstd ). Some other codecs designs a context-adaptive entropy coder to exploit context information. For example, PPM1096090PPM utilize a simple suffix-based context model to count symbol frequencies adaptively, while PAQMahoney2005AdaptiveWO utilize the context mixing scheme to combine predictions from many specially designed context models. Recent works also explores using neural networks to learn prior information of data or exploit context information more robustly, such as CMIXCMIX , and DzipDBLP:conf/dcc/GoyalTCO20 .

However, for scene graph data, such codecs ignore hidden redundancy between different data elements. Furthermore, context information in general-purpose compression methods are usually explored in adjacent string of symbols, while for graph-structured data like scene graphs, adjacency usually exists along graph edges. Therefore, applying general-purpose codecs is an inefficient solution for scene graph lossless compression.

Graph is a common form of data in many areas, such as web-page database, social network and biological data. Therefore, graph structure lossless compression has also been studied in many previous works, as reviewed in DBLP:journals/corr/abs-1806-01799 . For example, WebGraphDBLP:conf/www/BoldiV04 is a framework designed for web graph compression. Based on the locality and sparsity features of web-page URLs, WebGraph utilizes multiple compression techniques including reference compression and differential compression to exploit such features. Another compression method built on such features was $k^{2}$-treeDBLP:conf/spire/BrisaboaLN09 , which iteratively divides the graph adjacency matrix into $k^{2}$ sub-matrices, and stops when the sub-matrix is a all-zero matrix.

Although such methods could be applied for graph structure compression, the compression of node data and edge data is usually ignored. In this work, in addition to graph structure compression, node data compression and edge data compression should also be considered.

Recently, many compression related works has been using neural networks. A large portion of those works are related to image compression. Most recent approaches of deep-learning based image compression utilize an autoencoder-based end-to-end framework DBLP:conf/iclr/BalleLS17 . Following DBLP:conf/iclr/BalleLS17 , DBLP:conf/iclr/BalleMSHJ18 proposed a hyperprior autoencoder that enables better latent prediction. Afterwards, a few works propose to improve the hyperprior model by exploiting context information from neighbouring pixels (DBLP:conf/nips/MinnenBT18 ; Chen9359473 ) or improving the distribution module with more complex distributions (DBLP:conf/cvpr/ChengSTK20 ; DBLP:journals/corr/abs-2107-06463 ). Other frameworks also explores different network structures such as context-based convolutional networks for entropy modelingLi9067005 or wavelet-like convolutional networks for image transformationDBLP:journals/tmm/Ma0X020 .

For lossless or near-lossless image compression, a straightforward solution is to apply a predictor and compress the residual 9277919 . Also, some works apply autoencoder as likelihood generative networks DBLP:conf/cvpr/MentzerATTG19 ; DBLP:conf/icml/KingmaAH19 to estimate pixel-wise probability for entropy coding in order to achieve lossless compression. Other works deploy invertable neural networks DBLP:conf/nips/HoogeboomPBW19 ; 9204799 to generate a lossless transformation of the original data for easier entropy coding.

The proposed framework is inspired by many successful learning-based compression frameworks DBLP:conf/iclr/BalleMSHJ18 ; DBLP:conf/nips/MinnenBT18 ; DBLP:conf/cvpr/MentzerATTG19 ; DBLP:conf/icml/KingmaAH19 . However, the proposed framework is specially designed for compression of the complete scene graph. To achieve this, some novel modules are proposed and designed, such as the adaptive element predictors with graph context models.

The unified prior extractor is expected to provide a coarse perception of the entire graph in the compressed bitstream for element predictors. We implement it with the popular autoencoder strcture. Specifically, the encoder first takes the entire preprocessed graph as input and produces a latent vector as the prior information. Then, the decoder receives the prior information and outputs an initial prediction that is fed into the element predictors to participate in the final prediction of symbol probability for lossless entropy coding. The above process is illustrated in Fig. 3a.

Furthermore, as the latent vector should be finally compressed into a bitstream, we need to calculate the entropy of the latent vector and utilize it to estimate and optimize the length of compressed unified prior bitstream during training. For latent vector $\mathbf{y}$, its entropy is estimated by:

where $P(\mathbf{y})$ is the probability of latent vector $\mathbf{y}$. To obtain $P(\mathbf{y})$ , we adopt the same factorized-prior model as DBLP:conf/iclr/BalleMSHJ18 . It learns a fixed factorized prior distribution to fit the latent vector’s distribution during training.

In scene graphs, the data shapes, contextual relationships, and distributions vary from data elements. Based on this observation, we propose the element-adaptive predictors for scene graph lossless compression, which empower our framework to cope with diverse data elements adaptively in scene graphs. Specifically, each predictor consists of two modules: the parameter estimation module and the distribution module. The former obtains element-specific information according to data shape and contextual relationship, while the latter fits different data distributions of data elements based on the element-specific information.

Specifically, the parameter estimation module first merges the context information extracted from the data buffer and the initial prediction to have a complete perception of data elements. Then, it can estimate distribution parameters for data elements according to the perception. To this end, we implement parameter estimation module with three sub-models to accomplish the above process: the context model, the prior model, and the merge model. Note that we design an element-specific context model for each graph element. As shown in Fig. 3d, it has basically three options in scene graphs: the node context model, edge context model, and graph structure context model. Detailed algorithms and implementations of the parameter estimation module, especially the context model, will be discussed in Section 4.

The distribution module contains a random distribution model to fit different data distributions. It receives the estimated parameters given by the parameter estimation module and predicts symbol-wise probabilities. As shown in Fig. 3d, two forms of distribution models are available for different data distribution. One is a direct distribution model for categorical data such as object and relation types. Another is an indirect distribution model for numerical data such as object locations. Moreover, we propose a learned distribution model to tackle the complicated distribution of numerical data. Details will be discussed in Section 5.

The calculation process of the element predictor could be represented as Eq. 2.

where $f_{D}$ represents the distribution model, $\mathbf{x}$ represents the data element, and $\mathbf{\omega}$ represents distribution parameters calculated by the parameter estimation module, represented as Eq. 3.

$f_{PEM}$ is the parameter estimation module, and $f_{P}$,$f_{C}$,$f_{M}$ are its prior model, context model and merge model respectively.

Note that there may exist other data elements than those 4 data elements in Fig. 1 in different scene graphs. For example, the ”relation weights”, namely the certainty level of the estimated relation, and the ”human pose”, namely the skeleton points on a human object. The proposed element predictor can generalize to various forms of data elements in scene graphs by reusing the proposed parameter estimation modules and distribution modules. An example of the implementation of the element predictor for some possible data elements in scene graph is described in Table 1. For example, object locations in scene graph belong to node data in graph, and is numerical data. Therefore, the element predictor for object locations should apply node context model and indirect distribution model. Relation types belong to edge data in graph, and is categorical data. Therefore, the element predictor for relation types should apply edge context model and direct distribution model. Implementations for other data elements such as object type and relation weight could also be inferred accordingly.

Lossless coding for a scene graph is achieved by directly applying entropy coders on the scene graph data with the probabilities predicted by element predictors. The whole coding is free from any processes that may introduce data loss, such as quantization. Note that although quantization is applied in our unified prior extractor, its initial prediction output is used for probability prediction, and thus it will not affect the losslessness of the original data. Moreover, the preprocessor shown in Fig. 3a only performs node sorting for better context modeling and will not introduce any data loss. More details about this issue will be discussed in Section 4.

During training, the predicted probability is used to calculate the data entropy to estimate the length of the compressed data bitstream according to Eq. 4.

where $P(\mathbf{x})$ is provided by the element predictor as in Eq. 2 .

For entropy coding, the entropy value indicates the length of compressed data. Consequently, the entropy of the unified prior and all data elements should be utilized to calculate the training loss, which is formulated as Eq. 5:

where $\mathcal{H}_{d,i}$ is the entropy value of the $i$-th data element. $N$ is the totoal number of nodes, which is used for normalization. The above loss function encourages our framework to reduce the entropy, i.e., the compression ratio.

We begin by devising the context model for the node data predictor. To satisfy the context condition, the node data predictor should be a causal system. Specifically, when predicting any node, we can aggregate information from previous nodes to predict the current one. On the other hand, information from latter nodes should be excluded from aggregation. Based on the graph convolution, we devise the Graph Context Convolution (GCC), which can well aggregate context information in node data while satisfying the context condition.

Vanilla graph convolution passes messages according to the adjacency matrix of the graph. A straightforward revision to satisfy the context condition is to cut the message passing flow from the latter to former nodes. In this way, we can obtain a causal message passing flow. To this end, the solution for GCC is proposed by removing the lower triangle part of the adjacency matrix before performing message passing, represented by Eq. 6:

where $\mathbf{X_{n}}$ is node features at layer n, and $H$ is the function that generates the message passing matrix, which varies with different graph convolution operations. $\mathbf{\hat{A}}$ is the adjacency matrix removing the lower triangle part, and $\mathbf{\theta}$ denotes the convolution kernel parameters. Note that any self-loop in the message passing matrix $H(\mathbf{\hat{A}})$ should also be removed since the decompressor cannot gather information from the node itself when decoding. Specifically, for GCN-basedDBLP:conf/iclr/KipfW17 GCC operation, it can be formulated by:

where $\mathbf{D}$ is the degree matrix of $\mathbf{\hat{A}}$.

Fig. 4 illustrates an example of the GCC for node data context modeling. During decompression, when decompressing node-1, node-4 is unknown, so $\text{edge}_{4\rightarrow 1}$ is unavailable in message passing. When decompressing node-2, node-1 has already been decompressed while node-4 has still not, so $\text{edge}_{1\rightarrow 2}$ is used for message passing and $\text{edge}_{4\rightarrow 2}$ is unavailable. When decoding node-3&4, all connecting edges are available. By adding all message passing matrices used in the decompressing steps, we can obtain $\mathbf{\hat{A}}$ (in Eq. 6) for compression.

Although the implementation of GCC is relatively simple and may introduce some context information losses in practice , it’s still a lightweight and effective operation to extract node contextual information by causal message passing. Moreover, we further apply a pre-processor to address the issue of context information loss (details will be discussed in Section 4.4).

For graph structure, existing works usually apply graph autoencodersDBLP:journals/corr/KipfW16a to embed graph structure into a latent vector, which could be used to predict its adjacency matrix, as illustrated in Fig. 5a. However, they could only process undirected graphs, thus cannot be applied to directed graph structure in most scene graphs. Therefore, we devise the Directed Graph Context Autoencoder (DGCAE) based on GCC to embed context information for the directed graph structure in scene graphs.

First, in order to process directed graph structure in scene graph data, we design the Directed Graph Autoencoder (DGAE). For directed graphs, the graph convolution passes messages from the out-nodes to the in-nodes along the edge direction. Therefore, the node messages are passed into the nodes pointed by directed edges. For this reason, we call such features obtained by node message passing as ‘node-in’ latent feature (denoted by $\mathbf{Z}_{i}$). Furthermore, in order to properly decode $\mathbf{Z}_{i}$ as an asymmetric prediction matrix for a directed graph, another ‘node-out’ latent feature (denoted by $\mathbf{Z}_{o}$) is then required to decode $\mathbf{Z}_{i}$ to obtain the prediction of the directed graph adjacency matrix. $\mathbf{Z}_{o}$ can be obtained by directly transforming the node data with an fully-connected encoder. The decoder can be implemented by a dot-product operation between $\mathbf{Z}_{o}$ and $\mathbf{Z}_{i}$:

where $\sigma$ is the sigmoid function. The whole pipeline of DGAE is illustrated in Fig. 5b.

Second, to model context information for directed graphs, we further propose a DGAE variant by replacing the GCN encoder in DGAE with a GCC-based encoder. In this way, the graph structure context information can be also embedded into a latent vector (denoted as $\mathbf{Z}_{c}$), replacing $\mathbf{Z}_{i}$ in DGAE. However, since the proposed GCC disallows the message passing from latter nodes to former nodes, $\mathbf{Z}_{c}$ can only be used to decode edges that satisfy the context condition. Therefore, the lower triangle part of the decoded adjacency matrix cannot utilize $\mathbf{Z}_{c}$. To deal with the lower triangle part, another autoencoder composed of fully-connected layers should be applied to obtain a pseudo node-in feature $\mathbf{Z}_{pi}$. By combining the matrices produced by both decoder, the final prior estimation result for the adjacency matrix are decoded. The decoding process could then be represented as in Eq. 9.

We call such variation of DGAE as Directed Graph Context Autoencoder (DGCAE), as shown in Fig. 5c.

By applying DGCAE, prediction of the graph structure in the upper half of the adjacency matrix could utilize both context information and prior information, and the lower half is predicted with prior information only. Note that DGCAE already includes the context model, prior model and merge model, as illustrated with dashed boxes in Fig. 5c, and forms a complete parameter estimation module.

Since scene graphs also contain numerous edge data, an edge data context modeling approach is necessary for the edge data predictor. However, different from node data, whose shape is independent of the graph structure, the shape of edge data is closely related to the graph structure. Similarly, the proposed GCC-based predictor cannot be directly applied to edge data context modeling. A naive idea is to convert edge data to adjacency matrices and apply the proposed DGCAE to process them. However, since most scene graphs are relatively sparse, the converted matrices may occupy much more memory than the original edge data.

Similar to graph nodes, edges that share the same node also possess a close semantic correlation. Therefore, we can also model the context information for edge data by message passing. To this end, we propose an algorithm to convert edge data to node data, which enables the GCC-based network to model edge data context information. We call such conversion as Edge Graph Construction (EGC). Specifically, it constructs a new graph by keeping the original edge data on new nodes and adding connections between edges that share the same node. Fig. 6 illustrates an example of EGC. Also, the detailed process of EGC is described in Algorithm 1.

Note that in the node pairing step, both node and edge data in the original graph can be used to construct the node data in the new graph. If node data are not available at the stage of edge data predictor, we use the unified prior information instead. In the edge connection step, the newly constructed graph is generated according to the edge adjacency in the original graph. As shown in Fig. 6, for adjacent edges that forms a path (e.g., $\text{edge}_{1\rightarrow 2}$ and $\text{edge}_{2\rightarrow 3}$), EGC will connect them with directed edges. In contrast, adjacent edges that share the same out-node or in-node (e.g., $\text{edge}_{2\rightarrow 3}$ and $\text{edge}_{2\rightarrow 5}$ share the same out-node 2) will be connected with undirected edges.

After applying EGC, we can reuse the Graph Context Convolution (GCC) to aggregate the context information in edge data.

As discussed in Section 4.1, the Graph Context Convolution (GCC) follows causal message passing flows. Therefore, all edges that start from latter nodes to former nodes (or non-causal edges) will be dismissed in message passing, which may introduce losses to context information. In order to address this issue for GCC-based context models, a preprocessing step is proposed to preserve more context information.

Specifically, we can preserve these non-causal edges by performing a topology sorting on the original graph, which makes all edges to be causal. An example is shown in Fig. 7a, we can observe that node-3 and node-4 are swapped by topology sorting, and $\text{edge}_{4\rightarrow 3}$ that should have been originally removed by GCC is preserved as $\text{edge}_{3\rightarrow 4}$ after sorting. However, topology sorting requires that the graph should be an acyclic graph. To satisfy this requirement while preserving more context information, we need to remove as few edges as possible before topology sorting, namely the maximum spanning directed acyclic graph problem. Such a problem has been studied in previous researches DBLP:conf/starsem/Schluter15 and has been acknowledged as an NP-hard problem. To address this issue, we further propose a simple but efficient solution.

For scene graphs, a cycle tends to appear within a small group of objects that perform similar actions. It indicates that components with cycles, which form strongly connected components, are relatively small compared to the whole graph, as shown on the left side of Fig. 7b. Based on this observation, a simple workaround is applied to bypass the acyclic requirement in topology sorting. We present an example of this workaround in Fig. 7b. Specifically, by extracting all strongly connected components (SCC) from the graph, we can construct an acyclic condensed graph according to all extracted SCC, which can be sorted in topological order. In this way, we can perform topology sorting for most of the original graph nodes and obtain a reordered graph with more causal edges. The reordered graph is then compressed by our proposed framework. By virtue of the above prepossessing, we can significantly reduce the context losses resulting from GCC and thus preserve more context information of scene graphs. Note that the sorting indices are not included in the compressed data, because by reordering nodes in the graph, the topology of the new graph still equals the original one.

We adopt Kosaraju’s algorithmSHARIR198167 to implement the extraction of the strongly connected components. The reason for using this algorithm is that it could extract components and sort the extracted components in topological order in one step. The sorted components are then lined up to form a sorting sequence, which is used to reorder the original graph. The detailed process is described in Algorithm 2.

Naturally, further sorting could be performed on the extracted strongly connected components to achieve further improvement with a brute-force method, but in many cases this step is unnecessary, as most nodes have already been sorted by the proposed method, and further sorting will not bring remarkable improvement. More details can be shown in Section 6.4.2.

In this section, according to the type of competitors, we report the overall performances from two aspects.

In order to study the effectiveness of different modules in our proposed framework, we conduct experiments by removing some modules or replacing some of them with other solutions to gain further insight. First, the proposed joint compression framework could be replaced by other learning-based general-purpose compression frameworks such as DzipDBLP:conf/dcc/GoyalTCO20 . Second, the graph context modeling techniques as well as the corresponding preprocessor for context improvement discussed in Section 4 can be removed so that each element predictor uses only the initial prediction result from the unified prior extractor. Finally, the learned distribution model discussed in Section 5 could be replaced by a simple Gaussian distribution model. Experiments are then conducted with the above variants. Results are shown in Table 5.

We can see that the proposed joint compression framework outperforms the general-purpose compression framework DzipDBLP:conf/dcc/GoyalTCO20 in all cases, even without the graph context model and learned distribution model. This is achieved by extracting vital information from the scene graph data with the unified prior extractor and designing adaptive element predictors according to different data elements. DzipDBLP:conf/dcc/GoyalTCO20 though, is designed for general-purpose data compression with a serial context model that does not fit graph data well and ignoring different data distributions, thus does not perform well on scene graph compression.

The graph context models also contribute significantly to the proposed framework in all datasets. The main reason is that our proposed context models are designed for all element predictors to reduce context redundancy for all data elements. Especially, the relative improvement is more distinct in VG-Gen, VG-GenRel, and HiEve, both of them contain generated scene graphs. It indicates that the adopted scene graph generators are more context-aware. Therefore, those generated scene graphs could benefit more from the proposed graph context models.

The proposed learned distribution model contributes less than the context model, as this module could only benefit object location compression in those datasets. Therefore, for datasets that contain more node data (e.g., VG and VG-Gen), the learned distribution model brings larger improvements than datasets that own fewer node data (e.g., VG-GenRel and HiEve).

In order to study the effectiveness of each key module (graph context model , preprocessor and distribution model) in our proposed framework, a series of ablation studies are conducted.

The proposed framework contains a few hyperparameters that can be adjusted for different datasets to achieve better compression results. The hyperparameter set mainly includes the unified prior extractor model parameters and the compression order of the element predictors. A series of ablation studies are conducted to reveal the effect of different hyperparameters on our proposed framework.

The datasets generated during and/or analysed during the current study are all available publicly. Please refer to Section 6.1.1 for further details.