HIGGS: HIerarchy-Guided Graph Stream Summarization

The architecture of HIGGS is essentially an aggregated B-tree with the following properties:

• •

  Each tree node corresponds to a time interval and is with at most $\theta$ children, along with a compressed matrix, summarizing the graph stream for that interval.

• •

  A non-leaf node with $k$ children contains $k-1$ keys, which are represented by timestamps acting as separation values dividing its subtrees. All leaves appear on the same layer, i.e., the bottom layer.

• •

  Each bucket, essentially an element in the compressed matrix, contains a set of entries, each representing the tuple $(f(s_{i}),f(d_{i}),t_{i},w_{i})$ for an edge $e_{i}$.

  • –

    $f(s_{i})$ and $f(d_{i})$ are fingerprints (compact identifiers) of the source and destination vertices, respectively;

  • –

    $t_{i}$ is the offset relative to the matrix’s start time;

  • –

    $w_{i}$ is the weight of $e_{i}$.

The non-leaf nodes of HIGGS, like leaf nodes, store multiple keys and pointers to compressed matrices. These keys represent the aggregated temporal scope of their child nodes. Each non-leaf node matrix summarizes data from its child nodes, devoid of timestamps, containing entries as $(f(s_{i}),f(d_{i}),w_{i})$.

Construction. The construction of HIGGS is item-based and bottom-up, utilizing the sequence in which graph streams arrive. When an edge arrives, it is inserted into the newest compression matrix at the first level in $O(1)$ time. If the matrix is full, a new one is constructed to store the edge, and its timestamp is propagated upwards. If the root becomes full, a new root is created with the original one as its child. Filler nodes ensure that matrices connected to leaf nodes remain at the leaf layer, keeping the tree balanced.

Insertion. Algorithm 1 outlines the edge insertion procedure for HIGGS. Initially, HIGGS consists of a single root node, linked to an empty compression matrix $M$. When an incoming edge $e_{i}=(s_{i},d_{i},w_{i},t_{i})$ arrives, it is processed using a hash function $H(\cdot)$ to obtain the hash values $H(s_{i})$ and $H(d_{i})$ for $s_{i}$ and $d_{i}$, respectively. The $F_{1}$-bit suffixes of $H(s_{i})$ and $H(d_{i})$ are then concatenated to create a fingerprint pair $(f(s_{i}),f(d_{i}))$. The remaining bits are used to compute the address pair $(h(s_{i}),h(d_{i}))$. For efficiency, we derive bit operations for calculating a vertex’s fingerprint and address:

Based on the calculated address, we can quickly locate the bucket $M[h(s_{i})][h(d_{i})]$. We then check each entry in this bucket that has a value. If an entry’s fingerprint pair matches $(f(s_{i}),f(d_{i}))$ and the recorded timestamp is $t_{i}$, we increase its stored weight by $w_{i}$. If no such entry is found, we insert $(f(s_{i}),f(d_{i}),t_{i},w_{i})$ into an empty entry. If all entries are occupied without a match, the insertion fails, prompting the creation of a new leaf node that connects to a matrix of the same size as $M$, into which the edge is inserted in the same manner. The timestamp $t_{i}$ is then transmitted to the parent node to serve as an key during queries.

Aggregation. The aggregation between a parent node and its children plays a vital role in enhancing query accuracy and efficiency. The aggregation process follows a bottom-up way:

• •

  Leaf Nodes: Each leaf node’s compressed matrix is directly computed based on raw stream items.

• •

  Non-Leaf Nodes: Matrices of upper-level nodes aggregate those of their children, encompassing the descendant nodes within their subtrees.

Algorithm 2 describes the details of the aggregation process. For a node at the $(i+1)$-th layer, we construct a $\sqrt{\theta}d_{i}\times\sqrt{\theta}d_{i}$ matrix to aggregate the $\theta$ matrices of size $d_{i}\times d_{i}$ from its children at the $i$-th layer. We observe that a larger matrix increases the number of address bits. An intuitive approach is to shift additional fingerprint bits into the address, thereby reducing the storage required for fingerprints. Binary representation, which is conducive to data compression and expansion, facilitates data aggregation through simple shift operations. To avoid introducing additional errors during the aggregation process, we maintain the matrix size as a power of two. Assuming that the aggregation process reduces the fingerprint storage by $R$ bits, increasing the address bits by $R$, the matrix size becomes $4^{R}$ times its original size. Therefore, $\theta$ needs to be a power of four. At the $i$-th layer, the fingerprint length is restricted to $F_{1}-(i-1)R$ bits. Here, $\theta$ denotes the maximum number of child nodes connected to a node, and $d_{i}$ and $F_{i}$ represent the matrix size and fingerprint length at the $i$-th level, respectively. It is noteworthy that the aggregation operation does not involve the storage of timestamps.

TRQ Evaluation Framework. For any specified temporal range query, including edge, vertex, path, and subgraph queries, it can be broken down into a series of sub-range queries using the boundary search algorithm (Algorithm 3) run on HIGGS. Each sub-range query is performed on its respective compressed matrix, thereby transforming the problem into querying across a set of matrices. Different types of query primitives, such as edge and vertex queries, correspond to different methods of accessing the compressed matrices. The boundary search algorithm involves two main steps:

First, starting from the root node of HIGGS, the algorithm identifies the child nodes entirely covered by the queried range $[t_{s},t_{e}]$ and adds them to the query list $X$. If $[t_{s},t_{e}]$ falls within the range represented by a child node, further queries are made into the child node, continuing until $t_{s}$ and $t_{e}$ no longer reside within the range of the same child node.

Second, the algorithm then identifies the child nodes containing $t_{s}$ or $t_{e}$ and adds the child nodes of parent nodes that are contained by $[t_{s},+\infty]$ or $[-\infty,t_{e}]$ to $X$. If $t_{s}$ or $t_{e}$ exactly matches the start or end point of a node’s range, the corresponding query can be directly terminated; otherwise, the search extends to the next level of child nodes containing $t_{s}$ or $t_{e}$. Upon reaching the leaf nodes, in addition to adding nodes that meet the previously mentioned conditions, nodes containing $t_{s}$ or $t_{e}$ are also added to $X$. This concludes the decomposition process of the temporal range.

Next, we proceed to discuss how the query framework is implemented for TRQ primitives, i.e., edge and vertex queries.

Edge Queries. For a temporal query range $[t_{s},t_{e}]$, an edge query from $s$ to $d$ is conducted on matrix $M$. Using Formula (1) and a shift operation, we derive the fingerprint $(f(s),f(d))$ and address $(h(s),h(d))$, locating the bucket $M[h(s)][h(d)]$.

Two cases arise: 1) Non-leaf node: If $M$ is a non-leaf, we check if any entry’s fingerprint matches $(f(s),f(d))$—if so, return its weight; otherwise, it returns $0$. 2) Leaf node: If $M$ is a leaf, we additionally check if the timestamp falls within $[t_{s},t_{e}]$. If both the fingerprint and timestamp match, the entry is considered valid, and we proceed as in the non-leaf case.

Vertex Queries. For the temporal query range $[t_{s},t_{e}]$, queries are made on a source or destination vertex $v$ based on matrix $M$ of size $d^{\prime}$. Using Formula (1) and a shift operation, we ascertain $v$’s fingerprint $f(v)$ and address $h(v)$, then locate the corresponding row in $M$:

Each entry in the row’s buckets is traversed. For non-leaf nodes, if the source vertex’s fingerprint matches $f(v)$, the weight is added to the result. For leaf nodes, both the fingerprint must match and the timestamp must fall within $[t_{s},t_{e}]$ for the weight to be included.

We propose three optimizations: multiple mapping buckets to enhance space efficiency, overflow blocks to improve accuracy, and parallelization to boost insertion throughput.

Multiple Mapping Buckets (MMB). When inserting an edge $e$, using only a single mapping bucket can lead to severe conflicts, potentially causing the insertion to fail and wasting significant space if the matrix is underutilized. To address this issue, we employ multiple mapping buckets, giving each edge multiple potential insertion positions. Specifically, we use the linear congruence method [47] to generate address sequences $\{h_{i}(s)|1\leq i\leq r\}$ for the source vertex $s$ and $\{h_{j}(d)|1\leq j\leq r\}$ for the destination vertex $d$, where $r$ represents the number of mapping positions of a vertex. The address sequences of the source and destination vertices are then combined pairwise to produce $r\times r$ mapping buckets. To track the position of each bucket in the address sequence, we record an index pair $(i,j)$. In this way, insertion only fails if all mapping buckets conflict, significantly reducing the likelihood of failure and improving space utilization of the matrix.

Overflow Blocks (OB). In the basic HIGGS framework, if an edge insertion at a leaf node fails, a new leaf is created to store the edge, and its timestamp is transmitted to the parent as a key to distinguish between leaf nodes. However, if multiple edges with identical timestamps have already been stored, this key becomes ineffective, leading to errors. To resolve this, we employ overflow blocks to aggregate edges that arrive simultaneously into a unified time range. If an edge insertion fails and has the same timestamp as the previous edge, an overflow block is created on the current leaf to store it; otherwise, a new leaf is created. The overflow block is essentially a small-scale compressed matrix, enabling more precise partitioning of the graph stream by timestamps and thereby enhancing query accuracy.

Parallelization. To enhance throughput, we adopt a parallel updating strategy, assigning each layer a separate thread that updates only the latest node. To maintain data consistency across the subtree, each element is first updated by the thread corresponding to leaf nodes before being processed by other threads. Since order preservation is needed only at the element level, parallel efficiency remains high. By utilizing parallelism, the throughput of HIGGS is increased substantially.

Space Savings through Aggregation. Compared to the storage methods employed in existing works, our design significantly reduces space costs.

Matrix Utilization Rate. The utilization rate of a matrix refers to the proportion of elements stored within the matrix relative to its total capacity. As discussed in Section IV-C, for a $d\times d$ compressed matrix, we enhance the utilization rate of the bucket by setting the number of mapping buckets for each edge to $p$. Simultaneously, each bucket contains $b$ entries. An insertion failure indicates that all $p$ mapping buckets have encountered conflicts, while all previous edges were successfully inserted. Let $A_{i}$ denote the probability that $i$-th edge is inserted successfully, and $1-A_{i}$ represent the probability of its failure. If $k$-th edge causes the first insertion failure, the probability of this occurrence, according to the geometric distribution, is:

Given that the compressed matrix containing $b\times d\times d$ elements, the expected utilization $E(\alpha)$ of the matrix is calculated as follows:

Here, $E(k)$ represents the expected number of elements successfully inserted into the matrix. Therefore, the formula quantifies the average utilization of the storage block by considering the successful insertions relative to the total number of available elements of its associated compressed matrix.

Space Complexity Analysis. The number of layers in HIGGS primarily depends on the quantity of leaf nodes and the out-degree of each node. Since the space occupied by a compressed matrix to which a non-leaf node points equals the cumulative space of the matrices of all its corresponding child nodes, the space overhead for each layer in HIGGS is fundamentally consistent. Let the average space utilization rate of each matrix pointed to by the leaf nodes be $\alpha$, and let the space occupied by the keys in HIGGS be $I$. The number of matrices at the leaf nodes is given by $n_{1}=\frac{|E|}{\alpha bd_{1}^{2}}$, leading to a space complexity for HIGGS of approximately $O(|E|\log\frac{|E|}{\alpha bd_{1}^{2}}+I)$. Since the space required for storing keys is significantly less than that needed for storing graph stream data, the overall space cost is $O(|E|\log\frac{|E|}{\alpha bd_{1}^{2}})$. After optimization with overflow blocks, assuming the average time span represented by each matrix is $L^{\prime}$ (where $1\leq L^{\prime}=\frac{\alpha bLd_{1}^{2}}{|E|}\leq L$), the formula can be derived as follows:

$O(E\log L)$ represents the space complexity of the existing framework. This establishes that the space complexity of HIGGS is superior to that of the existing framework. Additionally, the fewer edges arrive per unit time in the graph stream, that is, the larger the value of $L^{\prime}$, the lower the space cost becomes.

The time complexity of HIGGS depends on the number of matrices accessed at each layer. Assuming the length of a given temporal range query is $L_{q}$, at most $2(\theta-1)\log_{\theta}\frac{L_{q}}{L^{\prime}}$ matrices are queried, leading to a time complexity approximates O($\log\frac{L_{q}}{L^{\prime}}$).

Furthermore, the query at each layer in the existing framework targets the data structure that stores the entire graph stream, resulting in larger scales. Consequently, when addressing temporal range queries related to topological structures, this framework accesses a greater number of buckets, which leads to lower query efficiency. In contrast, the total size of all matrices to be queried in HIGGS is consistent with the size of the graph stream within that time range, making it smaller in scale and more efficient to access.

As streaming edges continuously arrive, HIGGS increases the number of leaf nodes to store data while aggregating at non-leaf nodes. The number of layers in HIGGS is $O(\log n_{1})$, with a space complexity of $O(E\log n_{1})$, an update time complexity of $O(E\log n_{1})$, and a query time complexity of $O(\log\frac{L_{q}}{L^{\prime}})$, where $n_{1}$ is the number of leaf nodes, $L_{q}$ is the query range length, and $L^{\prime}$ is the average time range length represented by each leaf node. To accelerate updates, we employ parallel optimization (see Section IV-C), where each layer is assigned a dedicated thread.

Collision Rate Analysis. In this section, we analyze the collision rates for nodes and edges maintained in HIGGS. Moreover, we show that HIGGS has one-sided error, i.e., it only overestimates the true query results; it never underestimates results.

1) Node Collision. For a given temporal query range $[t_{s},t_{e}]$ and an edge $e_{i}=(s_{i},d_{i},w_{i},t_{i})$, conflicts for $e_{i}$’s source (or destination) node arise only from other edges within this time range. Specifically, if there exists an edge $e_{j}=(s_{j},d_{j},w_{j},t_{j})$ conflicting with $e_{i}$, the following conditions must be met:

• •

  $s_{i}\neq s_{j}$ (or $d_{i}\neq d_{j}$);

• •

  $h(s_{i})=h(s_{j})$ (or $h(d_{i})=h(d_{j})$);

• •

  $t_{s}\leq t_{i},t_{j}\leq t_{e}$.

Suppose that within the time range $[t_{s},t_{e}]$, the number of source (or destination) nodes different from $s_{i}$ (or $d_{i}$) is denoted as $k$. The total number of distinct source (or destination) nodes in the entire graph stream is $K$, and the size of the value range of the hash function is $Z$, where $Z=d_{1}2^{F_{1}}$. The probability of a node conflicting with $s_{i}$ is $\frac{1}{Z}$. Given that there are $k$ such nodes within $[t_{s},t_{e}]$, the node conflict probability for $e_{i}$ is,

From the equation, it is evident that increasing the number of fingerprint bits $F_{1}$ or the size of the compression matrix $d_{1}$ in the first layer will reduce the probability of node conflicts.

2) Edge Collision. Similar to node collision, when provided with a specified time range $[t_{s},t_{e}]$ and an edge $e_{i}=(s_{i},d_{i},w_{i},t_{i})$ for a query, edges outside beyond the time range will not conflict with $e_{i}$. If there is an edge $e_{j}$ that conflicts with $e_{i}$, it must satisfy the following conditions:

• •

  $s_{i}\neq s_{j}$ or $d_{i}\neq d_{j}$

• •

  $h(s_{i})=h(s_{j})$ and $h(d_{i})=h(d_{j})$

• •

  $t_{s}\leq t_{i},t_{j}\leq t_{e}$

Then, the collision probability of edge $e_{i}$ is:

Here, $S$ denotes the number of edges that, within the range $[t_{s},t_{e}]$, share the same source or destination node as $e_{i}$ and have a probability of conflict with $e_{i}$ of $\frac{1}{Z}$. Conversely, the probability of conflict for the remaining edges, which are distinct from $e_{i}$, is $\frac{1}{Z^{2}}$. Additionally, $C^{\prime}$ denotes the number of edges distinct from $e_{i}$ within this range. $\Phi_{o}$ and $\Phi_{i}$ respectively signify the maximum out-degree and in-degree across the entire graph stream, while $C$ denotes the count of distinct edges throughout the graph stream. Similar to node collision, increasing the fingerprint size or matrix size in the first layer proves effective in reducing the probability of conflicts.

Error Bounds. Next, we analyze the error bounds for node and edge queries in HIGGS and the configuration of the fingerprint length $F_{1}$ and the matrix size $d_{1}$ of the leaf nodes.

1) Error Bound for Vertex Queries. We use $\hat{w}_{node}$ and $w_{node}$ to represent the estimated value and the actual value for vertex queries within the range $[t_{s},t_{e}]$ of length $L_{q}$, respectively. Given a parameter $\varepsilon$, we set $F_{1}=\log\left(\frac{e}{d_{1}\varepsilon}\right)$, resulting in $Z=\frac{e}{\varepsilon}$. We denote the number of edges within the range $[t_{s},t_{e}]$ as $E^{\prime}$ and the sum of their weights as $\|\mathit{w}\|^{\prime}$.

2) Error Bound for Edge Queries. Similar to vertex queries, We use $\hat{w}_{edge}$ and ${w}_{edge}$ to respectively denote the estimated value and the actual value for edge queries within the range $[t_{s},t_{e}]$ of length $L_{q}$.

Baselines. In the field of graph stream summarization supporting temporal range queries, SOTA methods include PGSS and Horae, with Horae excelling in query accuracy. However, their scalability is limited. Therefore, we create stronger baselines by extending the SOTA graph stream summarization structure, i.e., Auxo, to support temporal range queries through the incorporation of Horae’s range decomposition scheme, yielding AuxoTime. Additionally, Horae has an optimized variant, Horae-cpt, aimed at reducing space overhead, which prompted us to also develop AuxoTime-cpt. In the empirical study, we consider five competitive baselines, including PGSS, Horae, Horae-cpt, AuxoTime, and AuxoTime-cpt.

Datasets. We consider 3 real datasets commonly used for evaluating graph stream summarization, namely Lkml, Wiki-talk, and Stackoverflow, as shown in Table II. The Lkml dataset [48] is a communication network of the Linux kernel mailing list, where nodes represent users (email addresses) and directed edges denote replies, encompassing 63,399 users and 1,096,440 replies. The Wikipedia talk (WT in short) [48] dataset captures the communication network of the English Wikipedia, with nodes representing users and edges indicating messages written on other users’ talk pages, comprising 2,987,535 nodes and 24,981,163 messages. The Stackoverflow (SO in short) [48] dataset, collected from 2009 to 2016, represents an interaction network between users on Stack Overflow, featuring 2,601,977 nodes and 63,497,050 edges²²2The time slice for each dataset is set to 1 second to align with the time unit settings of each dataset..

Metrics. The metrics primarily include average absolute error (AAE), average relative error (ARE), average query time, throughput, and space cost. AAE and ARE are commonly used for evaluating the accuracy of graph summarization, quantifying the error between the true values and the queried values. If there are $p$ queries, each with a true value $f_{i}$ and an estimated value $\hat{f}_{i}$, then:

Average query time reflects the system’s performance on average query response for a series of query operations. Insertion throughput measures the number of elements processed per unit time, while insertion latency indicates the speed of processing insertions. Space overhead indicates the amount of main memory required to store or process data in the system.

Configuration. Experiments were done on a machine equipped with a 32-core 2.6 GHz Xeon CPU, 384 GB RAM, and a 960 GB SSD. Baseline parameters were set following corresponding papers. For HIGGS, we set the number of optional addresses for each node to $4$ and $b$ to $3$. Therefore, for each edge, $4$ bits are required to store the index pair, indicating the position. Due to the close correlation between $Z$ and the degree of conflicts, we configured parameter $d_{1}$ to $16$ and $F_{1}$ to $19$ to ensure that the $Z$ value of HIGGS aligns with those of the baselines. For vertex and edge queries, we vary the query range length $L_{q}$ from $10^{1}$ to $10^{7}$. For each $L_{q}$, we randomly generated 100K edge queries and 10K vertex queries. For path and subgraph queries, the path length is set to $[1,7]$ and subgraph size is set to $[50,350]$. For each path length and subgraph size, 1K queries are randomly generated. Each value reported is the average of 1,000 runs.

1) AAE vs. L: Fig. 15 (a–c) examine the accuracy of edge queries for all competitors in terms of AAE. In all experiments, the AAE of HIGGS is notably lower than those of other baselines, and this is consistent across all the three real graph stream datasets, i.e., Lkml, Wikipedia talk, and StackOverflow. For example, when the length of the query range $L_{q}$ equals $10^{4}$, the AAE of HIGGS is about $5$ orders of magnitude lower than that of the second-best competitor on Stackoverflow (Fig. 10(c)). Remarkably, the AAE of HIGGS is almost zero on the Lkml dataset (Fig. 10(a)), implying 100% query accuracy. The superior performance of HIGGS is attributed to the elimination of conflicts of streaming items in the upper layers. In contrast, other methodologies experience an accumulation of errors as the query range expands, resulting in inaccuracies caused by the error accumulation in the query processing. Since the number of structure layers and the length of the query range are logarithmically related, the AAE of these methods increases logarithmically as the query range expands. However, our approach avoids the inter-layer error accumulation. Even when the temporal range spans the entire graph stream, the conflicts in HIGGS are confined to the layer of leaf nodes. This also explains why, when the query range is exceptionally large, the advantage of HIGGS over other competitors becomes smaller, since the task of TRQ degenerates into the task of count estimations of conventional (non-temporal) graph summaries like TCM. Although the accuracy of HIGGS is slightly lower, it remains orders of magnitude better than the accuracies of its competitors. Furthermore, the AAE of Horae-cpt and AuxoTime-cpt are higher than those of Horae and AuxoTime, respectively, as they decompose the temporal query range into more sub-ranges, leading to increased query conflicts and, consequently, a relative increase in AAE.

2) ARE vs. L: Fig. 15 (d–f) depict the ARE results for edge queries across three datasets, as the query range length varies. Similar to the result observed in Fig. 15 (a–c), HIGGS consistently outperforms other baselines and maintains near-lossless performance across all three datasets, showcasing its advantage. It is noteworthy that in Fig. 15 (d–f), the ARE of the baselines initially increase and then decrease. This pattern is due to the relative error decreasing as the true edge weights increase with the lengthening of the query range.

3) Latency vs. L: Fig. 15 (g–i) illustrate the relationship between the query latency and the length of the query range, across all datasets. It shows that HIGGS and PGSS markedly outperform other baselines: when the query range length $L_{q}$ is set to $10^{4}$, HIGGS surpasses AuxoTime by an order of magnitude and improves on Horae by two orders of magnitude. The superior performance of HIGGS stems primarily from its ability to access fewer summary layers given the same query range, and each layer accesses fewer nodes/buckets compared to AuxoTime and Horae. PGSS exhibits competitive query latency (Fig. 15(g–i)), but it falls short in query accuracy (Fig. 15(a–f)). Additionally, Horae’s query efficiency is compromised by excessive accesses to the buffer structure, degrading its performance, as is the case with AuxoTime. Moreover, the query latency of AuxoTime-cpt and Horae-cpt exceed those of AuxoTime and Horae, respectively, because they decompose the query range into more sub-ranges, increasing the query complexity from $O(\log L)$ to $O(\log^{2}L)$.