How Do Hyperedges Overlap in Real-World Hypergraphs? - Patterns, Measures, and Generators

We review the concept of hypergraphs and then the Chung-Lu model, which our null model is based on.

While the CL model flips a coin for all possible $O(|\bar{V}|^{2})$ node pairs, the fast CL (FCL) model (Pinar et al., 2012) samples two nodes independently with probability proportional to the degree of each node. Then, it creates an edge between the sampled pair of nodes. This process is repeated $|\bar{E}|$ times, and the total time complexity is $O(|\bar{E}|)$. Even in graphs generated by the FCL model, the expected degree of each node $i$ is equal to $d_{i}$.

We use thirteen real-world hypergraphs from six different domains (Benson et al., 2018a) after removing duplicated or singleton hyperedges. Refer to Table 3 for some statistics of the hypergraphs.

• •

  email (email-Enron (Klimt and Yang, 2004) and email-Eu (Leskovec et al., 2005; Yin et al., 2017)): Each node is an email account and each hyperedge is a set of the sender and receivers of an email.

• •

  contact (contact-primary (Stehlé et al., 2011) and contact-high (Mastrandrea et al., 2015)): Each node is a person, and each hyperedge is a group interaction among individuals.

• •

  drugs (NDC-classes and NDC-substances): Each node is a class label (in NDC-classes) or a substances (in NDC-substances) and each hyperedge is a set of labels/substances of a drug.

• •

  tags (tags-ubuntu and tags-math): Each node is a tag, and each hyperedge is a set of tags attached to a question.

• •

  threads (threads-ubuntu and threads-math): Each node is a user, and each hyperedge is a group of users participating in a thread.

• •

  co-authorship (coauth-DBLP, coauth-geology (Sinha et al., 2015), and coauth-history (Sinha et al., 2015)): Each node is an author and each hyperedge is a set of authors of a publication.

We introduce HyperCL, a random hypergraph generator that extends the FCL model (see Section 3.1) to hypergraphs. We use random hypergraphs generated by HyperCL as null models throughout this work. As described in Algorithm 1, the degree distribution of nodes and the size distribution of hyperedges in a considered real-world hypergraph are given as inputs. For each $i$-th hyperedge $\tilde{e}_{i}$, its nodes are sampled independently, with probability proportional to the degree of each node (i.e., the probability is $d_{v}/\sum_{j=1}^{|V|}d_{j}$ for each node $v$) until the size of the hyperedge reaches $s_{i}$ (lines 1-1). Note that duplicated nodes are ignored so that each $i$-th hypergraph contains $s_{i}$ distinct nodes.

In hypergraphs generated by HyperCL, the size distribution of hyperedges is exactly the same as the input size distribution, and the degree distribution of nodes is also expected to be similar to the input degree distribution. Specifically, if we assume $\sum_{j=1}^{|V|}d_{j}\gg(\max_{k\in\{1,\cdots,|E|\}}s_{k})\cdot(\max_{k\in\{1,\cdots,|V|\}}d_{k}$) and let $\tilde{d_{v}}$ be the in a generated hypergraph,

Specifically, as seen in the figures in the first row of Table 4, when considering the egonets with the same number of hyperedges, they tend to contain fewer nodes in real-world hypergraphs than in randomized ones. Thus, the density, which is defined as the ratio of the number of hyperedges to the number of nodes tends to be higher in real-world hypergraphs than in randomized ones. In the figures, the slopes of the regression lines, which are close to the average egonet density, are steeper in real-world hypergraphs than in randomized ones.

To address this issue, we first present three axioms that any reasonable measure of the hyperedge overlaps should satisfy. Then, we propose overlapness, a new measure that satisfies all the axioms. The three axioms are formalized in Axioms 1, 2, and 3.

Note that density and the four additional widely-used measures listed in Table 5 do not satisfy all the axioms. Thus, we propose overlapness (see Definition 2) as a measure of the degree of hyperedge overlaps, and it satisfies all the axioms, as formalized in Theorem 1.

In overlapness, the sum of sizes of hyperedges, instead of the number of hyperedges, is considered. Notably, the overlapness of a hyperedge set is equivalent to the average degree of the distinct nodes in the set. In addition, overlapness is equivalent to weighted density if we assign the size of each hyperedge as its weight. Overlapness agrees with our intuition in the previous example. That is, for $\mathcal{E}_{1}=\{\{a,b,c\},\{a,b,c,d\},\{a,b,c,d,e\}\}$ and $\mathcal{E}_{2}=\{\{v,w,x\},\{x,y\},\{y,z\}\}$, $o(\mathcal{E}_{1})=12/5>o(\mathcal{E}_{2})=7/5$.

Given a pair or triple of nodes, how many hyperedges do overlap at them? In other words, how many hyperedges do contain the pair or triple? While the degree is generally defined as the number of hyperedges that contains each individual node, here we extend the concept to pairs and triples of nodes. Specifically, if we let $E_{S}:=\{e\in E:S\subseteq e\}$ be the set of hyperedges overlapping at a subset $S\subseteq V$ of nodes, then the degree of each node pair $\{i,j\}$ is defined as $d^{(2)}(\{i,j\}):=|E_{\{i,j\}}|$, and the degree of each node triple $\{i,j,k\}$ is defined as $d^{(3)}(\{i,j,k\}):=|E_{\{i,j,k\}}|$. The degree of a pair or triple can also be interpreted as the structural similarity between the nodes in the pair or triple. Intuitively, nodes are structurally more similar as they are included together in more hyperedges.

Examining the degree distributions of pairs and triples of nodes, instead of that of individual nodes, gives higher-order insights on how nodes as a set form hyperedges. In the third and fourth columns of Table 4, we provide the distributions of $d^{(2)}$ and $d^{(3)}$ in real-world hypergraphs and those in a corresponding randomized hypergraph. Our findings are summarized in Observations 3 and 4.

In addition to the visual inspection, we compute the log-likelihood ratio of three representative heavy-tailed distributions (power-law, truncated power-law, and log normal) against the exponential distribution, as suggested in (Alstott et al., 2014; Clauset et al., 2009). If the ratio is greater than $0$, the given distribution is more similar to the corresponding heavy-tailed distribution than an exponential distribution. As reported in Table 6, except for one case, at least one heavy-tailed distribution has a positive ratio, and in most cases the ratio is highest for truncated power-law distributions. These result support the claim that the degree distributions of pairs and triples of nodes is heavy-tailed and similar to truncated power-law distributions.

In fact, these results are intuitive. The more often a pair or triple of nodes interact together, the more likely they are to interact together again. For example, researchers that have co-authored multiple papers are likely to share common interests, which can lead to more collaborations in the future.

How are nodes that form hyperedges together related to each other? It is unlikely in real-world hypergraphs that each hyperedge is formed by nodes chosen independently at random. It is expected to exist a strong dependency among the nodes forming a hyperedge together. In order to investigate the dependency, we use the homogeneity of hyperedge, defined in Definition 3, to measure how structurally similar such nodes are.

The figures in the last row of Table 4 show the homogeneity of the hyperedges in real-world hypergraphs and corresponding randomized hypergraphs. As summarized in Observation 5, there is a tendency that the homogeneity of each hyperedge in real-world hypergraphs is greater than that in randomized ones. Moreover, we verify that the distribution of homogeneity is heavy-tailed (see Table 7), as in the previous subsection.

The homogeneity of hyperedges plays a key role in generating realistic hypergraphs, as described in the following section.

We propose \method, a realistic hypergraph generative model whose pseudocode is described in Algorithm 2. The key idea behind \methodis to extend HyperCL to multiple levels. Recall that HyperCL itself cannot accurately reproduce realistic overlapping patterns, as shown in Section 4.

Proof. Given any hyperedge $e$, \methodfirst randomly selects a suitable level with probability proportional to the given weight. Thus, the probability for the level $\ell$ to be selected is $w_{\ell}/W_{e}$. Once the level is determined, any of the $2^{\ell-1}$ groups in level $\ell$ is selected uniformly at random, i.e., with probability $1/2^{\ell-1}$. The probability for $e$ to be generated from $S_{g}^{(\ell)}$ is the product of the two probabilities, and thus Eq. (2) holds. $\blacksquare$

For each node $v$, let $\hat{d}_{v}^{(\ell)}$ be the number of hyperedges that contain the node $v$ among those generated at level $\ell$. Then, the degree $\hat{d}_{v}$ of $v$ in an output hypergraph is the sum of $\hat{d}_{v}^{(\ell)}$ over all levels, i.e., $\hat{d}_{v}=\sum_{\ell=1}^{L}\hat{d}_{v}^{(\ell)}$. Let $d_{max}:=\max_{k\in\{1,\cdots,|V|\}}d_{k}$ and $s_{max}=\max_{k\in\{1,\cdots,|E|\}}s_{k}$. Assume $|V|\gg 2^{L-1}\cdot d_{max}$ and $\sum_{j\in S_{g}^{(\ell)}}d_{j}\gg d_{max}\cdot s_{max}$ for all $S_{g}^{(\ell)}$.³³3If $|V|\gg 2^{L-1}\cdot d_{max}$, then $\sum_{k\in S_{g}^{(\ell)}}d_{k}/\sum_{j=1}^{|V|}d_{j}\approx 1/2^{\ell-1}$ for all $\ell\in\{1,\cdots,L\}$. Then,

where $S_{g}^{(\ell)}(v)$ is the group at level $\ell$ containing $v$. That is, $\hat{d}_{v}$ is expected to be close to $d_{v}$, as we confirm empirically in (onl, 2021).

• •

  For a pair or triple of nodes belonging to the same small group, the number of hyperedges overlapping at them is expected to be high. Thus, the distribution of the number of overlapping hyperedges at each pair or triple is expected to be skewed.

• •

  As hyperedges can be formed within a small group, which contains structurally similar nodes, the homogeneity of each hyperedge is expected to be high. Moreover, as the size of groups varies, the homogeneity of hyperedges is expected to vary depending on the size of the groups that they are generated from.

• •

  As the hyperedges in the egonet of each node $v$ are likely to contain nodes belonging to the same small group with $v$, their density and overlapness are expected to be high.  
  

Given an input hypergraph $G$, how can we set the parameters of \method(i.e., the number of levels $L$ and the weight of each level $\{w_{1},\cdots,w_{L}\}$) so that it generates a synthetic hypergraph $\hat{G}$ especially similar to a target real-world hypergraph? The parameters should be carefully tuned since the structural properties of the generated hypergraphs vary depending on their settings. To this end, we propose \methodauto, which automatically tunes the parameters.

Since there are infinitely many combinations of level weights $w_{1},\cdots,w_{L}$, we propose an efficient greedy optimization scheme described in Algorithm 3, where some fraction of hyperedges created at a lower level are replaced with those newly created at a higher level, repeatedly, until Eq. (3) converges.

Specifically, \methodautofirst generates a hypergraph by HyperCL, which is equivalent to \methodwith $L=1$ (line 3). This is equivalent to set $w_{1}$ to $1$ and set $w_{\ell}$ to $0$ for all $\ell>1$. Then at each level $\ell$ from $2$ to $L$, we search for an optimal fraction of hyperedges created at level $\ell-1$ to be replaced with those newly created at level $\ell$ (line 3). Note that only hyperedges of size $\frac{|V|}{2^{\ell-1}}$ or smaller can be replaced. If the replacement strictly decreases the hyperedge homogeneity distance, then \methodautoupdates the current synthetic hypergraph (line 3). This is equivalent to decrease $w_{\ell-1}$ and increase $w_{\ell}$ by the same amount. Otherwise, we return the current synthetic hypergraph (line 3). We fix the update resolution $p$ to $0.05$ throughout this work. We note that the quality of generated hypergraphs is empirically insensitive to the choices of $p$.

How well do the hypergraphs generated by \methodautoreproduce the structural properties of the input hypergraphs? We evaluate its effectiveness by comparing them with four strong baselines: HyperCL, HyperPA (Do et al., 2020), HyperFF (Kook et al., 2020), and naïvely tuned \method.⁴⁴4We set the number of levels $L$ same as \methodautoand assign the weights $\{w_{1},\cdots,w_{L}\}$ uniformly equal, i.e., $w_{i}=1/L$ $\forall 1\leq i\leq L$. We describe the detailed experimental settings at Appendix B.

To measure the similarity between the distributions derived from the real-world hypergraph and the generated hypergraph, we use the Kolmogorov-Smirnov D-statistic, defined as $D=\max_{x}\{|F^{\prime}(x)-F(x)|\}$, where $x$ is a value of the considered random variable, and $F^{\prime}$ and $F$ are the cumulative distribution functions of the real and corresponding generated distributions.

In this subsection, we analyze the scalability of \methodand \methodautoboth theoretically and experimentally. Noteworthy, we show empirically that both \methodand \methodautoscale almost linearly with the size of the considered hypergraph.

In fact, while some baselines are intractable in particular datasets, \methodand \methodautoare scalable enough to be executed in all considered datasets. The scalability of HyperPA heavily depends on the sizes of hyperedges, and thus does not work in hypergraphs that includes large-sized hyperedges (i.e., email-Eu, NDC-substances, coauth-DBLP, coauth-geology, and coauth-history). HyperFF depends on the number of nodes, and does not work in large datasets with many nodes (i.e., threads-ubuntu, threads-math, coauth-DBLP, coauth-geology, and coauth-history).

Given the number of levels and weights of each level, how much time does it take to run \method? Assume that all sets and maps are implemented using hash tables. For each hyperedge $e$, level $\ell$ and group $g$ are selected in $O(1)$ time. In addition, since each node is sampled independently, $|e|$ nodes are selected in $O(|e|\cdot(1+\epsilon))$ time, where $\epsilon$ is due to the possibility of collisions (i.e., nodes selected multiple times for a hyperedge). The term $\epsilon$ depends on the degrees of nodes and the sizes of hyperedges. We note that $\epsilon$ is empirically very small in the considered datasets. Hence, generating $|E|$ hyperedges takes $O(\sum_{e\in E}(|e|\cdot(1+\epsilon)))$ time. In \methodauto, we consider the replacement step. At each level, at most $\frac{1}{p}\cdot|E|=O(|E|)$ hyperedges are (temporarily) replaced, taking $O(\sum_{e\in E}(|e|\cdot(1+\epsilon)))$ time. Since the maximum number of levels is $\log_{2}|V|$, \methodautotakes $O(\log_{2}|V|\cdot\sum_{e\in E}(|e|\cdot(1+\epsilon)))$ time in total.