Strongly Local Hypergraph Diffusions for Clustering and Semi-supervised Learning

Two common scenarios in graph-based data analysis are: (i) What are the clusters, groups, modules, or communities in a graph? and (ii) Given some limited label information on the nodes of the graph, what can be inferred about missing labels? These statements correspond to the clustering and semi-supervised learning problems respectively, and while there exists a strong state of the art in algorithms for these problems on graphs (Zhou et al., 2003; Andersen et al., 2006; Gleich and Mahoney, 2015; Yang et al., 2020; Veldt et al., 2016, 2019; Joachims, 2003; Mahoney et al., 2012), research on these problems is currently highly active for hypergraphs (Yoshida, 2019; Li and Milenkovic, 2018; Veldt et al., 2020b; Ibrahim and Gleich, 2020; Yin et al., 2017; Chitra and Raphael, 2019; Zhang et al., 2017) building on new types of results (Hein et al., 2013; Li and Milenkovic, 2017; Veldt et al., 2020a) compared to prior approaches (Karypis et al., 1999; Zhou et al., 2006; Agarwal et al., 2006). The lack of flexible, diverse, and scalable hypergraph algorithms for these problems limits the opportunities to investigate rich structure in data. For example, clusters can be relevant treatment groups for statistical testing on networks (Eckles et al., 2017) or identify common structure across many types of sparse networks (Leskovec et al., 2009). Likewise, semi-supervised learning helps to characterize subtle structure in the emissions spectra of galaxies in astronomy data through characterizations in terms of biased eigenvectors (Lawlor et al., 2016). The current set of hypergraph algorithms are insufficient for such advanced scenarios.

Hypergraphs, indeed, enable a flexible and rich data model that has the potential to capture subtle insights that are difficult or impossible to find with traditional graph-based analysis (Benson et al., 2016; Yin et al., 2017; Li and Milenkovic, 2018; Veldt et al., 2020b; Yoshida, 2019). But, hypergraph generalizations of graph-based algorithms often struggle with scalability and interpretation (Agarwal et al., 2006; Hein et al., 2013) with ongoing questions of whether particular models capture the higher-order information in hypergraphs. Regarding scalablility, an important special case for that is a strongly local algorithm. Strongly local algorithms are those whose runtime depends on the size of the output rather than the size of the graph. This was only recently addressed for various hypergraph clustering and semi-supervised learning frameworks (Veldt et al., 2020b; Ibrahim and Gleich, 2020). This property enables fast (seconds to minutes) evaluation even for massive graphs with hundreds of millions of nodes and edges (Andersen et al., 2006) (compared with hours). For graphs, perhaps the best known strongly local algorithm is the Andersen-Chung-Lang (henceforth, ACL) approximation for personalized PageRank (Andersen et al., 2006) with applications to local community detection and semi-supervised learning (Zhou et al., 2003). The specific problem we address is a mincut-inspired hypergraph generalization of personalized PageRank along with a strongly local algorithm to rapidly approximate solutions. Our formulation differs in a number of important ways from existing Laplacian (Zhou et al., 2006) and quadratic function-based hypergraph PageRank generalizations (Li and Milenkovic, 2018; Li et al., 2020; Takai et al., 2020).

Although our localized hypergraph PageRank is reasonably simple to state formally (§3.2), there are a variety of subtle aspects to both the problem statement and the algorithmic solution. First, we wish to have a formulation that enables the richness of possible hypergraph cut functions. These hypergraph cut functions are an essential component to rich hypergraph models because they determine when a group of nodes ought to belong to the same cluster or obtain a potential new label for semi-supervised learning. Existing techniques based on star expansions (akin to treating the hypergraph as a bipartite graph) or a clique expansion (creating a weighted graph by adding edges from a clique to the graph for each hyperedge) only model a limited set of cut functions (Agarwal et al., 2005; Veldt et al., 2020a). More general techniques based on Lovász extensions (Li and Milenkovic, 2018; Li et al., 2020; Yoshida, 2019) pose substantial computational difficulties. Second, we need a problem framework that gives sparse solutions such that they can be computed in a strongly local fashion and then we need an algorithm that is actually able to compute these—the mere existence of solutions is insufficient for deploying these ideas in practice as we wish to do. Finally, we need an understanding of the relationship between the results of this algorithm and various graph quantities, such as minimal conductance sets as in the original ACL method.

To address these challenges, we extend and employ a number of recently proposed hypergraph frameworks. First, we show a new result on a class of hypergraph to graph transformations (Veldt et al., 2020a). These transformations employ carefully constructed directed graph gadgets, along with a set of auxiliary nodes, to encode the properties of a general class of cardinality based hypergraph cut functions. Our simple new result highlights how these transformations not only preserve cut values, but preserve the hypergraph conductance values as well (§3.1). Then we localize the computation in the reduced graph using a general strategy to build strongly local computations. This involves a particular modification often called a “localized cut” graph or hypergraph (Liu and Gleich, 2020; Veldt et al., 2020b; Andersen and Lang, 2008; Fountoulakis et al., 2020). We then use a squared $2$-norm (i.e. a quadratic function) instead of a $1$-norm that arises in the mincut-graph to produce the hypergraph analogue to strongly local personalized PageRank. Put another way, applying all of these steps on a graph (instead of a hypergraph) is equivalent to a characterization of personalized PageRank vector (Gleich and Mahoney, 2014).

Once we have the framework in place (§3.1,§3.2), we are able to show that an adaptation of the push method for personalized PageRank (§3.3) will compute an approximate solution in time that depends only on the localization parameters and is independent of the size of a hypergraph with fixed maximum hyperedge size (Theorem 3.5). Consequently, the algorithms are strongly local.

The final algorithm we produce is extremely efficient. It is a small factor (2-5x) slower than running the ACL algorithm for graphs on the star expansion of the hypergraph. It is also a small factor (2-5x) faster than running an optimized implementation of the ACL algorithm on the clique expansion of the hypergraph. Nevertheless, for many instances of semi-supervised learning problems, it produces results with much larger F1 scores than alternative methods. In particular, it is much faster and performs much better with extremely limited label information than a recently proposed flow-based method (Veldt et al., 2020b).

Summary of additional contributions. In addition to providing a strongly local algorithm for the squared $2$-norm (i.e. a quadratic function) in §3.2, which gives better and faster empirical performance (§7), we also discuss how to use a $p$-norm (§6) instead. Finally, we also show a Cheeger inequality that relates our results to the hypergraph conductance of a nearby set (§4).

Our method is the first algorithm for hypergraph clustering that includes all of the following features: it is (1) strongly-local, (2) can grow a cluster from a small seed set, (3) models flexible hyperedge cut penalties, and (4) comes with a conductance guarantee.

A motivating case study with Yelp reviews. We begin by illustrating the need and utility for the methods instead with a simple example of the benefit to these spectral or PageRank-style hypergraph approaches. For this purpose we consider a hypothetical use case with an answer that is easy to understand in order to compare our algorithm to a variety of other approaches. We build a hypergraph from the Yelp review dataset (https://www.yelp.com/dataset). Each restaurant is a vertex and each user is a hyperedge. This model enables users, i.e. hyperedges, to capture subtle socioeconomic status information as well as culinary preferences in terms of which types of restaurants they visit and review. The task we seek to understand is either an instance of local clustering or semi-supervised learning. Simply put, given a random sample of 10 restaurants in Las Vegas Nevada, we seek to find other restaurants in Las Vegas. The overall hypergraph has around 64k vertices and 616k hyperedges with a maximum hyperedge size of 2566. Las Vegas, with around 7.3k restaurants, constitutes a small localized cluster.

We investigate a series of different algorithms that will identify a cluster nearby a seed node in a hypergraph: (1) Andersen-Chung-Lang PageRank on the star and clique expansion of the hypergraph (ACL-Star, ACL-Clique, respectively), these algorithms are closely related to ideas proposed in (Zhou et al., 2006; Agarwal et al., 2006); (2) HyperLocal, a recent maximum flow-based hypergraph clustering algorithm (Veldt et al., 2020b); (3) quadratic hypergraph PageRank (Li et al., 2020; Takai et al., 2020) (which is also closely related to (Hein et al., 2013)), and (4) our Local Hypergraph-PageRank (LHPR). These are all strongly local except for (3), which we include because our algorithm LHPR is essentially the strongly local analogue of (3).

The results are shown in Figure 1. The flow-based HyperLocal method has difficulty finding the entire cluster. Flow-based methods are known to have trouble expanding small seed sets (Veldt et al., 2016; Fountoulakis et al., 2020; Liu and Gleich, 2020) and this experiment shows that same behavior. Our strongly local hypergraph PageRank (LHPR) slightly improves on the performance of a quadratic hypergraph PageRank (QHPR) that is not strongly local. In particular, it has 10k non-zero entries (of 64k) in its solution.

This experiment shows the opportunities with our approach for large hypergraphs. We are able to model a flexible family of hypergraph cut functions beyond those that use clique and star expansions and we equal or outperform all the other methods. For instance, another more complicated method (Ibrahim and Gleich, 2020) designed for small hyperedge sizes showed similar performance to ACL-Clique (F1 around 0.85) and took much longer.

Let $G=(V,E,w)$ be a directed graph with $|V|=n$ and $|E|=m$. For simplicity, we require weights $w_{ij}\geq 1$ for each directed edge $(i,j)\in E$. We interpret an undirected graph as having two directed edges $(i,j)$ and $(j,i)$. For simplicity, we assume the vertices are labeled with indices $1$ to $n$, so that we may use these labels to index matrices and vectors. For instance, we define $\boldsymbol{\mathrm{d}}$ as the length-$n$ out-degree vector where its $i$th component $d_{i}=\sum_{j\in V}w_{ij}$. The incidence matrix $\boldsymbol{\mathrm{B}}\in\{0,-1,1\}^{m\times n}$ measures the difference of adjacent nodes. The $k$th row of $\boldsymbol{\mathrm{B}}$ corresponds to an edge, say $(i,j)$, and has exactly two nonzero values, 1 for the node $i$ and -1 for the node $j$. (Recall that we have directed edges, so the origin of the edge is always $1$ and the destination is always -1.)

Let $\mathcal{H}=(V,\mathcal{E})$ be a hypergraph where each hyperedge $e\in\mathcal{E}$ is a subset of $V$. Let $\zeta=\max_{e\in\mathcal{E}}|e|$ be the maximum hyperedge size. With each hyperedge, we associate a splitting function $f_{e}$ that we use to assess an appropriate penalty for splitting the hyperedge among two labels or splitting the hyperedge between two clusters. Formally, let $S$ be a cluster and let $A=e\cap S$ be the hyperedge’s nodes inside $S$, then $f_{e}(A)$ penalizes splitting $e$. A common choice in early hypergraph literature was the all-or-nothing split, which assigns a fixed value if a hyperedge is split or zero if all nodes in the hyperedge lie in the same cluster (Hadley, 1995; Ihler et al., 1993; Lawler, 1973): $f_{e}(A)=0$ if $A=e$ or $A=\emptyset$ and $f_{e}(A)=1$ otherwise (or an alternative constant). More recently, there have been a variety of alternative splitting functions proposed (Li and Milenkovic, 2018, 2017; Veldt et al., 2020a) that provide more flexibility. We discuss more choices in the next section (§3.1). With a splitting function identified, the cut value of any given set $S$ can be written as $\text{cut}_{\mathcal{H}}(S)=\sum_{e\in E}f_{e}(e\cap S)$. The node degree in this case can be defined as $d_{i}=\sum_{e:i\in e}f_{e}(\{i\})$ (Li and Milenkovic, 2017; Veldt et al., 2020b), though other types of degree vectors can also be used in both the graph and hypergraph case. This gives rise to a definition of conductance on a hypergraph

where $\textbf{vol}(S)=\sum_{i\in S}d_{i}$. This reduces to the standard definition of graph conductance when each edge has only two nodes ($\zeta=2$) and we use the all-or-nothing penality.

Diffusion algorithms for semi-supervised learning and local clustering. Given a set of seeds, or what we commonly think of as a reference, set $R$, a diffusion is any method that produces a real-valued vector $\boldsymbol{\mathrm{x}}$ over all the other vertices. For instance, the personalized PageRank method uses $R$ to define the personalization vector or restart vector underlying the process (Andersen et al., 2006). The PageRank solution or the sparse Andersen-Chung-Lang approximation (Andersen et al., 2006) are then the diffusion $\boldsymbol{\mathrm{x}}$. Given a diffusion vector $\boldsymbol{\mathrm{x}}$, we round it back to a set $S$ by performing a procedure called a sweepcut. This involves sorting $\boldsymbol{\mathrm{x}}$ from largest to smallest and then evaluating the hypergraph conductance of each prefix set $S_{j}=\{[1],[2],\ldots,[k]\}$, where $[i]$ is the id of the $i$th largest vertex. The set returned by sweepcut picks the minimum conductance set $S_{j}$. Since the sweepcut procedures are general and standardized, we focus on the computation of $\boldsymbol{\mathrm{x}}$. When these algorithms are used for semi-supervised learning, the returned set $S$ is presumed to share the label as the reference (seed) set $R$; alternatively, its value or rank information may be used to disambiguate multiple labels (Zhou et al., 2003; Gleich and Mahoney, 2015).

Our overall goal is to compute a hypergraph diffusion that will help us perform a sweepcut to identify a set with reasonably small conductance nearby a reference set of vertices in the graph. We explain our method: localized hypergraph quadratic diffusions (LHQD) or also localized hypergraph PageRank (LHPR) through two transformations before we formally state the problem and algorithm. We adopted this strategy so that the final proposal is well justified because some of the transformations require additional context to appreciate. Computing the final sweepcut is straightforward for hypergraph conductance, and so we do not focus on that step.

We give a local conductance guarantee that results from solving (6). Because of space, we focus on the case $\kappa=0$. We prove that a sweepcut on the solution $\boldsymbol{\mathrm{x}}$ of (6) leads to a Cheeger-type guarantee for conductance of the hypergraph $\mathcal{H}$ even when the seed-set size $|R|$ is $1$. It is extremely difficult to guarantee a good approximation property with an arbitrary seed node, and so we first introduce a seed sampling strategy $\mathbb{P}$ with respect to a set $S^{*}$ that we wish to find. Informally, the seed selection strategy says that the expected solution mass outside $S^{*}$ is not too large, and more specifically, not too much larger than if you had seeded on the entire target set $S^{*}$.

Note that $\textbf{vol}(S^{*})$ is just to normalize the effect of using different numbers of seeds. For an arbitrary $S^{*}$, a good sampling strategy $\mathbb{P}$ for the standard graph case with $c=1$ is to sample nodes from $S^{*}$ proportional to their degree. Now, we provide our main theorem and show its proof in Appendix B .

The proof is in the appendix. This implies that for any set $S^{*}$, if we have a sampling strategy that matches $S^{*}$ and tune $\gamma$, our method can find a node set with conductance $O(\sqrt{\phi_{\mathcal{H}}(S^{*})\bar{\delta}\log(\textbf{vol}(S^{*}))})$. The term $\bar{\delta}$ is the additional cost that we pay for performing graph reduction. The dependence on $\bar{\delta}$ essentially generalizes the previous works that analyzed the conductance with only all-or-nothing penalty (Li et al., 2020; Takai et al., 2020), as our result matches these when $\bar{\delta}=1$. But our method gives the flexibility to choose other values $\delta_{e}$ and while $\bar{\delta}$ in the worst case could be as large as $|e|/2$, in practice, $\bar{\delta}$ can be chosen much smaller (See §7). Also, although we reduce $\mathcal{H}$ into a directed graph $G$, the previous conductance analysis for directed graphs (Yoshida, 2016; Li et al., 2020) is not applicable as we have degree zero nodes in $G$. Those degree zero nodes introduce challenges.

We have discussed most related work in-situ throughout the paper. Here, we address a few related hypergraph PageRank vectors directly. First, Li et al. (2020) defined a quadratic hypergraph PageRank by directly using Lovász extension of the splitting function $f_{e}$ to control the diffusion instead of a reduction. Both Li et al. (2020) and Takai et al. (2020) simultaneously proved that this PageRank can be used to partition hypergraphs with an all-or-nothing penalty and a Cheeger-type guarantee. Neither approach gives a strongly local algorithm and they have complexity $O(|\mathcal{E}||V|\min\{|\mathcal{E}|,|V|\}\text{poly}(\zeta))$ or in terms of Euler integration or subgradient descent.

In the context of the local graph clustering, the quadratic cut objective can sometimes “over-expand” or “bleed out” over natural boundaries in the data. (This is the opposite problem to the maxflow-based clustering.) To solve this issue, (Liu and Gleich, 2020) proposed a more general $p$-norm based cut objective, where $1<p\leq 2$. The corresponding p-norm diffusion algorithm can not only grow from small seed set, but also capture the boundary better than 2-norm cut objective. Moreover, (Yang et al., 2020) proposed a related p-norm flow objective that shares similar characteristics. Our hypergraph diffusion framework easily adapts to such a generalization.

Definition: p-norm local hypergraph diffusions. Given a hypergraph $\mathcal{H}=(V,\mathcal{E})$, seeds $R$, and values $\gamma,\kappa$. Let $\boldsymbol{\mathrm{B}},\boldsymbol{\mathrm{w}}$ again be the incidence matrix and weight vector of the localized reduced directed cut graph. A $p$-norm local hypergraph diffusion is:

Here $\ell(x)=\frac{1}{p}x^{p}$, $1<p\leq 2$. And the corresponding residual function is $\boldsymbol{\mathrm{r}}(\boldsymbol{\mathrm{x}})=-\frac{1}{\gamma}\boldsymbol{\mathrm{B}}^{T}\text{diag}(\ell^{\prime}((\boldsymbol{\mathrm{B}}\boldsymbol{\mathrm{x}})_{+}))\boldsymbol{\mathrm{w}}$.

The idea of solving (13) is similar to the quadratic case, where the goal is to iteratively push values to $x_{i}$ as long as node $i$ violates the optimality condition, i.e. $r_{i}>\kappa d_{i}$. The challenge of solving a more general p-norm cut objective is that we no longer have a closed form solution even if the ordering of adjacent nodes is known. Thus, we need to use binary search to find $\Delta x_{i}$, $\Delta x_{a}$ and $\Delta x_{b}$ up to $\varepsilon$ accuracy at every iteration. This means that in the worst case, the general push process can be slower than 2-norm based push process by a factor of $O(\text{log}(1/\varepsilon))$. We defer the details of the algorithm to a longer version of the paper, but we note that a similar analysis shows that this algorithm is strongly local.

In the experiments, we will investigate both the LHQD (2-norm) and 1.4-norm cut objectives with the $\delta$-linear threshold as the splitting function (more details about this function in §3.3). Our focus in this experimental investigation is on the use of the methods for semi-supervised learning. Consequently, we consider how well the algorithms identify “ground truth” clusters that represent various known labels in the datasets when given a small set of seeds. (We leave detailed comparisons of the conductances to a longer version.)

In the plots and tables, we use LH-2.0 to represent our LHQD or LHPR method and LH-1.4 to represent the 1.4 norm version from §6. The other four methods we compare are:
(i) ACL (Andersen et al., 2006), which is initially designed to compute approximated PageRank on graphs. Here we transform each hypergraph to a graph using three different techniques, which are star expansion (star+ACL), unweighted clique expansion (UCE+ACL) and weighted clique expansion (WCE+ACL) where a hyperedge $e$ is replaced by a clique where each edge has weight $1/|e|$ (Zhou et al., 2006). ACL is known as one of the fastest and most successful local graph clustering algorithm in several benchmarks (Veldt et al., 2016; Liu and Gleich, 2020) and has a similar quadratic guarantee on local graph clustering (Andersen et al., 2006; Zhu et al., 2013).
(ii) flow (Veldt et al., 2020b), which is the maxflow-mincut based local method designed for hypergraphs. Since the flow method has difficulty growing from small seed set as illustrated in the yelp experiment in §1, we will first use the one hop neighborhood to grow the seed set. (OneHop+flow) To limit the number of neighbors included, we will order the neighbors using the BestNeighbors as introduced in (Veldt et al., 2020b) and only keep at most 1000 neighbors. (Given a seedset $R$, BestNeighbors orders nodes based on the fraction of hyperedges incident to $v$ that are also incident to at least one node from $R$.)
(iii) LH-2.0+flow, this is a combination of LH-2.0 and flow where we use the output of LH-2.0 as the input to the flow method to refine.
(iv) HGCRD (Ibrahim and Gleich, 2020), this is a hypergraph generalization of CRD (Wang et al., 2017), which is a hybrid diffusion and flow.¹¹1Another highly active topic for clustering and semi-supervised learning involves graph neural networks (GNN). Prior comparisons between GNNs and diffusions shows mixed results in the small seed set regime we consider (Ibrahim and Gleich, 2019; Liu and Gleich, 2020) and complicates doing a fair comparison. As such, we focus on comparing with the most directly related work.

In order to select an appropriate $\delta$ for different datasets, Veldt et al. found that the optimal $\delta$ is usually consistent among different clusters in the same dataset (Veldt et al., 2020b). Thus, the optimal $\delta$ can be visually approximated by varying $\delta$ for a handful of clusters if one has access to a subset of ground truth clusters in a hypergraph. We adapt the same procedure in our experiments and report the results in App. C. Other parameters are in the reproduction details footnote.²²2Reproduction details. The full algorithm and evaluation codes can be found here https://github.com/MengLiuPurdue/LHQD. We fix the LH locality parameter $\gamma$ to be 0.1, approximation parameter $\rho$ to be 0.5 in all experiments. We set $\kappa=0.00025$ for Amazon and $\kappa=0.0025$ for Stack Overflow based on cluster size. For ACL, we use the same set of parameters as LH. For LH-2.0+flow, we set the flow method’s locality parameter to be 0.1. For OneHop+flow, we set the locality parameter to be 0.05, 0.0025 on Amazon and Stack Overflow accordingly. For HGCRD, we set $U=3$ (maximum flow that can be send out of a node), $h=3$ (maximum flow that an edge can handle), $w=2$ (multiplicative factor for increasing the capacity of the nodes at each iteration), $\alpha=1$ (controls the eligibility of hyperedge), $\tau=0.5$ and 6 maximum iterations.

This paper studies the opportunities for strongly local quadratic and $p$-norm diffusions in the context of local clustering and semi-supervised learning.

One of the distinct challenges we encountered in preparing this manuscript was comparing against the ideas of others. Clique expansions are often problematic because they can involve quadratic memory for each hyperedge if used simplistically. For running the baseline ACL PageRank diffusion on the clique expansion, we were able to use the linear nature of this algorithm to implicitly model the clique expansion without realizing the actual graph in memory. (We lack space to describe this though.) For others the results were less encouraging. We, for instance, were unable to set integration parameters for the Euler scheme employed by QHPR (Takai et al., 2020) to produce meaningful results in §7.

Consequently, we wish to discuss the parameters of our method and why they are reasonably easy to set. The values $\gamma$ and $\kappa$ both control the size of the result. Roughly, $\gamma$ corresponds to how much the diffusion is allowed to spread from the seed vertices and $\kappa$ controls how aggressively we sparsify the diffusion. To get a bigger result, then, set $\gamma$ or $\kappa$ a little bit smaller. The value of $\rho$ corresponds only to how much one of our solutions can differ from the unique solution of (6). Fixing $\rho=0.5$ is fine for empirical studies unless the goal is to compare against other strategies to solve that same equation. The final parameter is $\delta$, which interpolates between the all-or-nothing penalty and the cardinality penalty as discussed in §3.2. This can be chosen based on an experiment as we did here, or by exploring a few small choices between $1$ and half the largest hyperedge size.

In closing, flow-based algorithms have often been explained or used as refinement operations to the clusters produced by spectral methods (Lang, 2005) as in LH-2.0+flow. As a final demonstration of this usage on the Yelp experiment from §1, we have precision, recall, and F1 result of $0.87,0.998,0.93$, respectively, which demonstrates how these techniques may be easily combined to even more accurately find target clusters.