Local Hyper-Flow Diffusion

Hypergraphs [8] generalize graphs by allowing a hyperedge to consist of multiple nodes that capture higher-order relationships in complex systems and datasets [36]. Hypergraphs have been used for music recommendation on Last.fm data [10], news recommendation [29], sets of product reviews on Amazon [37], and sets of co-purchased products at Walmart [1]. Beyond the internet, hypergraphs are used for analyzing higher-order structure in neuronal, air-traffic and food networks [6, 30].

In order to explore and understand higher-order relationships in hypergraphs, recent work has made use of cut-cost functions that are defined by associating each hyperedge with a specific set function. These functions assign specific penalties of separating the nodes within individual hyperedges. They generalize the notion of hypergraph cuts and are crucial for determining small-scale community structure [30, 33]. The most popular cut-cost functions with increasing capability to model complex multiway relationships are the unit cut-cost [28, 27, 23], cardinality-based cut-cost [43, 44] and general submodular cut-cost [31, 47]. An illustration of a hyperedge and the associated cut-cost function is given in Figure 1. In the simplest setting, all cut-cost functions take value either 0 or 1 (e.g., the case when $\gamma_{1}=\gamma_{2}=1$ in Figure 1(b)), we obtain a unit cut-cost hypergraph. In a slightly more general setting, the cut-costs are determined solely by the number of nodes in either side of the hyperedge cut (e.g., the case when $\gamma_{1}=1/2$ and $\gamma_{2}=1$ in Figure 1(b)), we obtain a cardinality-based hypergraph. We refer to hypergraphs associated with arbitrary submodular cut-cost functions (e.g., the case when $\gamma_{1}=1/2$ and $0\leq\gamma_{2}\leq 1$ in Figure 1(b)) as general submodular hypergraphs.

Hypergraphs that arise from data science applications consist of interesting small-scale local structure such as local communities [33, 42]. Exploiting this structure is central to the above mentioned applications on hypergraphs and related applications in machine learning and applied mathematics [7]. We consider local hypergraph clustering as the task of finding a community-like cluster around a given set of seed nodes, where nodes in the cluster are densely connected to each other while relatively isolated to the rest of the graph. One of the most powerful primitives for the local hypergraph clustering task is the graph diffusion. Diffusion on a graph is the process of spreading a given initial mass from some seed node(s) to neighbor nodes using the edges of the graph. Graph diffusions have been successfully employed in the industry, for example, both Pinterest and Twitter use diffusion methods for their recommendation systems [16, 17, 22]. Google uses diffusion methods to perform clustering query refinements [40]. Let us not forget PageRank [9, 39], Google’s model for their search engine.

Empirical and theoretical performance of local diffusion methods is often measured on the problem of local hypergraph clustering [45, 20, 33]. Existing local diffusion methods only directly apply to hypergraphs with the unit cut-cost [26, 42]. For the slightly more general cardinality-based cut-cost, they rely on graph reduction techniques which result in a rather pessimistic edge-size-dependent approximation error [46, 26, 33, 43]. Moreover, none of the existing methods is capable of processing general submodular cut-costs. In this work, we are interested in designing a diffusion framework that (i) achieves stronger theoretical guarantees for the problem of local hypergraph clustering, (ii) is flexible enough to work with general submodular hypergraphs, and (iii) permits computationally efficient algorithms. We propose the first local diffusion method that simultaneously accomplishes these goals.

In what follows we describe our main contributions and previous work. In Section 2 we provide preliminaries and notations. In Section 3 we introduce our diffusion model from a combinatorial flow perspective. In Section 4 we discuss the local hypergraph clustering problem and Cheeger-type quadratic approximation error. In Section 6 we perform experiments using both synthetic and real datasets.

Submodular function. Given a set $S$, we denote $2^{S}$ the power set of $S$ and $|S|$ the cardinality of $S$. A submodular function $F:2^{S}\rightarrow\mathbb{R}$ is a set function such that $F(A)+F(B)\geq F(A\cup B)+F(A\cap B)$ for any $A,B\subseteq S$.

Submodular hypergaph. A hypergraph $H=(V,E)$ is defined by a set of nodes $V$ and a set of hyperedges $E\subseteq 2^{V}$, i.e., each hyperedge $e\in E$ is a subset of $V$. A hypergraph is termed submodular if every $e\in E$ is associated with a submodular function $w_{e}:2^{e}\rightarrow\mathbb{R}_{\geq 0}$ [31]. The weight $w_{e}(S)$ indicates the cut-cost of splitting the hyperedge $e$ into two subsets, $S$ and $e\setminus S$. This general form allows us to describe the potentially complex higher-order relation among multiple nodes (Figure 1). A proper hyperedge weight $w_{e}$ should satisfy that $w_{e}(\emptyset)=w_{e}(e)=0$. To ease notation we extend the domain of $w_{e}$ to $2^{V}$ by setting $w_{e}(S):=w_{e}(S\cap e)$ for any $S\subseteq V$. We assume without loss of generality that $w_{e}$ is normalized by $\vartheta_{e}:=\max_{S\subseteq e}w_{e}(S)$, so that $w_{e}(S)\in[0,1]$ for any $S\subseteq V$. For the sake of simplicity in presentation, we assume that $\vartheta_{e}=1$ for all $e$.¹¹1This is without loss of generality. In the appendix we show that our method works with arbitrary $\vartheta_{e}>0$. A submodular hypergraph is written as $H=(V,E,\mathcal{W})$ where $\mathcal{W}:=\{w_{e},\vartheta_{e}\}_{e\in E}$. Note that when $w_{e}(S)=1$ for any $S\in 2^{e}\backslash\{\emptyset,e\}$, the definition reduces to unit cut-cost hypergraphs. When $w_{e}(S)$ only depends on $|S|$, it reduces to cardinality-based cut-cost hypergraphs.

Vector/Function on $V$ or $E$. For a set of nodes $S\subseteq V$, we denote $\mathit{1}_{S}$ the indicator vector of $S$, i.e., $[\mathit{1}_{S}]_{v}=1$ if $v\in S$ and 0 otherwise. For a vector $x\in\mathbb{R}^{|V|}$, we write $x(S):=\sum_{v\in S}x_{v}$, where $x_{v}$ in the entry in $x$ that corresponds to $v\in V$. We define the support of $x$ as $\textnormal{supp}(x):=\{v\in V|x_{v}\neq 0\}$. The support of a vector in $\mathbb{R}^{|E|}$ is defined analogously. We refer to a function over nodes $x:V\rightarrow\mathbb{R}$ and its explicit representation as a $|V|$-dimensional vector interchangeably.

Volume, cut, conductance. Given a submodular hypergraph $H=(V,E,\mathcal{W})$, the degree of a node $v$ is defined as $d_{v}:=|\{e\in E:v\in e\}|$. We reserve $d$ for the vector of node degrees and $D=\mbox{diag}(d)$. We refer to $\textnormal{vol}(S):=d(S)$ as the volume of $S\subseteq V$. A cut is treated as a proper subset $C\subset V$, or a partition $(C,\bar{C})$ where $\bar{C}:=V\setminus C$. The cut-set of $C$ is defined as $\partial C:=\{e\in E|e\cap C\neq\emptyset,e\cap\bar{C}\neq\emptyset\}$; the cut-size of $C$ is defined as $\textnormal{vol}(\partial C):=\sum_{e\in\partial C}\vartheta_{e}w_{e}(C)=\sum_{e\in E}\vartheta_{e}w_{e}(C)$. The conductance of a cut $C$ in $H$ is $\Phi(C):=\frac{\textnormal{vol}(\partial C)}{\min\{\textnormal{vol}(C),\textnormal{vol}(V\setminus C)\}}$.

Flow. A flow routing over a hyperedge $e$ is a function $r_{e}:e\rightarrow\mathbb{R}$ where $r_{e}(v)$ specifies the amount of mass that flows from $\{v\}$ to $e\setminus\{v\}$ over $e$. To ease notation we extend the domain of $r_{e}$ to $V$ by identifying $r_{e}(v)=0$ for $v\not\in e$, so $r_{e}$ is treated as a function $r_{e}:V\rightarrow\mathbb{R}$ or equivalently a $|V|$-dimensional vector. The net (out)flow at a node $v$ is given by $\sum_{e\in E}r_{e}(v)$. Given a routing function $r_{e}$ and a set of nodes $S\subseteq V$, a directional routing on $e$ with direction $S\rightarrow e\setminus S$ is represented by $r_{e}(S)$, which specifies the net amount of mass that flows from $S$ to $e\setminus S$.  A routing $r_{e}$ is called proper if it obeys flow conservation, i.e., $r_{e}^{T}\mathit{1}_{e}=0$. Our flow definition generalizes the notion of network flows to hypergraphs. We provide concrete illustrations in Figure 2.

In this section we provide details of the proposed local diffusion method. We consider diffusion process as the task of spreading mass from a small set of seed nodes to a larger set of nodes. More precisely, given a hypergraph $H=(V,E,\mathcal{W})$, we assign each node a sink capacity specified by a sink function $T$, i.e., node $v$ is allowed to hold at most $T(v)$ amount of mass. In this work we focus on the setting where $T(v)=d_{v}$, so that a high-degree node that is part of many hyperedges can hold more mass than a low-degree node that is part of few hyperedges. Moreover, we assign each node some initial mass specified by a source function $\Delta$, i.e., node $v$ holds $\Delta(v)$ amount of mass at the start of the diffusion. In order to encourage the spread of mass in the hypergraph, the initial mass on the seed nodes is larger than their capacity. This forces the seed nodes to diffuse mass to neighbor nodes to remove their excess mass. In Section 4 we will discuss the choice of $\Delta$ to obtain good theoretical guarantees for the problem of local hypergraph clustering. Details about the local hypergraph clustering problem are provided in Section 4.

Given a set of proper flow routings $r_{e}$ for $e\in E$, recall that $\sum_{e\in E}r_{e}(v)$ specifies the amount of net (out)flow at node $v$. Therefore, the vector $m=\Delta-\sum_{e\in E}r_{e}$ gives the amount of net mass at each node after routing. The excess mass at a node $v$ is $\textnormal{ex}(v):=\max\{m_{v}-d_{v},0\}$. In order to force the diffusion of initial mass we could simply require that $\textnormal{ex}(v)=0$ for all $v\in V$, or equivalently, $\Delta-\sum_{e\in E}r_{e}\leq d$. But to provide additional flexibility in the diffusion dynamics, we introduce a hyper-parameter $\sigma\geq 0$ and we impose a softer constraint $\Delta-\sum_{e\in E}r_{e}\leq d+\sigma Dz$, where $z\in\mathbb{R}^{|V|}$ is an optimization variable that controls how much excess mass is allowed on each node. In the context of numerical optimization, we show in Section 5 that $\sigma$ allows a reformulation which makes the optimization problem amenable to efficient alternating minimization schemes.

Note that so far we have not yet talked about how specific higher-order relations among nodes within a hyperedge would affect the flow routings over it. Apparently, simply requiring that the $r_{e}$’s obey flow conservation (i.e., $r_{e}^{T}\mathit{1}_{e}=0$) similar to the standard graph setting is not enough for hypergraphs. An important difference between hyperedge flows and graph edge flows is that additional constraints on $r_{e}$ are in need. To this end, we consider $r_{e}=\phi_{e}\rho_{e}$ for some $\phi_{e}\in\mathbb{R}_{+}$ and $\rho_{e}\in B_{e}$, where

is the base polytope [4] for the submodular cut-cost $w_{e}$ associated with hyperedge $e$. It is straightforward to see that $r_{e}(v)=0$ for every $v\not\in e$ and $r_{e}^{T}\mathit{1}_{e}=0$, so $r_{e}$ defines a proper flow routing over $e$. Moreover, for any $e\subseteq V$, recall that $r_{e}(S)$ represents the net amount of mass that moves from $S$ to $e\setminus S$ over hyperedge $e$. Therefore, the constraints $\rho_{e}(S)\leq w_{e}(S)$ for $S\subseteq e$ mean that the directional flows $r_{e}(S)$ are upper bounded by a submodular function $\phi_{e}w_{e}(S)$. Intuitively, one may think of $\phi_{e}$ and $\rho_{e}$ as the scale and the shape of $r_{e}$, respectively.

The goal of our diffusion problem is to find low cost routings $r_{e}\in\phi_{e}B_{e}$ for $e\in E$ such that the capacity constraint $\Delta-\sum_{e\in E}r_{e}\leq d+\sigma Dz$ is satisfied. We consider the (weighted) $\ell_{2}$-norm of $\phi$ and $z$ as the cost of diffusion. In the appendix we show that one readily extends the $\ell_{2}$-norm to $\ell_{p}$-norm for any $p\geq 2$. Formally, we arrive at the following convex optimization formulation (input: the source function $\Delta$, the hypergraph $H=(V,E,\mathcal{W})$, and a hyper-parameter $\sigma$):

We name problem (1) Hyper-Flow Diffusion (HFD) for its combinatorial flow interpretation we discussed above. The dual problem of (1) is:

where $f_{e}$ in (2) is the support function of the base polytope $B_{e}$ given by $f_{e}(x):=\max_{\rho_{e}\in B_{e}}\rho_{e}^{T}x$. $f_{e}$ is also known as the Lovász extension of the submodular function $w_{e}$.

We provide a combinatorial interpretation for (2) and leave algebraic derivations to the appendix. For the dual problem, one can view the solution $x$ as assigning heights to nodes, and the goal is to separate/cut the nodes with source mass from the rest of the hypergraph. Observe that the linear term in the dual objective function encourages raising $x$ higher on the seed nodes and setting it lower on others. The cost $f_{e}(x)$ captures the discrepancy in node heights over a hyperedge $e$ and encourages smooth height transition over adjacent nodes. The dual solution embeds nodes into the nonnegative real line, and this embedding is what we actually use for local clustering and node ranking.

In this section we discuss the performance of the primal-dual pair (1)-(2), respectively, in the context of local hypergraph clustering. We consider a generic hypergraph $H=(V,E,\mathcal{W})$ with submodular hyperedge weights $\mathcal{W}=\{w_{e},\vartheta_{e}\}_{e\in E}$. Given a set of seed nodes $S\subset V$, the goal of local hypergraph clustering is to identify a target cluster $C\subset V$ that contains or overlaps well with $S$. This generalizes the definition of local clustering over graphs [19]. To the best of our knowledge, we are the first one to consider this problem for general submodular hypergraphs. We consider a subset of nodes having low conductance as a good cluster, i.e., these nodes are well-connected internally and well-separated from the rest of the hypergraph. Following prior work on local hypergraph clustering, we assume the existence of an unknown target cluster $C$ with conductance $\Phi(C)$. We prove that applying sweep-cut to an optimal solution $\hat{x}$ of (2) returns a cluster $\hat{C}$ whose conductance is at most quadratically worse than $\Phi(C)$. Note that this result resembles Cheeger-type approximation guarantees of spectral clustering in the graph setting [2], and it is the first result that is independent of hyperedge size for general hypergraphs. We keep the discussion at high level and defer details to the appendix, where we prove a more general, and stronger, i.e., constant approximation error result when the primal problem (1) is penalized by the $\ell_{p}$-norm for any $p\geq 2$.

In order to start a diffusion process we need to provide the source mass $\Delta$. Similar to the $p$-norm flow diffusion in the graph setting [20], we let

where $S$ is a set of seed nodes and $\delta\geq 1$. Below, we make the assumptions that the seed set $S$ and the target cluster $C$ have some overlap, there is a constant factor of $\textnormal{vol}(C)$ amount of mass trapped in $C$ initially, and the hyper-parameter $\sigma$ is not too large. Note that Assumption 2 is without loss of generality: if the right value of $\delta$ is not known apriori, we can always employ binary search to find a good choice. Assumption 3 is very weak as it allows $\sigma$ to reside in an interval containing 0.

Let $\hat{x}$ be an optimal solution for the dual problem (2). For $h>0$ define the sweep sets $S_{h}:=\{v\in V|\hat{x}_{v}\geq h\}$. We state the approximation property in Theorem 1.

One of the challenges we face in establishing the result in Theorem 1 is making sure that our diffusion model enjoys both good clustering guarantees and practical algorithmic advantages at the same time. This is achieved by introducing the hyper-parameter $\sigma$ to our diffusion problem. We will demonstrate how $\sigma$ helps with algorithmic development in Section 5, but from a clustering perspective, the additional flexibility given by $\sigma>0$ complicates the underlying diffusion dynamics, making it more difficult to analyze. Another challenge is connecting the Lovász extension $f_{e}(x)$ in (2) with the conductance of a cluster. We resolve all these problems by combining a generalized Rayleigh quotient result for submodular hypergraphs [31], primal-dual convex conjugate relations between (1) and (2), and a classical property of the Choquet integral/Lovász extension.

Let $(\hat{\phi},\hat{r},\hat{z})$ be an optimal solution for the primal problem (1). We state the following lemma on the locality (i.e., sparsity) of the optimal solutions, which justifies why HFD is a local diffusion method.

We use a simple Alternating Minimization (AM) [5] method that efficiently solves the primal diffusion problem (1). For $e\in E$, we define a diagonal matrix $A_{e}\in\mathbb{R}^{|V|\times|V|}$ such that $[A_{e}]_{v,v}=1$ if $v\in e$ and 0 otherwise. Denote $\mathcal{C}:=\{(\phi,r):r_{e}\in\phi_{e}B_{e},~{}\forall e\in E\}$. The following Lemma 1 allows us to cast problem (1) to an equivalent separable formulation amenable to the AM method.

The AM method for problem (4) is given in Algorithm 1. The first sub-problem corresponds to computing projections to a group of cones, where all the projections can be computed in parallel. The computation of each projection depends on the choice of base polytope $B_{e}$. If the hyperedge weight $w_{e}$ is unit cut-cost, $B_{e}$ holds special structures and projection can be computed with $O(|e|\log|e|)$ [32]. For general $B_{e}$, a conic Fujishige-Wolfe minimum norm algorithm can be adopted to obtain the projection [32]. The second sub-problem in Algorithm 1 can be easily computed in closed-form. We provide more information about Algorithm 1 and its convergence properties in the appendix.

We remark that the reformulation (4) for $\sigma>0$ is crucial from an algorithmic point of view. If $\sigma=0$, then the primal problem (1) has complicated coupling constraints that are hard to deal with. In this case, one has to resort to the dual problem (2). However, problem (2) has a nonsmooth objective function, which prohibits applicability of optimization methods for smooth objective functions. Even though subgradient method may be applied, we have observed empirically that its convergence rate is extremely slow for our problem, and early stopping results in a bad quality output.

Lastly, as noted in Lemma 2, the number of nonzeros in the optimal solution is upper bounded by $\|\Delta\|_{1}$. In Figure 3 we plot the number of nodes having positive excess (which equals the number of nonzeros in the dual solution $\hat{x}$) at every iteration of Algorithm 1. Figure 3 indicates that Algorithm 1 is strongly local, meaning that it works only on a small fraction of nodes (and their incident hyperedges) as opposed to producing dense iterates. This key empirical observation has enabled our algorithm to scale to large datasets by simply keeping track of all active nodes and hyperedges. Proving that the worst-case running time of AM depends only on the number of nonzero nodes at optimality as opposed to size of the whole hypergraph is an open problem, which we leave for future work.

In this section we evaluate the performance of HFD for local clustering. First, we carry out experiments on synthetic hypergraphs with varying target cluster conductances and varying hyperedge sizes. For the unit cut-cost setting, we show that HFD is more robust and has better performance when the target cluster is noisy; for a cardinality-based cut-cost setting, we show that the edge-size-independent approximation guarantee is important for obtaining good recovery results. Second, we carry out experiments using real-world data. We show that HFD significantly outperforms existing state-of-the-art diffusion methods for both unit and cardinality-based cut-costs. Moreover, we provide a compelling example where specialized submodular cut-cost is necessary for obtaining good results. Code that reproduces all results is available at https://github.com/s-h-yang/HFD.