Toward Data-centric Directed Graph Learning: An Entropy-driven Approach

Although the HKT construction process may superficially resemble hierarchical clustering, it is crucial to emphasize that HKT is fundamentally distinct. Unlike traditional clustering methods, HKT leverages topology-driven structural entropy—a dynamic metric grounded in the information theory of structured data—to extract deeper structural insights from the graph. This methodology transcends the limitations of static clustering techniques, offering a more nuanced and context-aware understanding of the underlying graph structure. Moreover, EDEN incorporates profile-oriented node MI through neural estimation as a pivotal criterion for HKT construction. This integration enables a more fine-grained and profile-aware analysis of node relationships, uncovering intricate patterns that static methods may overlook. As a result, the multi-granularity quantification criteria established by our approach not only deviate significantly from conventional hierarchical clustering but also introduce a novel and innovative perspective for understanding complex graph data (see Sec. 3.1 for more details).

While traditional hierarchical clustering can reveal the layered structure of a network, it is not directly applicable to the complexities of (di)graph learning. EDEN, on the other hand, utilizes HKT as a foundational framework to enable the development of learnable knowledge generation and transfer mechanisms that can be seamlessly integrated with existing (Di)GNN architectures. This integration provides a novel way to enhance model learning by effectively capturing and utilizing the hierarchical structure of directed graphs. Furthermore, we have designed a random walk-based leaf prediction mechanism, tailored to various graph-based downstream tasks, ensuring that our approach is robust and adaptable to different application scenarios (for more technical details, refer to Sec. 3.2-3.3).

Graph KD typically follows a model-level, offline teacher-student framework. In this setup, knowledge is transferred from a large, pre-trained teacher GNN to a more compact and efficient student model, such as a smaller GNN or MLP. The teacher captures complex patterns and representations within the graph. The student, rather than learning directly from ground truth labels, learns from the teacher’s soft predictions or intermediate representations. This approach allows the student model to replicate the teacher’s performance while significantly reducing computational complexity.

With the rapid advancement of KD, it has expanded into multiple model-level KD variants. These include self-distillation, where a single model simultaneously acts as both the teacher and student, enhancing its own learning process (Chen et al., 2021; Zhang et al., 2023), and online distillation, where both teacher and student models are continuously updated throughout the training process (Zhang et al., 2021b; Feng et al., 2022). These innovations reflect the growing diversity in how knowledge transfer can be applied beyond the initial teacher-student (large model to lightweight model) framework.

In this paper, we focus specifically on data-centric graph KD, which emphasizes uncovering the latent knowledge embedded in graph structures, using data samples as the medium for distillation (Zhang et al., 2020; Zhu et al., 2024). In the EDEN framework, parent and child nodes within the HKT assume the roles of teacher and student, respectively. This enables knowledge transfer through their representations in a hierarchical manner. Our approach aligns with the principles of data-level online KD, leveraging the topological relationships between nodes to drive more effective distillation.

To define high-dimensional measurements of directed structural information, we introduce a partition tree $\mathcal{T}$ of digraphs, which can also be regarded as the coarse-grained HKT without profile-oriented refinement (i.e., knowledge discovery (a) from a topology perspective only). Notably, community detection or clustering can be understood as a hierarchical structure, specifically a 3-layer partition tree. In this structure, the leaf nodes represent the individual nodes from the original graph, while their parent nodes serve as virtual nodes that represent entire communities. To make it easier to understand, we first give an example of a two-dimensional directed structural measurement of the graph, $\mathcal{H}^{2}(\mathcal{G})$, where we consider a digraph $\mathcal{G=(V,E)}$ and its 2-order partition $\mathcal{P}=\left\{\mathcal{X}_{1},\cdots,\mathcal{X}_{\mathcal{C}}\right\}$ of node sets $\mathcal{V}$. Building upon this, we interpret $\mathcal{P}$ through a $2$-height partition tree $\mathcal{T}$ as follows.

To begin with, we introduce the root node $\lambda$ and define a set of nodes $T_{\lambda}=\mathcal{V}$ as a subset of the root node $\lambda$ in the $2$-height partition tree $\mathcal{T}$. Notably, in this two-dimensional directed structural measurement, the nodes in the $2$-height partition tree have only three types of identity information:

(1) the root node ($h=0$), which does not exist in the original digraph but is used to describe the partition tree;

(2) the successor nodes ($h=1$), which are not present in the original digraph but are employed to characterize leaf nodes;

(3) the leaf nodes ($h=2$), which represent the original digraph nodes.

Then, we introduce ${\mathcal{C}}$ immediate successors for the root denote $\phi_{i}=\lambda\langle i\rangle$, $i=1,\cdots,{\mathcal{C}}$. Naturally, we can extend the concept associated with the root to successor nodes $\phi_{i}$, which are directly related to the coarse partitioning of leaf nodes $\mathcal{X}_{i}$. Thus, we define $T_{\phi_{i}}=\mathcal{X}_{i}$. Now, for each $\phi_{i}$, we introduce $\left|\mathcal{X}_{i}\right|$ immediate successors denoted $\phi_{i}\langle j\rangle$ for all $j\in\left\{1,\cdots,\left|\mathcal{X}_{i}\right|\right\}$, and each successor $\phi_{i}\langle j\rangle$ is associated with an element in $\mathcal{X}_{i}$. Thus, we define $T_{\phi_{i}\langle j\rangle}$ as the singleton of a node in $T_{\phi_{i}}=\mathcal{X}_{i}$.

To this point, $\mathcal{T}$ is a partition tree of height 2, and all its leaves are associated with singletons. For any node $\alpha\in\mathcal{T}$, $T_{\alpha}$ is the union of $T_{\beta}$ for all $\beta$ values (immediate successors) of $\alpha$, and the union of $T_{\alpha}$ for all nodes with $\alpha$ values at the same level of the partition tree $\mathcal{T}$ constitutes a partition of $\mathcal{V}$. Hence, the partition tree of a digraph is a set of nodes, each associated with a nonempty subset of nodes in digraph $\mathcal{G}$, and can be defined as follows:

According to Definition A.1, for a given digraph $\mathcal{G}$, we compute the $h$-dimensional directed structural information measurement $\mathcal{H}^{h}(\mathcal{G})$ of $\mathcal{G}$ by Eq. (3) while simultaneously identifying a $h$-height coarse-grained HKT $\mathcal{T}$. The above process adheres to the following principles:

(1) The $h$-dimensional structural information measurement $\mathcal{H}^{h}(\mathcal{G})$ of a digraph $\mathcal{G}$ is achieved or approximated through the $h$-dimensional hierarchical partition tree $\mathcal{T}$ of $\mathcal{G}$;

(2) $\mathcal{H}^{h}(\mathcal{G})$ serves as the guiding principle for the formation of the $h$-dimensional coarse-grained HKT $\mathcal{T}$ by minimizing the uncertainty or non-determinism inherent in the $h$-dimensional structures of $\mathcal{G}$;

(3) $\mathcal{T}$, functioning as a coarse-grained HKT for $\mathcal{G}$, encompasses the rules, regulations, and orders governing $\mathcal{G}$. This HKT is derived by minimizing the random variations present in the $h$-dimensional structures of the digraphs, with these variations being determined by our $h$-dimensional directed structural information measurement.

Based on the above principles, the $h$-dimensional structural measurement of digraphs, provided by the $h$-height partition tree, serves as a metric enabling us to comprehensively or maximally identify the $h$-dimensional structure while mitigating the impact of random variations in the digraphs. Meanwhile, $\mathcal{H}^{h}(\mathcal{G})$ excellently facilitates the complete extraction of order from unordered digraphs, allowing us to discern order from disorder within structured data. Remarkably, our definition retains all properties of the digraphs, providing robust support for the thorough analysis of structured data.

Utilizing simple random walks (SRW) on digraphs introduces unique challenges due to the inherent structure of these graphs. A common issue arises when the random walk encounters nodes with no outgoing edges, causing the walk to terminate prematurely. To better understand and visualize this limitation, we apply SRW starting from each node across four different digraphs. As the walk length increases, we track the proportion of complete paths relative to the total sequences, as shown in Fig. LABEL:fig:_empirical_1_(a). To further assess the impact of graph cycles, we design a modified SRW that excludes cycles and conduct the same experiment, with results presented in Fig. LABEL:fig:_empirical_1_(b).

This investigation highlights a key limitation of random walks on digraphs: strictly following edge directions leads to frequent interruptions in the walk. Due to the non-strongly connected nature of most digraphs, the proportion of complete walks drops sharply after just five steps. This indicates that random walks on digraphs typically fail to gather information beyond the immediate neighborhood of the starting node, limiting their ability to capture long-range dependencies. Moreover, when we eliminate the influence of cycles, the proportion of uninterrupted sequences declines even further, underscoring the difficulty of maintaining continuous paths in digraphs and further highlighting the limitations of SRWs (forward-only) in exploring deeper graph structures. It is evident that this significantly hinders the ability of structural entropy to capture topological uncertainty, reducing the effectiveness of $\mathcal{H}^{h}(\mathcal{G})$ and leading to sub-optimal coarse-grained HKT.

The primary impetus for developing the greedy partition tree construction algorithm lies in the quest for an effective method to construct hierarchical tree structures from digraph data while simultaneously minimizing the complexity and uncertainty associated with the underlying relationships. In complex systems represented by digraphs, directed structural entropy serves as a key metric to gauge the disorder and intricacy within the network. By harnessing the concept of directed edge structural entropy minimization, the algorithm aims to derive hierarchical trees that capture essential structural characteristics while promoting simplicity and interpretability. In a nutshell, the design principles of our proposed algorithm are as follows

(1) Directed edge structural entropy definition: The algorithm hinges on a rigorous definition of directed edge structural entropy within the context of the digraph mentioned in Sec. 3.1. This metric quantifies the uncertainty and disorder associated with the relationships between nodes in the digraph.

(2) Greedy selection strategy: At its core, the algorithm employs a greedy strategy, iteratively selecting directed edges that contribute most significantly to the reduction of directed structural entropy. This strategy ensures that each step in the tree construction process maximally minimizes the overall disorder in the evolving hierarchy.

(3) Hierarchical tree construction: The selected directed edges are systematically incorporated into the growing tree structure, establishing a hierarchical order that reflects the inherent organization within the graph. This process continues iterations until a coherent and informative tree representation is achieved.

(4) Complexity considerations: The algorithm balances the trade-off between capturing essential structural information and maintaining simplicity. By prioritizing directed edges that significantly impact entropy reduction, it aims to construct trees that are both insightful and comprehensible.

In conclusion, the greedy partition tree construction algorithm for digraph data, rooted in the minimization of directed edge structural entropy, presents a promising avenue for extracting hierarchical structures from the network with intricate topology. To clearly define a greedy partition tree construction algorithm, we introduce the following meta-operations in Alg. 1.

These meta-operations collectively define the intricate logic underlying the greedy partition tree construction algorithm, providing a comprehensive framework for constructing hierarchical structures in graph data while adhering to the principles of minimizing directed edge structural entropy. Building upon these foundations, we employ meta-operations to present the detailed workflow of the greedy structural tree construction algorithm. This facilitates the coarse-grained HKT construction from a topological perspective, ultimately achieving digraph data knowledge discovery (i.e., Step 1 Knowledge Discovery (a) in our proposed EDEN as illustrated in Fig. 2).

The Alg. 2 outlines the construction of a height-limited partition tree algorithm, emphasizing the minimization of directed structural uncertainty. It begins by sorting input data in non-decreasing order. Subsequently, it constructs an initial partition tree, using a greedy approach that iteratively combines nodes until the root has only two children. After that, it enters a phase of height reduction, wherein nodes contributing to excess height are detached iteratively until the tree attains height $h$. To stabilize the structure, it inserts filler nodes for any node with a height discrepancy exceeding 1. This three-phase process ensures the efficient construction of a height-limited partition tree while minimizing directed structural measurement.

As discussed in Sec. 3.1, node profiles in a digraph act as essential identifiers. These profiles are not only instrumental in distinguishing nodes but also play a critical role in the construction of data knowledge. Recognizing this, our proposed partition-based node MI neural estimation seeks to further refine the coarse-grained HKT, which is initialized by the greedy algorithm. This refinement is achieved by quantifying the correlations between node profiles within the partition tree, thereby enhancing the granularity of the HKT. The refined tree provides a more accurate and nuanced representation of the graph, laying a robust foundation for subsequent KD. This process ensures that both topological structure and node profile information are effectively leveraged in the distillation, leading to improved model performance.

Considering a digraph $\mathcal{G=(V,E)}$ and its coarse-grained partition tree $\mathcal{T}$, where $\mathcal{V}$ encompasses all nodes in the digraph, along with the corresponding feature and label matrix represented as $\mathbf{X}$ and $\mathbf{Y}$. For current partition $\mathcal{X}_{p}$ given by $\mathcal{T}$, we employ a sampling strategy to obtain a candidate node subset $\Omega_{p}$ with $K_{p}$ nodes from the current partition $\mathcal{X}_{p}$ and other partitions $\mathcal{X}_{q}$. Notably, different partitions used for sampling should be at the same height within the HKT (e.g., the current partition $\mathcal{X}_{p}$ and other partitions $\mathcal{X}_{q}$ should satisfy $h(\mathcal{X}_{p})=h(\mathcal{X}_{q})$). Building upon this, to reduce the computational complexity, we adopt a computation-friendly sampling strategy. Specifically, considering the number of nodes in the current partition is $\left|\mathcal{X}_{p}\right|$, we include all of them in the candidate set $\Omega_{p}$. Additionally, we perform random sampling for partition-by-partition until the total non-duplicated nodes in the $\Omega_{p}$ satisfy $\kappa\left|\mathcal{X}_{p}\right|$, where $\kappa\geq 1$ is used to control the knowledge domain expansion come from the other partitions $\mathcal{X}_{q}$. This subset $\Omega_{p}$ is used to generate knowledge that represents the current partition $\mathcal{X}_{p}$, formally represented as the parent representation of this partition in the HKT. Notably, we assign distinct identifiers to the sampled nodes based on their partition affiliations, denoting them as $v\in\mathcal{X}_{p}$ and $u\in\mathcal{X}_{q}$, providing clarity in illustrating our method and derivation process.

Building upon this foundation, given the node $v$ as an example, a random variable $f_{v}$ is introduced to represent the node feature when randomly selecting a node from $\Omega_{p}$ within the current partition $\mathcal{X}_{p}$. Then, the probability distribution of $f_{v}$ is formally defined as $P_{f_{v}}=P(f_{v}=\mathbf{X}_{v}),\forall v\in\Omega_{p}\cap\mathcal{X}_{p}$. Similarly, we can generalize $P_{f_{v}}$ to scenarios originating from other partitions to obtain $P_{f_{u}}=P(f_{u}=\mathbf{X}_{u}),\forall u\in\Omega_{p}\cap\mathcal{X}_{q}$. In $\Omega_{p}$, the definition of the generalized neighborhoods for any node is closely tied to the partition provided by the HKT, rather than relying on the traditional definition based on the adjacency matrix $\mathbf{A}$ from directed edge sets $\mathcal{E}$.

Specifically, for nodes belonging to the current partition, denoted as $v\in\mathcal{X}_{p}$, their generalized neighborhoods are defined as $\mathcal{N}_{v}^{\mathcal{T}}=\mathcal{X}_{p}$. This is done to identify nodes with sufficient information to efficiently represent the current partition i.e., (measure MI between $v$ and $\mathcal{N}_{v}^{\mathcal{T}}$). As for nodes belonging to other partitions, denoted as $u\in\mathcal{X}_{q}$, their generalized neighborhoods are defined as $\mathcal{N}_{u}^{\mathcal{T}}=\mathcal{X}_{p}\cup\mathcal{X}_{q}$. This is intended to address the limitations of the coarse-grained partition tree produced by considering only topological metrics. In other words, we aim to identify sets of nodes within other partitions that effectively capture the representation of both the current partition $\mathcal{X}_{p}$ (explore potential correlation from the profile perspective) and their own partition $\mathcal{X}_{q}$ (inherit their own partition criteria about directed structural information measurement), thereby refining the HKT through MI measurement between $u$ and $\mathcal{N}_{u}^{\mathcal{T}}$.

Notably, we chose $\mathcal{N}_{v}^{\mathcal{T}}=\mathcal{X}_{p}$ for the following reasons: (1) We aim to calculate the MI neural estimation between the current node $v$ and its generalized neighborhoods $\mathcal{X}_{p}$ as a criterion for quantifying affinity scores. This approach ensures that nodes representative of the current partition receive higher affinity scores. Therefore, the generalized neighborhood of the current node needs to be closely related to the partition to which the node belongs, leading us to impose this restriction rather than defining the neighborhood as all nodes $\mathcal{V}$. For more on the motivation, intuition, and theory behind this mechanism, please refer to Sec. 2.2. As for the details on the calculation of affinity scores, we recommend referring to Sec. 3.2 on knowledge generation. (2) In general, the number of partitions $\mathcal{X}_{p}$ is considerably smaller than the total set of nodes $\mathcal{V}$. As a result, one of the key motivations for imposing this neighborhood restriction is to minimize computational overhead and improve overall runtime efficiency. By limiting the scope of the calculations, we are able to streamline the process without sacrificing performance, making the method more scalable for large-scale graphs. In summary, expanding the neighborhood to include all nodes would result in higher computational costs and poorer performance. Therefore, we restrict the definition of the generalized neighborhood based on the partition obtained by HKT.

In either case, the generalized neighborhoods are subgraphs containing nodes from $\mathcal{V}$. These nodes may not be directly connected in the original topology but reveal inherent correlations at a higher level through the measurement of directed structural information. Therefore, this representation transcends the topological exploration of the digraph by $\mathbf{A}$ and reflects intrinsic knowledge at a higher level. Building upon this, considering a node $v$ as an example, let $f_{\mathcal{N}_{v}^{\mathcal{T}}}$ be a random variable representing the generalized neighborhood feature selected from $\Omega_{p}$, originating from the current partition $\mathcal{X}_{p}$. We define the probability distribution of $f_{\mathcal{N}_{v}^{\mathcal{T}}}$ as $P_{f_{\mathcal{N}_{v}^{\mathcal{T}}}}=P(f_{\mathcal{N}_{v}^{\mathcal{T}}}=\mathbf{X}_{\mathcal{N}_{v}^{\mathcal{T}}})$.

Therefore, considering a node $v\in\mathcal{X}_{p}$ as an example, we define the joint distribution of the random variables of node features and its generalized neighborhood features within partition $\mathcal{X}_{p}$ given by HKT, which is formulated as:

where the joint distribution reflects the probability that we randomly pick the corresponding node feature and its generalized neighborhood feature of the same node $v$ within partition $\mathcal{X}_{p}$ together. Building upon this, the MI between the node features and the generalized neighborhood features within the current partition $\mathcal{X}_{p}$ is defined as the KL-divergence between the joint distribution $P\left(f_{v},f_{\mathcal{N}_{v}^{\mathcal{T}}}\right)$ and the product of the marginal distributions of the two random variables $P_{f_{v}}\otimes P_{f_{\mathcal{N}_{v}^{\mathcal{T}}}}$. The above process can be formally defined as:

This MI measures the mutual dependency between the selected node and its generalized neighborhoods in $\Omega_{p}$. The KL divergence adopts the $f$-representation (Belghazi et al., 2018) is defined as:

where $\mathcal{F}$ is an arbitrary class of functions that maps a pair of selected node features and its generalized neighborhood features to a real value. Here, we use $F(\cdot,\cdot)$ to compute the dependency. If we explore any possible function $F\in\mathcal{F}$, it can serve as a tight lower bound for MI. Building upon this, we can naturally extend the above derivation process to the scenario of sampling nodes belonging to other partitions, specifically $u\in\Omega_{p}\cap\mathcal{X}_{q}$. At this point, we can assess the shared contribution of nodes $v$ and $u$ with different affiliations in generating knowledge for the current partition $\mathcal{X}_{p}$.

The primary objective here is to introduce a node selection criterion that is grounded in quantifying the dependency between the selected node and its generalized neighborhoods. This dependency serves as the foundation for assessing the relevance and influence of each node within its local structure. The key insights behind using this dependency as a guiding principle are central to the formulation of the criterion function. By leveraging this approach, we aim to enhance the process of knowledge generation for the current partition $\mathcal{X}_{p}$, ensuring that both local and global relationships are effectively captured and utilized in the knowledge distillation process. The detailed reasoning and benefits of this approach are outlined as follows:

(1) In our definition, the generalized neighborhoods of the selected node are closely tied to the current partition $\mathcal{X}_{p}$ and their own partition $\mathcal{X}_{i}$. Thus, measuring this dependency is equivalent to quantifying the correlation between the representation of the selected node and the knowledge possessed by the current partition and their own partition.

(2) The node-selection criterion is essentially a mechanism for weight allocation. Since the candidate node set is fixed by the sampling process, this step aims to assign higher affinity scores to nodes that better represent the current and their own partition. This guides the knowledge generation process to acquire the parent node representation for the current partition.

Building upon this, instead of calculating the exact MI based on KL divergence, we opt for non-KL divergences to offer favorable flexibility and optimization convenience. Remarkably, both non-KL and KL divergences can be formulated within the same $f$-representation framework. We commence with the general $f$-divergence between the joint distribution and the product of marginal distributions of vertices and neighborhoods. The above process can be formally defined as follows:

where $f(\cdot)$ represents a convex and lower-semicontinuous divergence function. When $f(x)=x\log x$, the $f$-divergence is specified as the Kullback-Leibler (KL) divergence. The function $f(\cdot)$ has a convex conjugate function, denoted as $f^{\star}(\cdot)$, where $f^{\star}(t)=\sup_{x\in\operatorname{dom}_{f}}\left\{tx-f(x)\right\}$, and $\operatorname{dom}_{f}$ is the domain of $f(\cdot)$. It’s important to note that these two functions, $f(\cdot)$ and $f^{\star}(\cdot)$, are dual to each other. According to the Fenchel conjugate (Hiriart-Urruty & Lemaréchal, 2004) and node sampling space $\Omega_{p}$ based on different affiliations given by HKT, the $f$-divergence can be modified as: