IRWE: Inductive Random Walk for Joint Inference of Identity and Position Network Embedding

As discussed in Section 3, it is possible to map RWs with different sets of nodes to a common AW. For instance, $(0,1,2,0)$ is the common AW of RWs $(v_{1},v_{2},v_{3},v_{1})$ and $(v_{1},v_{4},v_{5},v_{1})$ in Fig. 3. Given a fixed length $l$, RWs on all possible topology structures can only be mapped to a finite set of AWs $\Omega_{l}$. Namely, $\Omega_{l}$ and its induced statistics are shared by all possible nodes and graphs, thus having the potential to support the inductive embedding inference. Based on this intuition, IRWE maintains an AW embedding $\varphi(\omega)\in\mathbb{R}^{d}$ for each AW $\omega\in\Omega_{l}$. In this setting, $\{\varphi(\omega)\}$ can be used as a special embedding lookup table for the derivation of inductive features regarding graph topology.

We also consider an additional constraint on $\{\varphi(\omega)\}$, where two AWs with more common elements in corresponding positions should have closer representations. For instance, $(0,1,2,1,2)$ and $(0,1,0,1,2)$ should be closer in the AW embedding space than $(0,1,2,1,2)$ and $(0,1,0,2,3)$. To apply this constraint, we transform each AW $\omega$ with length $l$ to a one-hot encoding $\rho(\omega)\in\{0,1\}^{(l+1)^{2}}$, where $\rho(\omega)_{jl:(j+1)l}$ (i.e., subsequence from the $jl$-th to the $(j+1)l$-th positions) is the one-hot encoding of the $j$-th element in $\omega$. For instance, we have $\rho(\omega)=[0000~{}0100~{}0010~{}0001]$ for $\omega=(0,1,2,3)$ in Fig. 3. An auto-encoder is then introduced to derive and regularize $\{\varphi(\omega)\}$, including an encoder and a decoder. Given an AW $\omega$, the encoder ${\rm{Enc}}_{\varphi}(\cdot)$ and decoder ${\rm{Dec}}_{\varphi}(\cdot)$ are defined as

which are both multi-layer perceptrons (MLPs). The encoder takes $\rho(\omega)$ as input and derives AW embedding $\varphi(\omega)$. The decoder reconstructs $\rho(\omega)$ with $\varphi(\omega)$ as input. Since similar AWs have similar one-hot encodings, similar AWs can have close embeddings by minimizing the reconstruction error between $\{\rho(\omega)\}$ and $\{{\hat{\rho}}(\omega)\}$.

IRWE derives identity embeddings $\{\psi(v)\}$ via the combination of AW embeddings $\{\varphi(\omega)\}$ inspired by the following Theorem 1 (Micali & Zhu, 2016).

Theorem 1. Let $\mathcal{G}_{s}(v,r)$ be the rooted subgraph induced by nodes with a distance less than $r$ from $v$. Let $q(v,l)$ be the distribution of AWs w.r.t. RWs starting from $v$ with length $l$. One can reconstruct $\mathcal{G}_{s}(v,r)$ in time $O(n^{2})$ with $O(n^{2})$ access to $[q(v,1),\cdots,q(v,l)]$, where $l=O(m)$; $n$ and $m$ are the numbers of nodes and edges in $\mathcal{G}_{s}(v,r)$.

For a given length $l$, let $\eta_{l}$ be the number of AWs. $q(v,l)$ can be represented as an $\eta_{l}$-dimensional vector, with the $j$-th element as the occurrence probability of the $j$-th AW. Since AWs with length $l$ include sequences of those with length less than $l$ (e.g., $(0,1,2,3)$ provides information about $(0,1,2)$), one can derive $q(v,k)$ ($k<l$) based on $q(v,l)$. Therefore, $q(v,l)$ can be used to characterize $\mathcal{G}_{s}(v,r)$ according to Theorem 1.

As defined in Section 3, nodes with similar rooted subgraphs are expected to play similar structural roles and thus have similar identities. For instance, in Fig. 1, $\mathcal{G}_{s}(v_{1},1)$ and $\mathcal{G}_{s}(v_{8},1)$ have the same topology structure, which is consistent with the same identity they have. Hence, $q(v,l)$ can characterize the identity of node $v$.

To estimate $q(v,l)$, we extract AW statistic $s(v)$ for each node $v$ using Algorithm 1. We first sample RWs with length $l$ starting from $v$ via the standard unbiased strategy (Perozzi et al., 2014) (see Algorithm 6 in Appendix A). Let $\mathcal{W}^{(v)}$ be the set of sampled RWs starting from $v$. Each RW $w\in\mathcal{W}^{(v)}$ is then mapped to its AW. Let $\Omega_{l}$ be an AW lookup table including all the $\eta_{l}$ AWs with length $l$, which is fixed and shared by all possible topology. We define the AW statistic as $s(v):=[c(\omega_{1}),\cdots,c(\omega_{\eta_{l}})]\in\mathbb{Z}_{+}^{\eta_{l}}$, where $c(\omega_{j})$ is the frequency of the $j$-th AW in $\Omega_{l}$ as illustrated in Fig. 3.

Although $\eta_{l}$ grows exponentially with the increase of length $l$, $\{s(v)\}$ are usually sparse. Fig. 4.1.2 visualizes the example AW statistics $\{s(v)\}$ derived from RWs on the Brazil dataset (see Section 5.1 for details) with $l=4$ and $|\mathcal{W}^{(v)}|=1,000$. The $i$-th row in Fig. 4.1.2 is the AW statistic $s(v_{i})$ of node $v_{i}$. Dark blue indicates that the corresponding element is $0$. There exist many AWs $\{\omega_{j}\}$ not observed during the RW sampling (i.e., $\forall v\in\mathcal{V}$ s.t. $s(v)_{j}=0$). We then remove terms w.r.t. these unobserved AWs in $\Omega_{l}$ and $\{s(v)\}$. Let ${\tilde{\Omega}}_{l}$, ${\tilde{s}}(v)$, and ${\tilde{\eta}}_{l}$ be the reduced $\Omega_{l}$, $s(v)$, and $\eta_{l}$. Table 4.1.2 shows the variation of $\eta_{l}$ and ${\tilde{\eta}}_{l}$ on Brazil as $l$ increases from $4$ to $9$, where $\eta_{l}$ is significantly reduced.

In addition to $\{{\tilde{s}}(v)\}$, one can also characterize node identities from the view of node degrees (Ribeiro et al., 2017; Wu et al., 2019) based on the following Hypothesis 1.

Hypothesis 1. Nodes with the same degree are expected to play the same structural role. This concept can be extended to high-order neighbors of nodes. Namely, nodes are expected to have similar identities if they have similar node degree statistics (e.g., distribution over all degree values) w.r.t. their high-order neighbors.

Based on this motivation, we extract high-order degree feature $\delta(v)$ for each node $v$ using Algorithm 2. Given a node $u$, one can construct a bucket one-hot encoding $\rho_{d}(u)\in\{0,1\}^{e}$ w.r.t. its degree $\deg(u)$, where only the $j$-th element $\rho_{d}(u)_{j}$ is set to $1$ with the remaining elements set to $0$ and $j=\left\lfloor{(\deg(u)-{\deg_{\min}})e/({\deg_{\max}}-{\deg_{\min}})}\right\rfloor$; $\deg_{\min}$ and $\deg_{\max}$ are the minimum and maximum degrees. Since high-order neighbors of a node $v$ can be explored by RWs ${\mathcal{W}}^{(v)}$ starting from $v$, we define $\delta(v)\in\mathbb{Z}^{(l+1)e}$ as an $(l+1)e$-dimensional vector, where the subsequence $\delta(v)_{ie:(i+1)e}$ is the sum of bucket one-hot degree encodings w.r.t. nodes occurred at the $i$-th position of RWs in ${\mathcal{W}}^{(v)}$. Fig. 3 gives a running example to derive $\delta(v_{1})$ (with $e=5$) for node $v_{1}$ in Fig. 1.

Following the aforementioned discussions regarding Theorem 1 and Hypothesis 1, IRWE derives identity embeddings $\{\psi(v)\}$ via the adaptive combination of AW embeddings $\{\varphi(\omega)\}$ w.r.t. AW statistics $\{{\tilde{s}}(v)\}$ and degree features $\{\delta(v)\}$. The multi-head attention is applied to automatically determine the contribution of each AW embedding $\varphi(\omega)$ in the combination, where we treat $\{\varphi(\omega)\}$ as the key and value; the concatenated feature $[\tilde{s}(v)||\delta(v)]$ is used as the query. Before feeding $[\tilde{s}(v)||\delta(v)]\in\mathbb{R}^{({\tilde{\eta}}_{l}+le)}$ to the multi-head attention, we introduce a feature reduction unit ${\rm{Red}}_{s}(\cdot)$, an MLP, to reduce its dimensionality to $d$:

The multi-head attention that derives identity embeddings $\{\psi(v)\}$ is defined as

where ${\mathop{\rm Att}\nolimits}(\cdot,\cdot,\cdot)$ is the standard multi-head attention unit (see Appendix D for details), with ${\bf{Q}}$, ${\bf{K}}$, and ${\bf{V}}$ as inputs of query, key, and value. In (3), we have ${\bf{Q}}_{i,:}={\bar{g}}(v_{i})$, ${\bf{K}}_{j,:}={\bf{V}}_{j,:}=\varphi(\omega_{j})$, and ${\bf{Z}}_{i,:}=\psi(v_{i})$.

An identity embedding decoder ${\rm{Dec}}_{\psi}(\cdot)$ is introduce to regularize identity embeddings $\{\psi(v)\}$ using statistics $\{[\tilde{s}(v)||\delta(v)]\}$. It takes the $\psi(v)$ of a node $v$ as input and reconstruct corresponding $[\tilde{s}(v)||\delta(v)]$ via

where ${\hat{g}}(v)$ is the reconstructed statistic. It can force $\{\psi(v)\}$ to capture node identities hidden in $\{[\tilde{s}(v)||\delta(v)]\}$ by minimizing the reconstruction error between $\{{\hat{g}}(v)\}$ and $\{[\tilde{s}(v)||\delta(v)]\}$. Note that we only apply ${\rm{Dec}}_{\psi}(\cdot)$ to optimize $\{\psi(v)\}$ and do not need this unit in the inference phase.

The input fusion unit extracts $\{\pi_{l}(j),\pi_{g}(v)\}$ and derives inputs of the transformer encoder combined with $\{\psi(v)\}$. Since the RW length $l$ is usually not very large (e.g., $\leq 10$ in our experiments), we define the local position encoding $\pi_{l}(j)\in\{0,1\}^{l+1}$ as the standard one-hot encoding of index $j$. Inspired by previous studies (Perozzi et al., 2014; Grover & Leskovec, 2016; Zhu et al., 2021) that validated the potential of RW for exploring local community structures, we extract the global position encoding $\{\pi_{g}(v)\}$ for each node $v$ w.r.t. RW statistic $r(v)$ using Algorithm 3.

Given a node $v$, we maintain $r(v)\in\mathbb{Z}^{|\mathcal{V}|}$, with the $j$-th element $r(v)_{j}$ as the frequency that node $v_{j}$ occurs in RWs ${\mathcal{W}}^{(v)}$ starting from $v$. For instance, we have $r({v_{1}})=[8,2,1,2,3,2,1,0,0,0,0,0,1]$ for the running example in Fig. 3 with $13$ nodes. Since nodes in a community are densely connected, nodes within the same community are more likely to be reached via RWs. Therefore, nodes $(v,u)$ with similar positions are expected to have similar statistics $(r(v),r(u))$. We then derive $\pi_{g}(v)$ by mapping $r(v)$ to a $d$-dimensional vector via the following Gaussian random projection, an efficient dimension reduction technique that can preserve the relative distance between input features with a rigorous guarantee (Arriaga & Vempala, 2006):

In this setting, the non-Euclidean node positions in a graph topology are encoded in terms of the relative distance between $\{\pi_{g}(v)\}$. Hence, $\pi_{g}(v)$ has the initial ability to encode the position of node $v$. IRWE integrates the relation between node identities and positions based on the following Hypothesis 2.

Hypothesis 2. In a community (i.e., node cluster with dense linkages), nodes with different structural roles may have different contributions in forming the community structure.

For instance, in a social network, an opinion leader (e.g., $v_{1}$ and $v_{8}$ in Fig. 1) is expected to have more contributions in forming the community it belongs to than an ordinary audience (e.g., $v_{2}$ and $v_{9}$ in Fig. 1). Based on this intuition, we use identity embeddings $\{\psi(v)\}$ to reweight global position encodings $\{\pi_{g}(v)\}$, with the reweighting contributions determined by a modified attention operation. Concretely, we set identity embeddings $\{\psi(v)\}$ as the query and let global position encodings $\{\pi_{g}(v)\}$ as the key and value (i.e., ${\bf{Q}}_{i,:}=\psi(v_{i})$ and ${\bf{K}}_{i,:}={\bf{V}}_{i,:}=\pi_{g}(v_{i})$). The modified attention operation is defined as

where ${\bf{\tilde{Q}}}:={\mathop{\rm BN}\nolimits}({\bf{Q}})$, ${\bf{\tilde{K}}}:={\mathop{\rm BN}\nolimits}({\bf{K}})$, and ${\bf{\tilde{V}}}:={\mathop{\rm BN}\nolimits}({\bf{V}})$; ${\mathop{\rm BN}\nolimits}(\cdot)$ and $\odot$ are the batch normalization and element-wise multiplication. In (6), we apply two MLPs to derive nonlinear mappings of the normalized $\{{\bf{Q}},{\bf{K}}\}$ and use their sum to support the element-wise reweighting of the normalized ${\bf{V}}$. For convenience, we denote the reweighted vector w.r.t. a node $v_{i}$ as ${\bar{\pi}}_{g}(v_{i})={\bf{Z}}_{i,:}$. Given an RW $w=(w^{(0)},w^{(1)},\cdots,w^{(l)})$, IRWE concatenates the reweighted vector ${\bar{\pi}}_{g}(w^{(j)})$ and local position encoding $\pi_{l}(j)$ for the $j$-th node and feeds its linear mapping to the transformer encoder:

where ${\bf{W}}_{t}\in\mathbb{R}^{(d+l+1)\times d}$ is a trainable parameter.

IRWE uses the transformer encoder ${\mathop{\rm TransEnc}\nolimits}(\cdot)$ to handle an RW $w=(w^{(0)},\cdots,w^{(l)})$:

It takes the corresponding sequence of vectors $(t(w^{(0)}),\cdots,t(w^{(l)}))$ as input and derives another sequence of vectors $(\bar{t}({w^{(0)}}),\cdots,\bar{t}(w^{(l)}))$ with the same dimensionality. ${\mathop{\rm TransEnc}\nolimits}(\cdot)$ follows a multi-layer structure, with each layer including the self-attention, skip connections, layer normalization, and feedforward mapping. Due to space limit, we omit details of ${\mathop{\rm TransEnc}\nolimits}(\cdot)$ that can be found in (Vaswani et al., 2017).

For an RW $w$ starting from a node $v$, the first output vector ${\bar{t}}(w^{(0)})={\bar{t}}(v)$ can be a representation of $v$. As we sample multiple RWs ${\mathcal{W}}^{(v)}$ starting from each node $v$, one can obtain multiple such representations based on ${\mathcal{W}}^{(v)}$. However, we only need one unique positon embedding $\gamma(v)$ for $v$. Let ${{\bar{t}}^{(v)}}:=\{\bar{t}({w^{(0)}})|w\in{\mathcal{W}^{(v)}}\}$. A naive strategy to derive $\gamma(v)$ is to average representations in ${{\bar{t}}^{(v)}}$. Instead, we develop the following attentive readout function to compute the weighted mean of ${{\bar{t}}^{(v)}}$, with the weights determined by attention:

In (9), ${\mathop{\rm Att}\nolimits}(\cdot,\cdot,\cdot)$ is the standard multi-head attention unit (see Appendix D for details), where we let the global position encoding $\pi_{g}(v)$ be the query (i.e., ${\bf{Q}}=\pi_{g}(v)\in\mathbb{R}^{1\times d}$) and ${{\bar{t}}^{(v)}}$ be the key and value (i.e., ${\bf{K}}_{j,:}={\bf{V}}_{j,:}={\bar{t}}_{j}^{(v)}$). $\gamma(v)$ and ${\bar{\gamma}}(v)$ are the (i) position embedding and (ii) auxiliary context embedding of node $v$, with $\{{\bf{W}}_{\gamma},{\bf{b}}_{\gamma},{\bf{W}}_{\bar{\gamma}},{\bf{b}}_{\bar{\gamma}}\}$ as trainable parameters.

The position embedding decoder is introduced to optimize position embeddings $\{\gamma(v)\}$ together with auxiliary context embeddings $\{\bar{\gamma}(v)\}$. Some of existing embedding methods (Perozzi et al., 2014; Tang et al., 2015; Hamilton et al., 2017) are optimized via the following contrastive loss with negative sampling:

where $D$ denotes the training set including positive and negative samples in terms of node pairs $\{(v_{i},v_{j})\}$; $p_{ij}$ is defined as the statistic of a positive node pair $(v_{i},v_{j})$ (e.g., the frequency that $(v_{i},v_{j})$ occurs in the RW sampling); $Q$ is the number of negative samples; ${n_{j}}$ is usually set to be the probability that $(v_{i},v_{j})$ is selected as a negative sample; $\sigma(\cdot)$ is the sigmoid function; $\tau$ is a temperature parameter to be specified. We follow prior work (Tang et al., 2015) to let ${p_{ij}}:={{\bf{A}}_{ij}}/{{\deg}(v_{i})}$ (i.e., the probability that there is an edge from $v_{i}$ to $v_{j}$ with ${\bf{A}}\in\{0,1\}^{|\mathcal{V}|\times|\mathcal{V}|}$ as the adjacency matrix) and ${n_{j}}\propto(\sum\nolimits_{i:({v_{i}},{v_{j}})\in\mathcal{E}}{{p_{ij}}{)^{0.75}}}$. In the next section, we demonstrate that the contrastive loss (10) can be converted to a reconstruction loss such that the joint optimization of IRWE only includes several reconstruction objectives.