Understanding Negative Sampling in Graph Representation Learning

Many network embedding algorithms, such as DeepWalk (Perozzi et al., 2014), LINE (Tang et al., 2015) and node2vec (Grover and Leskovec, 2016), actually assign each node two embeddings ¹¹1Others, for example GraphSAGE (Hamilton et al., 2017) use a unique embedding., node embedding and contextual embedding. When sampled as context, the contextual embedding, instead of its node embedding is used for inner product. This implementation technique stems from word2vec (Mikolov et al., 2013), though sometimes omitted by following papers, and improves performance by utilizing the asymmetry of the contextual nodes and central nodes.

We can split each node $i$ into central node $v_{i}$ and contextual node $u_{i}$, and each edge $i\rightarrow j$ becomes $v_{i}\rightarrow u_{j}$. Running those network embedding algorithms on the transformed bipartite graph is strictly equivalent to running on the original graph.²²2For bipartite graphs, always take nodes from $V$ part as central node and sample negative nodes from $U$ part. According to the above analysis, we only need to anaylze graphs with unique embeddings without loss of generality.

We unify a large part of graph representation learning algorithms into the general SampledNCE framework, shown in Algorithm 1. The framework depends on an encoder $E_{\theta}$, which can be either an embedding lookup table or a variant of GNN, to generate node embeddings.

In the training process, $m$ positive nodes and $k\cdot m$ negative nodes are sampled. In Algorithm 1, we use the cross-entropy loss to optimize the inner product where $\sigma(x)=\frac{1}{1+e^{-x}}$. And the other choices of loss function work in similar ways. In link prediction or recommendation, we recall the top $K$ nodes with largest $E_{\theta}(v)\cdot E_{\theta}(u)$ for each node $u$. In classification, we evaluate the performance of logistic regression with the node embeddings as features.

Edges in natural graph data can be assumed as sampled from an underlying positive distribution $p_{d}(u,v)$. However, only a few edges are observable so that numerous techniques are developed to make reasonable assumptions to better estimate $p_{d}(u,v)$, resulting in various $\hat{p_{d}}$ in different algorithms to approximate $p_{d}$.

Network embedding algorithms employ various positive distributions implicitly or explicitly. LINE (Tang et al., 2015) samples positive nodes directly from adjacent nodes. DeepWalk (Perozzi et al., 2014) samples via random walks and implicitly defines $\hat{p_{d}}(u|v)$ as the stationary distribution of random walks (Qiu et al., 2018). node2vec (Grover and Leskovec, 2016) argues that homophily and structural equivalence of nodes $u$ and $v$ indicates a larger $p_{d}(u,v)$. These assumptions are mainly based on local smoothness and augment observed positive node pairs for training.

Graph neural networks are equipped with different encoder layers and perform implicit regularizations on positive distribution. Previous work (Li et al., 2018) reveals that GCN is a variant of graph Laplacian smoothing, which directly constrains the difference of the embedding of adjacent nodes. GCN, or its sampling version GraphSAGE (Hamilton et al., 2017), does not explicitly augment positive pairs but regularizes the output embedding by local smoothness, which can be seen as an implicit regularization of $p_{d}$.

Proof  Let us consider the conditional data and the negative distribution of node $v$. To maximize $J$ is equivalent to minimize the following objective function for each $v$,

For each $(u,v)$ pair, if we define two Bernoulli distributions $P_{u,v}(x)$ where $P_{u,v}(x=1)=\frac{p_{d}(u|v)}{p_{d}(u|v)+kp_{n}(u|v)}$ and $Q_{u,v}(x)$ where $Q_{u,v}(x=1)=\sigma(\vec{\bm{u}}^{\top}\vec{\bm{v}})$, the equation above is simplified as follows:

where $H(p,q)=-\sum_{x\in\mathcal{X}}p(x)\log q(x)$ is the cross entropy between two distributions. According to Gibbs Inequality, the maximizer of $L^{(v)}$ satisfies $P=Q$ for each $(u,v)$ pair, which gives

The above theorem clearly shows that the positive and negative distributions $p_{d}$ and $p_{n}$ have the same level of influence on optimization. In contrast to abundant research on finding a better $\hat{p_{d}}$ to approximate $p_{d}$, negative distribution $p_{n}$ is apparently under-explored.

In this section, we only consider optimization of node $v$, which can be directly generalized to the whole embedding learning. More precisely, we equivalently consider $\bm{\theta}=[\vec{\bm{u_{0}}}^{\top}\vec{\bm{v}},...,\vec{\bm{u}}_{N-1}^{\top}\vec{\bm{v}}]$ as the parameters to be optimized, where $\{u_{0},...u_{N-1}\}$ are all the N nodes in the graph. Suppose the optimal parameter for $J^{(v)}_{T}$ and $J^{(v)}$ are $\bm{\theta}_{T}$ and $\bm{\theta}^{*}$ respectively. The gap between $\bm{\theta}_{T}$ and $\bm{\theta}^{*}$ is described as follows:

Proof  Our analysis is based on the Taylor expansion of $\nabla_{\theta}J_{T}^{(v)}(\bm{\theta})$ around $\bm{\theta}^{*}$. As $\bm{\theta}_{T}$ is the minimizer of $J_{T}^{(v)}(\bm{\theta})$, we must have $\nabla_{\theta}J_{T}^{(v)}(\bm{\theta}_{T})=\bm{0}$, which gives

Up to terms of order $O(\|\bm{\theta}^{*}-\bm{\theta}_{T}\|^{2})$, we have

Next, we will analyze $-\big{(}\nabla^{2}_{\theta}J_{T}^{(v)}(\bm{\theta}^{*})\big{)}^{-1}$ and $\sqrt{T}\nabla_{\theta}J_{T}^{(v)}(\bm{\theta}^{*})$ respectively by the following lemmas.

Proof  First, we calculate the gradient of $J_{T}^{(v)}(\bm{\theta})$ and Hessian matrix $\nabla_{\theta}^{2}J^{(v)}_{T}(\bm{\theta})$. Let $\bm{\theta}_{u}$ be the parameter in $\bm{\theta}$ for modeling $\vec{\bm{u}}^{\top}\vec{\bm{v}}$ and $\bm{e}_{(u)}$ be the one-hot vector which only has a 1 on this dimension.

According to equation (3), $\sigma(\bm{\theta}_{u_{i}})=\frac{p_{d}(u|v)}{p_{d}(u|v)+kp_{n}(u|v)}$ at $\bm{\theta}=\bm{\theta}^{*}$. Therefore,

Proof  According to equation (3)(8),

Let us continue to prove Theorem 2 by substituting (9)(12) into equation (7) and derive the covariance of $\sqrt{T}(\bm{\theta}_{T}-\bm{\theta}^{*})$.

According to Theorem 2, the mean squared error, aka risk, of $\vec{\bm{u}}^{\top}\vec{\bm{v}}$ is

To determine an appropriate $p_{n}$ for a specific $p_{d}$, a trade-off might be needed to balance the rationality of the objective (to what extent the optimal embedding fits for the downstream task) and the expected loss, also known as risk (to what extent the trained embedding deviates from the optimum).

A simple solution is to sample negative nodes positively but sub-linearly correlated to their positive sampling distribution, i.e. $p_{n}(u|v)\propto p_{d}(u|v)^{\alpha},0<\alpha<1$.

(Monotonicity) From the perspective of objective, if we have $p_{d}(u_{i}|v)>p_{d}(u_{j}|v)$,

where $c$ is a constant. The sizes of inner products is in accordance with positive distribution $p_{d}$, facilitating the task relying on relative sizes of $\vec{\bm{u}}^{\top}\vec{\bm{v}}$ for different $u$s, such as recommendation or link prediction.

(Accuracy) From the perspective of risk, we mainly care about the scale of the expected loss. According to equation (13), the expected squared deviation is dominated by the smallest one in $p_{d}(u|v)$ and $p_{n}(u|v)$. If an intermediate node with large $p_{d}(u|v)$ is negatively sampled insufficiently (with small $p_{n}(u|v)$), the accurate optimization will be corrupted. But if $p_{n}(u|v)\propto p_{d}(u|v)^{\alpha}$, then

The order of magnitude of deviation only negatively related to $p_{d}(u|v)$, meaning that with limited positive samples, the inner products for high-probability node pairs are estimated more accurate. This is an evident advantage over evenly negative sampling.

Although we deduced that $p_{n}(u|v)\propto p_{d}(u|v)^{\alpha}$, the real $p_{d}$ is unknown and its approximation $\hat{p_{d}}$ is often implicitly defined. – How can the principle help negative sampling?

We therefore propose a self-contrast approximation, replacing $p_{d}$ by inner products based on the current encoder, i.e.

The resulting form is similar to a technique in RotatE (Sun et al., 2019), and very time-consuming. Each sampling requires $O(n)$ time, making it impossible for middle- or large-scale graphs. Our MCNS solves this problem by our adapted version of Metropolis-Hastings algorithm.

As one of the most distinguished Markov chain Monte Carlo (MCMC) methods, the Metropolis-Hastings algorithm (Metropolis et al., 1953) is designed for obtaining a sequence of random samples from unnormalized distributions. Suppose we want to sample from distribution $\pi(x)$ but only know $\tilde{\pi}(x)\propto\pi(x)$. Meanwhile the computation of normalizer $Z=\sum_{x}\tilde{\pi}(x)$ is unaffordable. The Metropolis-Hastings algorithm constructs a Markov chain $\{X(t)\}$ that is ergodic and stationary with respect to $\pi$ —- meaning that, $X(t)\sim\pi(x),t\rightarrow\infty$.

The transition in the Markov chain has two steps:
(1) Generate $y\sim q(y|X(t))$ where the proposal distribution $q$ is arbitrary chosen as long as positive everywhere.
(2) Pick $y$ as $X(t+1)$ at the probability $\min\left\{\frac{\tilde{\pi}(y)}{\tilde{\pi}(X(t))}\frac{q(X(t)|y)}{q(y|X(t))},1\right\}$, or $X(t+1)\leftarrow X(t)$. See tutorial (Chib and Greenberg, 1995) for more details.

The main idea for MCNS is to apply the Metropolis-Hastings algorithm for each node $v$ with $\tilde{\pi}(u|v)=\big{(}E_{\theta}(u)\cdot E_{\theta}(v)\big{)}^{\alpha}$. Then we are sampling from the self-contrast approximated distribution. However, a notorious disadvantage for the Metropolis-Hastings is a relatively long $burn-in$ period, i.e. the distribution of $X(t)\approx\pi(x)$ only when $t$ is large, which heavily affects its efficiency.

Fortunately, the connatural property exists for graphs that nearby nodes usually share similar $p_{d}(\cdot|v)$, as evident from triadic closure in social network (Huang et al., 2014), collaborative filtering in recommendation (Linden et al., 2003), etc. Since nearby nodes have similar target negative sampling distribution, the Markov chain for one node is likely to still work well for its neighbors. Therefore, a neat solution is to traverse the graph by Depth First Search (DFS, See Appendix A.6) and keep generating negative samples from the last node’s Markov chain (Figure 2).

The proposal distribution also influences convergence rate. Our $q(y|x)$ is defined by mixing uniform sampling and sampling from the nearest $k$ nodes with $\frac{1}{2}$ probability each. The hinge loss pulls positive node pairs closer and pushes negative node pairs away until they are beyond a pre-defined margin. Thus, we substitute the binary cross-entropy loss as a $\gamma$-skewed hinge loss,

where $(u,v)$ is a positive node pair and $(x,v)$ is negative. $\gamma$ is a hyperparameter. The MCNS is summarized in Algorithm 2.

In this section, we demonstrate the results in the 19 settings. Note that all values and standard deviations reported in the tables are from ten-fold cross validation. We also leverage paired t-tests (Hsu and Lachenbruch, 2007) to verify whether MCNS is significantly better than the strongest baseline. The tests in the 19 settings give a max p-value (Amazon, GraphSAGE) of 0.0005 $\ll 0.01$, quantitatively proving the significance of our improvements.