Dropout Training of Matrix Factorization and Autoencoder for Link Prediction in Sparse Graphs

Link prediction is one of the fundamental problems of network analysis, as pointed out in [12], ”a network model is useful to the extent that it can support meaningful inferences from observed network data.” Given a graph $G(V,E)$ together with its adjacency matrix $A\in\{0,1\}^{N\times N}$, the set of nodes $V$, and the set of edges $E$, link prediction can be considered as a matrix completion problem on $A$. The problem is challenging because $A$ is often large and sparse, which means that only a small fraction of the links are observed. As a result, a good model should have enough capacity to accommodate the complex connectivity pattern between all $N^{2}$ pairs of nodes, as well as strong generalization ability to make accurate predictions on unobserved pairs.

Among the large number of models proposed over the decade, Matrix Factorization (MF) is one of the most popular ones in network modeling and relational learning in general [27, 24, 14, 22]. In its simplest form, MF directly models the interaction between a pair of nodes as the inner product of two latent factors, one for each node. This assumes that the large square adjacency matrix $A$ can be factorized into the product of a tall, thin matrix and a short, wide matrix as $A\approx W_{2}W_{1}$, where $W_{2}\in R^{N\times K},W_{1}\in R^{K\times N}$. Each row of $W_{1}$ and each column of $W_{1}$ are often called a latent factor, as they often capture the community membership information of the nodes. Training of such a model is usually conducted with stochastic gradient descent, which makes it easily scalable to large datasets [27, 24].

Bayesian models, such as MMSB [2, 15, 26] are another family of latent factor models for studying network structures. Compared with MF which directly learns the latent factors as free parameters by solving an optimization problem, Bayesian models treat the latent factors as random variables and model the stochastic process of the creation of a link. As a result, they can be considered as a stochastic version of MF. By putting various priors on the latent factors, link prediction is reduced to the inference problem of the posterior distribution of the link status. While Bayesian models can often significantly reduce overfitting compared with their deterministic counterparts, the inference processes such as MCMC and Variational Inference are usually much slower to run than direct optimization. As a result, their application has been limited to only moderate sized datasets.

Autoencoder (AE) together with its variants such as Restricted Boltzman Machine (RBM) has recently achieved great success in various machine learning applications, and is well recognized as the building block of Deep Learning [5, 8]. AE learns useful representations by learning a mapping from an input to itself, which makes it different from the above mentioned approaches. Surprisingly, it is not until recently that AE finds its application to modeling graphs [7, 11]. In this paper we investigate the effectiveness of AE on the link prediction problem. In particular, we show that AE is closely connected to MF when unified in the same architecture. Motivated by this observation, we argue that MF and AE are indeed two complementary views of the same problem, and propose a novel model MF+AE where MF and AE are jointly trained with shared parameters.

To prevent overfitting, we train the model with Dropout [19] combined with stochastic gradient descent. We highlight the effect of dropout training to MF+AE, and show that when approximated by the second order Taylor expansion, the dropout training effectively penalizes a scaled $\ell_{2}$ norm of (the combination of) the rows or columns of the weight matrices. We evaluate MF+AE on six real world sparse graphs and show that dropout significantly mitigates overfitting on both MF and AE, and that MF+AE outperforms the competing methods on all the datasets.

Matrix Factorization: Given an undirected graph $G(V,E)$ with $N$ nodes and $|E|$ edges, MF approximates its adjacency matrix $A\in\{0,1\}^{N\times N}$ with the product of two low rank matrices $W_{2}$ and $W_{1}$ by solving the optimization problem:

(2.1)

$$\begin{split}&\min\sum_{i,j}{L(A_{i,j},\;g(W_{2}^{j}W_{1,i}+b_{1,i}+b_{2,j}))}\end{split}$$

where $W_{1}\in R^{K\times N},W_{2}\in R^{N\times K}$; $b_{1}\in R^{N},b_{2}\in R^{N}$ are the biases, $W_{1,i}$ is the $i^{th}$ column of $W_{1}$, $W_{2}^{j}$ is the $j^{th}$ row of $W_{2}$, $g$ is the link function, $L$ is the loss function defined on each pair of nodes, and $K$ is the dimension of the latent factor. Since $A$ is symmetric for undirected graph, it is sometimes useful to adopt tied weights, i.e., $W_{1}=W_{2},b_{1}=b_{2}$. We refer to the model with tied weights as the symmetric version of MF in this paper.

Autoencoder: AE is originally proposed as an unsupervised representation learning method. Given an example represented by a feature vector $x$, an AE learns a reconstruction of itself by a function $\tilde{x}=F(x)$. The typical form of $F(x)$ can be characterized by a neural network with one hidden layer, where $x$ is first mapped to a hidden layer representation $h$, then a reconstruction $\tilde{x}$ is obtained by mapping $h$ to the original feature space:

(2.2)

$$\begin{split}h&=f(W_{1}x+b_{1})\\ \tilde{x}&=g(W_{2}h+b_{2})\end{split}$$

with $W_{1}\in R^{K\times N},b_{1}\in R^{K},W_{2}\in R^{N\times K},b_{2}\in R^{N}$. The parameters are learned by solving the following optimization problem:

(2.3)

$$\begin{split}\min\sum_{i}{L(x_{i},\tilde{x}_{i};W_{1},b_{1},W_{2},b_{2})}\end{split}$$

Here we have slightly overloaded the loss function $L$ by defining it on the column vector $x_{i}$. The natural way of applying AE to modeling graphs is to represent each node as the set of its neighbors; in other words, set $x_{i}=A_{i}$. This is analogous to the bag of words representation prevalent in the document modeling community, and we call it the bag of nodes representation. Note that this representation is sufficient since when only the topological structure is available, we can learn an unseen node if we know all its neighbors.

The Joint Model: To better see the connection and difference between MF and AE, we now rewrite (2.3) by substituting $x_{i}$ with $A_{i}$:

(2.4)

$$\begin{split}&\min\sum_{i,j}L(A_{i,j},g(W_{2}^{j}h_{i}+b_{2,j}))\\ &s.t.\;h_{i}=f(W_{1}A_{i}+b_{1})\end{split}$$

And we rewrite (2.1) by omitting the $b_{1}$term:

(2.5)

$$\begin{split}&\min\sum_{i,j}L(A_{i,j},g(W_{2}^{j}h_{i}+b_{2,j}))\\ &s.t.\;h_{i}=W_{1}\delta_{i}\end{split}$$

where $\delta_{i}\in R^{N}$ is the indicator vector of node $i$, which is a binary vector with $1$ at the $i^{th}$ entry and $0$ for all the rest. We deliberately organize (2.4) and (2.5) in such a way that in the high level, they share the same architecture. Both models first learn a hidden representation $h_{i}$, which is then fed through a classifier with link function $g$ and loss function $L$. The main difference is only in the form of the hidden representation. For each node $i$, MF only looks at its id, and the hidden layer representation is learned by simply extracting the $i^{th}$ column of $W_{1}$. For AE, we first sum up the columns of $W_{1}$ indicated by $i$’s neighbors, and then pass the sum through an activation function $f$. As a result, in MF, two nodes propagate ”positive” information to each other only if they are directly connected; in AE, however, two nodes can do so as long as they appear in the same neighborhood of some other node, even if they are not directly connected. The different ways of the information propagation between that two models indicates that MF and AE are complementary to each other to model different aspects of the same topological structure.

We can also interpret the distinction between MF and AE as two different views of the same entity: MF uses $\delta_{i}$, and AE uses $A_{i}$. We note that the two views are disjoint and sufficient: they do not overlap with each other, but each of them can sufficiently represent a node. This perspective motivates us to build a unified architecture where we train the two models jointly, and require both of them to be able to uncover the graph structure. The idea is similar to co-training [23] in the semisupervised learning setting, where one trains two classifiers on two sufficient views such that the two views can ”teach” each other on unlabeled data. While in our problem, there is no ”unlabeled data” available, we argue that the model can still benefit from the co-training idea by requiring the two views to ”agree with” each other. We then formulate our model, which we call MF+AE, as follows:

(2.6)

$$\begin{split}\min&\sum_{i}{L(A_{i},g(W_{2}h_{1,i}+b_{2}))}\\ &+\rho\sum_{i}{L(A_{i},g(W_{2}h_{2,i}+b_{4}))}\\ s.t.\;&h_{1,i}=f(W_{1}A_{i}+b_{1}),\\ &h_{2,i}=f(W_{1}\delta_{i}+b_{3})\end{split}$$

In (2.6), the objective function is composed of two parts, one for AE and the other for MF. $h_{1,i}$ and $h_{2,i}$ are the hidden representation learned by the AE and MF part, respectively, and the ”agreement” is achieved by using the same set of weights $W_{1}$ and $W_{2}$ for both AE and MF. We modify the architecture of the MF objective by adding the same activation function $f$ and a corresponding bias term $b_{3}$. $\rho$ is a positive real number which could be simply set to $1$ in practice. We show the architecture of MF+AE in Figure 1.

In Figure 1, the bottom part and top part correspond to the first and second line of (2.6), respectively. The color of each entry of the vector indicates its value, with white for $0$ and black for $1$. We see that the MF module is trained to reconstruct the neighborhood structure based on $\delta_{i}$, while the AE module learns to reconstruct $A_{i}$ from itself. The reconstructions from AE and MF are denoted as $\tilde{A}^{(1)}_{i}$ and $\tilde{A}^{(2)}_{i}$, respectively. Note that although the $5^{th}$ node does not appear in the neighbor of $i$ ($A_{i,5}=0$), the reconstruction is close to $1$ (dark gray). This means that we can make a confident prediction of the link of $i$ to node $5$. After the two reconstructions are obtained, the final prediction is calculated as the geometric mean of the two:

(2.7)

$$\tilde{A}_{i}=\sqrt[1+\rho]{\tilde{A}^{(1)}_{i}\odot(\tilde{A}^{(2)}_{i})^{\rho}}$$

where $\odot$ denotes the element-wise product of two vectors.

Activation Function and Loss Function: For the activation function $f$, we choose the Rectified Linear Units ($ReLU$), which is defined as:

(2.8)

$$f(x)=\max(0,x)$$

$ReLU$ provides nonlinearity by zeroing out all the negative inputs and keeping positive ones intact. We choose $ReLU$ over other popular choices such as $Sigmoid$ or $\tanh$ for two reasons. First, it does not squash the output to a fixed interval as $Sigmoid$ or $tanh$ does. As a result, it is closest to our intuition of approximating the behavior of MF. In fact, from the point of view of MF, the effect of $ReLU$ can be considered as putting a non-negativity constraint on $h$, which is closely related to the idea of Non-negative Matrix Factorization. Secondly, $ReLU$ is fast to compute compared with its alternatives, and still provides enough nonlinearity to significantly improve the model capacity over linear structures.

For $g$ and $L$, we choose the $Sigmoid$ function combined with cross entropy loss:

(2.9)

$\displaystyle g(x)=\frac{1}{1+e^{-x}}$

(2.10)

$$L(x,\tilde{x})=-x\log{\tilde{x}}-(1-x)\log(1-\tilde{x})$$

The saturating property of the $Sigmoid$ function endows the model much flexibility since $h_{1,i}$ and $h_{2,i}$ need only to have similar activation patterns to both achieve good reconstructions; cross entropy is naturally a more appropriate choice than square loss for binary matrix. Moreover, as A is often extremely sparse, reconstructing the whole matrix incurs the class imbalance problem, which means that the loss of the positive entries is dominated by the negative entries. As a result, it is important to reweight the cost of the positive and negative classes by utilizing the cost sensitive strategy. Consequently, our final form of the loss function becomes:

(2.11)

$$L(A_{i},\tilde{A}_{i})=\sum_{j\approx i}{-\log\tilde{A}_{i,j}-\eta\sum_{j\not\approx i}\log(1-\tilde{A}_{i,j})}$$

where $\tilde{A}_{i}=g(W_{2}h_{i}+b_{2})$ is the reconstruction of $A_{i}$; $j\approx{i}$ if $A_{i,j}=1$, otherwise $j\not\approx i$. In practice, it is sufficient to approximate the second part of $L$ by a few samples; that is to say that at each training iteration we only need to sample part of the non-links for each node. Doing this may greatly speed up the training process on large graphs without sacrificing the performance. $\eta$ is the weight for the loss of the negative entries, which can be simply set as $\frac{\#nonlink\,samples}{\#links}$.

Dropout As Regularization: Regularization is critical for most machine learning models to generalize well to unseen data. Instead of putting explicit constraints on the parameters or hidden units, we use dropout training [19, 21] as an implicit regularization. Dropout is a technique originally proposed for training feed forward neural networks to avoid overfitting. It works as follows: in the stochastic gradient training, we randomly drop out half of the hidden units (for both AE and MF) and half of the feature units (for AE) for each node in the current iteration. Mathematically, the effect of dropping out can be simulated as applying an element-wise random dropout mask as follows:

(2.12)

$$\begin{split}\min\sum_{i}&\mathrm{E}_{\xi_{h,i},\xi_{in,i}}\{L(A_{i},g(W_{2}(\xi_{h,i}\odot h_{1,i})+b_{2}))\\ &+L(A_{i},g(W_{2}(\xi_{h,i}\odot h_{2,i})+b_{4}))\}\\ s.t.\;&h_{1,i}=f(W_{1}(\xi_{in,i}\odot A_{i})+b_{1}),\\ &h_{2,i}=f(W_{1}\delta_{i}+b_{3})\end{split}$$

Here $\xi_{h,i}\in\{0,1\}^{K}$ and $\xi_{in,i}\in\{0,1\}^{N}$ are the dropout masks for the hidden and input units, respectively; each element of them is an iid draw from the Bernoulli distribution. And $\odot$ is the element-wise product of two vectors. Note that we use the same dropout mask $\xi_{h,i}$ for both the AE and MF modules. This is to ensure that dropout does not cause any difference in the architecture between the two views.

For the AE module, randomly dropping out the input can be considered as a ”denoising” technique, which was exploited by the previous work [20, 6], and also was applied to link prediction [7]. The motivation is that a model should be able to make a good reconstruction under a noisy or partial input. This property is particularly interesting to our problem because this is exactly the same link prediction problem: prediction of the whole based on parts.

While theoretically explaining the effect of dropout is difficult for complex models, we can still gain an insight by looking at an approximate surrogate. Previously, [21, 6] used the second order Taylor expansion to explain the effect of feature noising in generalized linear models and AE, respectively. We can borrow the same tool to showcase a simplified version of MF+AE.

Dropout for Matrix Factorization: We consider the effect of dropout training on (2.1). For a concise articulation we ignore the bias terms; the resulting model is described as the following objective function:

(2.13)

$$O=\sum_{i,j}\mathrm{E}_{\xi_{i}}\{L(A_{i,j},g(W_{2}^{j}(\xi_{i}\odot W_{1,i}))\}$$

When $Sigmoid$ activation function with the cross entropy loss is used, we compute the second order approximation of (2.13) in a closed form as:

(2.14)

$$\begin{split}\tilde{O}=\sum_{i,j}&L(A_{i,j},g(\frac{1}{2}W_{2}^{j}W_{1,i}))\\ &+\mathrm{E}_{\xi_{i}}\{(W_{2}^{j}\odot W_{1,i}^{T})(\xi_{i}-\frac{1}{2}e)(g_{i,j}-A_{i,j})\}\\ &+\frac{1}{2}\mathrm{E}_{\xi_{i}}\{((W_{2}^{j}\odot W_{1,i}^{T})(\xi_{i}-\frac{1}{2}e))^{2}g_{i,j}(1-g_{i,j})\}\\ =\sum_{i,j}&L(A_{i,j},g(\frac{1}{2}W_{2}^{j}W_{1,i}))\\ &+\frac{1}{8}(W_{2}^{j})^{2}(W_{1,i})^{2}g_{i,j}(1-g_{i,j})\\ =\sum_{i}&L(A_{i},g(\frac{1}{2}W_{2}W_{1,i}))\\ &+\frac{1}{8}\underbrace{(\sum_{j}(W_{2}^{j})^{2}g_{i,j}(1-g_{i,j}))}_{\text{$\lambda^{i}$}}(W_{1,i})^{2}\end{split}$$

where $e$ is a column vector of all $1$s, and $(W_{2}^{j})^{2}$ and $(W_{1,i})^{2}$ are the element-wise square of the row and column vectors, respectively; $g_{i,j}$ is short for $g(\frac{1}{2}W_{2}^{j}W_{1,i})$. The first equality of (2.14) is the result of the second order Taylor expansion; the second equality performs the expectation over the random variable $\xi_{i}$ whose $K$ entries are iid Bernoulli variables; the third equality is just a reorganization. We see that with the second order approximation, the dropout effect can be split into two factors. The first term is equivalent to the original objective except that the activation of each pair is scaled down by a half. The second part is more interesting; it can be considered as the product of a row vector $\lambda^{i}$ and the square of the column vector of $W_{1}$. Note that if we set $\lambda^{i}\propto e$ , the second term is reduced to the ordinary $\ell_{2}$ regularization on each column of $W_{1}$. In the case of dropout, however, $\lambda^{i}$ is equivalent to a weighted sum of the square of the rows of $W_{2}$, where the weight of each row of $W_{2}$ is determined by the degree of uncertainty of the prediction $g_{i,j}$. The overall effect of this regularization is two folds. First, it encourages the model to make confident predictions everywhere by minimizing $g_{i,j}(1-g_{i,j})$; secondly, it performs a scaled version of $\ell_{2}$ regularization on the columns of $W_{1}$: coordinates that are highly active in the rows of $W_{2}$ are less penalized in the columns of $W_{1}$, and vice versa. In other words, the penalization on the column vectors of $W_{1}$ is adapted both to the structure of $W_{2}$ and the uncertainty of the predictions. This is in stark contrast to $\ell_{2}$ regularization where the penalization is uniformly put on each column of $W_{1}$. Finally, note that since the roles of $W_{1}$ and $W_{2}$ are exchangeable, the discussion of the regularization on the columns of $W_{1}$ also applies to the rows of $W_{2}$ by symmetry.

Dropout for Autoencoder: The nonlinear and nonsmooth nature of the ReLU activation function makes it difficult to analyze the behavior of dropout. We thus only show the case when $f$ is set as the identity function. Unsurprisingly, the effect of dropping out the hidden layer is similar to that of MF; the only difference is that we replace $W_{1,i}$ in (2.14) with $W_{1}A_{i}$ . Following similar reasoning, it is obvious to see that dropping out the hidden layer in AE again penalizes the scaled $\ell_{2}$ norm of rows of $W_{2}$ in the same way. Its effect on $W_{1}$ is more subtle: instead of penalizing the norms of the columns of $W_{1}$ directly, the regularization is performed on the linear combinations of them.

Next we proceed to study the effect of dropping out the input. Let us now denote $W=W_{2}W_{1}$ , and we have the dropout version of objective function:

(2.15)

$$O=\sum_{i}\mathrm{E}_{\xi_{i}}\{L(A_{i},g(W(\xi_{i}\odot A_{i})))\}$$

The second order approximation immediately follows as:

(2.16)

$$\begin{split}\tilde{O}=\sum_{i}&L(A_{i},g(\frac{1}{2}WA_{i}))\\ &+\sum_{j}\mathrm{E}_{\xi_{i}}\{(W^{j}\odot A_{i}^{T})(\xi_{i}-\frac{1}{2}e)(g_{i,j}-A_{i,j})\}\\ &+\frac{1}{2}\sum_{j}\mathrm{E}_{\xi_{i}}\{((W^{j}\odot A_{i}^{T})(\xi_{i}-\frac{1}{2}e))^{2}g_{i,j}(1-g_{i,j})\}\\ =\sum_{i}&L(A_{i},g(\frac{1}{2}WA_{i}))\\ +\frac{1}{8}&\sum_{j}(W^{j})^{2}\underbrace{\sum_{i}g_{i,j}(1-g_{i,j})(A_{i})^{2}}_{\text{$\lambda_{j}$}}\end{split}$$

where $W^{j}=W_{2}^{j}W_{1}$ is the $j^{th}$ row of $W$, $g_{i,j}$ is short for $g(\frac{1}{2}W^{j}A_{i})$. Recall that dropping out the hidden units in both MF and AE performs a scaled $\ell_{2}$ norm regularization on (the linear combinations of) columns of $W_{1}$; dropping out the input performs a scaled $\ell_{2}$ regularization on the linear combinations of rows of $W_{1}$. The regularization on $W_{2}$ is also very different from the case of dropping out the hidden units, but in a less clear way.

To summarize, we show that in the simplified case, dropping out the hidden units and inputs can be both interpreted as an adaptive regularization. They both push the model to make confident predictions by minimizing the factor $g_{i,j}(1-g_{i,j})$, while they penalize different aspects of the parameter matrices. When combined in the joint training architecture, they provide complementary regularization to prevent MF+AE from overfitting.

The link prediction problem can be considered as a special case of relational learning and recommender systems [27, 24], and a lot of techniques proposed are directly applicable to link prediction as well. Salakhutdinov et al. [30] first apply RBM to movie recommendation. Recently, Chen and Zhang [7] propose a variant of linear AE with marginalized feature noise for link prediction, and Li et al. [11] apply RBM to link prediction in dynamic graphs.

MF+AE is also related to the supervised learning based methods [3, 13]. While these approaches directly train a classifier on manually collected features, MF+AE directly learns the appropriate features from the adjacency matrix.

The utilization of dropout training as an implicit regularization also contrasts with Bayesian models [2, 15]. While both dropout and Bayesian Inference are designed to reduce overfitting, their approaches are essentially orthogonal to each other. It would be an interesting future work to investigate whether they can be combined to further increase the generalization ability. Dropout has also been applied to training generalized linear models [21], log linear models with structured output [28], and distance metric learning [25].

This work is also related to graph representation learning. Recently, Perozzi et al. [16] propose to learn node embeddings by predicting the path of a random walk, and they show that the learned representation can boost the performance of the classification task on graph data. It would also be interesting to evaluate the effectiveness of MF+AE in the same setting.

We propose a novel model MF+AE which jointly trains MF and AE with shared parameters. We show that dropout can significantly improve the generalization ability of both MF and AE by acting as an adaptive regularization on the weight matrices. We conduct experiments on six real world sparse graphs, and show that MF+AE outperforms all the competing methods, especially on datasets with strong non-cohesive structures.

This work is supported in part by NSF CCF-1017828, the National Basic Research Program of China (2012CB316400), and Zhejiang Provincial Engineering Center on Media Data Cloud Processing and Analysis in China.