Progressive Graph Convolutional Networks
for Semi-Supervised Node Classification

Convolutional Neural Networks (CNNs) [1], as an end-to-end deep learning paradigm, have been very successful in many machine learning and computer vision tasks [2, 3, 4]. While CNNs have high ability to extract latent representations and local meaningful statistical patterns from data, they can only operate on Euclidean data structures such as audio, images and videos which have a form based on $1$D, $2$D and $3$D regular grids, respectively. Recently, there has been an increasing research interest in applying deep learning approaches on non-euclidean data structures, like graphs, which lack geometrical properties, but have high ability to model complex irregular data such as social networks [5, 6] citation networks and knowledge graphs [7].

Motivated by CNNs, the notion of convolution was generalized from grid data to graph structures, which correspond to locally connected data items, by aggregating the node’s neighbors’ features with its own features [8]. Combining such convolutional operator with a data transformation process and hierarchical structures Graph Convolutional Networks (GCNs) are obtained, which can be trained in an end-to-end fashion to optimize an objective defined on individual graph nodes, or the graph as a whole. As an example, the GCN method proposed in [6] is a semi-supervised node classification method consisting of two convolutional layers. Each graph convolutional layer learns nodes’ features by applying aggregation rule on their corresponding first-order neighbors. The graph structure can also be learned dynamically [9]. GCNs, have been successful in various graph mining tasks like node classification, [10, 11], graph classification [12], link prediction [13], and visual data analysis [14].

One of the main drawbacks of existing GCN-based methods is that they require an extensive hyper-parameter search process to determine a good topology of the network. This process is commonly based on extensive experimentation involving the training of multiple GCN topologies and monitoring their performance on a hold-out set. To avoid such a computationally demanding process, a common practice is to empirically define a network topology and use it for all examined problems (data sets). As an example, a two layer GCN model cannot gain sufficient global information by aggregating the information from just a two-hop neighborhood for each node. Meanwhile, in the reported results of [6] it has been shown that adding more layers makes the training process harder and does not necessarily improve the classification performance. It was also shown that the topology of the GCN plays a crucial role in performance which is related to the underlying difficulty of the problem to be solved [15].

Problem-specific design of the neural networks’ architecture contributes in improving the performance and the efficiency of a learning system. Recently, methods of finding an optimized network topology have been receiving much attention and many works were proposed to define compact network topologies by employing various learning strategies, such as compressing pre-trained networks, adding neurons progressively to the network, pruning the network’s weights and applying weight quantization [16, 17, 18, 19]. However, all these methods work with grid data in Euclidean spaces. In this regard, learning a compact topology for GCNs makes a step towards increasing the training efficiency and reducing the computational complexity and storage needed while achieving comparable performance with existing methods.

In this paper, we propose a method to jointly define a problem-specific GCN topology and optimize its parameters, by progressively growing the network’s structure both in width and depth. The contributions of our work are:

• •

  We propose a method to learn an optimized and problem-specific GCN topology progressively without user intervention. The resulting networks are compact in terms of number of parameters, while performing on par, or even better, compared to other recent GCN models.

• •

  We provide a convergence analysis for the proposed approach, showing that the progressively building GCN topology is guaranteed to converge to a (local) minimum.

• •

  We conduct experiments on widely-used graph datasets and compare the proposed method with recently proposed GCN models to demonstrate both the efficiency and competitive performance of the proposed method. Our experiments include an analysis of the effect of the network’s complexity with respect to the underlying complexity of the classification problem, highlighting the importance of network’s topology optimization.

The rest of the paper is organized as follows: Section II provides a description of graph-based semi-supervised classification, along with the terminology used in this paper. Section III reviews the GCN method [6] as the baseline method of our work. The proposed method is described in detail in section IV. The conducted experiments are described in section V, and conclusions are drawn in Section VI.

Let $G=(\mathcal{V},\mathcal{E})$ be an undirected graph where $\mathcal{V}$ and $\mathcal{E}$ denote the set of nodes $\nu_{i}\in\mathcal{V},\>i=1,\dots,N$ and the set of edges $(\nu_{i},\nu_{j})\in\mathcal{E}$, respectively. $\mathbf{A}\in R^{N\times N}$ denotes the adjacency matrix of $G$ encoding the node connections. The elements $\mathbf{A}_{ij}$ can be either binary values, indicating presence or absence of an edge between nodes $\nu_{i}$ and $\nu_{j}$, or real values encoding the similarity between $\nu_{i}$ and $\nu_{j}$, based on a similarity measure. Using $\mathbf{A}$, we define the degree matrix $\mathbf{D}$ which is a diagonal matrix with elements equal to $\mathbf{D}_{ii}=\sum_{j}\mathbf{A}_{ij}$. Each node of the graph $\nu_{i}$ is also equipped with a representation $\mathbf{x}_{i}\in\mathbb{R}^{D}$, which is used to form the feature vector matrix $\mathbf{X}\in R^{N\times D}$. When such a feature vector for each graph node is not readily available for the problem at hand (e.g. in the case of processing citation graphs), vector-based node representations are learned by using some node embedding method, like [20, 21].

Traditional graph-based semi-supervised node classification methods [22, 23, 24], learn a mapping from the nodes’ feature vectors to labels, which for a $C$-class classification problem are usually represented by $C$-dimensional vectors following the $1$-of-$C$ encoding scheme. This mapping exploits a graph-based regularization term and it commonly has the form:

where $f(\cdot)$ denotes the learnable function mapping the $D$-dimensional node representations to the $C$-dimensional class vectors, $\mathbf{T}_{L}$ is a matrix formed by the class label vectors of the labelled nodes which form the matrix $\mathbf{X}_{L}$, and $\mathbf{L}=\mathbf{D}-\mathbf{A}$ is the unnormalized graph Laplacian matrix. The first term in (1) is the classification loss of the trained model measured on the labelled graph nodes, and the second term corresponds to graph Laplacian-based regularization incorporating a smoothness prior to the optimization function $\mathcal{L}$. $\lambda>0$ expresses the relative importance of the two terms. By following this approach the labels’ information of the labelled graph nodes is propagated over the entire graph.

GCNs are mainly categorized into spatial-based and spectral-based methods. The spatial-based methods update the features of each node by aggregating its spatially close neighbors’ feature vectors. In these methods, the convolution operation is defined on the graph with a specified neighborhood size which propagates the information locally [10, 25, 26, 5].

The spectral-based GCN methods follow a graph signal processing approach [6]. Let us denote by $G_{\theta}$ a filter and by $\mathbf{X}$ a multi-dimensional signal defined over the $N$ nodes of the graph. The signal transformation using $G_{\theta}$ is given by:

where $\star$ denotes the convolution operator, $\mathbf{U}$ is the matrix of eigenvectors of the normalized graph Laplacian $\tilde{\mathbf{L}}=\mathbf{I}_{N}-\mathbf{D}^{-\frac{1}{2}}\mathbf{A}\mathbf{D}^{-\frac{1}{2}}=\mathbf{U}\boldmath{\Lambda}\mathbf{U}^{T}$ with $\boldsymbol{\Lambda}$ being a diagonal matrix having as elements the corresponding eigenvalues, and $\mathbf{U}^{T}\mathbf{X}$ being the graph Fourier transform of $\mathbf{X}$. Since computing the eigen-decomposition of $\tilde{\mathbf{L}}$ is computationally expensive, low-rank approximations using truncated Chebyshev polynomials have been proposed [27]. The transformation in (2) corresponds to the building block of a GCN. Followed by an element-wise activation it forms a GCN layer.

The multilayer GCN for semi-supervised node classification was proposed in [6] by stacking multiple GCN layers. To achieve a fast and scalable operation a first-order approximation of the spectral graph convolution is proposed leading to:

where $\tilde{\mathbf{A}}=\mathbf{I}_{N}+\mathbf{A}$ and $\tilde{\mathbf{D}}_{ii}=\sum_{j}\tilde{\mathbf{A}}_{ij}$. Let us denote by $\mathbf{H}^{(l)}$ the graph node representations at layer $l$ of the multi-layer GCN. The propagation rule for calculating the graph node representations at layer $l+1$ is given by:

where $\mathbf{W}^{(l)}$ is the $l^{th}$ layer weight matrix and $\sigma(\cdot)$ denotes the activation function, such as $ReLU(\cdot)$, or $softmax(\cdot)$ used for the output layer. For a two layer GCN model this leads to:

where $\hat{\mathbf{A}}=(\tilde{\mathbf{D}}^{-\frac{1}{2}}\tilde{\mathbf{A}}\tilde{\mathbf{D}}^{-\frac{1}{2}})$ and $\mathbf{Y}\in\mathbb{R}^{N\times C}$ denotes the predicted feature vectors for all the $N$ graph nodes in $C$ classes. The model parameters ($\mathbf{W}^{(0)},\mathbf{W}^{(1)}$) are finetuned by minimizing the cross entropy loss over the labeled nodes.

One of the drawbacks of all existing GCN methods is that they use a predefined network topology, which is selected either based on the user’s experience, or empirically by testing multiple network topologies. In the next section, we describe a method for automatically determining a problem-specific compact GCN topology based on a data-driven approach.

PGCN follows a data-driven approach to learn a problem-dependant compact network topology, in terms of both depth and width, by progressively building the network’s topology based on a process guided by its performance. That is, the learning process of GCN jointly determines the network’s topology and estimates its parameters to optimize the cost function defined at the output of the network.

The learning process starts with a single hidden layer formed by one block of $B$ neurons equipped with an activation function (e.g. $ReLU(\cdot)$) and an output layer with $C$ neurons. At iteration $t=1$, the synaptic weight matrix $\mathbf{W}_{1}^{(1)}\in\mathbb{R}^{D\times B}$ connecting the input layer to the hidden-layer neurons is initialized randomly. The graph nodes’ representations defined at the outputs of the hidden layer $\mathbf{H}_{1}^{(1)}\in\mathbb{R}^{N\times B}$, where the index indicates that the one block of $B$ hidden-layer neurons is used, are obtained by using graph convolution:

By setting $\mathbf{H}^{(1)}=\mathbf{H}_{1}^{(1)}$, the network’s output for all graph nodes $\mathbf{Y}\in\mathbb{R}^{N\times C}$ is calculated using a linear transformation as follows:

where $\mathbf{O}\in\mathbb{R}^{B\times C}$ denotes the weight matrix connecting hidden layer to the output layer. The transformation matrix $\mathbf{O}$ can be calculated by minimizing the regression problem:

where $Tr(\cdot)$ denotes the trace operator, $\lambda_{1}>0$ denotes the relative importance of model loss, $\mathbf{H}_{L}^{(1)}$ denotes the hidden layer representations of the labeled graph nodes and $\mathbf{T}_{L}\in\mathbb{R}^{N\times C}$ is a matrix formed by the labeled nodes’ target vectors.

To exploit the information in both the labeled and unlabeled nodes’ feature vectors in $\mathbf{H}^{(1)}$, we can replace the linear regression problem in (8) by a semi-supervised regression problem exploiting the smoothness assumption of semi-supervised learning [28, 24] expressed by the term:

Minimization of (9) with respect to $\mathbf{O}$ leads to node feature vectors at the output of the network which are similar for nodes connected in the graph. To incorporate this property in the optimization problem of (8) the term in (9) is added as a regularizer. Thus, $\mathbf{O}$ is obtained by minimizing:

where $N$ is the number of all (labeled and unlabeled) graph nodes and $\lambda_{2}$ denotes the relative importance of Laplacian regularization and is set to $\frac{1}{N^{2}}$ which is the natural scale factor to estimate the Laplacian operator empirically [22].

The optimal solution of (10) is obtained by setting ${\partial\mathcal{J}_{2}}/{\partial\mathbf{O}}=0$, and is given by:

where $\mathbf{I}_{N}\in\mathbb{R}^{N\times N}$ and $\mathbf{I}_{D^{(1)}}\in\mathbb{R}^{{D^{(1)}}\times{D^{(1)}}}$ are identity matrices. ${D^{(1)}}$ denotes the feature dimension of the first hidden layer which is equal to $B$ in the first iteration.

After the initialization of both the hidden layer and output layer weights, these are finetuned based on Backpropagation using the loss of the model on the labeled graph nodes. Finally, the model’s performance (classification accuracy on the labeled nodes) $\mathcal{A}_{1}$ is recorded.

At iteration $t=2$, the network’s topology grows by adding a second block of $B$ hidden layer neurons. We initialize the weights $\mathbf{W}_{1}^{(1)}$ connecting the input layer to the first block of hidden layer neurons with finetuned values obtained at iteration $t=1$, while the weight matrix of the newly added block $\mathbf{W}_{2}^{(1)}$ is initialized randomly. The hidden layer representations corresponding to the newly added neurons $\mathbf{H}_{2}^{(1)}$ are calculated as in (6) by replacing $\mathbf{W}_{1}^{(1)}$ with $\mathbf{W}_{2}^{(1)}$. Then, we combine $\mathbf{H}^{(1)}=[\mathbf{H}_{1}^{(1)},\mathbf{H}_{2}^{(1)}]$ and the output weight matrix $\mathbf{O}$ is calculated by using (11). At this iteration, $D^{(1)}=B+B$ and $\mathbf{H}^{(1)}$, $\mathbf{O}$ are of size $N\times D^{(1)}$ and $D^{(1)}\times C$, respectively.

After finetuning all the parameters $\mathbf{W}_{1}^{(1)}$, $\mathbf{W}_{2}^{(1)}$, $\mathbf{O}$, and recording the model’s performance, the network’s progression is evaluated based on the rate of performance improvement given by:

where $\mathcal{A}_{1}$ and $\mathcal{A}_{2}$ denote the model’s performance before and after adding the second block of hidden layer neurons. If the addition of the second GCN block does not improve the model’s performance, i.e. when $r<\epsilon$, the newly added block is removed, and all the model parameters are set to the finetuned values which were obtained at previous iteration. At this point, the progression in the current hidden layer terminates.

After stopping the progressive learning in the first hidden layer, a new hidden layer is formed which takes as input the previous hidden layers’ output. The block-based progression of the newly added hidden layer starts by using a single block of $B$ neurons and repeats in the same way as for the first hidden layer until model’s performance converges.

Let us assume that at iteration $t=k$, the network’s topology comprises of $l$ layers (the input layer corresponds to $l=0$) and it is growing at the ${(l+1)}^{th}$ layer giving as outputs $\mathbf{H}^{(l+1)}$. Before adding the $b^{th}$ block formed by $B$ neurons, the weights of all the existing blocks in the network are set to the finetuned values obtained in the previous step. The newly added block in the ${(l+1)}^{th}$ hidden layer takes the output of the previous hidden layer $\mathbf{H}^{(l)}$ as input, and the graph convolutional operation for the $b^{th}$ block is given by:

where $\mathbf{H}_{b}^{(l+1)}$ and $\mathbf{W}_{b}^{(l+1)}$ denote the hidden representation of the newly added block and the randomly initialized synaptic weights connecting the $l^{th}$ hidden layer to $b^{th}$ block of ${(l+1)}^{th}$ layer, respectively. Given $\mathbf{H}^{(l+1)}=[\mathbf{H}^{(l+1)},\mathbf{H}_{b}^{(l+1)}]$ which denotes the hidden representations formed by using both the existing blocks and the newly added block in the ${(l+1)}^{th}$ hidden layer, the models’ output $\mathbf{Y}\in\mathbb{R}^{N\times C}$ is calculated using a linear transformation as follows:

where $\mathbf{O}\in\mathbb{R}^{D^{(l+1)}\times C}$ denotes the transformation matrix which is calculated based on semi-supervised linear regression:

Similar to other GCN-based methods, the linear transformation for calculating the models’ output in (14) (and (7), respectively) can also be followed by softmax activation (on a node basis) as follows:

and instead of Mean Squared Error (MSE), the Cross-Entropy (CE) loss function can be employed for finetuning the model.

After the initialization step, the synaptic weights of the existing blocks and all the weight parameters of the newly added block are finetuned with respect to the labeled data using (8). The model diagram is shown in Fig. 1 which indicates the progressive learning process in ${(l+1)}^{th}$ layer where a new block is added and after initializing its weights (dashed lines), all the model parameters are finetuned together. Fig. 2 indicates the graph convolution process which is applied on graph node by a block of the $B$ neurons. This convolution process gets as input the hidden representation matrix of nodes at layer $l$, $\mathbf{H}^{(l)}$, and outputs the representation matrix $\mathbf{H}^{(l+1)}$ at layer $l+1$.

To evaluate the network’s progression, the model’s performance is recorded and the rate of the improvement is given by:

where $\mathcal{A}_{b-1}$ and $\mathcal{A}_{b}$ denote the classification accuracy before and after adding the $b^{th}$ block, respectively. If the addition of a new GCN block in step $k$ does not improve the model’s performance, i.e. when $r_{b}<\epsilon_{b}$, the progression of the $(l+1)^{th}$ layer terminates and all the network parameters are set to the finetuned values obtained in the previous step with $b-1$ blocks. After stopping the block progression for the $(l+1)^{th}$ layer, the algorithm evaluates whether the network’s performance converged using the rate of the performance before and after adding the new hidden layer:

When $r_{l}<\epsilon_{l}$, the last hidden layer $l+1$ is removed and the algorithm stops growing the network’s topology. Here we should note that it is also possible to use other performance metrics, such as model loss, to evaluate the network progression process in (17) and (18).

Algorithm 1 summarizes the PGCN algorithm. In A, we show that the proposed approach of building GCN layers in a progressive manner converges.

In this paper, we proposed a method for progressively training graph convolutional networks for semi-supervised node classification which jointly defines a problem-dependant compact topology and optimizes its parameters. The proposed method employs a learning process which utilizes the input to each layer data to grow the network’s structure both in terms of depth and width. This is achieved by operating an efficient layer-wise propagation rule leading to a self-organized network structure exploiting data relationships expressed by their vector representations and the adjacency matrix of the corresponding graph structure. Experimental results on three commonly used datasets for evaluating graph convolutional networks on semi-supervised classification indicate that the proposed method outperforms the baseline method GCN and performs on par, or even better, compared to more complex recently proposed methods in terms of classification performance and efficiency.