Attention-based Graph Convolution Fusing Latent Structures and Multiple Features for Graph Neural Networks

In the spectral methods, the aggregation operation is carried out in the graph Fourier domain. Eigenvectors of graph Laplacian are known as the graph Fourier bases [1, 19, 8]. In order to reduce the computational complexity for large graphs, polynomial approximations of graph filters are often utilized [10, 20, 11].

GCN [12] further reduces computational complexity through the first-order graph filters. It can be formulated as follows:

where $H:=\{\textit{h_v}\}_{\textit{v}\in\mathcal{V}}$ is the set of node-wise features, $\widetilde{A}=A+I_{n}$ is the adjacency matrix with self-loops, and $W$ is a learnable weight matrix.

Spatial methods are a counterpart of spectral methods. As we described above, spatial methods can be classified into direct and attention-based methods. Direct spatial methods directly use the node-wise features and the aggregation operation is carried out spatially. A representative method for direct spatial methods is GraphSAGE [13], which treats each neighbor equally with mean aggregation.

Later, attention-based spatial methods are proposed. Instead of treating all neighboring nodes equally, attention-based methods calculate an attention weight for each neighboring node. Then, they use the weighted sum to aggregate features of neighboring nodes. GAT [16] is a representative method for attention-based spatial methods. It is composed of three steps. In the first step, the learnable weights are multiplied by the node-wise features, i.e., $h^{\prime}_{\mathcal{V}}=\{W\cdot h_{v}\}_{v\in\mathcal{V}}$, where $W$ is the learnable weights. The second step computes attention weights as follows:

where $W_{a}$ is learnable weights. In the third step, the node-wise features of a target node $v$ is updated as follows:

However, as mentioned earlier, existing methods ignore the structural information of neighboring nodes in the high-dimensional space, and use only one type of attention function when calculating an attention weight for each neighboring node.

In contrast to the existing methods, we first focus on computing a representation of the structure of neighboring nodes in high-dimensional feature space and installing them into a spatial GC. Second, we use multiple types of attention functions simultaneously in an AGC. We use both of these two methods simultaneously to achieve SFAGC.

Feature angle describes the local structure of neighboring nodes. First, a set of structure vectors pointing from target node $v$ to the neighboring nodes are calculated as $\text{SV}_{\textit{N(v)}}={\{\text{sv}_{uv}:=h_{u}-h_{v}\}}_{u\in N(v)}$. Then, a base structure vector $\text{sv}_{b}$ is learned from $\text{SV}_{N(v)}$ as follows:

where W_b is learnable weights. An example of a base structure vector ${sv}_{\textit{b}}$ is shown in Fig. 3

Finally, the cosine of the angle between $\text{sv}_{uv}$ and $\text{sv}_{b}$ is calculated to obtain the feature angle $\text{fa}_{uv}$ as follows:

An example is shown in Fig. 4 (a).

The second structural feature is feature distance. It is the absolute difference between the node-wise features of $h_{u}$ and $h_{v}$ represented as follows:

An example is shown in Fig. 4 (b).

The third structural feature is relational embedding. It can be learned from $\{\text{sv}_{uv}\}$ as follows:

where $W_{\text{re}}$ is learnable weights. An example of it is shown in Fig. 4 (c).

The inputs of this step are the feature angle, feature distance and relational embedding, i.e., $\text{fa}_{uv}$, $\text{fd}_{uv}$ and $\text{re}_{uv}$, introduced in Section II-B. We first compute structure vectors as follows:

where $W_{\text{se}}$ is a learnable weight matrix.

Then, we project each $\text{s}_{uv}$ as follows:

Finally, we calculate the summation of the projected structure vectors as follows:

Fig. 7 illustrates an example of the local structure projection aggregation.

In this step, we fuse the $\text{af}_{v}$ with the $h_{v}$ as follows:

where $W_{s}$ is learnable weights.

As we described above, we simultaneously use multiple-type attention functions to calculate attention weights. In existing methods [23, 24, 25], the subtraction or dot-production attention function is often utilized to calculate attention weights. Instead of the single attention function, we use these attention functions simultaneously. The subtraction attention function is defined as

where $W_{q1}$ and $W_{k1}$ are learnable weights. The dot-production attention function is represented as

where $W_{q2}$ and $W_{k2}$ are learnable weights.

Then, the two types of attention functions are added with the position embedding p_u as follows:

where $W_{c}$ is learnable weights that also converts $\text{a}_{2vu}$ into the same dimensions as $\text{a}_{1vu}$.

Finally, $\text{qk}_{vu}$ is input into a small network to calculate the attention weights between the target node $v$ and the neighboring node $u$ as follows:

where $W_{a1}\in\mathbb{R}^{d_{\text{in}}\times d_{out}}$ and $W_{a2}\in\mathbb{R}^{d_{out}\times 1}$ are learnable weights, $d_{in}$ and $d_{out}$ are the dimensions of the $W_{a1}$.

For $h^{\prime}_{v}$ and $h^{\prime}_{u},u\in N(v)$, weighted sum aggregation is calculated as follows:

where $W_{v}$ is a learnable matrix.

$h_{v}$, $co_{v}$ and $\widetilde{h_{v}}$ are integrated as follows:

where $W$ is learnable weights.

We use a layer of PointNet++ [29] to generate a low-resolution point cloud. Both global and local information of the point cloud can be obtained through the multi-resolution structure.

Effective graph pooling methods are a hot topic in GNNs and graph signal processing [30, 31]. Early work has been done by global pooling of all node-wise features or by using graph coarsening algorithms.

Recently, trainable graph pooling operations DiffPool [32], GraphU-net [33], and AttPool [34] have been proposed. Inspired by the ResNeXt [35], we extend the score-based graph pooling module proposed in SAMGC [22] by introducing the multi-branch architecture. The score-based graph pooling module has three branches: score-based sampling, integration, and SFAGC branches. The architecture of the score-based graph pooling is shown in Fig. 9. In the following, we introduce their details.

In the score-based sampling branch, we propose a scroe-based sampling to find the indices of the best $t$ nodes according to their scores. The score associated with each node is first computed as follows:

where $W_{1}$ is learnable weights. We then sort the nodes in descending order according to their scores. We find the indices of the top $t$ nodes as follows:

where $\text{rank}(\cdot)$ is the ranking operation, and it finds the indices of the $t$ highest scores.

In the integration branch, node-wise features are multiplied by the node scores as follows:

In the SFAGC branch, the input graph is processed using SFAGC as follows:

Finally, the subset of $\hat{h}_{\mathcal{V}}$ and the subset of $\{h^{\prime}_{\mathcal{V}},{co}^{\prime}_{\mathcal{V}}\}$ are found using $\text{idx}_{select}$ as follows:

$\hat{h}_{\mathcal{V}_{sub}}$ and $h^{\prime}_{\mathcal{V}_{sub}}$ are merged using learnable weights as follows:

where $W_{pl}$ is learnable weights. The score-based graph pooling can be summarized as follows:

Here, we also use the intermediate supervision technique [36] and the hierarchical prediction architecture (Fig. 8 (a)), as in SAMGC [22]. The advantage of this architecture is that, by combining the outcomes of the different phases, more reliable and robust predictions can be produced [22]. The details are presented below.

We use two SFAGC modules in phase 1 to phase 5. One PointNet++ [29] layer is used in phase 6. Each phase connects with a max pooling layer and an average pooling layer. The outputs are then concatenated and input into a fully connected layer. We calculate the prediction and the classification loss for each phase. The total classification loss is obtained by adding the losses of several phases. Meanwhile, the total prediction is also increased by the predictions of several phases. The following is a representation of this processing:

where $prediction_{i}$ is the prediction of the $i$th phase, $loss_{i}$ is the cross-entropy loss of the $i$th phase. $prediction$ and $loss$ are the total prediction and classification loss, respectively. $P$ is the number of phases.

For point cloud segmentation, we use the graph pooling module to reduce the overall computation cost. Here, we propose the farthest point sampling-based graph pooling (FPS-based graph pooling) by modifying the score-based graph pooling. The architecture of the FPS-based graph pooling is illustrated in Fig. 9. In the following, we introduce its details.

The FPS-based graph pooling has a multi-branch architecture like the score-based graph pooling in Section IV-C2. In contrast to the score-based method, the FPS-based graph pooling has two branches, i.e., the FPS and SFAGC branches.

In the FPS branch, we perform FPS on nodes to obtain indices of the best $t$ nodes according to their coordinates. FPS algorithm is widely utilized in 3D point cloud processing [29, 37]. The mechanism of FPS is to iteratively select the node that is farthest from the existing sampled nodes. Therefore, the sampled nodes with the FPS-based sampling are expected to be more evenly distributed than those with the score-based sampling. This branch can be summarized as follows:

where $\text{idx}_{select}$ is the indices of the $t$ selected nodes, $\text{FPS}(\cdot)$ is the farthest point sampling algorithm.

The SFAGC branch is the same as the one that is used in the score-based graph pooling represented as

Finally, the subset of $\{h^{\prime}_{\mathcal{V}},{co}^{\prime}_{\mathcal{V}}\}$ are extracted using $\text{idx}_{select}$ as follows:

The FPS-based graph pooling is represented as follows:

Graph pooling can be regarded as a downsampling operation. For point cloud segmentation, the network also needs to perform upsampling in order to maitain the number of points. Therefore, the feature propagation used in PointNet++ [29] is also used in our network as the upsampling operation.

The ModelNet dataset [17] is used in our point cloud classification experiment. 12,308 computer-aided design (CAD) models in 40 categories are included in ModelNet40. In addition, 9,840 CAD models are utilized for training and 2,468 CAD models are used for testing. 4,899 CAD models are included in ModelNet10. They are divided into 3,991 for training and 908 for testing from ten categories. For each CAD models, the CAD mesh faces were evenly sampled with 1,024 points. Initially, all point clouds were normalized to be in a unit sphere.

The settings of hyperparameters are summarized in Table I. We use the average accuracy of all test instances (OA) and the average accuracy of all shape classes (mAcc) to evaluate the performance of our network.

Table II summarizes the results for point cloud classification. The results of existing methods are taken from their original papers. In terms of both OA and mAcc, our method performs better than the others.

In the following, we focus on the comparison of our method and graph-based methods. The GCs utilized in DPAM [45] are GCN [12]. Therefore, it only uses node-wise features of one-hop neighboring nodes. RGCNN [46], 3DTI-Net [47], PointGCN [48], and LocalSpaceGCN [49] are the spectral methods. Although they can design the global spectral response, they may neglect the local spatial information. In comparison with the direct spatial methods [28, 50, 27, 51, 52, 53, 54], our method can obtain the local structural information of the graph in the feature space, and the information of neighboring nodes can be utilized efficiently using attention-based aggregation. In comparison with the direct spatial methods, i.e., SAGConv [21] and SAMGC [22], the proposed method can better aggregate the structural information of the neighboring nodes using the local structure projection aggregation. Furthermore, the information of neighboring nodes can be utilized efficiently using attention-based aggregation. These are possible reasons for the performance improvement of the proposed method.

The ShapeNet dataset [18] is used in the experiment. It contains 16,846 computer-aided design (CAD) models in 40 categories and each point cloud has 2,048 points. 2,874 point clouds are used for testing, and 13,998 of them are taken for training.

Table III shows the hyperparameter settings. For this experiment, we re-perform existing methods with their corresponding codes available online. The hyperparameters (epoch, batch size, learning rate, and drop out) used in the existing methods experiments are the same with those shown in Table III.

We evaluated the average mIoU of all test instances with other neural networks designed specifically for point cloud segmentation.

Experimental results for point cloud segmentation are summarized in Table IV. It is observed that our method has higher mIoU than the existing methods.

Here, we discuss the possible reasons of the improvement. We first focus on the comparison between our method and DGCNN [27]. DGCNN corresponds to a graph-based direct spatial method. In contrast to the direct method, our method can utilize the local structural information in the feature space, and it also collects the information of neighboring nodes through attention-based aggregation.

We also compare our method with transformer-based methods PointASNL [37] and PCT [39]. While the transformer basically has a large number of parameters, they are restricted to use only node-wise features and the single dot-production attention function to calculate attention weights. In contrast, SFAGC utilizes the local structural information in the feature space and the multiple-type attention functions.