Unsupervised Domain Adaptive Graph Classification

A graph is denoted as $G=(\mathcal{V},\mathcal{E})$, where $\mathcal{V}$ is the set of nodes and $\mathcal{E}$ is the set of edges. The graph node feature is $\bm{X}\in\mathbb{R}^{|V|\times d}$, each row $\bm{x}_{v}\in\mathbb{R}^{d}$ represents the feature vector of node $v\in\mathcal{V}$ and $d$ is the dimension of node features, $|\mathcal{V}|$ is the number of nodes. In our settings, we are given a labeled source domain $\mathcal{D}^{s}=\{(G_{i}^{s},y_{i}^{s})\}_{i=1}^{N_{s}}$ containing $N_{s}$ labeled samples and an unlabeled target domain $\mathcal{D}^{t}=\{G_{j}^{t}\}_{j=1}^{N_{t}}$ with $N_{t}$ examples. $\mathcal{D}^{s}$ and $\mathcal{D}^{t}$ sharing the same label space $\mathcal{Y}=\{1,2,\cdots,C\}$ but have distinct data distributions in the graph space. Our purpose is to train the graph classification model on the labeled $\mathcal{D}^{s}$ and unlabeled $\mathcal{D}^{t}$ to obtain the competitive result on the target domain.

Graph Convolutional Network Branch. We use a graph convolutional network to implicitly capture the topology of the graph. For each node, we combine the embeddings of its neighbors from the previous layer. Then we update the node embedding by merging it with the combined neighbor embedding. The embedding of node $v$ at layer $l$ is $\bm{h}_{v}^{(l)}$:

$$\bm{h}_{v}^{(l)}=\operatorname{COM}^{(l)}_{\theta}\left(\bm{h}_{v}^{(l-1)},\operatorname{AGG}^{(l)}_{\theta}\left(\left\{\bm{h}_{u}^{(l-1)}\right\}_{u\in\mathcal{N}(v)}\right)\right),$$

where $\mathcal{N}(v)$ denotes the neighbors of $v$. $\operatorname{AGG}^{(l)}_{\theta}$ and $\operatorname{COM}^{(l)}_{\theta}$ denote the aggregation and combination operations parameterized by $\theta$ at the $l$-th layer, respectively. Then, we utilize the $\operatorname{READOUT}$ function to summarize the node representations, and introduce a multilayer perception (MLP) classifier $H(\cdot)$ for final prediction:

$$z=F(G)=\operatorname{READOUT}\left(\bm{h}_{v}^{(L)}\right),\;\bm{p}=H(z),$$

(1)

where $z$ denotes the graph-level representation and $\bm{p}$ is final prediction.

Graph Kernel Network Branch. Given two graph samples $G_{1}=(V_{1},E_{1})$ and $G_{2}=(V_{2},E_{2})$, graph kernels calculate their similarity by comparing their local-substructure using a kernel function. In formulation,

$$K\left(G_{1},G_{2}\right)=\sum_{v_{1}\in V_{1}}\sum_{v_{2}\in V_{2}}\kappa\left(l_{G_{1}}\left(v_{1}\right),l_{G_{2}}\left(v_{2}\right)\right)$$

(2)

where $l_{G_{1}}(v_{1})$ represents the local substructure centered at node $v_{1}$ and $\kappa(\cdot,\cdot)$ is a pre-defined similarity measurement. We omit $l_{G}(\cdot)$ and leave $\kappa(u_{1},u_{2})$ in Eq. 2 for simplicity.

In this part, we apply the Weisfeiler-Lehmah (WL) Subtree Kernels as the default kernel branch. WL subtree kernels compare all subtree patterns with limited depth rooted at every node. Given the maximum depth $l$, we have:

	$\displaystyle K_{\text{subtree }}^{(i)}\left(G_{s}^{i},G_{t}^{j}\right)$	$\displaystyle=\sum_{v_{1}\in V_{s}^{i}}\sum_{v_{2}\in V_{t}^{j}}\kappa_{\text{subtree }}^{(i)}\left(u_{1},u_{2}\right)$		(3)
	$\displaystyle K_{WL}\left(G_{s}^{i},G_{t}^{j}\right)$	$\displaystyle=\sum_{i=0}^{l}K_{\text{subtree }}^{(i)}\left(G_{s}^{i},G_{t}^{j}\right)$

where $\kappa_{\text{subtree }}^{(i)}\left(u_{1},u_{2}\right)$ is derived by counting matched subtree pairs of depth $i$ rooted at node $u_{1}$ and $u_{2}$, respectively. We would assign the source label to the similar target graphs.

In this part, our goal is to use perturbations on source graphs to mitigate the impact of domain inconsistencies on feature representations with target semantics. We employ adversarial training to identify promising perturbation directions. Taking the GCN branch as an example, we obtain feature representations and label predictions for both the source and target domains. We train a domain discriminator $D(\cdot)$ on each branch to differentiate between the two domains. For each source graph $G^{s}_{i}$, we introduce noise $\bm{\delta}(\bm{H}^{s}_{i})$ to the node embedding $\bm{H}^{s}_{i}$. The direction of perturbation is determined through a minimax optimization process, and $\bm{\delta}$ is updated based on the gradient descent direction for each source graph $G^{s}_{i}$.

	$\displaystyle\min_{||\delta(\cdot)||_{F}\leq\epsilon}\max_{\theta_{d}}\mathcal{L}_{DA}^{C}$	$\displaystyle=\mathbb{E}_{G^{s}_{i}\in\mathcal{D}^{s}}\log D(F(G^{s}_{i};\bm{H}^{s}_{i}+\bm{\delta}(\bm{H}^{s}_{i})),{\bm{p}}^{s}_{i})$
		$\displaystyle+\mathbb{E}_{G^{t}_{j}\in\mathcal{D}^{t}}\log(1-D(F(G^{t}_{j}),{\bm{p}}^{t}_{j})),$
	$\displaystyle\bm{\delta}_{t+1}$	$\displaystyle=\bm{\delta}_{t}-\epsilon\phi\left(\bm{\delta}_{t}\right)/\left\|\phi\left(\bm{\delta}_{t}\right)\right\|_{F},$		(4)

where $\theta_{d}$ is the parameters of the domain discriminator and $\epsilon$ is the maximum of perturbation size. $\phi\left(\bm{\delta}_{t}\right)=\nabla_{\bm{\delta}}\log D(F(G^{s}_{i};\bm{H}^{s}_{i}+\bm{\delta}(\bm{H}^{s}_{i})))$ is the gradient of the loss for $G^{s}_{i}$ with respect to the perturbation $\bm{\delta}$. Similarity, we can get the perturbation $\bm{\zeta}$ on the GKN branch by optimize the minimax objective, i.e., $\min_{||\zeta(\cdot)||_{F}\leq\epsilon}\max_{\theta_{d}^{\prime}}\mathcal{L}_{DA}^{K}$.

The primary training objective of DAGRL combines adversarial perturbation loss and target data classification loss. Additionally, minimizing the expected source error for labeled source samples is essential.

$$\mathcal{L}_{S}=\mathbb{E}_{G_{i}^{s}\in\mathcal{D}^{s}}\mathcal{E}(H(\hat{\bm{z}}_{i}^{s}),y_{i}^{s}),$$

(5)

where $\hat{\bm{z}}_{i}^{s}$ denotes the representations of source data under perturbations. Consequently, to update the network parameters, we minimize the ultimate objective as follows:

$$\mathcal{L}=\mathcal{L}_{S}-\lambda_{1}\mathcal{L}_{DA}^{C}-\lambda_{2}\mathcal{L}_{DA}^{K},$$

(6)

where $\lambda_{1}$ and $\lambda_{2}$ are hyper-parameters to balance the domain adversarial loss and classification loss. Meanwhile, we need to update the perturbations of source samples, i.e., $\bm{\delta}$ and $\bm{\zeta}$, and then update the model parameters interactively.

Datasets. To evaluate the proposed method’s effectiveness, we conducted experiments using our method on several real-world datasets: Mutagenicity (M) [10], and Tox21 ¹¹1https://tripod.nih.gov/tox21/challenge/data.jsp. (T) from the TUDataset [11]. Mutagenicity dataset consists of 4337 molecular structures and their corresponding Ames test data. Tox21 dataset is used to assess the predictive ability of models in detecting compound interferences. To address domain distribution variations within each dataset, we divided them into four sub-datasets: $D$0, $D$1, $D$2, and $D$3 ($D$ represents each respective dataset) based on edge density [8].

Baselines. To evaluate the effectiveness of our method, we compare the proposed method with a large number of state-of-the-art methods, including five graph approaches (WL subtree [12], GCN [13], GIN [14], CIN [15] and GMT [16]), and three recent domain adaptation methods (CDAN [17], ToAlign [18], MetaAlign [19]) and two graph domain adaptation methods (DEAL [7], CoCo [8]).

In our implementation, a two-layer GIN [14] is employed in the GCN branch, while a two-layer network with the Weisfeiler-Lehman (WL) kernel [12] is used in the GKN branch. The Adam optimizer with a learning rate of $10^{-4}$ is used by default. Both the embedding dimension of hidden layers and the batch size are set to 64.

Table 1 and 2 display the performance of graph classification in different unsupervised domain adaptation scenarios. The results indicate the following observations: 1) Domain adaptation methods outperform graph kernel and GNN methods, suggesting that current graph classification models lack transfer learning capabilities. Therefore, an effective domain adaptive framework is crucial for graph classification. 2) While domain adaptation methods achieve competitive results in simple transfer tasks, they struggle to make significant progress in challenging transfer tasks compared to GIN. This difficulty is attributed to the complexity of acquiring graph representations, making direct application of current domain adaptation approaches to GCNs unwise. 3) DAGRL achieves superior performance compared to all baselines in most cases. This improvement is attributed to two key factors: (i) Dual branches (GCN and GKN) enhances graph representation in situations with limited labels. (ii) Adversarial perturbation learning facilitates effective domain alignment.

To evaluate the impact of each component in our DAGRL, we introduce several model variants as follows: 1) DAGRL/P1: It removes the perturbation on the GCN branch; 2) DAGRL/P2: It removes the perturbation on the GKN branch; 3) DAGRL-GIN: It uses two different GINs to generate graph representations; 4) DAGRL-GKN: It uses two different GKN to generate graph representations.

We conducted performance comparisons on Mutagenicity, and the results are presented in Table 3. Based on the observations from the table, the following findings can be summarized: 1) DAGRL/P1 and DAGRL/P2 exhibit worse performance compared to DAGRL. This confirms that adaptive adversarial perturbation learning is effective in achieving domain alignment. 2) Both DAGRL-GIN and DAGRL-GKN produce similar results but perform worse than DAGRL. This suggests that relying solely on ensemble learning in either implicit or explicit manner does not significantly enhance graph representation learning when labels are scarce.