DeepGS: Deep Representation Learning of Graphs and Sequences for Drug-Target Binding Affinity Prediction

Figure 1 shows the overview of DeepGS. It takes the symbolic sequences of target/protein and drug, as well as the molecular structure of the drug as the input. It outputs the binding affinity for the drug-target pair. Remind that the central idea of DeepGS is to consider both local chemical context and the molecular structure, by using some embedding techniques (i.e., Smi2Vec and Prot2Vec) to encode the amino acids and atoms to a distributed representation. Therefore, we design DeepGS as a three-step framework for DTA prediction: {enumerate*}

Encoding symbolic tokens in target/drug sequences;

Encoding the whole drug/target sequences and the molecular structure of the drug;

Predicting the binding affinity value based on the encodings of the drug and the target.

Specifically, motivated by Word2Vec [20], in the first step we encode the symbols in the sequence of the target/protein and the drug to a distributed representation, by using Prot2Vec and Smi2Vec, respectively. Then, the sequences can be transformed into matrices, where each row is the representation of a symbol in the sequences. In the second step we extract features, from the drug/target matrices and the molecule graph, to encode the whole sequences and graph. For the target/protein, we consider the local chemical context of the amino acids, by using a convolutional neural network (CNN). For the drug, we consider both the molecular structure and the local chemical context. Particularly, since the molecular structure can be represented by a graph, we suggest a graph attention network (GAT) based approach to extract the topological information of the drug. In the meantime, the local chemical context of atoms in the drug is captured, by using a bi-directional gated recurrent unit (BiGRU). As a result, we obtain a latent representation for the target and two latent presentations for the drug. To predict the binding affinity, in the third step DeepGS inputs the concatenation of the three latent representations to a stack of fully connected layers, and outputs a real value binding affinity. Next, we present the details of our method.

Targets/proteins are often represented as a sequence of amino acids (e.g., MKKFFDSRREQ… shown in Figure 1). Similar to the SMILES string, we propose to first encode the amino acids into a $d$-dimensional vector following Prot2Vec [2], which allows us to capture local chemical information in targets/proteins. As a single amino acid often makes no sense, we apply a fixed-length $N$-gram splitting approach to partition the sequence into meaningful “biological words”. Note that, here the sequence refers to the fixed-length input protein sequence (instead of the full sequences), which is preprocessed as similar as we handle the SMILES strings (recall Section 2.2). Compared to the commonly used label encoding methods, the fixed-length $N$-gram divides the sequence into a sequence of $N$-grams. Each $N$-gram is considered as a “biological word”. Intuitively, it can generates more “words context” than label encoded by one-hot encoding.

Considering that there are generally 20 kinds of amino acids, rendering that the maximum number of possible N-grams is 20^N. To make trade-off between the training feasibility and vocabulary size, in our paper we define $N=3$. Specifically, given a protein or target sequence $L=\{x_{i}|(i=1,2,...,|l|)\}$, where $x_{i}$ represents the $i$-th amino acid and $|l|$ represents the sequence length, the fixed-length 3-gram splitting method partitions the sequence into the following 3-grams, each of which is a biological word consisting of 3 amino acids: $[x_{1};x_{2};x_{3}],[x_{4};x_{5};x_{6}],...,[x_{|l|-2};x_{|l|-1};x_{|l|}]$. For each biological word, we map it to an embedding vector by looking up a pre-trained embedding dictionary for 9048 words [2], which is obtained from Swiss-Prot (https://www.uniprot.org/) with 560,118 manually annotated sequences. As a result, we transform each target sequence to a matrix, in which each row is the embedding of a biological word. The matrix is then fed into a CNN to extract the local chemical context of the target. It is worth noting that, different from the early ligand-based approach [13] that neglects the local context information in targets/proteins, our solution above leverages the embedding technique to learn the representation from the protein sequence.

In this study, we look on drug-target prediction as a regression task by predicting the binding affinity values. With the representation learned from the previous sections, we can integrate all the information from drugs and targets to predict the binding affinity. In brief, we concatenate all the representations and feed them to three fully-connected dense layers to output the affinity value. More precisely, for the GAT block, we use two graph attention layers to update the node vectors in a graph considering their neighbor nodes. For the CNN block, we use three consecutive 2D-convolutional layers. And for the BiGRU block, we use one BiGRU layer. Besides, we use Rectified Linear Unit (ReLU) [22] as the activation function, which has been commonly adopted in deep learning research. Given a set of drug-target pairs and the ground-truth affinity values in the training dataset, we can use the mean square error (MSE) as the loss function: $\mathcal{L}_{MSE}=\frac{1}{N}\sum_{i=1}^{N}{(\hat{y}_{i}-y_{i})^{2}}$, where $\hat{y}_{i}$ is the predicted value, $y_{i}$ is the ground-truth value, and $N$ represents the number of drug-target pairs.

To examine the competitiveness of the proposed model, we compared DeepGS with state-of-the-art models (including classic and deep learning models) used for DTA prediction. Table 2 reports the average CI, MSE, $r_{m}^{2}$ and AUPR scores on the Davis dataset.

From this table, we can see that, on the whole classic methods such as KronRLS perform worse than deep learning-based methods. This is because classic methods rely heavily on hand-crafted features and the similarity matrices of drugs and targets. In contrast, deep learning-based approaches capture more information via automatic feature engineering with CNN and GNN. In addition, we find that our method performs better than other two deep learning-based methods. The reason could be that (i) compared to DeepCPI, our method jointly considers topological structures and local chemical context, which is benefit to the performance; (ii) compared to DeepDTA, we incorporate GAT model to obtain the topological information of drug and advanced embedding techniques which bring more contextual information than one-hot vectors for modeling both drugs and targets.

Overall, this set of experiments demonstrate that our proposed method DeepGS outperforms all these baselines in all metrics. This is a very encouraging result. It is worth noting that, although the improvements seem to be small at the first glance, it is essentially a non-trivial achievement in terms of DTA prediction.

Besides the comparison on the Davis dataset, we also conduct the comparison on the KIBA dataset. Table 3 shows the comparison results. It can be seen that, the overall performance tendency is similar to that on the Davis dataset. For example, the performance of KronRLS is inferior to that of deep learning-based approaches, the performance of DeepCPI is inferior to that of DeepDTA, and our method exhibits better performance on almost all these metrics. This further demonstrates the competitiveness of DeepGS. Note that, in terms of CI metric, our method still has the comparable performance to DeepDTA, since the value of our method is only slightly smaller than that of DeepDTA. The possible reason is that, the KIBA dataset comes from multiple sources (e.g., $K_{i}$, $K_{d}$ and $IC_{50}$, recall Section 3.1), the data heterogeneity in KIBA dataset may make a negative effect on the CI metric of our model.

In this section, we conduct more experiments to analyze our model. In the first experiment, we further examine the prediction performance of our model based the predicted value $(p)$ and measured value $(m)$. In the second experiment, we examine the sensitiveness of our model by using various sequence lengths.

Figure 3 plots the predicted value $(p)$ and measured value $(m)$ on these two datasets. Note that, a good model should hold that predicted value $(p)$ is close to the measured value $(m)$, and thus the samples should fall close to the dashed ($p=m$) line. One can see that, for the Davis dataset, the dense area of the $pK_{d}$ value is in the range of 5 to 6 in terms of x-axis. This is because the $pK_{d}$ value of 5 constitutes more than half of the dataset (i.e., 20,931 out of 30,056, as reported from [23]). In addition, we observe that the dense area of the KIBA score is in the range of 10 to 14 in terms of x-axis. The reason is similar to that for the Davis dataset. Particularly, for both datasets, the samples are close to the dashed ($p=m$) line. This justifies, from another perspective, that the proposed solution has a good prediction performance.

To investigate the sensitiveness of our model, a simple way is to remove some information of the input sequences, and then to test the model’s prediction performance. In this paper, we use the following scheme that not only can remove some information of input sequences but also can partially reflect the impact of sequence length. Specifically, we fix the length of the input sequences at some value, say $l$, when training BiGRU and CNN. To achieve this, we cut the input sequence if the length of the input sequence is longer than $l$, and otherwise we use the zero-padding at the end of the input sequence. As for the drug’s input sequence (i.e., SMILES), we set its length to [50, $100$, 500], as shown in Table 1. Note that, the average length of SMILES sequences in the Davis dataset is 64 and the bold refers to the default value. Correspondingly, we set the length of target/protein’s input sequence (i.e., amino acids) to [500, $1000$, 2000]. Here, the average length of protein sequences in the Davis dataset is 788. For clarity, we use $L_{ds}$ and $L_{ps}$ to denote the Length of drug sequence and that of protein sequence, respectively.

Figure 2 shows the results when we set $L_{ds}$ to 50, 100 and 500, respectively. Meanwhile, Figure 4 shows the results when we set $L_{ps}$ to 500, 1000 and 2000, respectively. We observe that, (i) the performance gap between $L_{ds}=100$ and $L_{ds}=500$ is very tiny, and the performance gap between $L_{ps}=1000$ and $L_{ps}=2000$ is not so obvious; (ii) the performance gap between $L_{ds}=50$ and $L_{ds}=100$ can be easily perceived, and the performance gap between $L_{ps}=500$ and $L_{ps}=1000$ is obvious. This phenomenon is a little bit strange at the first glance. To dig out the reason behind it, we plot the distribution of sequence lengths, as shown in Figure 6. It can be seen that the lengths of most SMILES sequences are less than 100 and larger than 50. Thus, it is natural that the performance gap between $L_{ds}=100$ and $L_{ds}=500$ is very tiny, since almost all SMILES sequences do not need to be cut even if $L_{ds}=100$, i.e., few information is missing. However, when $L_{ds}=50$, many SMILES sequences may need to be cut, and so the performance degrades. With the similar argument, it is not hard to understand that the performance gap between $L_{ps}=500$ and $L_{ps}=1000$, since most protein sequences need to be cut when $L_{ps}=500$. The reason for the relatively small performance gap between $L_{ps}=1000$ and $L_{ps}=2000$ can be inferred with the similar argument. This result may imply that when the sequence length $l$ is set to a value larger than the average length of sequences in the dataset, the performance degradation could be trivial.

As mentioned before, existing models for DTA prediction have leveraged the topological structure to learn the representation for drug and target/protein, while they often ignored the local chemical context. Thereby, this work considers both local chemical context and topological structure to learn the interaction between drugs and targets. More precisely, the core idea of DeepGS is to fully leverage local chemical context, by using advanced embedding techniques, to better learn the drug and target representations. To study the effectiveness of the central idea, we implemented a variant of our model, called DeepGS1. This variant model removes both drug’s and protein’s local chemical context information obtained by Smi2Vec and Prot2Vec from the framework. The detailed configuration of DeepGS1 is illustrated in Table 4. In the following experiments, we use the same experimental settings mentioned in Table 1.

Figure 5 shows the comparison results. It can be seen that DeepGS1 is basically inferior to DeepGS in terms of CI, MSE, $rm^{2}$ and AUPR. These results demonstrate that combining the local chemical context information, which reflects the functionalities of the atoms, is benefit to learning a good representation for drugs and proteins, improving the prediction performance.

In addition to examining the effectiveness of the central idea, we also conduct another experiment, which is used to answer the following interesting question. Recall Section 2.2.2, we develop a GAT-based molecular structure modeling approach. One could argue that, GNN (Graph Neural Network) can also map a graph to a vector that encodes the topological structure of the graph, since recent work [31] have showed that GNN can effectively model drugs. To address it, we implemented another variant of our model, called DeepGS2. Compared to our model, the major difference is that it uses a GNN-based molecular structure modeling approach (cf., Table 4).

The experimental results are also shown in Figure 5. We can see that, the variant DeepGS2 is obviously poorer than our model. These results justify our choice in Section 2.2.2. The reasons could be two-fold: (i) the molecular structure may contribute a lot to the prediction performance; and (ii) The changes to the molecular structure modeling approach are sensitive to the model, especially when local chemical context is also considered in the model.