Keywords Guided Method Name Generation

Høst and Østvold [5] propose a heuristic based approach to check name consistency and suggest appropriate method names. They construct several explicit rules based on the prior knowledge of naming conventions. An inconsistency indicates a conflict between method name and the extracted rules. Liu et al. [17] design a deep learning based model to check the name consistency, and suggest method names based on the idea of information retrieval (IR), in such a way that two methods with similar implementations should have similar names. Allamanis et al. [18] introduce a machine learning based language model, which can explore the semantic similarity of method names by embedding them into a high-dimensional continuous space.

Source code is unambiguous and highly structured, which can be reconstructed definitely according to given AST. Alon et al. [7] suggest method names via a general path-based approach which samples different paths in corresponding AST. Two methods that share similar bag-of-paths structure are close to each other. Later, Alon et al. [8] propose Code2vec, a neural network model to aggregate the semantic information of AST paths, and produce a code vector using attention mechanism. Then the code vector can be used to generate method names. However, Code2vec cannot represent unseen paths in training dataset. Alon et al. [9] further develop Code2seq to represent paths node-by-node so that each unique path can be decomposed to a collection of nodes. Additionally, Code2seq can generate neologisms which have not appeared in training corpus by splitting full tokens into subtokens. Jiang et al. [6] carry out extensive experiments to evaluate the effectiveness of Code2vec, the evaluation results reveal that Code2vec rarely works with more rigorous settings. They also propose a heuristic based approach to recommend method names.

With the powerful representation ability of neural network, many deep learning based models are devised to generate method names. Allamanis et al. [12] consider method names as the extreme summarization of source code, they propose a convolutional attention neural network that employs convolution on the code tokens to capture the structure information of source code. Nguyen et al. [11] adopt a sequence-to-sequence model with attention to predict method names, and the input tokens are collected from the three contexts including its body, the interface and the enclosing class name. Xu et al. [13] tackle method naming task by employing a hierarchical neural attention network to represent method implementation with the key idea that tokens form basic blocks and basic blocks form a code snippet. Fernandes et al. [14] propose a model for extending sequence encoders with a graph neural network. GNN-based model can mitigate the long-term dependencies in source code by specific edges. Besides, to resolve the out of vocabulary problem, they also incorporate pointer network [19] into their sequence decoder.

Code comment generation task is much similar to the abstractive sentence summarization of NLP. Thus, many strong models can be migrated to the code summarization task. Iyer et al. [20] establish the first Long Short Term Memory (LSTM) networks with attention to produce sentences that describe code snippets and SQL queries. Ahmad et al. [21] explore a transformer based model with relative position encoding to summarize source code. Transformer model has shown to be effective in capturing long-term dependencies [22]. Wei et al. [23] propose a dual training model to train the code summarization task and code generation task simultaneously to exploit the intuitive correlation between two tasks.

Above approaches just view source code as plain text, thus the structure information contained in AST are omitted. To address this problem, many studies have been investigated to incorporate the structure information into their models. Hu et al. [24] propose a structure based traversal method to traverse the AST. In such way, the structure information can be kept since the original AST can be unambiguously reconstructed from the traversal sequence. Wan et al. [25] apply an independent tree-based LSTM component to model the AST and solve the exposure bias issue by reinforcement learning. LeClair et al. [26] process code token sequence and code structure of AST as separate inputs, in this way the model can learn code structure independent of the token sequence. Furthermore, LeClair et al. [27] replace the gated recurrent unit with a graph neural network based component to model the AST. The experiment results show that graph neural network indeed improve the performance of the code summarization task.

The above approaches of method name generation and code comment generation have achieved remarkable performance. However, these deep learning models just implicitly learn the relations of important tokens in the source code with method name, while our keywords guided model can explicitly focus more on the valuable input.

The overall architecture of our keywords guided method name generation approach is depicted in Fig.2. We decompose the method naming task into two related tasks, and we propose a keywords extractor and a keywords guided method name generator to solve the two tasks respectively.

The motivation of our approach is the high proportions of common shared tokens between method name and method implementation, which can provide necessary clues for the key point of code snippet. For training, these keywords can be easily acquired by labeling the tokens appearing both in the input code token sequence and reference method name. For testing, we cannot directly obtain the reference method name to guide the labeling of keywords. Thus we propose a keywords extraction task, and we solve this task by training an independent keywords extractor in supervised way. Specifically, we apply a graph neural network based component to learn the representation of each token in the input graph, and a sigmoid layer over the node representation to decide whether a token is a keyword or not.

Our keywords guided method name generator is based on a graph-to-sequence framework, which employs a same graph component as the keywords extractor to encode the representations of input tokens. Additionally, a standard LSTM based component is developed to learn the representation of each keyword. We introduce a dual selective gate to explore the semantic relationship between input tokens and keywords before decoding. For decoding, the copy mechanism is integrated in our sequence decoder, and we apply a dual attention mechanism to combine the semantics of input code sequence and keywords, while traditional attention mechanism only works on the input token sequence.

The input of our keywords extractor is a graph constructed by source code tokens and corresponding AST. We label the overlapping tokens between the code tokens and the reference method name as 1, and other tokens as 0. The target of keywords extractor is essentially to classify each token into 1 or 0. Then we obtain the keywords by removing all the duplicated tokens predicted as 1. The keywords paired with the input graph of keywords extractor are fed into the method name generation model. We briefly introduce the details about the construction of input graph here.

An example for constructing the input graph is presented in Fig.2(a), following the works of Fernandes et al. [14], we split all code tokens into subtokens according to the naming conventions of Java. For each code snippet, the node set is formed by all the tokens, subtokens and nodes of AST, with the original method name masked by a special symbol. Besides, we utilize several meaningful edges to connect each node based on the works of Fernandes et al. [14]. Firstly, we connect each node using NextToken edge according to the order of the original source code sequence. Then the SubToken edge is added to connect each token with its subtokens. To leverage the structure information, we also connect the nodes of the parse tree with its corresponding leaves by the Child edge. Finally, we add LastLexicalUse edge to connect identifiers to their most recent use in the source code.

Unlike natural language, source code contains complicated and easy-to-acquire structure information, which can be definitively represented by AST. Traditional sequence encoder views source code as plain text, and the rich structure information is neglected. While graph neural network can be directly applied to the input graph, thus the syntactic and semantic information of source code can be captured sufficiently.

Our graph encoder is based on the graph component proposed by Fernandes et al. [14], which is developed according to the Gated Graph Neural Network (GGNN) [28]. A graph $G=(V,E,X)$ is consists of a node set $V$, a list of edge sets $E=(E_{1},...,E_{k})$, where $k$ is the number of edge types, and node embeddings $X$. Each node $u\in V$ corresponds to a node embedding $X_{u}\in R^{d_{h}}$, where $d_{h}$ is the dimension of the node embedding. The detailed procedure of message passing are as follows:

• 1.

  Each node $u$ needs to send the message to its neighbors. The message $m_{u}^{(t)}$ of each node $u$ is converted by an edge-type dependent function $f_{k}^{(t)}$ working on its current hidden state $h_{u}^{(t)}$. In our works, we use a simple linear layer to represent $f_{k}^{(t)}$ of edge type $k$ at timestep $t$. Note that we initialize the hidden state $h_{u}^{(0)}$ of each node $u$ by its associated node embedding $X_{u}$.

  $$m_{u}^{(t)}=f_{k}^{(t)}(h_{u}^{(t)})$$

  (1)

• 2.

  Each node $u$ aggregates the received messages from its neighbors by element-wise summation. $N(u)$ means the neighbors of node $u$.

  $$M_{u}^{(t)}=\sum_{v\in N(u)}m_{v}^{(t)}$$

  (2)

• 3.

  Each node $u$ updates its current state based on the aggregated messages $M_{u}^{(t)}$ , the update function is the gated recurrent unit (GRU) [29].

  $$h_{u}^{(t+1)}=GRU(M_{u}^{(t)},h_{u}^{(t)})$$

  (3)

The procedure of message passing is rolled out for timesteps $T$, and the hidden state of each node $u$ at the final timestep $h_{u}^{(T)}$ is used as node representation. In the following sections of this paper, we use $h_{u}$ to denote node representation to simplify the formula.

Besides, we obtain the global graph state $r_{g}$ by summing all the weighted node representations. The weight of each node is calculated based on the concatenation of the node representation $h_{u}$ and node embedding $X_{u}$, the details are given as follows:

$W_{i}\in R^{2d_{h}\times 1}$, $W_{j}\in R^{d_{h}\times d_{h}}$ are two learnable parameter matrices, $\sigma(\cdot)$ is sigmoid function. Then we apply element-wise multiplication on the two outputs, and at last sum all the weighted node representations.

After obtaining the node representations, the keywords extractor can determine whether a node is a keyword or not based on a simple dense layer with the sigmoid activation. We compute the cross entropy loss according to the output of sigmoid function $\hat{y}$ and ground truth labels $y$. The calculation details are given as follows:

where $W_{e}$ is a projection matrix, $\sigma(\cdot)$ is sigmoid function, and $|V|$ is the size of node set. We minimize this loss function through Adam optimizer [30].

We are inspired by the works of Li et al.[31] to encode the node representations and keywords in the dual selective way. The dual selective gate is expected to explore the relevance between the node representations and keywords. Therefore, the unnecessary information contained in node representations can be filtered out by the key selective gate since the keywords state contains more salient semantics. While the keywords representations can be adaptively adjusted by the graph selective gate since the graph state contains more global semantics.

In our works, we use a standard bidirectional LSTM to encode the keyword representations. The BiLSTM reads the keyword embeddings and encode keywords forwardly and backwardly to generate hidden states:

$X_{i}$ is the embedding of $i_{th}$ keyword, and the $i_{th}$ keyword representation $h^{k}_{i}$ is the concatenation of the forward and backward hidden states:

Then we concatenate the representation of the first keyword $h^{k}_{1}$ and the last keyword $h^{k}_{n}$ to obtain the keywords state $r_{k}$:

dual selective gate The encoded keywords state contains more salient semantics of input, which can guide the encoder to pay more attention on the important node representations. Meanwhile, the information contained in the keywords is incomplete, so more global semantic information contained in graph state is required to adaptively adjust the keyword representations. Therefore we apply a dual selective gate to explore the latent relevance between graph nodes and keywords. The calculation details are given as follows:

where $W_{k},U_{k},W_{g},U_{g}\in R^{d_{h}\times d_{h}}$ are parameters, $GraphGate_{i}$ is the selective gate for $i_{th}$ keyword, and $KeyGate_{u}$ is the selective gate for node $u$. Note that the selective gate vector has the same dimension as the node representations as well as keyword representations. Then, we encode the node representations and keyword representations by computing the element-wise multiplication on the selective gate and original representations.

Our decoder is based on the LSTM unit, and we use the graph state $r_{g}$ to initialize the decoder state. To sufficiently exploit the salient information contained in encoded keyword representations, we apply a dual attention mechanism to integrate the node information in input graph and keyword information. Besides, Out-of-vocabulary (OOV) tokens are ubiquitous in source code, thus we incorporate copy mechanism employed in pointer generator [16] to solve the OOV problem.

A general sequence decoder calculates hidden state $s_{t}$ according to the embedding of current input token $X_{t}$ and previous hidden state $s_{t-1}$ at timestep $t$.

Then we compute the attention scores of nodes in input graph based on the $s_{t}$ and node representations $h^{\prime}_{u}$, and obtain the context vector $c^{n}_{t}$ of nodes by weighted summing all the node representations.

where $W^{\prime}_{a}\in R^{d_{h}\times 1},W_{a}\in R^{2d_{h}\times d_{h}}$ are parameters, $b_{a}$ is bias. $\alpha^{t}$ is the attention score of current hidden state to each node representation.

Similarly, we use hidden state $s_{t}$ and keyword representations $h^{k^{\prime}}_{i}$ to calculate the attention scores of keywords, and compute the context vector $c^{k}_{t}$ of keywords. It is noteworthy that the parameters of keywords attention mechanism are independent to the nodes attention mechanism.

Then we perform a linear transformation $U_{c}\in R^{2d_{h}\times d_{h}}$ on the concatenation of the nodes context vector $c^{n}_{t}$ and the keywords context vector $c^{k}_{t}$ to obtain the final context vector of current decoding step.

The output probability distribution of current timestep $t$ can be calculated based on the hidden state $s_{t}$ and context vector $c_{t}$.

where $W^{\prime}_{v}\in R^{d_{h}\times|vocab|}$, $W_{v}\in R^{2d_{h}\times d_{h}}$, $b^{\prime}_{v}\in R^{|vocab|}$, $b_{v}\in R^{d_{h}}$ are learnable parameters, $|vocab|$ denotes the vocabulary size. $P_{vocab}$ is a probability distribution of the output, and the output dimension corresponds to vocabulary size which informs decoder which token in the vocabulary should be used as the output at current decoding step.

Since the OOV tokens are common in source code, $P_{vocab}$ cannot contain the probability of OOV tokens. We refer to pointer generator [16] to copy the OOV tokens based on the attention distribution of input tokens. The pointer generator uses a soft switch to decide whether to copy a token from input token sequence or to output a token from vocabulary. The calculation details of soft switch are given as follows:

where $W_{s},W_{c},W_{x}\in R^{d_{h}\times 1},b_{gen}\in R^{1}$. At decoding step $t$, decoder reads hidden state $s_{t}$, context vector $c_{t}$ and embedding of the input token $X_{t}$ to produce a scaler $P_{gen}^{t}\in[0,1]$. $P_{gen}^{t}$ denotes the probability for generating a token according to the vocabulary distribution $P_{vocab}$, and $1-P_{gen}^{t}$ denotes the probability for copying a token based on the attention distribution $P_{copy}$ where each attention score points to a token in source code. Thus the final output probability distribution is calculated as follows:

At each decoding step, decoder outputs a token based on the $P(w)$, the loss function is the negative log likelihood of the generated token. Then the whole encoder-decoder model is trained in an end-to-end way, the overall loss for generated method names is:

generalization ability of the model evaluated across projects is better than the model verified across files. To ensure the fairness of comparative experiments, we reuse the train-validation-test splits originally divided, 9 for training, 1 for validation and 1 for testing. More statistics about the Java-small dataset are presented in Table I.

We compare our approach with following baselines:

• •

  ConvAttention. Allamanis et al. [12] use a convolutional attention neural network to summarize the code snippet.

• •

  Code2seq. Alon et al.[9] aggregate several sampled paths of AST and use attention mechanism to select relevant paths to suggest method names.

• •

  BiLSTM+Copy. Fernandes et al.[14] generate method names by incorporating sequence-to-sequence model with the copy mechanism.

• •

  GNN+Copy. Fernandes et al.[14] propose a graph-to-sequence framework to model the syntactic and semantic information of source code.

• •

  BiLSTM+GNN+Copy. Fernandes et al.[14] extend sequence-to-sequence model with a graph component to capture the structure information, which achieves state-of-the-art results on the method naming task.

In our experiment, we use 256-dimensional hidden state to represent input tokens and keywords. The gated graph neural network is rolled out for 4 timesteps. We remove all the tokens whose frequency of occurrence is less than five to build the vocabulary. When training, the correct keywords can be easily acquired, but the mismatching problem between training and testing leads to poor performance. Therefore we use the keywords predicted by the keywords extractor as the input to train the whole model. We initialize the Adam optimizer with the learning rate 0.0005 and decay the learning rate

by 0.95 per 3000 steps. More experiment settings about our approach can be found in publicly available code repository ¹¹1https://github.com/css518/Keywords-Guided-Method-Name-Generation.

To ensure the fairness of the comparative experiments, we keep the hyper parameters consistent with our approach and train the state-of-the-art model with their original implementations. The evaluation results are similar as their original paper reported, thus we reuse the experiment results presented in their paper.

Our proposed approach is referred to KG-MNGen (Keywords Guided Method Name Generator). We compare KG-MNGen with several strong baselines under the ROUGE-1, ROUGE-2 and ROUGE-L evaluation metrics. The evaluation results are presented in Table II. We have run the KG-MNGen for many times and the fluctuation of experiment results is slight, around 0.3%. Therefore the average results are reported.

We can see from the table that ConvAttention obtains the lowest results among several baselines, which might due to the split-by-projects choice. ConvAttention is devised to predict method names across-files in its original paper. Additionally, all the graph based baselines achieve higher results than sequence based baselines. This observation indicates that effective modeling of structure information indeed boost the generation of accurate method names. Finally, KG-MNGen achieves the highest results on all the evaluation metrics. Compared with the state-of-the-art model BiLSTM+GNN+Copy, KG-MNGen improves absolute ROUGE-1, ROUGE-2 and ROUGE-L score about 2.0%, 1.4% and 3.6% respectively. We argue that the reason is the definition of ROUGE-L, which uses longest common subsequences to calculate the evaluation score. KG-MNGen can locate the key information effectively, thus KG-MNGen generates method names centered around the keywords, and the common subsequence is relatively longer.

In summary, KG-MNGen obtains the highest results among several strong baselines on all the evaluation metrics. The significant gains brought by KG-MNGen show that keywords guidance can indeed boost the effectiveness of method name generation.

To make full use of the positive signals brought by keywords, we adopt two strategies in the

encoder and decoder of KG-MNGen. One is the dual selective gate applied in encoder, which consists of two gates, including key gate and graph gate. Another is the dual attention mechanism applied in decoder. Therefore, to sufficiently explore the different effects of each strategy, we compare KG-MNGen with several variants to understand the influence of dual attention mechanism and dual selective gate. The detailed evaluation results are reported in Table III.

As we can see from Table III, firstly, the complete model KG-MNGen is superior to all the comparative variants under all the evaluation metrics, which indicates the effectiveness of the dual selective gate and dual attention mechanism.

Secondly, compared to the w/o DualGate+DualAtten, both the dual selective gate equipped model w/o DualAtten and the dual attention mechanism equipped model w/o DualGate have made significant improvements under all the evaluation metrics. The experiment results prove that dual selective gate and dual attention mechanism can be applied separately, both the dual selective gate and dual attention mechanism are helpful to integrate the keywords signals.

Thirdly, we compare the influence of KeyGate and GraphGate respectively. The evaluation results of w/o DualGate are slightly lower than w/o KeyGate and w/o GraphGate, which proves the usefulness of each selective gate. And the performance improvements brought by KeyGate are slightly higher than GraphGate, which might due to the key information contained in the keywords, while GraphGate contains more comprehensive information.

In summary, both the dual selective gate and dual attention mechanism are effective to incorporate the positive keywords guidance.

We notice that the length of samples in the test dataset differ seriously, thus we compare

KG-MNGen with the state-of-the-art model BiLSTM+GNN+Copy under different input length in terms of the f1, precision and recall of ROUGE-1 to explore the fine-grained effects of different input lengths. The detailed results are presented in Fig.3. We use the number of the code tokens to denote the input length.

Firstly, all models show a natural descent as the input length increases. KG-MNGen outperforms BiLSTM+GNN+Copy almost across all input lengths, especially when input length between 20 and 30 tokens. Actually, input less than 30 tokens accounts for 47.3% of the test dataset. Secondly, as we can see from Fig.3 (b) and Fig.3 (c), the improvements of precision score are higher than recall score, which might due to the key information contained in keywords, the decoder can generate more accurate tokens. Thirdly, we also count the number of generated tokens, which are reported in Fig.3 (d). BiLSTM+GNN+Copy is prone to generate longer method name as the input length increases, while KG-MNGen tends to generate stable length for method name. This observation can explain why two models obtain similar recall score. Higher recall means more tokens in reference method name can be found in generated method name, thus it becomes simple when more tokens are generated.

In summary, our approach KG-MNGen outperforms the strong baseline almost across all input lengths on the f1, precision and recall of ROUGE-1. The keywords can provide useful guidance under different input lengths.

To explore the compatibility of keywords guidance methodology to other encoder-decoder frameworks, we equip two baselines with keywords signals, including BiLSTM+Copy and BiLSTM+GNN+Copy. The results are shown in Fig.4. All the evaluation metrics of two baselines have improved differently under the keywords guidance. In terms of BiLSTM+Copy, ROUGE-1 score improvements are roughly 3%, while ROUGE-2 and ROUGE-L score improvements are slight. In terms of BiLSTM+GNN+Copy, the improvement of evaluation metrics are 1.1%, 1.2% and 2.6% respectively. The experiment results are approximately linear related to the keywords.

In summary, the keywords can provide effective guidance to other encoder-decoder framework, instead of just suitable for our proposed model.

Obviously, the performance of our approach is closely related to the quality of extracted keywords. Therefore, we extract keywords of method in different ways to explore the influence of the quality of keywords. The keywords selection strategies are as follows:

• •

  Random. We randomly select 4 tokens from code token sequence as keywords. Note that we define several regular expressions to filter some inappropriate tokens, such as symbols and numbers, etc.

• •

  TextRank[33]. We select top 4 keywords according to the importance score computed by TextRank of each token.

• •

  TF-IDF. We use TF-IDF to calculate the importance score of each token, instead of Textrank.

• •

  Keywords Extractor. We train a GNN based keywords extactor in supervised way to extract the keywords of input.

• •

  Reference. We directly use the reference method name of test dataset to label the common shared tokens as keywords, which is impossible in practice.

The evaluation results are reported in Table IV. We use the ROUGE score of extracted keywords against ground truth method name to indicate the quality of keywords. The experiment results are consistent with our intuition, random strategy obtains the worst results, TF-IDF strategy is slightly better than TextRank strategy, while keywords extractor strategy achieves a significant performance improvement compared to the unsupervised TF-IDF and TextRank strategy. However, there is still a very large performance gap relative to the reference strategy, which can be regarded as the upper bound of extracted keywords. Additionally, it is normal that ROUGE-2 scores of all experiments are extremely poor since the extracted keywords are independent of each other, thus they cannot form a highly readable method name.

Then we input KG-MNGen with above different keywords to explore the correlation between the quality of keywords and model performance. The evaluation results are recorded in Table IV Predictions against Ground Truth. Firstly, it can be concluded from the experiment results that the performance of the model is roughly proportional to the quality of the keywords, which also reveals the effectiveness of our keywords guided method name generation approach. Secondly, it is a little surprise to us that random strategy achieves slightly worse experiment results with TextRank. This is contrary to our assumptions since we originally think that random keywords might provide misleading guidance for the model and reduce the performance of the model. Thirdly, ROUGE-2 score of all experiments obtain a significant improvement, which due to the sequence decoding of language model. Higher ROUGE-2 score means highly readable and fluent method name. Lastly, reference strategy achieves promising results, far superior to other keywords selection strategy. However, reference strategy cannot be applied in actual since the reference method name of test dataset are unavailable. Nevertheless, we can optimize our keyword extractor to generate more accurate keywords. Note that our keywords extractor is relatively simple, only using graph neural network for representation. Thus there are still many aspects of keywords extractor can be enhanced, as a result more accurate keywords lead to better performance.

In summary, firstly, the keywords guided method name generation approach is highly related to the quality of keywords. Secondly, inspired by the high performance brought by reference strategy, the method name generation task can be improved significantly by optimizing the keywords extraction task. Additionally, compared to the method name generation task, the keywords extraction task is easier to optimize. But optimizing the keywords extraction task has a definite upper bound on performance.

In this subsection, we present the discussion on the comparison of generated method names between KG-MNGen and the state-of-the-art baseline BiLSTM+GNN+Copy to qualitatively evaluate the superiority of our approach.

KG-MNGen is based on the common shared token information between method body and reference method name. Therefore, we split the test dataset into five parts according to the number of shared tokens. Then we compute the ROUGE-1 score on each part of the test dataset. The detailed results are reported in Table V.

Firstly, the performance of two approaches shows a stable increasing tendency when the number of shared tokens is less than 4. We can conclude that it is relatively easy to generate method names when more tokens are common shared since high proportions of key information are already explicitly contained in method body. Besides, it is normal that the ROUGE-1 score decreases when the number of shared tokens is greater than 4 since the average length of generated method names of two approaches is less than three tokens. Therefore, the ROUGE-1 score shows a natural descent as the lengths of method name increase.

Secondly, both approaches obtain poor performance when there are no shared tokens. It means that key information is not explicitly shown in method body. Therefore a descriptive method name requires the model to fully comprehend the program as well as developers. Unfortunately, it is much harder to achieve this. Besides, the performance of KG-MNGen is slightly higher than BiLSTM+GNN+Copy when the number of shared tokens is greater than 2. As aforementioned, it is relatively easy to understand the program when more common tokens are shared. Therefore, BiLSTM+GNN+Copy can achieve similar results as our approach in relatively simple cases.

Thirdly, it is hard to generate method names when just one common token is shared. A little key information is explicitly contained in method body. Thus the model needs to comprehend the program based on the key information. As we can see from Table V, BiLSTM+GNN+Copy obtains poor results and has a wide margin compared with other experiments of BiLSTM+GNN+Copy, about 9.04% and 10.7% lower than the experiments of 2 and 3 shared tokens. While KG-MNGen bests BiLSTM+GNN+Copy by significant margins of 6.5%. This finding indicates that KG-MNGen is much superior than BiLSTM+GNN+Copy in hard cases. Additionally, the size of each part of the test dataset shows that condition with only one shared token is more common. Therefore, we can conclude that our approach can generate more accurate method names in hard cases than BiLSTM+GNN+Copy.

Finally, we perform a more detailed comparison in the condition with only one shared token to evaluate how the 6.5% improvements are presented in generated method names. As we all know, there are some template methods in Java program, such as setXxx, getXxx, toString, equals and hashCode,etc. These methods are easily to predict method names since they share similar structures. So we further verify our approach after removing above simple methods. The evaluation results are reported in Table VI. Our approach gains an absolute ROUGE-1 score about 7.8% over the baseline after removing above listed template methods. The wide margin indicates that our approach is more effective with rigorous settings compared with BiLSTM+GNN+Copy.

To sum up, in terms of the generated method names between KG-MNGen and BiLSTM+GNN+Copy, KG-MNGen can generate more accurate method names in different code cases. Especially in complex programs, KG-MNGen outperforms BiLSTM+GNN+Copy by significant margins of 7.8%.

We present some examples of generated method names by KG-MNGen and BiLSTM+GNN+Copy in Fig.5, and we mark the extracted keywords in red. Firstly, although some noise are contained in our keywords, KG-MNGen is able to focus more on the accurate keywords and generate correct method name. While BiLSTM+GNN+Copy is easily affected by the messy information in the source code, thus misunderstanding the key function of program, such as example 3. The function of example 3 is to test whether the program can read the input normally, but BiLSTM+GNN+Copy only focuses on the smallBuffer. Secondly, none of tokens are shared in example 4. This case needs model to fully comprehend the program as well as developers. While KG-MNGen predicts the method name of example 4 accurately. Thirdly, example 1 denotes the condition with one shared token. BiLSTM+GNN+Copy might focus on the boolean and ret$\textgreater$=0, thus mistaking the key function of this program for compare. While KG-MNGen predicts the accurate method name under the keywords guidance.