A Comprehensive Evaluation of Parameter-Efficient Fine-Tuning on Software Engineering Tasks

Transformer (Vaswani et al., 2017) is a deep learning model to solve sequence-to-sequence tasks. It can handle long text dependencies and has the flexibility to handle variable-length input and output sequences, making it suitable for tasks such as machine translation, text summarization, and sentiment analysis. Transformer has significantly influenced the NLP field as well as many other fields, such as SE.

The transformer has many key components, including a multi-head attention (MHA) layer, a fully connected feed-forward network (FFN), layer normalization, and residual connection. The Transformer architecture with these key components is shown in Figure 2.

The core of Transformer is the self-attention mechanism, which allows it to weigh the importance of different elements in a sequence. The self-attention mechanism enables the model to capture relationships between words or tokens. The calculation of attention function depends on the three components of queries $Q\in\mathbb{R}^{n\times d}$, keys $K\in\mathbb{R}^{m\times d}$ and values $V\in\mathbb{R}^{m\times d}$, where $n$ and $m$ are the number of queries and key-value pairs, and $d$ denotes the hidden size of the Transformer. The scaled dot-product attention can be formulated as follows:

Based on self-attention, MHA performs the attention function in parallel over $N_{h}$ heads, where $i$th head is parameterized by $W^{i}_{q},W^{i}_{k},W^{i}_{v}\in\mathbb{R}^{d\times d_{h}}$ to project inputs to queries, keys, and values. In MHA, $d_{h}$ is typically set to $\frac{d}{N_{h}}$ to save parameters, which indicates that each attention head is operating on a lower dimensional space. Given a sequence of $m$ vectors $X\in\mathbb{R}^{m\times d}$ and a query vector $x\in X$ (the inputs of queries, keys, and values of self-attention are the same), the attention in the $i$th head can be computed as:

Then, given the initial parameters $W_{0}\in\mathbb{R}^{d\times d}$, the MHA concatenates the attention results of different heads as follows:

FFN consists of two linear transformations with an activation function of ReLU in between, formulated as follows:

where $W_{1}\in\mathbb{R}^{d\times d_{m}},W_{2}\in\mathbb{R}^{d_{m}\times d}$ are trainable parameters, and $d_{m}$ is typically set to 4d and acts as a scaling factor.

Transformer has become the foundation model and has been adapted to various models, which are called PTMs. PTMs are usually trained on extensive datasets over pre-training tasks. Pre-training tasks are specially designed general tasks that his somewhat related to downstream tasks, allowing PTMs to learn effective representations that can be applied to a wide range of downstream tasks. Transformer consists of encoders and decoders. Encoders process the input sequence, while decoders generate the output sequence based on the encoder’s context and the previously generated tokens. PTMs are built upon the Transformer architecture and often use multiple encoders and/or decoders. They can be divided into encoder-only, decoder-only, and encoder-decoder models, as is shown in Figure 3.

PTMs are becoming more important because non-pre-training methods require large labeled datasets for training, which may result in slow training, information loss, and model instability. The common paradigm of using PTMs is to pre-train and then fully fine-tune, and it has proven advantages in many SE downstream tasks, such as clone detection and code search. Full fine-tuning requires tuning all parameters of a PTM, including embedding layers, MHA layers, and FFN layers, which enables the PTM to transfer learned text representations to unknown tasks. However, as PTMs become larger, the cost of full fine-tuning increases. Many techniques have been proposed to reduce the heavy computational demands of large models (Gou et al., 2021; Chang et al., 2021; Shi et al., 2022; Svyatkovskiy et al., 2021) and a popular strategy is PEFT. PEFT focuses on optimizing fine-tuning that only tunes a small number of parameters or learns external modules for new tasks while keeping most of the PTMs’ parameters frozen. PEFT can achieve similar effects with full fine-tuning and is particularly useful when computing resources and memory are limited.

Next, we introduce the PEFT methods used in this paper, including Houlsby, Pfeiffer, Prefix, LoRA, and Parallel, as illustrated in Figure 4. Houlsby and Pfeiffer use adapters in similar ways, so we only show the adapter architecture of Houlsby in the figure. Houlsby and Pfeiffer belong to “sequential” computation that uses $h$, the output of the MHA/FFN layer to compute $\Delta h$. In contrast, there is another way of “parallel” computation. The “parallel” methods contain Prefix, LoRA, and Parallel, and directly use $x$, the input of the MHA/FFN layer, to compute $\Delta h$.

Houlsby (Houlsby et al., 2019) approach inserts small modules, called adapters, after MHA and FFN layers. The adapter layer uses a down-projection $W_{down}\in\mathbb{R}^{d\times r}$ to project the input vector to a lower-dimensional vector of dimensional size $r$, a nonlinear activation function $f$, and a up-projection $W_{up}\in\mathbb{R}^{r\times d}$ to project lower-dimensional vector to the dimension of input vector. Given an output vector $h\in\mathbb{R}^{d}$ of MHA/FFN layers, the output after an adapter can be represented by:

Pfeiffer (Pfeiffer et al., 2020b) is a more efficient variant in which adapters are inserted only after the FFN layers.

Prefix (Li and Liang, 2021) introduces new parameters in the MHA layers. Specifically, it adds $l$ trainable prefix vectors to the keys and values of the MHA and concatenates prefix vectors $P_{k}\in\mathbb{R}^{l\times d}$ and $P_{v}\in\mathbb{R}^{l\times d}$ to the original keys $K$ and values $V$, respectively. Then MHA is performed on the new prefixed keys and values and the computation of $head_{i}$ in Eq. 2 becomes:

where $P_{k}$ and $P_{v}$ are split into $N_{h}$ head vectors and $p^{i}_{k}\in\mathbb{R}^{l\times d/N_{h}}$, $p^{i}_{v}\in\mathbb{R}^{l\times d/N_{h}}$ denote the $i$th head vector of $P_{k}$ and $P_{v}$. Following He et al. (2021b), the Eq. 6 can be rewrote as:

where $\lambda(x)$ is a scalar representing the sum of normalized attention weights over prefixes and can be represented by:

LoRA (Hu et al., 2021) is short for Low-Rank Adaptation. It injects trainable low-rank decomposition matrices into transformer layers. This can be applied to any weight matrix, but LoRA only applies to the query $Wq$ and value $Wv$ projection matrices of MHA layers. For any layer expressed as a matrix multiplication of the form $h=xW_{0}$, $x\in\mathbb{R}^{d}$, $W_{0}\in\mathbb{R}^{d\times k}$, it performs a reparameterization such that:

where $s\geq 1$ is a trainable scalar hyperparameter, $W_{down}\in\mathbb{R}^{d\times r}$, $W_{up}\in\mathbb{R}^{r\times k}$ are trainable parameters, and $r$, the low-dimensional rank of the decomposition, is the most important hyperparameter.

Parallel (He et al., 2021b) inserts adapter layers in parallel to Transformer layers, and the design of Parallel is similar to Eq. 7 and Eq. 9. Different from sequential adapters, the input of parallel adapters is the input of the MHA/FFN layers rather than the output of the MHA/FFN layers. According to Eq. 5, given the input $x$ and the output $h$ of MHA/FFN layer, the output after Parallel can be expressed by:

He et al. (2021b) also proposed two variants of parallel adapters. The multi-head parallel adapter applies the parallel adapter to modify the head attention outputs. The scaled parallel adapter transfers the composition and insertion form of LoRA into an adapter. In this paper, we use the original parallel adapters to fine-tune PTMs.

We evaluate five PEFT methods, Houlsby, Pfeiffer, Prefix, LoRA, and Parallel, and all the PEFT methods are from AdapterHub (Pfeiffer et al., 2020a). In the paper, we focus on their effectiveness on various PTMs, aiming to answer the following three research questions:

• •

  RQ1: How do text-only PTMs perform compared to code-specific PTMs of the same architecture and size?

  (Liu et al., 2023a) have studied the effectiveness of PEFT methods, but they only focus on code-specific PTMs. In addition to using code-specific PTMs on code-related tasks, another common strategy is to use PTMs pre-trained on other datasets (Ma et al., 2023; Akbar et al., 2023), such as natural language (NL). The PEFT methods have the potential for cross-modal transfer learning (Weiss et al., 2016; Zhang et al., 2022; Shen et al., 2022). It is worth discussing the effectiveness of PEFT methods on text-only PTMs as they may show different effects from the code-specific PTMs. Furthermore, designing code-specific PTMs and collecting code-related datasets for pre-training is labor-intensive. If PEFT methods can help text-only PTMs achieve good results on SE downstream tasks, the required resources can be reduced.

• •

  RQ2: How do the same text-only/code-specific PTMs with different sizes perform?

  The same PTM with different sizes usually have different learning abilities for code representation, and larger PTMs usually have better performance (Zan et al., 2023). The size of a PTM is also an important factor for PEFT, as PEFT methods help PTMs with fewer parameters to achieve results that are close to or better than those with full fine-tuning. PEFT methods may close the performance gap of PTMs with different sizes, and we can use PTMs with fewer parameters to reduce the cost. Since text-only PTMs and code-specific PTMs may have inconsistent results, we discuss them with different sizes respectively.

• •

  RQ3: How do text-only/code-specific PTMs in different architectures perform?

  There is evidence that different model architectures have preferences for SE downstream tasks (Niu et al., 2023). For example, PTMs with encoder-only architecture are better at code understanding tasks, while PTMs with encoder-decoder architecture are better at code generation tasks. Similar conclusions may exist on PEFT methods, and we want to explore the impact of different architectures and which architecture is more suitable for PEFT methods. The same with RQ2, we discuss text-only PTMs and code-specific PTMs in different architectures respectively.

For each research question, we fine-tune PTMs on four SE downstream tasks: clone detection, defect detection, code search, and code translation.

Clone detection (Svajlenko et al., 2014) is a code understanding (Wang et al., 2022a) task that involves predicting whether pairs of code are functionally equivalent and the output is a binary classification value. We use the BigCloneBench (BCB) (Lu et al., 2021) dataset, which contains over 6 million true clone pairs and 260k false clone pairs.

Defect detection (Steenhoek et al., 2022) is another code understanding task to predict whether a function is vulnerable. Unlike clone detection, which considers pairs of code snippets, defect detection focuses on individual code snippets. We use the Devign (Zhou et al., 2019) dataset, which consists of functions collected from two open-source projects and contains approximately 22k training data. Compared with clone detection, the dataset of defect detection is low-resource.

Code search (Yan et al., 2020) is a code retrieval task (Mou et al., 2016) to find the most relevant code snippets from a given set based on an NL query, where the NL usually refers to the source code comments. We utilize the CodeSearchNet (CSN) (Husain et al., 2019) dataset, which includes Go, Java, JavaScript (JS), PHP, Python (Py), and Ruby languages. Low-quality queries are filtered following the UniXcoder.

Code translation (Lu et al., 2021) is a code generation (Yang et al., 2023) task that requires translating code from one programming language to another. We use the CodeTrans (Lu et al., 2021) dataset, which contains parallel code snippets in Java and C#. The PTMs need to generate C#/Java code for the given corresponding Java/C# code.

We give a brief introduction to datasets of these downstream tasks and the statistics of the datasets are listed in Table 1. Fine-tuning uses the training and validation datasets, and the evaluation of the downstream task uses the testing datasets. These tasks are representative of various SE downstream tasks, including code classification, code retrieval, and code generation, and the datasets include high-resource, low-resource, monolingual, and multilingual. All the datasets for fine-tuning are sourced from CodeXGLUE (Lu et al., 2021), and we use their provided code to pre-process the datasets for all tasks.

For classification and retrieval tasks, metrics such as Accuracy (Acc), F1 (Nafi et al., 2019), Precision (P), Recall (R), Mean Reciprocal Rank (MRR), and Mean Average Precision (MAP) are often used. For generation tasks, metrics such as BLEU (B.) (Papineni et al., 2002), and CodeBLEU (C.B.) (Ren et al., 2020), are usually used. Some generation tasks have also used variants of Accuracy, such as Exact Match (EM), indicating whether the sequence generated by the model perfectly matches the correct answer, and Computational Accuracy (CA), computing the number of times the hypothesis function generates the same output as the reference when given the same inputs. Following Niu et al. (2023), the evaluation metrics of clone detection, defect detection, and code search are F1, Acc, and MRR, separately. For code translation, since B. and C.B. cannot accurately indicate whether the predicted code and correct answer are semantically equivalent, we use EM as the primary evaluation metric and only discuss the results of EM in this paper.

In the paper, the used PTMs to fine-tune are CodeBERT, RoBERTa, CodeT5, T5, CodeT5-large, T5-large, UniXcoder and BART. A brief description of some important information about the used PTMs is listed in Table 2.

BERT is one of the most famous text-only PTMs, which is a bidirectional encoder-only architecture model for text representation from Transformer. It uses masked language modeling and next sentence prediction methods to train the model. Derived from BERT, RoBERTa (Liu et al., 2019) is an optimization encoder-only model with just the masked language modeling task and benefits from additional training, larger batch sizes, and longer sequences. BART and T5 are two classic different encoder-decoder architecture models. BART (Lewis et al., 2019) uses denoising autoencoder tasks, such as token masking and token deletion, to construct the model. T5 (Raffel et al., 2020) treats every task as a text-to-text problem, which involves feeding input text into the model and generating target text. CodeBERT (Feng et al., 2020) is one of the first encoder-only code-specific PTMs. It adapts BERT for SE downstream tasks and proposes a new replaced token detection task, training on both natural languages and programming languages. Based on T5, CodeT5 (Wang et al., 2021) introduces identifier-aware pre-training tasks to leverage code identifiers. This strategy improves the performance on SE downstream tasks compared with T5. In addition to individual architectures, UniXcoder (Guo et al., 2022) implements a unified cross-modal architecture and uses prefixes and different self-attention masks to control the behavior of the model. It is pre-trained on multi-modal data, which includes code, comments, and AST.

All these PTMs come from Hugging Face¹¹1https://huggingface.co, and we select different PTMs for each research question. For RQ1, we employed two text-only PTMs (RoBERTa and T5) and code-specific PTMs (CodeBERT and CodeT5) for fine-tuning. CodeBERT and RoBERTa have the same size and are BERT-based architecture. T5 and CodeT5 share the same size and T5-based architecture. For RQ2, we selected the T5 and CodeT5 with different sizes: basic and large. We used T5 and CodeT5 to refer to the basic versions to maintain consistency with RQ1, and T5-large and CodeT5-large to refer to the large versions of T5 and CodeT5 respectively. For RQ3, we applied three code-specific PTMs (CodeBERT, UniXcoder, and CodeT5) and text-only PTMs (RoBERTa, BART, and T5). CodeBERT and UniXcoder are both encoder-only architectures, but UniXcoder can use the input prefixes to learn encoder-decoder and decoder-only architectures, so they can be regarded as different architectures. Similarly, T5 uses the end-to-end pre-training strategy, while BART uses denoising autoencoding to pre-train. For convenience, they can also be regarded as different encoder-decoder architectures. In the following sections, we refer to the architectures of CodeBERT, UniXcoder, BART, and T5 as BERT-based encoder-only, cross-modal encoder-only, BART-based encoder-decoder and T5-based encoder-decoder architecture respectively.

Our implementation for fine-tuning is derived from two open-source repositories. We adopt code from the UniXcoder²²2https://github.com/microsoft/CodeBERT/tree/master/UniXcoder repository on the code search task and CodeXGLUE³³3https://github.com/microsoft/CodeXGLUE repository on the other three tasks. We use their original code directly for fine-tuning encoder-only models and modify the original code to accommodate encoder-decoder models. The key hyperparameters for PTMs to fine-tune on different downstream tasks are listed in Table 3 and other hyperparameters are set to follow their original release. All experiments rely on pure Python packages and are conducted on servers with 2/4 GPUs of NVIDIA Tesla V100 32GB.

Clone Detection: From Table 4 and Table 5, Parallel outperforms other PEFT methods on all the four PTMs, no matter the code-specific or text-only PTMs. Compared with CodeBERT, PEFT methods applied to RoBERTa are more effective in keeping the performance of PTMs. The results on the best PEFT and full fine-tuning also show that PEFT methods applied to T5 are more effective than CodeT5. Thus, PEFT methods are more useful for text-only PTMs than code-specific PTMs on clone detection.

Defect Detection: From Table 4 and Table 5, Parallel ranks first on CodeBERT and CodeT5. Furthermore, Parallel achieves the best results on RoBERTa and T5, and it outperforms full fine-tuning by 2.41% and 2.3%, separately. CodeBERT and CodeT5 are pre-trained on code-related datasets, and their full fine-tuning results are better than RoBERTa and T5. However, the best PEFT methods are worse on CodeBERT/CodeT5 than that on RoBERTa/T5. Thus, PEFT methods are more effective for text-only PTMs than code-specific PTMs. LoRA achieves the worst results on CodeBERT and CodeT5. Houlsby and Pfeiffer perform the least effectively on RoBERTa and T5. Compared with the results of clone detection, the performance among different fine-tuning methods is more different. This implies that for tasks that perform poorly with full fine-tuning, the results of PEFT methods will be more unstable.

Code Translation: From Table 4 and Table 5, Parallel outperforms other PEFT methods for all PTMs on both sub-tasks, Prefix is the worst on two scenarios and LoRA performs worst for the other scenarios. Comparing full fine-tuning and PEFT, the results of Java $\rightarrow$ C# translation mostly mirror those of C# $\rightarrow$ Java translation, with the difference that full fine-tuning has a better effect on CodeBERT on C# $\rightarrow$ Java translation. It could be that CodeBERT has learned Java, and requires less trainable parameters to learn the representation on Java $\rightarrow$ C# translation. For other scenarios, text-only PTMs all perform better than code-specific PTMs. As a generation task, code translation is more difficult and requires the PTMs to comprehend code representation. CodeBERT and CodeT5 need full fine-tuning to extend to unlearned C# language, while PEFT of RoBERTa and T5 can achieve comparable or even better results with full fine-tuning. It implies that PEFT is better at cross-modal transfer learning than cross-language transfer learning.

Code Search: From Table 6 and Table 7, Parallel performs best on all languages, while LoRA performs worst. The results of CodeBERT and CodeT5 are stable for various PEFT methods, while the results of PEFT methods on RoBERTa and T5 are quite different. The PEFT method performs worse on RoBERTa compared with CodeBERT, while the results of CodeT5 and T5 show that T5 is better when using PEFT methods. Whether a PTM is trained specifically on source code does not seem to have a significant impact on the code search task. It may be that code search is a cross-modal matching task that requires learning natural language and programming language knowledge at the same time. Other factors like the model architecture of the PTM may be more important, which we will discuss in RQ3.

Answer to RQ1: Among PEFT methods, Parallel always performs best on code-specific and text-only PTMs. Compared with the results of code-specific PTMs, PEFT methods applied to text-only PTMs are more effective in maintaining the performance of full fine-tuning PTMs on clone detection, defect detection, and code translation. Whether a PTM is trained specifically on source code or natural language text has no impact on the performance of code search. This shows that the PEFT methods are more advantageous in cross-modal transfer learning for PTMs.

Clone Detection: From Table 8 and Table 9, Parallel performs the best on T5, CodeT5, and CodeT5-large, and Pfeiffer performs best on T5-large. The results show that increasing trainable parameters of T5 and CodeT5 will reduce the performance of PEFT methods. Among PTMs, the performance on T5-large is the worst compared with full fine-tuning. The difference between the best PEFT and full fine-tuning is almost the same between CodeT5 and CodeT5-large, while the performance of T5-large drops a lot compared to T5. Therefore, size has a greater negative impact on T5 compared to CodeT5.

Defect Detection: From Table 8 and Table 9, Parallel can exceed the performance of full fine-tuning on T5 by up to 2.3%. Parallel also ranks first on CodeT5 and CodeT5-large, and Prefix is the best method on T5-large. The PEFT method on CodeT5-large brings improvements over CodeT5, but the performance on T5-large is worse than T5. It shows that increasing trainable parameters hurts T5 and has a positive impact on CodeT5. Besides, T5 can perform better with PEFT than CodeT5, but T5-large underperforms CodeT5-large with PEFT. This shows that code-related pre-training tasks can help PTM perform better with more trainable parameters.

Code Translation: From Table 8 and Table 9, for all the scenarios of two sub-tasks and four PTMs, Parallel is the best PEFT method. Prefix is the worst on C# $\rightarrow$ Java for T5-large and CodeT5, and LoRA achieves the worst for other scenarios. Parallel outperforms the full fine-tuning method for T5 and T5-large and provides comparable results for CodeT5 and CodeT5-large. Increasing the trainable parameters only improves CodeT5 on Java $\rightarrow$ C# translation, while resulting in negative improvements on C# $\rightarrow$ Java translation. Since the improvement is larger, we believe that increasing trainable parameters is better for CodeT5 with PEFT. On the other hand, increasing the trainable parameters brings negative improvements to T5 on both sub-tasks.

Code Search: From Table 10 and Table 11, Parallel achieves the best results on all languages among the four PTMs, while LoRA is the worst method for T5 and T5-large and Prefix performs worst for CodeT5 and CodeT5-large. For T5 and T5-large, the best PEFT method outperforms the full fine-tuning method, and for some languages, the improvement is large, such as Ruby. Furthermore, the PEFT method provides comparable or even better results to the full fine-tuning method for CodeT5 and CodeT5-large. Compared with CodeT5-large, the PEFT method performs better on CodeT5, especially outperforming the full fine-tuning method in JavaScript, Python, and Ruby. As the trainable parameters increase, the PEFT method also has a negative improvement on T5, but the gap is smaller compared with CodeT5.

Answer to RQ2: Among PEFT methods, Parallel always performs best on different model sizes. The results of CodeT5 with different sizes are more stable than those of T5. Increasing the trainable parameters of T5 reduces the performance of the PEFT method compared to full fine-tuning. CodeT5 with more trainable parameters performs better on defect detection and code translation and performs worse on clone detection and code search.