Identifying Source Code File Experts

MacDonald and Ackerman (McDonald and Ackerman, 2000) use a heuristic called Line 10 rule that prioritizes the developer who last changed a module in solving problems. Following the same premise, Hossen et al. (Hossen et al., 2014) presented an approach called iMacPro that identifies experts associated with a change request based on who last changed certain files. Other works count the number of changes made on source code elements (Hattori and Lanza, 2009, 2010; Mockus and Herbsleb, 2002; Bird et al., 2011; Canfora et al., 2012). There are also studies that use information from files present in development branches to identify experts who perform merge operations involving these files (Costa et al., 2016, 2019). Other models, such as the one proposed by Sülün, Tüzün and Dogrusöz (Sülün et al., 2019), use the number of commits in the artifact of interest and in related artifacts for the calculation of knowledge, in order to recommend code reviewers. In summary, these studies are based mainly on information about changes such as the number of commits and who made the last change to identify expertise. However, based on past works (Krüger et al., 2018; Avelino et al., 2018), we suspect that these variables alone are not enough. For this reason, in this study, we analyze more variables and their relationship with developers’ knowledge.

Other studies try to model the knowledge flow in the history of the source code. The Degree of Knowledge (DOK) model proposed by Fritz et al. (Fritz et al., 2014) uses the information related to the degree of authorship (DOA) that the developer has with the code artifact, and the number of interactions (selections and edits) that the developer had with the artifact, named the degree of interest (DOI). However, the calculation of the DOI requires the use of special plugins in the development environment, which makes its usage impractical in a large study as the one we present in this paper. Regarding the differences for the models studied in this work, DOK does not deal with recency directly and does not consider the size of the file when estimating knowledge. These two variables were pointed out as important factors in the calculation of knowledge in previous works (Krüger et al., 2018; Avelino et al., 2018).

Other techniques model the impact of time on the knowledge that developers have with source code artifacts. Silva et al. (da Silva et al., 2015) presented a model that computes the developer’s expertise in an entire (atomic) artifact, and also in its subparts (internal classes and methods), based on the number of changes made by a developer. The expertise analysis can be done using time windows that divide the history of an artifact into subsets of commits. Other approaches that consider the recency of changes appear in studies focused on the recommendation of developers for the resolution of change requests. Kagdi et al. (Kagdi et al., 2012) proposed an approach that locates source code files relevant for a given change request and identifies experts in those files using the xFinder (Kagdi et al., 2008; Kagdi and Poshyvanyk, 2009) approach, which prioritizes developers who made most commits in a given file. Tüzün and Dogrusöz (Sülün et al., 2021), extend a previous work (Sülün et al., 2019), by adding information on modification recency for the calculation of knowledge, aiming to recommend code reviewers. On one hand, these studies consider some measure of recency for identifying experts in source code files. On the other hand, they did not present an in-depth and large analysis that shows how the variables used are suitable for this identification.

In comparison to the data source used to extract knowledge information in source code, in this work we focus only on data contained in version control systems. Some works use other sources such as: number of interactions with a file (Fritz et al., 2007), code reviews(Thongtanunam et al., 2016; Jabrayilzade et al., 2022), numbers of meeting related to commits (Jabrayilzade et al., 2022). While these are valid data sources, they depend on specific tools, such as plugins installed in the development environment, the use of company-specific tools, and development culture. Due to the universality of version control systems in current software development (Zolkifli et al., 2018), its use as a data source becomes easier in practice.

Regarding the use of machine learning, Montandon and colleagues (Montandon et al., 2019) investigated the performance of supervised and unsupervised classifiers in identifying experts in three open-source libraries. Even though we followed a similar process for data collection and analysis, our work has a distinct purpose. We rely on classifiers for identifying experts at the level of source code files, while Montandon target the identification of experts in the use of libraries and frameworks, therefore using different variables than the ones used in this work. Other examples of machine learning applications in the context of developer expertise target the bug assignment problem (Sajedi-Badashian and Stroulia, 2020), which is also a distinct problem than the one investigated here. In summary, we have not identified studies that investigate the performance of machine learning in the classification of file experts based on VCS information, such as our key goal in this work.

Krüger and colleagues (Krüger et al., 2018) analyzed the impact of forgetfulness of the developer about the code, using data from ten open-source repositories. They studied whether the forgetting curve described by Ebbinghaus (Ebbinghaus, 1885) can be applied in the context of software development, and which variables influence the developer’s familiarity with source code. They analyzed variables such as number of commits, changes made by other developers, percentage of code written by a developer in the current version of the file, and the behavior of tracking changes made by other developers.

Other works used techniques and models to identify expertise. Avelino and colleagues compared the performance of Commits, Blame, and Degree-of-Authorship (DOA) techniques in identifying source code file maintainers (Avelino et al., 2018). A survey similar to the one presented in this paper was made to create a dataset with data from eight open-source repositories and two private ones. The results showed that all three techniques have similar performance in identifying source code maintainers. However, the results also pointed out the importance of considering the recency of the modifications and the file size as a possible strategy to improve these techniques.

There are also papers that studied other types of expertise. For example, Hannebaur et al. (Hannebauer et al., 2016) compared the performance of eight algorithms to recommend code reviewers. Out of these algorithms, six are based on expertise by modification and two are based on review expertise. The six algorithms based on expertise by modification are Line 10 Rule (McDonald and Ackerman, 2000), Number of Changes, Expertise Recommender (McDonald and Ackerman, 2000), Code Ownership (Girba et al., 2005), Expertise Cloud (Alonso et al., 2008), and DOA. The algorithms based on review expertise are File Path Similarity (FPS) (Thongtanunam et al., 2014), and a model proposed by the authors, called Weighted Review Count (WRC). They used data from four FLOSS projects: Firefox, AOSP, OpenStack, and Qt. The algorithms based on review expertise performed better than the ones based on modification expertise, and the WRC algorithm achieved the best results.

Other works use information on expertise for the resolution of bug reports. Anvik and Murphy (Anvik and Murphy, 2007) compared two approaches to determine appropriate developers for resolving bug reports. One approach uses data from code repositories to define which developers are experts in the files associated with the bug report using the Line 10 heuristic. The other approach uses data from bug networks such as the carbon-copy list (cc:), comments, and information from who resolved previous bugs. Development data from the Eclipse¹¹1https://www.eclipse.org/eclipse/ platform was used. The authors concluded that the best approach depends on what users are looking for regarding precision and recall.

The work presented in this paper can be distinguished from the works described before in three key aspects: purpose, compared techniques, and scope. In relation to purpose, two works have the same objectives as ours: Avelino et al. (Avelino et al., 2018) and Anvik and Murphy (Anvik and Murphy, 2007) (but they do not use the same techniques, particularly machine learning classifiers).

Regarding the compared techniques, Avelino et al. (Avelino et al., 2018), Krüger et al. (Krüger et al., 2018), and Hannebauer et al. (Hannebauer et al., 2016) used the baseline techniques selected in this work. However, we also investigate the performance of machine learning models. Finally, regarding the scope of the studies, none of them used data from a similar number of repositories as in our study.

We selected open source repositories from the GitHub platform²²2https://github.com/. To select these repositories, we adopted a similar procedure to other studies that investigate GitHub data (Avelino et al., 2016; Yamashita et al., 2015; Ray et al., 2014). First, for each of the six most popular programming languages on GitHub (Java, Python, Ruby, JavaScript, PHP, and C++)³³3The six most popular programming languages in 2019 https://octoverse.github.com/#top-languages we selected the 50 most popular repositories as indicated by their number of stars. This measure is widely used by researchers in the selection of GitHub repositories (Avelino et al., 2016; Ray et al., 2014; Padhye et al., 2014; Hilton et al., 2016; Mazinanian et al., 2017; Jiang et al., 2017; Nielebock et al., 2019; Rigger et al., 2018; Castro and Schots, 2018), and perceived by developers as a reliable proxy of popularity (Borges and Valente, 2018). Then, after cloning the repositories, we performed a repository filtering step based on three metrics: number of commits, number of files, and number of developers. As was done in a previous work (Avelino et al., 2016), for each language, we removed repositories in the first quartile of the distribution of each metric, resulting in the intersection of the remaining sets. In other words, repositories with few commits, files, and developers were removed. Table 1 shows the first quartile of each of the three metrics for each programming language.

Additionally, we discarded repositories whose development history suggests that most of the software was developed outside of GitHub by removing repositories where more than half of its files were added in a few commits. As few commits, we considered the outliers of the distribution of the number of files added in each commit of a repository; we discarded repositories if most of their files (¿50%) were added by the set of outliers commits, following the procedure done by Avelino and others (Avelino et al., 2016).

The resulting dataset is composed of 111 popular GitHub repositories distributed over the six most popular programming languages, which have a relevant number of developers, files, and commits. Table 2 summarizes the characteristics of these 111 repositories.

We also used data from two industrial projects from Ericsson⁴⁴4https://www.ericsson.com/en. Both projects were developed in the Java programming language. The development history of Project #1 has 74,078 commits, 17,329 files, and 513 contributors. On the other hand, Project #2 has 26,678 commits, 15,930 files, and 262 contributors. Therefore, the two industrial projects are also relevant according to the criteria applied to select the open-source repositories.

First, we ran git log --no-merge --find-renames command to extract data from the commits logs of each repository. This command returns all commits that have no more than one parent (no-merge) and it automatically handles possible file renames.⁵⁵5https://git-scm.com/docs/git-log/1.5.6 From each commit, three pieces of information were extracted: (1) changed files; (2) name and email of the commit’s author (developer who performed the change); (3) type of the change: addition, modification, or rename.

After that, we discarded files that do not contain source code (e.g., images and documentation), third-party libraries, and files that are not part of one of the six programming languages considered in the study. To identify and discard these files we rely on the Linguist tool ⁶⁶6https://github.com/github/linguist/blob/master/lib/linguist/languages.yml.

Finally, we handled developer aliases by following the same procedure adopted in other works (Avelino et al., 2018, 2016, 2017, 2019). Aliases arise when a developer is associated with more than one pair (name-dev, email) on Git. For the purposes of this work, it is important to unify these contributors. This unification was performed in two stages. First, users who shared the same email, but with different names, were unified. Additionally, users with different emails, but with similar names were grouped. This similarity was computed using the Levenshtein distance (Navarro, 2001), using a maximum modification threshold of 30% on the number of letters.

1. (1)

   We randomly selected a file and retrieved the list of developers who touched (created or modified) it.

2. (2)

   We discarded the file if at least one of these developers reached the maximal limit of files we plan to send to a developer (file_limit), asking about his/her expertise on the file. Otherwise, we added the file to the list of each developer.

3. (3)

   We repeated steps 1-2 until there are no more files to be verified.

By discarding files in each one of the developers already reached her file_limit, step 2 warrants that a file will be added to the sample only if it is possible to add all developers who modified it. This step aims to maximize the possibility of obtaining answers from all developers who touched a file. We established a file limit of five, seeking to not discourage developers from responding to the survey, which can happen according to guidelines in the literature (Kasunic, 2005; Linaker et al., 2015).

At the end of this step, a list of pairs (developer, file) was created for each repository. For open-source repositories, 20,564 pairs were generated, with 7,803 developers, 2.64 files per developer on average. For the two private repositories, 394 pairs were generated, with 92 developers, and 4.34 files per developer.

For the open-source repositories, 7,803 emails were sent, and 501 responses were received, representing a response rate of 7%. For the private repositories, 92 emails were sent, 38 responses were received, representing a rate of 41%. From these responses, two datasets were created: one with 1,024 developer-file pairs coming from the answers of the open-source repositories, and a second dataset with 163 pairs extracted from the answers of the two private repositories. In the remainder of this paper, these two datasets will be named public and private datasets.

Figure 1 and 2 shows the distribution of responses in the public and private datasets respectively. The public dataset is composed of 54% of declared experts and 46% of declared non-experts, and the private dataset is composed of 47% of declared experts and 53% of declared non-experts. As we can see, the proportion of declared expert and declared non-expert sets are close in both datasets, and this class balance is an important property in classifier training (Weiss and Provost, 2001).

The variables Adds, Dels, Mods, and Conds are extracted using the command git diff ⁷⁷7https://git-scm.com/docs/git-diff. This command returns the lines added and removed between two versions of a file. In this work, a modification is defined as a set of removed lines followed by a set of added lines of the same size (Lira, 2016; Ibiapina et al., 2017). We use Levenshtein distance (Navarro, 2001) to identify which pairs (remove, add) are modifications between two versions of the files. The algorithm takes the removed and added line as input, and returns a value that represents the number of characters that must be modified to transform the removed line into the added line. In this work, a change is a modification if the returned value is less than a certain percentage (threshold) of the size of the removed line. A threshold of 40% was used, as suggested by Canfora et al. (Canfora et al., 2007).

This section describes the techniques used in this study to identify code experts.

where,

• •

  $FA$: 1 if developer d is the creator of the file f, 0 otherwise.

• •

  $DL$: is the number of changes made by developer d until version v of file f.

• •

  $AC$: is the number of changes made by other developers in the file f up to version v.

Five well-known machine learning classifiers are evaluated: Random Forest (Liaw et al., 2002), Support-Vector Machines (SVM) (Weston and Watkins, 1998), K-Nearest Neighbor (KNN) (Peterson, 2009), Gradient Boosting Machine (Friedman, 2001), and Logistic Regression (Hosmer Jr et al., 2013). We use 10-fold Cross-Validation to evaluate the performance of the machine learning classifiers. 10-fold Cross-Validation consists of partitioning the data set in ten complementary subsets, and the use of each of these subsets for model training and the rest for model validation. In the end, the fitness measures are combined to obtain a more accurate estimate of the model’s performance (Berrar, 2019). We use F-Measure as a performance measure. To find the best settings for these classifiers, we perform a grid search (Claesen and De Moor, 2015) to adjust the hyper-parameters and find the best settings for each classifier.

To evaluate NumCommits, Blame, and DOA in the identification of software experts, we adopt a similar procedure as the one followed in a previous related work (Avelino et al., 2018). First, we normalize the technique values. For this purpose, we set as $1$ the expertise of a developer d in file f if d has the highest value for a given technique in f; otherwise, we set a proportional value. For example, suppose that for a given file f developers d1, d2, and d3 have a Blame value of 10, 15, and 20. Their normalized values for blame regarding the file f are as follows: expertise(d1, f) = 10/20 = 0.5, expertise(d2, f) = 15/20 = 0.75, and expertise(d3, f) = 20/20 = 1. We apply this normalization process to all techniques.

After the normalization, a developer d is classified as an expert of a file f if its expertise (d, f) is higher or equal than a threshold k; otherwise he/she is considered a non-expert. In the example of the previous paragraph, considering a threshold k = 0.7, developers d2 and d3 would be considered experts of the f file accordingly with the Blame technique.

To this purpose, we first compute the performance of each technique by adopting 11 different thresholds, i.e., by varying the threshold k from 0 to 1, under 0.1 steps. At the end of this process, the best performance of the linear techniques is obtained together with their associated thresholds k. For each threshold, we use 10-fold Cross-Validation to compute the F-Measure (F1-Score) of the techniques by applying the following equations:

Figures 3 and 4 show the F-Measure results for each linear technique at each analyzed threshold, under the public and private datasets, respectively.

As we can see, Blame reaches its best performance when we use the lowest possible threshold ($\mathit{k=0}$), in both public and private data. In other words, by adopting $\mathit{k=0}$, Blame reaches its best performance when it is used to classify as an expert any developer who has at least one line of code in the last version of the file. Similarly, NumCommits achieves the best performance by adopting the thresholds k = 0.1 and k = 0.3, in the public and private datasets, respectively. The low thresholds indicate that both techniques perform better by relying on minimal changes to consider developers as file experts. On the other hand, DOA has it best performance with a threshold k = 0.1. on public data; and k = 0.7, on private data.

We analyze the correlations between the development variables and the developers’ knowledge, as obtained in the survey. In the remainder of this paper, the survey responses will be represented by a variable named Knowledge. We answer RQ1 and, consequently, identify which variables could be part of a prediction knowledge model.

Table 5 shows the directions and intensities of the correlations between the extracted variables and the Knowledge variable, by applying Spearman’s rho, since it is suitable for data sets that do not follow the normal distribution. We consider results to be statistically significant when p ¡ 0.05. Even though NumCommits is the most used variable for inferring file’s knowledge in the literature, it does not show the strongest positive correlation with the Knowledge variable. The variable with the highest positive correlation is FA, closely followed by Adds. This suggests that a more fine-grained measure of changes like Adds can be more important than NumCommits to compose a knowledge model. The variable with the lowest correlation with Knowledge is Mods. Finally, the variable that shows the highest negative correlation is NumDays, followed by NumModDevs and Size. These results reinforce the importance of recency for inferring the knowledge that a developer has about a source code file.

We also analyze the correlation between the extracted variables using Spearman’s rho (Schober et al., 2018). These correlations are represented in Figure 5, where colors close to 1 and -1 represent higher positive and negative correlations, respectively. This analysis is relevant because variables with a certain level of correlation can lead to inaccurate models (Yu and Liu, 2003).

As expected, NumCommits has a strong correlation with Adds, Dels, Mods, Amount, and AvgDaysCommits (rho $\geq$ 0.5). Therefore, any one of these five variables can be used to measure the number of changes made by a developer throughout the history of a file. The variables NumModDevs and NumDays also show a moderate positive correlation (rho $\geq$ 0.5) with each other.

The variable NumCommits is not included in the models. It can be viewed as just a macro way of accounting for changes made by a developer to a source code file, and the variable Adds has already been chosen, since it has a higher positive correlation with Knowledge. In addition, Adds has a moderate correlation with NumCommits ($rho\geq 0.5$, Figure 5). For a similar reason, AvgDaysCommits is not included in the model.

We also add the variable FA, which has the highest positive correlation with Knowledge among all variables. The variable NumModDevs is not included due to its moderate correlation with NumDays ($rho\geq 0.5$).

After identifying and selecting the variables with the highest correlations with source code knowledge, we train the machine learning models using them as features. After that, we compared the performance of these models with those of linear models in the identification of experts.

Table 6 presents the performance of linear techniques and machine learning classifiers, in the two analyzed scenarios. Regarding the public dataset (Table 6 - Public), DOA and NumCommits had the best F-Measure (70%), followed by Blame, with a F-Measure of 67%. Regarding recall, the technique with the best result was DOA (97%). The technique with the lowest recall in the same scenario is Blame (83%). Finally, the technique with the highest precision is NumCommits (60%), while DOA has the lowest precision (55%).

When using the private dataset (Table 6 - Private), DOA had the highest value for F-Measure (68%), closely followed by NumCommits (67%). As in the other scenario, Blame had the worst F-Measure (55%). The best Recall was from NumCommits (86%), with Blame presenting the lowest recall (62%). Regarding precision, DOA achieved the best result (61%), and Blame the lowest one (50%).

When we consider the scenario using the public dataset (Table 6 - Public), Random Forest, SVM, and GBM had the best F-Measure (73%), with the other two classifiers reaching close values. The classifiers with the highest recall were SVM and Logistic Regression (75%), and KNN obtained the lowest result (71%). In terms of precision, GBM and Random Forest have the highest values (73%). The lowest precision is from Logistic Regression (69%).

Regarding the private dataset (Table 6 - Private), the classifier with the highest F-Measure was KNN (67%) and Logistic Regression had the lowest one (58%). The best recall was also obtained by KNN (69%) and Logistic Regression had the lowest one (51%). Finally, the best precision was achieved by Logistic Regression and KNN (70%). Random Forest had the lowest precision (59%).