Fork Entropy: Assessing the Diversity of Open Source Software Projects’ Forks

The pull-based development model is a modern paradigm for software development that is particularly well-suited to geographically distributed teams [4]. This workflow can be broken down into seven steps, which include: forking, cloning, editing, syncing, pushing, submitting, and evaluating [2, 3]. By providing a code base and a set of tools for task management, code review, and DevOps, the pull-based model simplifies participation and lowers entry barriers for external contributors compared to traditional patch-based models [4].

Forks can be created for many different reasons or intentions. In [1], Jiang et al. summarizes developers’ reasons and preferences of creating forks, and OSS projects’ attracting characteristics of getting forked. They discover various reasons of forking including contributing back with pull-requests, fixing bugs, adding new features, keeping copies, etc. Stănciulescu et al. conduct a case study using the Marlin project to explore the reasons, benefits, are challenges of forks [30]. According to whether they will contribute back to the source repository, forks can be classified as “hard” forks [31, 32], independently developed forks (IDFs) [33], or divergent forks [34] that are not intended to be merged back, and “social” forks [32] that are created to make contributions.

The many forks created from a source repository (with different intentions above) form the fork population studied in this work. The concept is closely related to the concept of software family in existing work [35, 31, 34]. In [35], Brisson et al. study the communication in a software family with respect to following, pull-requests, and issues, and analyze the correlation between such communication and a project’s star counts. In [34], Businge et al. explore the characteristics including the size, package dependencies, categories, etc, of software families, as well as their code propagation practices. Software families composed of hard forks are studied in the above work [35, 34]. In [31], Hadian et al. study the evolution and communication between repositories in a family of projects (forks). They discover differences between hard and social forks in terms of activity, deviation of dependencies, and communication. In this work, we study the diversity of a projects’ contributors-owned fork population created on GitHub. Although we do not discriminate hard and social forks in our studies, we are more interested in contributing forks [33] by analyzing the correlation between the forks’ diversity and contributions received in the source repository.

For owners of contributing forks that follow the pull-based model, they can fork and develop in their own forked repositories before contributing back to the source repository through pull-requests. However, coordination issues such as misalignment and conflict among developers can lead to inefficiencies in forking practices, which include lost contributions, rejected pull requests, redundant development, and fragmented communities as summarized by Zhou et al in [6]. Consequently, it is essential to understand the factors, and develop tools to aid efficient forking practices. In [6], Zhou et al. explore characters of the source repository such as modularity, and coordination mechanisms on the efficiency of forking practices. To better understand the forks of OSS projects, tools are developed to visualize source code changes in forks [36], and identify features from commits in a projects forks [21]. In this work, we propose a new metric called fork entropy to facilitate studies of the diversity of a forks based on their modification to files, which offers a new perspective to understand an OSS project’s fork population.

In this work, we focus on forks owned, and contributions made by external contributors because we focus on the diversity and contributions related to OSS projects’ forks. Unlike core developers who have write access, and can directly commit into a source repository, external contributors make contributions to projects through patches or pull-requests [22, 4]. In [22], Padhye et al. categorize code committers into core, external, and mutant, and find that the number of external committers of projects developed by popular scripting languages is comparable with the number of core committers. After inspecting the pull-requests opened in 2013 in GitHub, Gousios et al. conclude that pull-requests is a way of projects to get external contributions because 73.07% of the pull-requests have been merged using facilities provided by GitHub [4]. In [8], Vasilescu et al. study the influence of adopting continuous integration (CI) on the of acceptance of core and external (non-core) developers’ pull-requests, as well as the impact on software quality. They discover an improvement in projects’ ability in integrating external contributions without sacrificing code quality after adopting CI. By inverstigating the Eclipse community, Sinha et al. discover factors, such as demonstration of knowledge and skill in bug repositories, can influence the promotion of external developers to core committers[37]. In brief, contributions made by external contributors through pull-requests play an important role in OSS development and maintenance which worth studying. Moreover, external contributors are a suitable group for our study on fork diversity and its correlation on OSS contributions because, unlike core developers with write access, they need to fork a repository before making contributions.

Most existing software engineering studies related to our work focus primarily on the diversity of team members in OSS projects. Gender diversity has been widely examined and found to have positive effects on project growth [14], team productivity [15], and community health [16]. Studies have also reported a positive correlation between the country diversity of team members and project growth [14], and the impact of tenure diversity on team productivity has been suggested [15]. Additionally, Daniel et al. measured the reputation and role diversity of participants and found positive effects on community engagement and market success [17]. However, as far as we know, there is currently a dearth of formal measures of fork diversity. Moreover, the diversity metrics presented above are typically measured using either the Blau index [38] or the coefficient of variation [39]. While the Blau index is well-suited for measuring diversity in categorical variables (such as gender [15] and country [14]), and the coefficient of variation can measure dispersion in numerical variables (like tenure and reputation), neither method is ideal for calculating the proposed metric of fork entropy due to limitations on the forms of data associated with understanding the fork populations of OSS projects.

Diversity is widely acknowledged as a multifaceted concept, which can be categorized into three main components: richness, evenness, and disparity [40]. Richness refers to the absolute number of species in a population, evenness describes the distribution of species abundances, and disparity denotes the differences among species [41]. Due to its complexity, diversity has been operationalized using different metrics, depending on the specific application. For instance, while species richness is a commonly used indicator of diversity that emphasizes the richness component, the Gini coefficient, which measures income inequality, highlights the disparity component [41]. In the context of OSS development, forks may naturally differ due to differences in expertise, experience, and intentions among developers [42]. Therefore, we focus on the disparity component of diversity and quantify the differences in file modifications among forks. To this end, we employ Rao’s quadratic entropy, which measures the expectation of dissimilarity between two samples randomly taken from a population [9]. This approach has been widely used in studies of population and ecological diversity [9, 10]. Further details on how we calculate fork entropy can be found in Section III.

Fig. 1 illustrates the overall process of calculating fork entropy to measure the diversity of forks created for an OSS project. As shown in Fig. 1(a), for each project involved in our study, we first collect data about its forks, and construct a series of snapshots of fork populations with predefined time intervals. We select projects from GitHub for easy data collection. We empirically set the time interval to one month in this work after consulting the literature [26]. Second, as shown in Fig. 1(b), we build a file modification matrix to contain the changes made by the forks in snapshot at time $t$, denoted as $M_{t}$. To keep the notations concise, we slightly abuse the notations to omit time index $t$ when referring to the file modification matrix for a snapshot in the rest of the paper. Finally, as shown in Fig. 1(c), we calculate the fork entropy with the Rao’s quadratic entropy defined on the pair-wise distance function that quantifies the difference between each pair of forks given a file modification matrix. Detailed steps are as follows.

The first step is to construct the fork population as shown in Fig. 1(a). Given an OSS project, we first locate its source repository, e.g., tensorflow/tensorflow, in the database. We then perform a breadth first search using the ‘forked_from’ key in the database to retrieve the direct forks and indirect forks (i.e., ‘forks of forks’) of the source repository in an iterative manner until all forks of the project are retrieved. The retrieved forks are segmented by time intervals with a fixed duration (empirically set to a month in this work) to get a series of snapshots. Each snapshot contain forks with their commits received during the time interval. For a fork to be included in a snapshot, it must: 1) have at least one commit with file modifications during the snapshot’s time interval; and 2) is owned by a external contributor who does not have write access to, or have privileges to close issues or pull-requests in the source repository [6]. We determine a contributor to be external if he / she has never directly commit into a repository, or performed any privileged actions such as closing issues opened by other users. A fork can be included in multiple snapshots if it meets the above criteria in different time intervals, but with different commits. We restrict our scope to forks owned by external members of the project to conduct a focused study as introduced in the beginning of the paper. It should be noted that fork ownership is not a part of the proposed fork entropy metric. Following the above approach, the set of forks in each snapshot forms a fork population of the project during the corresponding time interval.

Next, we build a file modification matrix to contain the changes made by the fork population in a snapshot. In the matrix, a fork, e.g., the $i$-th fork, is encoded as a row vector:

where $n$ is the number of files in the project that are modified by one of the forks in the population, and $c_{ij}\in\mathbb{R}$ is the count of lines in file $j$ which are modified by fork $i$ during the time interval of the snapshot. Intuitively, $\vec{\mathbf{c}}_{i}$ is the fingerprint of the $i$-th fork in terms of modifications to project files. Let $m$ be the number of forks in the population, we can obtain the file modification matrix, $M\in\mathbb{R}^{m\times n}$, by stacking the row vectors of all of the $m$ forks as shown in Fig. 1(b). Because the set of files modified by fork populations in different snapshots are likely to be different, it is common that the number of columns, $n$, varies with different time intervals. We include only files that are modified by the fork population in a snapshot to guarantee that the file modification matrix does not contain rows or columns that are all zeros.

For each snapshot, we use the Rao’s quadratic entropy with a distance function defined on the file modification matrix $M$ to calculate the average degree of difference between forks in the population following Eq. (1).

where $\mathrm{QE}(M)$ denotes the Rao’s quadratic entropy that estimates the expectation of difference between two individuals randomly selected from the population [9], $M$ is the file modification matrix correspond to the snapshot (with the time index $t$ omitted), $m$ is the number of forks in the population, $\vec{\mathbf{c}}_{i},\vec{\mathbf{c}}_{j}$ are the the $i$- and $j$-th row in $M$, respectively, and $\mathcal{D}$ is a distance function that quantifies the degree of difference between two vectors as defined in Eq. (2) and visualized in Fig. 1(c).

where $\exp(-\gamma\|\vec{\mathbf{c}}_{i}-\vec{\mathbf{c}}_{j}\|_{1})$ is the Laplacian kernel [43], $\|\cdot\|_{1}$ is the 1-norm, and $\gamma$ is the hyperparameter used to adjust the sensitivity of the function to differences. We adopt the Laplacian kernel because it is a non-linear transform sensitive to slight change and performs excellently in many detection tasks, e.g., character recognition [44].

Practical considerations also led us to adopt the Laplacian kernel in our study. Our analysis revealed that many forks in our dataset contain only minor changes to project files—some forks modifying only a single line in a single file, similar to findings from previous research [45, 46]. Consequently, we observed a substantial number of small differences, resulting in a fork entropy distribution that roughly followed a bell curve. After testing various distance functions, including the Laplacian and Gaussian kernels [43], we ultimately selected the distance function presented in Equation (2) based on its real-world performance.

By substituting Eq. (2) into Eq. (1), we have the Eq. (3) for fork entropy.

where $\gamma$ is set to 1 to compute the raw difference between two vectors in practice. With the above definition, it is trivial to see that $0\leq\mathcal{H}_{fork}<1$, and $\mathcal{H}_{fork}$ takes the minimum value when the numbers of changed lines for all files modified by different forks are identical.

Considering the basic axioms [41] of a diversity index and expectations in the particular context jointly, the following properties are met by the proposed fork entropy:

Symmetry. The fork entropy is not related to the order of forks during its calculation. It is straightforward to see $\mathcal{H}_{fork}$ satisfies the symmetry property because its distance function $\mathcal{D}$ is symmetric, i.e., $\mathcal{D}(\vec{\mathbf{c}}_{i},\vec{\mathbf{c}}_{j})=\mathcal{D}(\vec{\mathbf{c}}_{j},\vec{\mathbf{c}}_{i})$.

Continuity. The fork entropy is a continuous function. It is also easy to see that $\mathcal{H}_{fork}$ is in a continuous interval $\mathcal{H}_{fork}\in[0,1)$ by its definition.

Symmetry and continuity are two fundamental properties of a diversity index [41]. Next, we introduce monotonicity as a property to meet our goal of measuring fork diversity.

Monotonicity. Adding a redundant (or distinctive) fork will decrease (or increase) fork entropy. We first explain the terms before proving monotonicity. Given $m$ existing forks and a new fork $\vec{\mathbf{c}}_{m+1}$, Eq. (4) calculates the difference of the new fork to existing ones.

We say $\vec{\mathbf{c}}_{m+1}$ is redundant if its difference to existing forks is less than the average difference among the existing $m$ forks; in contrast, $\vec{\mathbf{c}}_{m+1}$ is distinctive if the new fork’s difference to existing forks exceeds the average difference among the existing $m$ forks.

Assuming that the vector for a new fork $\vec{\mathbf{c}}_{m+1}$ is added to an existing file modification matrix $M$ that contains $m$ forks to obtain a new matrix $M^{\prime}$, we derive the new fork entropy of $M^{\prime}$ in Eq. (5).

We denote $\mathcal{H}_{fork}(M^{\prime})-\mathcal{H}_{fork}(M)$ as $\Delta$ and obtain Eq. (6) by substituting $\Delta$ into Eq. (5).

According to Eq. (6), $\Delta$ is negative when $\vec{\mathbf{c}}_{m+1}$ is redundant and is positive when $\vec{\mathbf{c}}_{m+1}$ is distinctive. It means that fork entropy decreases after adding a redundant fork while increases after adding a distinctive fork. Consequently, fork entropy possesses the above properties and is valid to quantify the diversity of forks.

We measure a project’s outcomes, including external productivity, the acceptance rate of external pull-requests, and number of reported bugs. We also introduce control variables relevant to those outcomes.

Outcome: external productivity. The number of commits is a widely used indicator of the productivity of OSS projects [25, 15, 8, 26]. In this work, we focus on external productivity that quantifies external contributor’ contributions to a project, which is measured by the number of commits integrated into the project’s source repository through pull-requests from external contributor-owned forks. The monthly external productivity is obtained to perform regression analysis with fork entropy in each snapshot to answer RQ1. A possible threat lies in that maintainers may change the origins of commits, for example, by “cherry-picking” in pull-requests [4], which can cause us to miss some contributions made by external contributors. Fortunately, we find such cases are rare in our dataset after manual inspections.

It is important note that software developers’ productivity is a multifaceted concept which covers the activity, performance, efficiency, satisfaction and well-being, etc, of the developers as suggested in [47], and can be influenced by many factors beyond the ones studied in this paper such as the team sizes [48]. In this paper, we adopt the number of commits as a commonly used measure of productivity with respect to developers’ activity, which should not be considered as a comprehensive measure of external developers’ productivity.

Outcome: external pull-request acceptance rate. Researchers regard the acceptance rate of pull-requests as a crucial indicator of the development efficiency of OSS projects [6] because maintainers reject pull-requests that are obsolete, conflicting, duplicated, etc [4, 49, 50]. We focus on the acceptance rate of external pull-requests delivered from external contributor-owned forks to an OSS project’s source repository, measured as the proportion of the merged pull-requests among closed ones. The list of merged and closed pull-requests can be obtained from GitHub’s rest API. The monthly acceptance rate of external pull-requests is calculated to perform regression analysis with fork entropy in each snapshot to answer RQ2. As many developers integrate pull-requests via other mechanisms rather than GitHub interface, the status of pull-requests is not very reliable reported by GitHub [4, 6]. We follow the heuristics first proposed by Gousios et al. [4] and subsequently refined by Zhou et al. [6] to determine pull-requests’ status. A pull-request is considered been merged if any of the following conditions is met: 1) Maintainers perform a ‘merged’ action for the pull-request on GitHub. 2) The pull-request is closed by a commit using certain phrase conventions (e.g., fixes $\#$1234) advocated by GitHub²²2https://github.blog/2011-04-09-issues-2-0-the-next-generation/, or any of the last three comments of the pull-request refers to a commit indicating the merge of the pull-request³³3Comment matches regular expression (merg$|$apply$|$appl$|$pull$|$ push$|$integrat$|$land$|$cherry(-$|$$\backslash$s+)pick$|$squash)(ing$|$i?ed)., and the commit exists in the source repository’s commit history.

Outcome: number of reported bugs. The number of bugs per unit time is a popular proxy of code quality [27, 28, 8, 29, 26]. We refer to [8, 26] to count emerging bug-report issues in each snapshot. We do not distinguish bug reports from core members or external contributors. To identify bug-report issues, we process issue titles and labels by lowercasing and Porter stemming [51] then search bug-related keywords, including defect, error, bug, issue, mistake, incorrect, fault, and flaw. If the title or any label of an issue contains at least one keyword, we mark it as a bug-report issue. The monthly code quality is assessed to perform regression analysis with fork entropy in each snapshot to answer RQ3. Since the count-based assessment of code quality relies heavily on the issue base, a threat arises if projects rarely utilize GitHub’s default issue-trackers. Thus, we examine the number of issues in each project and exclude projects with few issues.

Control variables. Based on prior software engineering literature [4, 52, 8, 6] and our experience, we include the following factors potentially relevant to the above project outcomes as control variables.

• •

  NumForks and NumFiles: The two variables are the number of forks and that of modified files, respectively. They jointly describe the shape of a file modification matrix. The more forks a project has, the more pull-requests are submitted by non-core developers [8].

• •

  ProjectAge: The age of a project’s source repository in days. The older the project, the fewer external pull-requests maintainers merge or reject [8].

• •

  NumStars: The number of stars a project’s source repository receives. This variable usually refers to the popularity of OSS projects. External contributors are more likely to contribute to more popular projects [8].

• •

  RatioOldContributors: The ratio of external contributors with prior experience in successfully submitting pull-requests to a project’s source repository. Core developers prefer to trust contributors they have worked with before [4, 52, 6].

• •

  RatioPRsWithTests: The ratio of pull-requests that contain test cases. A pull-request has test cases if any file pathname contains ‘test’ [52]. Pull-requests that contain test cases are more likely to be merged [52].

• •

  RatioPRsTouchHotFiles: The ratio of pull-requests that touch hot files. A hot file is that one modified by any merged pull-request in the past three months [4]. Existing studies suggest pull-requests that modify hot files are more likely to be accepted [4].

We collect data through GHTorrent [53] and GitHub REST API⁴⁴4https://docs.github.com/en/rest, with the aid of the OSS Compass community [54]. We start by selecting the most popular five thousand projects from the May 2019 GHTorrent dump according to the number of stars a source repository receives. Then, we filter projects based on the following criteria.

• •

  Projects that do not develop software applications or frameworks are removed. We remove projects that serve for document storage or course teaching. We examine project names and ‘README’ files by searching keywords, including awesome, homework, assignment, course, note, and document. If any keyword is found, we remove the project after manually rechecking. We also delete projects with no programming-language-specific files by looking at the file extensions.

• •

  Projects whose number of active forks or external pull-requests are less than one hundred are removed. To ensure a project has sufficient forks and contributions, we retain projects that 1) contain at least one hundred active forks, i.e., forks that have pushed at least one commit into the forked repository [34], and 2) contain at least one hundred external pull-requests submitted by external contributors.

• •

  Projects whose issues are less than one hundred are also removed. To reduce the threat to RQ3, we conservatively exclude projects with less than one hundred issues to ensure the remaining projects actively use the default issue-trackers on GitHub.

Among the 2533 candidate projects selected with the above criteria, we sample fifty projects⁵⁵5A complete list of the fifty projects studied in this paper can be found at https://github.com/wangliang-cs/fork-entropy-ase-2023-repos. to cover different application domains, which include ten application software (e.g., Atom/atom), two system software (e.g., kubernetes/kubernetes), sixteen web libraries and frameworks (e.g., angular/angular.js), nine non-web libraries and frameworks (e.g., tensorflow/tensorflow), and thirteen software tools (e.g., Microsoft/vscode).

We calculate monthly variables for each project from its creation to May 2019. All variables in each project snapshot compose an independent unit for regression analyses. We omit units with empty fork populations because fork entropy is meaningless without forks, even if it has a value of zero. Before fitting models with the data, we perform log-transform on control variables under skewed distributions to stabilize variance and reduce heteroscedasticity [55], including NumForks, NumFiles, NumStars, and RatioOldContributors. Then, we standardize fork entropy and all control variables to make the mean of each one is zero and the standard deviation is one, which makes all estimated coefficients of the model are on the same scale. Additionally, we manually examine distributions of outcome variables and conservatively remove about 1% of values as outliers to ensure the models are robust against outliers [56].

We build generalized linear mixed models (GLMMs) to analyze the correlations between fork entropy and the outcome variables. GLMMs inherit from generalized linear models (GLMs) to allow response variables from non-normal distributions and extend GLMs to include both fixed and random effects [57]. After exploratory data analysis, GLMMs are appropriate because the response variables in our data are non-normal and show apparent variability among projects. Specifically, fork entropy, control variables, and interactions between fork entropy and each control variable are modeled as fixed effects. To capture the project-to-project variability in the response (e.g., some projects naturally attract more external contributors and receive more contributions than others), we add a random-effects term for projects into the models. We also allow for deviations in the slope of the number of a project’s forks from the population values (i.e., we accept the possibility that, for example, projects with higher initial external productivity may, on average, be less strongly affected by the increase in fork counts). We use the glmer function provided by the lme4 package [58] in R to build models. Following prior practices [59, 6], the Poisson and logistic regressions are specified for count (i.e., external productivity and number of reported bugs) and ratio (i.e., the acceptance rate of external pull-requests) response variables, respectively. We also explicitly set the denominator (i.e., the number of closed pull-requests) from the ratio as the weights parameter when modeling the acceptance rate of external pull-requests.

We check the collinearity among independent variables using the variance inflation factors (VIF below five is recommended [60]). All values are below 2 in our models, which means collinearity is not a problem in our data. We adopt the marginal R-squared ($R_{m}^{2}$) and the conditional R-squared ($R_{c}^{2}$) to assess the goodness-of-fit of the models. $R_{m}^{2}$ and $R_{c}^{2}$ describe the proportion of variance explained by the fixed effects alone and explained by the fixed and random effects together, respectively [61, 62]. In addition, we report the estimated effect, standard error, and significance level (i.e., $p$-value) for each model variable. We also report the results of ANOVA type-II analysis, and Akaike’s information criteria (AIC) [63] and Bayesian’s information criteria (BIC) [64] for each model using the car and performance package [65] in R, respectively.

To answer RQ1, we model the number of commits integrated from external contributor-owned forks into a project’s source repository as a function of fork entropy. In the regression, we control factors including the number of forks, the number of modified files, project age, the number of stars, and the ratio of external contributors with prior experience. Table I summarizes the regression results. We can see that the model is effective by explaining about 98% of the variance with the fixed and random effects together, which exceeds the fixed effects alone by approximately 58 percentage points.

The results in Table I suggest that fork entropy significantly and positively correlates to the external productivity of OSS projects ($p$-value below 0.001), with the third largest estimated coefficient just lower than the coefficients of NumForks and NumFiles. This result suggest that projects with more diverse fork populations integrate more commits created by external contributors. In addition, the controlled factors in the model are also significantly linked to external productivity. Table I shows that all controls except project age are positively correlated to the external productivity of OSS projects. As a result, OSS projects with more forks and modified files, higher popularity, younger in age, and more proportion of external contributors with prior experience generally correspond to higher external productivity.

We next analyze the interactions between fork entropy and the control variables in the model. As shown in Table I, all interactions show significance. The interactions between fork entropy and the number of forks and between fork entropy and project age negatively correlate to the external productivity of OSS projects, and the remaining interactions show positive correlation with the response. We can also observe from the table that the two interactions negatively correlated to the response have the two largest effects as discussed below.

Fig. 2(a) illustrates the trends of external productivity with the increasing number of forks at low, middle, and high levels of fork entropy, respectively. We define the low, middle, and high levels of fork entropy respectively to correspond to its values with mean minus one standard deviation (Mean - Std.), mean, and mean plus one standard deviation (Mean + Std.) after consulting the literature [66]. In general, external productivity raises with the increase in NumForks. And the interaction suggests a higher level of fork entropy is associated with a lower growth rate of external productivity with respect to NumForks. With a fixed number of forks, an increase in a project’s fork entropy is related to an increased external productivity. This result suggests that, a project with a larger and more diverse population of forks have a higher probability of receiving contributions from the external contributors.

Fig. 2(b) depicts the interaction between fork entropy and project age. We can see that the external productivity generally decreases with the increasing of project age. The ratio of decreasing in external productivity with increased project age is amplified with higher levels of fork entropy. A possible explanation to this result is, younger projects under active development are possibly more open to accepting diverse contributions from external contributors than older projects. And projects older in age may have possibly entered a stage of stable maintenance, or slowly dying for lacking the capacity of handling diverse external contributions.

Fork entropy shows a positive and significant interaction between each of the other control variables including NumFiles, NumStars, and RatioOldContributors, respectively, as reported in Table I. We omit the details due to page limits.

In summary, we answer RQ1 as follows. Fork entropy shows a significant, positive correlation with the external productivity of OSS projects. Further analysis uncovers significant interactions between fork entropy and other variables, including the number of forks, project age, number of files, number of stars, and ratio of old contributors. We observe that increasing fork entropy is related to a higher level of external productivity, particularly with respect to the number of forks. Additionally, our results indicate that for younger projects, increasing fork entropy shows a stronger positive relation with external productivity compared to older projects.

To answer RQ2, we study the correlation between fork entropy and the acceptance rate of pull-requests submitted by external contributors. Here we calculate fork entropy based on a variant of the file modification matrix that only includes modifications involved in pull-requests. We perform the adjustment because pull-requests allow us to figure out which modifications are submitted to source repositories in each time interval. We exclude modifications made in forks but not included in pull-requests because they are largely invisible to maintainers who make decisions. We apply logistic regression for the acceptance rate by considering fork entropy. We control other factors, including NumForks, NumFiles, ProjectAge, RatioOldContributors, RatioPRsWithTests, and RatioPRsTouchHotFiles. Interactions between fork entropy and each control variable are also involved in the model. We ignore the interaction between fork entropy and NumForks because the control variable does not show a significant correlation with the response. Table II summarizes the model’s results. The model explains about 53% of variability by including the fixed and random effects together, achieving a significant improvement compared to the fixed effects alone.

From Table II, we find fork entropy significantly and positively correlates to the acceptance rate of external pull-requests, despite that fork entropy has a smaller estimated effect of 0.179 compared with the estimated coefficients of RatioOldContributors, ProjectAge, and NumForks of 0.614, 0.49, -0.196, respectively. However, the effect of NumForks fails to show statistical significance. Except for RatioPRsTouchHotFiles, the other four control variables are positively linked to the acceptance rate of external pull-requests.

For the model’s interaction terms, only the interaction between fork entropy and the proportion of pull-requests that touch hot files show significant and positive correlation with the acceptance rate of external pull-requests. The remaining interactions are significantly and negatively correlated to the response. Furthermore, we analyze the interactions with relatively notable effects, i.e., the interaction between fork entropy and RatioPRsTouchHotFile, as well as its interaction with RatioPRsWithTests, as follows.

First, as illustrated in Fig. 3(a), the correlation between the proportion of pull-requests that touch hot files (RatioPRsTouchHotFiles) and external pull-requests’ acceptance rate differs with different levels of fork entropy. The acceptance rate declines with RatioPRsTouchHotFiles when fork entropy is at a median or low level, and increases when fork entropy is high. A possible explanation is that, when fork entropy is at the median or low level, pull-requests are touching the same or similar sets of hot files, which result in conflicts and duplication among them, leading to a reduced acceptance rate. On the contrary, when fork entropy is at a high level, the pull-requests are touching different, less overlapped, hot files in the recent history. As a result, an increase in RatioPRsTouchHotFiles also leads to a increase in acceptance rate.

Next, Fig. 3(b) illustrates the interaction between fork entropy and RatioPRsWithTests on the acceptance rate of external pull-requests. We observe that an increase in the proportion of pull-requests that contain test cases is correlated to a higher acceptance rate when fork entropy is at a low level, which is consistent with existing observations that including test cases are helpful to make a pull request being accepted [52]. However, the above direction of correlation inverses at a high level of fork entropy. From the perspective of varying fork entropy by fixing the other factor, i.e., comparing the left and right parts separated by the vertical dashed line in Fig. 3(b), the acceptance rate generally increases with an increasing fork entropy with respect to the number of pull-requests contain test cases. But when the majority of pull-requests contain test cases, as shown by the part to the right of the dashed line, the increase in fork entropy instead shows a negative correlation with the acceptance rate.

We omit the details about the interactions between fork entropy and other control variables including NumFiles, ProjectAge, and ProjectAge, which show significant and negative interactions as listed in Table II due to page limites.

In summary, our answer to RQ2 is as follows. Fork entropy has a statistically significant, positive correlation with the acceptance rate of external pull-requests, albeit with a limited effect. Our analysis indicates that fork entropy also positive interacts with the number of pull-requests that touch hot files on the response. We find that the acceptance rate of pull-requests touching hot files only exhibits an upward trend when fork entropy is at a high level.

In this section, we study the correlation between fork entropy and the number of reported bugs of OSS projects. We model the number of bug-report issues as a function of fork entropy with controlling other factors including the number of forks, the number of modified files, project age, and the number of stars. Interactions between fork entropy and each controlled factor are also involved in the model as fixed effects. Table III summarizes the results of the number of reported bugs model. The model fits the data well by explaining about 96% of the variability by including both fixed and random effects, achieving a considerable improvement compared to the fixed effects alone by about 88 percentage points.

The results in Table III suggest that fork entropy significantly and negatively correlates to the number of bug-report issues with an estimated coefficient of -0.086. The absolute value is the third largest among the factors, lower than the estimated coefficient of NumForks and ProjectAge, which are 0.356, and 0.115, respectively. The results also suggest that the number of modified files significantly and negatively correlates to the number of bug-reporting issues, while the other controlled factors show significant and positive correlations with the response. All interaction terms show significance, with the interaction of fork entropy and the number of forks negatively correlates to the response.

We observe an opposition between fork entropy and the number of forks, where the former negatively correlates to the response and the latter positively relates to the response, respectively. Fig. 4 shows the interaction. We find that more bug-report issues are submitted with the increasing in the number of forks regardless of fork entropy. There is a slower growth in bug-report issues when fork entropy is at a higher level. Combined with the results in in RQ 1 and RQ 2, a possible explanation to the above result is that increased fork entropy are related to more commits and pull-requests been accepted, including bug-fixing ones, before potential bugs are reported. With the above results, if we agree with the assumption that less reported bugs indicates a higher software quality [27, 26], we can state that a higher level of fork entropy is positively linked to an improved software quality.

In summary, we answer RQ3 as below. Our study reveals a significant, negative correlation between fork entropy and the number of reported bugs in OSS projects. Furthermore, it shows that the rising trend of bug-reporting issues associated with the increasing number of OSS projects’ forks is reduced when fork entropy is at high levels.

This section discuss the implications of our work.

Developing metrics and models that measure the health of OSS projects has received much attention from both researchers and practitioners of OSS recently. For example, communities such as CHAOSS [67] and OSS Compass [54] are founded to achieve an understanding, and provide services to measure the health of OSS communities, projects, and ecosystems through numerous qualitative and quantitative metrics. Fork related metrics, such as the number of technical forks [68], is considered as a useful indicator. The fork entropy proposed in this paper is a new metric about an OSS project’s forks, and can potentially be added to the collection of metrics provided by the above communities due to its correlations with projects’ external productivity, pull-request acceptance ratio, and number of reported bugs as shown in our studies.

The proposed fork entropy metric also provides opportunities for further research on OSS projects’ forks. For instance, one could utilize pattern mining and time series analysis technologies to examine the evolving patterns and future trends of fork entropy over time [69]. Furthermore, it is potentially important to identify and assess the various factors and events that contribute to the rise and fall of fork diversity in OSS projects, and analyze their impact on the sustainability and prosperity of OSS projects [12, 33]. With a thorough understanding of fork diversity trends and influencing factors, it is possible to develop monitoring tools and guidelines to facilitate effective forking and collaboration during the development and maintenance of OSS projects.

We discuss the threats to validity as follows.

Construct Validity. This work explores the diversity of OSS projects’ fork populations measured by the proposed fork entropy metric. The construct validity concerns about whether the fork entropy measures the diversity of the fork population. Because fork entropy is built on top of the well-established metric of Rao’s quadratic entropy [9] which has shown to be effective in measuring population diversity [10], and because we have shown the properties of for entropy including symmetry, continuity, and monotonicity, we argue that the proposed fork entropy is valid in measuring the diversity of an OSS project’s fork population.

Internal Validity. In this work, we conduct a data-driven approach to study the correlation between fork entropy and OSS projects’ external productivity, pull-request acceptance rate, and number of reported bugs. We also restrict our scope to forks owned, and contributions made, by external contributors. The restriction is designed to make our study more focused on fork-related properties about OSS projects. However, there are many other factors affecting the contributions made to an OSS project [13]. External contributors’ contribution are also partially determined by the core-members of OSS teams. As a result, the conclusions made in this paper do not fully explain how fork-related contributions made to OSS projects or imply a causal relationship. In addition, we set the time interval to one month when taking snapshots to build file modification matrices and calculate fork entropy. Although this size is widely used in previous OSS-related studies [26], changing the time granularity during the analysis may potentially change the results. It is our future work to test the results with different time intervals.

External Validity. We select fifty projects from GitHub that cover various application domains in our studies. However, the collection of projects studied is small compared to the number of projects hosted on OSS platforms. The conclusions made in this paper may not generalize well to other projects, or to projects hosted on other platforms. It is our future work to include more projects in our studies.