PRF: A Framework for Building Automatic Program Repair Prototypes for JVM-Based Languages

Automatic program repair (APR) (Goues et al., 2019) is one of the recent advances in automated software engineering that holds the promise of reducing debugging costs by suggesting high-quality patches that either directly fix the bugs or help the developers during manual debugging (Liang et al., 2020). In the last decade, APR has been the subject of intense research and still remains an active area of research (Gazzola et al., 2017; Monperrus, 2018).

Generate-and-validate (G&V) refers to the class of APR techniques that attempt to fix the bug by repeatedly generating patches to produce program variants that subsequently get validated against certain rules or checks. A patch is called plausible if it passes all the checks. Validating a patch can be accomplished via a spectrum of techniques ranging from sound formal verification techniques (van Tonder and Goues, 2018) to testing (Martinez and Monperrus, 2016; Ghanbari et al., 2019; Wen et al., 2018). However, in real-world situations, formal specification of software is usually absent and automating formal verification is theoretically impossible. Testing, on the other hand, remains as the prevalent, economic method of getting more confidence about the quality of software, and a vast majority of G&V APR techniques use tests as correctness criteria.

By examining the architecture of state-of-the-art test-based G&V APR tools (Gazzola et al., 2017), we observed that most of them have components for fault localization (FL) (Wong et al., 2016), patch generation, and patch filtering/prioritization. During fault localization a suspiciousness value is assigned to each executed program location. During patch generation, a subset of these locations are transformed, resulting in a set of patches. Since test cases do not perfectly specify the software, many of the generated patches happen to pass all the tests without fixing the bugs. Such patches are known as overfitted patches (Smith et al., 2015). Therefore, the APR process is usually followed by a post-processing phase to filter out overfitted patches or prioritize patches that are more likely to be correct. Different program repair techniques usually differ in the way they generate patches to produce program variants, while the other components remain more or less the same and are reused from one implementation to another or reinvented. Even reusing these processes takes a considerable amount of programming effort for the APR researchers when building their prototypes.

This paper presents PRF (Program Repair Framework), a framework for building APR prototypes by extending it with a patch generation plugin for transforming the identified suspicious locations. PRF provides the APR tool programmer with spectrum-based FL information at different levels of granularity (e.g., line-/statement-, method-, or class-level). The framework also provides a safe, fast, and configurable facility for patch validation, and a customizable fix report generation component for prioritization/filtering of plausible patches. The patch validation component employs an assortment of techniques to safely accelerate patch validation process; it reorders test cases and does test selection. The component also implements a novel work-stealing algorithm (Wikipedia contributors, 2020) so that the patches can be validated concurrently, thereby maximizing CPU utilization. This compensates for the overhead of repeated creation of Java Virtual Machine (JVM) processes to achieve safety in patch validation. Although PRF is shipped with a default fix report generation plugin, the APR tool programmers can construct their own patch prioritization/filtering plugins for a customized fix report generation.

PRF depends solely on runtime information, so it is useful not just for prototyping Java APR tools but also for building tools targeting other JVM-based programming languages such as Kotlin, Scala, etc. It also comes in the form of a Maven plugin so that it can be reliably applied on different Maven-based projects.

We have constructed a patch generation plugin by specializing CapGen APR tool proposed by Wen et al. (Wen et al., 2018). Our experiments with CapGen show that patch validation is responsible for 77.7% of the total repair time, on average, and our integration with PRF results in 3.1X speedup when reordering test cases while validating patches sequentially without any test selection. In addition, we obtained an 11.4X speedup in patch validation when using 8 CPU cores combined only with test reordering.

Constructing a plugin for CapGen is as simple as making a wrapper for the tool and disabling the patch validation and prioritization processes that are to be delegated to PRF. We expect that with the help of APR libraries like ASTOR (Martinez and Monperrus, 2016), APR researchers will be able to make fast and reliable prototypes easily. This will, in turn, greatly help the reproducibility of APR experiments. Such a customizable framework has the added benefit of simplifying studies that compare different algorithms used in certain aspects of APR (e.g., fault localization (Liu et al., 2019) or patch classification/prioritization (Ye et al., 2019)).

PRF and its documentation, together with experimental result for CapGen, are publicly available on GitHub (Ghanbari and Marcus, 2020).

Inspired by the steps common to state-of-the-art test-based G&V APR techniques, we build PRF with four components responsible for profiling, patch generation, patch validation, and fix report generation. Figure 1 shows PRF’s main components. We next describe each component in more details.

PRF comes in the form of a Maven plugin. After checking out the source code from the GitHub repository (Ghanbari and Marcus, 2020) and installing the Maven plugin and the core library for PRF on the local Maven repository, the framework will be ready to use. The following XML snippet shows the minimal amount of configuration in the POM file of the target buggy project.

Once mvn edu.utdallas:prf-maven-plugin:run is executed through the command-line, PRF shall use a default patch generation plugin named DummyPatchGenerationPlugin which simply looks for a directory named patches-pool under the base directory of the project. This directory is expected to contain the pool of the generated patches, and the class file(s) for each patch must reside in a separate sub-directory. Names of the sub-directories shall be used as patch identifiers during the fix report generation. Similarly, PRF uses a default patch prioritization plugin named DummyPatchPrioritizationPlugin that does not have any effect. Furthermore, by default, only test execution time and test result collector is active in the profiler, and patch validator uses all the available CPU cores to validate patches in parallel.

These default choices can be overridden by the user through the POM file for the program under repair. Such configurations go under the <configuration> tag in plugin description in the above XML code. Table 1 summarizes the options that the users can tune. More details about using PRF is available in the project documentation (Ghanbari and Marcus, 2020) and in the companion demo video https://bit.ly/3ehduSS.

FLAIR is a proprietary framework used in Fujitsu Labs America to construct the APR system ELIXIR (Saha et al., 2017). However, this system is not publicly available. PRF is the first open-source framework that provides all the functionalities offered by a framework like FLAIR, plus PRF offers an efficient patch validation facility.

ASTOR (Martinez and Monperrus, 2016) is a general-purpose library for developing source code level, Java-based APR tools. Unlike ASTOR, PRF is a customizable framework upon which different patch generation algorithms can be installed as plugins and different strategies for patch prioritization and/or filtering can be employed. PRF is not intended to replace ASTOR, instead, PRF complements the library in that it enables APR researchers focus on taking full advantage of ASTOR’s features to implement more reliable patch generation algorithms, and better understand the impact of different fault localization, patch prioritization and/or filtering strategies on the effectiveness of their implementation. As a toolset for prototyping APR techniques, PRF can be seen as a step similar in nature to (Gay and Just, 2020).

Le Goues et al. (Le Goues et al., 2013) highlight the high cost of patch validation in test based G&V APR. Mehne et al. (Mehne et al., 2018) report that patch validation can take between 40% to 92% of total repair time and propose to prune the patches needed to be tested as well as test case selection to reduce this cost. A recent line of research (Ghanbari et al., 2019; Guo et al., 2019), proposes to use the HotSwap trick offered by the JVM to validate the patches on-the-fly, without restarting the JVM. This was previously used in mutation testing systems like PIT (Coles, Henry, 2020). It has the benefit of avoiding the high costs of restarting JVM for each patch, but the patches that involve altering the class structure (e.g., addition or removal of class members) are not eligible for being HotSwapped. The approach in order to be effective needs proper isolation of test execution side-effects (e.g., (Bell and Kaiser, 2014)). However, such a method does not solve the current limitations of a HotSwap-based approach. In this work, we follow a different approach that proves to be fast, reliable, and more effective. PRF validates each patch in a fresh JVM session, thereby containing the side-effects of test execution and also avoiding restrictions of a HotSwap-based approach, and instead of dealing with the internals of JVM, we rely on test selection and reordering, as well as parallelism to speedup patch validation.

With PRF, APR researchers will be able to build research prototypes by simply providing a patch generation plugin and fine tuning the framework’s existing components for multi-granularity level fault localization, patch validation, and fix report generation. PRF uses a novel patch validation technique, relying on test selection and prioritization as well as parallelism, that achieves 11+X speedup compared to vanilla testing. PRF is publicly available at (Ghanbari and Marcus, 2020).

We are working on integrating our JVM language agnostic patch prioritization system ObjSim (Ghanbari, 2020) in PRF and release the framework with an effective built-in patch prioritization mechanism.