Parameter-Free Probabilistic API Mining across GitHub

Learning the application programming interface (API) of an unfamiliar library or software framework can be a significant obstacle for developers [32, 33]. This is only exacerbated by the fact that API documentation can often be incomplete or ambiguous [37]. Fortunately, an opportunity to address this problem has arisen out of the simultaneous growth in the amount of source code that is available online and the growth of large scale data mining and machine learning methods. This confluence has enabled the development of API mining methods [44, 40], which aim to automatically extract a set of API patterns, which are lists of API methods that are usually used together, and that together characterize how an API is used.

Despite a number of interesting proposed tools, including well known ones such as MAPO [44] and UPMiner [40], so far API mining tools have not yet gained wide-spread adoption in development environments such as Eclipse and Visual Studio. We suggest that the fundamental reason for this is that the quality of the extracted patterns is not yet high enough: the patterns returned by current methods are numerous and highly redundant (Section 5). For example, Figure 5 shows the top ten API patterns extracted by two state of the art methods. To a large extent, the patterns from both methods are variations on a theme, repeating different variations of ways to repeat the same few API methods.

The fundamental reason for this, we argue, is that current API mining methods are built on statistically shaky ground. Specifically, API mining algorithms largely employ frequent sequence mining, which is a family of techniques from the data mining literature that takes as input a database of sequences and from those attempts to identify a set of patterns that frequently occur as subsequences. In the context of API mining, each sequence in the database is the list of API methods called by some client method, and the subsequence patterns that are mined are candidates for API patterns. Frequent sequence mining methods are very good at their intended purpose, which is to efficiently enumerate subsequences that occur frequently. But they are not suitable for pattern mining all by themselves, for a simple reason: frequent patterns are often uninteresting patterns. This problem is well known in the data mining literature ([4], Chap. 5), and the technical reason for it is easy to understand. Events that are individually frequent but unrelated will also tend to frequently occur in a sequence, simply by chance. Imagine running a frequent sequence mining algorithm on the sequence of events that occur in the day of a typical academic. Perhaps the most frequent individual events would be SendEmail and DrinkCoffee. Then, a frequent sequence miner may well return the pattern (SendEmail, DrinkCoffee) even if the two actions have no direct statistical relationship, because frequent sequence miners do not consider any notion of statistical correlation or independence among items in the sequence. Within the API mining literature, methods like MAPO and UPMiner apply clustering methods precisely to reduce the number of redundant API patterns that would be returned from the frequent sequence miner; however, as we show in Section 5, even with this step, substantial redundancy remains.

We address this problem by developing new a mining algorithm that returns API patterns that not only occur often, but also that occur in a sequence more often than expected by chance, that is, the most interesting sequences ([4], Chap. 5). In order to quantify what “most interesting” means, we employ a powerful set of techniques from statistics and machine learning called probabilistic modelling. That is, we design a probability distribution over the list of API methods called by client methods, and we evaluate a proposed API pattern by whether it improves the quality of the model. The quality of a probabilistic model can be measured simply by whether it assigns high probability to a training set.

While numerous sequential probabilistic models of source code have appeared in the literature [18, 7, 28, 31] these all rely on $n$-gram language models. But an $n$-gram in this context is a contiguous sequence of method calls, whereas API patterns contain gaps, that is, the methods in an API pattern do not always occur contiguously in a client method, but instead can have other method calls interspersed between them. For example, consider the methods in Figure 1 which show real-world usage examples of a four line call sequence to set up a twitter client. The first four lines of the last method shows the basic call sequence but the other two methods have additional calls interspersed between them, e.g. the first method also turns on SSL support. Although frequent sequence mining algorithms handle gaps in sequences automatically, previous probabilistic models of code do not, so we need to introduce a new probabilistic model.

To address this unsatisfactory state of affairs, we propose PAM (Probabilistic API Miner)¹¹1https://github.com/mast-group/api-mining, a near parameter-free probabilistic algorithm for mining the most interesting API patterns. PAM makes use of a novel probabilistic model of sequences, based on generating a sequence by interleaving a group of subsequences. The list of component subsequences are then the mined API patterns. This is a fully probabilistic formulation of the frequent sequence mining problem, is able to correctly represent both gaps in sequences, and — unlike API mining approaches based on frequent sequence mining — largely avoids returning sequences of items that are individually frequent but uncorrelated.

Furthermore, another drawback of current methods for API mining is that they have multiple parameters that are very hard to tune in practice, rendering them difficult to use without expensive pre-training. First, frequent sequence mining methods depend on a user-specified dataset-specific minimum frequency threshold [5, 17]. This threshold is extremely difficult to tune in practice as it is prone to exponential blow-up: setting the threshold too low leads to billions of patterns. Conversely, setting it too high leads to no patterns at all. API mining algorithms also tend to employ hierarchical clustering techniques in an attempt to reduce the inherent redundancy of frequent sequence mining (which tends to produce highly repetitive patterns). Choosing the number of clusters/cutoff is considered somewhat of a black art and can only be reliably inferred through computationally expensive training procedures on a validation set [44, 40]. PAM, on the other hand, is near parameter free in the sense that our two user-specified parameters are independent of the dataset, have simple intuitive interpretations and sensible default values.

To showcase PAM, we apply it to mining API patterns for 17 of the most popular Java libraries on the GitHub hosting service. We collect all usages of these APIs on GitHub, yielding a data set of 54,911 client methods from 967 client projects totalling over 4 million lines of code (LOC). We focus on libraries that contain a specific examples/ subdirectory, which allows us to automatically collect API usage examples that the libraries’ developers thought were most salient. This yields a corpus of 3,385 Java files comprising 300,000 LOC solely of hand-written API usage examples. These two sources of information allow us to perform extensive evaluation of PAM compared to MAPO and UPMiner. We show that PAM significantly outperforms both MAPO and UPMiner, achieving 69% precision on our test set. Moreover, the set of patterns that PAM returns is dramatically less redundant than those from MAPO or UPMiner. To the best of our knowledge, we are the first to mine API calls across the entirety of GitHub, from a data set of 54,911 client methods totalling over 4 million LOC.

We examine whether the API patterns mined by PAM could be used to augment API examples that are provided with popular projects. This is a new method of evaluating API mining algorithms. We show that there is substantial overlap between mined patterns and developer-written examples, providing evidence that the mined patterns are meaningful, but that — even though our corpus averages 18,000 example LOC per project — the mined patterns describe many new patterns that appear in practice but are not covered by the examples.

The first algorithm for mining API usage patterns from source code was MAPO, proposed by Xie and Pei [41] and extended by Zhong et al. [44]. MAPO [44] mines method call sequences (that call a desired API) from code snippets retrieved by code search engines. Sequences of called API methods are first extracted from the code snippets and then clustered according to a distance metric, computed as an average of the similarity of method names, class names, and the called API methods themselves. For each cluster, MAPO mines the most frequent API calls using SPAM and feeds them to an API usage recommender that ranks them based on their similarity with the developer’s code context.

UP-Miner [40] extends MAPO in an attempt to further reduce the redundancy of the mined API call sequences. This is achieved through three principal approaches: using the BIDE closed frequent sequence miner that returns only the frequent sequences that have no subsequences with the same frequency; using a clustering distance metric based on the set of all API call sequence $n$-grams and an additional clustering step on the mined call sequences from each cluster. Unlike MAPO, the API call sequences are presented as probabilistic graphs, ranked according to their frequency.

Acharya et al. [3] extract API usage scenarios among multiple APIs as partial orders. Frequent API calls are first mined from inter-procedural program traces, API call ordering rules are then extracted from the frequent calls and presented as partial order graphs.

Buse and Weimer [10] generate API usage documentation by extracting and synthesizing code examples from API method call graphs annotated with control flow information. Their approach first extracts method control flow paths from files that use a desired API. Path predicates are then computed for each statement relevant to a static instantiation of a desired API type and assembled, along with the statements, into API usage graphs. The API usage graphs are then clustered based on their statement ordering and type information, and an abstract usage graph is extracted from each cluster by merging its concrete API usages. Finally, each abstract API usage graph is transformed into a code fragment by extracting the most representative abstract statement ordering from the graph and assigning each abstract statement a concrete name according to heuristics.

MUSE [25] uses static slicing to extract and rank code examples that show how to use a specific method. For each relevant method invocation, MUSE extracts an intra-procedural backward slice, representing a raw code example. The raw code examples are then clustered using type-2 clone detection and the resulting groups of clones ranked according to their popularity. A representative code example from each group of clones is selected based on its readability and ease of reuse, and each example is annotated with inline descriptions of method parameters mined from the Javadoc.

Other API mining papers include Uddin et al. [38] who detect API usage patterns in terms of their time of introduction into client programs and Li and Zhou [22] who automatically extract implicit programming rules in large software code. A related approach that has been well-studied in the literature is API code search, where text matching is used to find API usage examples from a large, often online, corpus. SNIFF [11] finds abstract code examples relevant to a natural language query expressing a desired task. SNIFF annotates publicly available source code with API documentation and the annotated code is then indexed for searching. In response to a query, matching code examples are retrieved and concise examples are extracted via a syntax-aware intersection, clustered and ranked based on their frequency of occurrence. Similarly, Keivanloo et al. [20] find abstract code examples by combining textual similarity and clone detection techniques, ranking the returned examples according to their similarity to the query and the completeness and popularity of their concrete usage patterns. Bajracharya et al. [9] find concrete API usage examples by combining heuristics based on structural and textual aspects of the code, based on the assumption that code containing similar API usages is also functionally similar.

Strathcona [19] is a code example recommendation tool that automatically generates queries from the developer’s current code context. The code examples relevant to the queries are identified using six heuristics that compare the structural context of the query against the structural context of the classes and methods within an example repository. The resulting code examples are ranked according to their frequency in the final set of top examples returned by each heuristic. Prompter [30] takes a similar approach, matching the generated context-specific queries to Stack Overflow discussions and automatically recommending the discussions most relevant to the developers’ task.

The API mining problem we consider (Section 3) is specifically to return sequences of API methods that are used together. Other important but distinct data mining problems that are related to API usage include mining preconditions of API methods [27], and mining code changes [23, 26]. An important recent piece of infrastructure for large-scale mining work on code is Boa [13]. Another interesting line of work is to mine existing code to measure the popularity of language constructs and APIs [14]; our work considers a different mining problem, that of discovering combinations of API methods that are used together, rather than measuring the popularity of known language features.

Sequential pattern mining is an extremely well studied problem with a huge number of published papers on the topic. The problem was first introduced by Agrawal and Srikant [5] in the context of market basket analysis, which led to a number of other algorithms for frequent sequence mining, including GSP [35], PrefixSpan [29], SPADE [43] and SPAM [8]. Frequent sequence mining suffers from pattern explosion: a huge number of highly redundant frequent sequences are retrieved if the given minimum support threshold is too low. One way to address this is by mining frequent closed sequences, i.e., those that have no subsequences with the same frequency, such as via the BIDE algorithm [39]. More recently, there has been work on sequence mining that directly addresses the pattern explosion issue, such as SQS-search [36] and GoKrimp algorithm [21]. Our proposed approach falls into this class of probabilistic sequential pattern mining algorithms, and returns patterns that are of a quality that is comparable to, if not better than, both SQS and GoKrimp (see [15] for details).

There have also been sequential probabilistic models of source code proposed in the software engineering literature. Hindle et al. [18] develop an $n$-gram language model for source code and apply it to a code completion task. Allamanis et al. [7] use an $n$-gram language model to learn and suggest variable naming conventions. Raychev et al. [31] develop an $n$-gram language model that synthesizes code completions for programs using APIs.

First we describe the specific API mining problem that we consider in this paper. For every client project that uses a given API, we extract the sequence of API calls used by each method in the project. The problem of mining API calls is then to infer from these sequences of API calls those subsequences that represent typical usage scenarios for the API. These could then be either supplied in example documentation to client developers, or suggested in real time as developers type.

For the purposes of this paper, we use a best-effort approach to extract API call sequences directly from source code files. Following MAPO [44], we parse each Java source file of interest using the Eclipse JDT parser and extract method calls to API methods from a specified library using a depth-first traversal of the AST. For simplicity, unlike MAPO, we do not attempt to enumerate all possible branches of conditional statements. For example, for the code snippet if(m1()) { m2(); } else { m3(); }, our method returns the call sequence m1,m2,m3 whereas MAPO would return the call sequences m1,m2 and m1,m3. If this example were indeed a common API usage pattern, we would argue that returning m1,m2,m3 is better in principle, because a subsequence like m1,m2 would provide only incomplete information about what the developer should write next.

Furthermore, unlike MAPO, we only consider method invocations and class instance creations, and approximately resolve their fully qualified names from the file import statements. For simplicity, superclass methods (and their return types), super method/constructor invocations and class cast expressions are not considered.

We use an approach similar to the original MAPO paper [41] to approximately resolve fully qualified method names. We keep track of field and local variable declarations so that we can resolve the fully qualified name of a method call on the field/variable. We also keep track of import statements so that we can resolve fully qualified names of classes that are explicitly imported, as well as those imported using a wildcard by scanning for specific imports from the wildcarded package in the corpus in a pre-processing step. Additionally, in the pre-processing step we find the return types of locally declared methods so that we are able to subsequently resolve any calls on them. Finally, we filter out any method names that cannot be fully resolved. Each API call sequence is then the sequence of fully qualified API method names that are called by a method in the source file.

For example, consider the client methods in Figure 1 that all share the common twitter4j API call sequence:

This is the minimum API call sequence required to set up OAuth authorization for a twitter client. All the methods in Figure 1 have added extra API calls for optional functionality (e.g. SSL encryption) but all contain the minimal API call sequence as a subsequence.

There are of course limitations to this approximation (as noted in the original MAPO paper [41]). In particular it is not possible to resolve external nested method calls (i.e., in the call method1().method2(), we cannot resolve method2() unless method1() is declared locally). However, for the purposes of this paper we are primarily interested in assessing the performance of PAM. Moreover, it is important to note that PAM is flexible and supports any API call extractor that returns sequences of calls, making it applicable to dynamically inferred call sequences as well as other programming languages. While we mine (possibly incomplete) API calls that are inferred statically from .java files in this paper, one can in principle extract fully resolved static or dynamic API call sequences using the BCEL bytecode library [1, 2]. The reason we did not perform dynamic call sequence extraction is that the idiosyncratic build process of most Java projects would have made compiling all $967$ open-source Java projects that used our chosen libraries in our dataset (see Table 1) prohibitive. Finally, note that any API call extractor that only extracts sequences of calls cannot handle conditional statements properly, as it is trying to approximate a graph with a sequence.

In this section we will describe our novel probabilistic model for API mining. Our model is a joint probability distribution over the list of API calls in a client method, which we observe in the data, and the underlying API patterns that the programmer intended to use, which we never observe directly. The model defines this probability assuming that the set of all possible true API patterns is known. Then, learning involves working backward: given the client methods that were observed, what set of true API patterns might have generated them? Specifically, we measure the quality of a proposed set of API patterns by supposing those were the true patterns, and measure the probability that the model assigns to all client methods in the database. We search for the set of API patterns that maximizes this probability, specifically we perform this search under the framework of a celebrated algorithm from statistics called expectation-maximization (EM) [12], which has seen an enormous number of applications. We use a particular variant called structural EM [16], as this deals with the specific setting of learning via search through a large combinatorial space, in our case, the space of all possible sets of API patterns.

In the next sections, we give a high-level overview of each of the aspects of the model. For ease of exposition, we do not describe some of the more theoretical aspects; for those we refer the reader to our paper [15] that describes a similar model and algorithm for general sequential pattern mining.

In this section we perform a comprehensive evaluation of PAM across GitHub, comparing and contrasting it against MAPO and UPMiner. In particular, we aim to answer the following three research questions.

Specifically, we first find all Java projects in the corpus that have an example directory (i.e., matching *example*|*Example*) containing more than 10K LOC. From these projects we then select those that are in the top $1,000$ projects in the corpus, ranked according to popularity. Popularity in the GitHub corpus is calculated as the sum of the number of project forks and watchers, where each is separately normalized into a z-score. From these top projects, we determine which of these are called from 50 or more methods belonging to other projects in the corpus, leaving us with the 17 projects in Table 1. We call this set of projects and associated client code the Example dataset, to emphasize the fact that we focus on libraries and frameworks that include examples.

Each of these 17 projects is a library or framework, which we will call a target project, for which we wish to extract API patterns. For each target project, we perform API mining separately, and all results are reported as an average over the 17 target projects. To extract a set of client methods for each target project, we search the entire GitHub Java corpus for all source files that import a class belonging to the respective package or one of its subpackages and this set of files (excluding duplicates) formed the Client file set. Extracting, for each project, all source files in the aforementioned example directory (excluding duplicates) formed the Example file set. Statistics on both file sets are given in Table 1.

In an absolute sense, the level of agreement between PAM and the hand-written examples, although substantial, might not seem especially high. This raises an interesting question: Does this level of disagreement occur because the PAM patterns are not representative of the client code they were mined from, or because the hand-written examples themselves are not fully representative of the client code? Although previous work has explored what it means for a single API example to be useful [42, 34], there seems to be much less work about what it means for a set of examples to be useful, and how well example directories in popular projects reflect all of the actual common uses of an API.

We can however make an initial assessment of this question by going back to the held-out test set of client code that we used in RQ1. We can measure how well the client test set and the handwritten examples agree by measuring sequence precision and recall if we take the handwritten examples as if they were API patterns and the client test set as the gold standard. When we do this, we find that the handwritten examples have a recall of $27\%$, meaning that three-quarters of client API method sequences are not contained within any of the handwritten examples. Turning to precision, the handwritten examples have a precision of $36\%$, meaning that two-thirds of API sequences from the example code are not used by any client method (where “used by” means “fully contained by”). This is significantly lower than the precision between the training set of client methods and the test set, suggesting that the training set is more representative of the test set than the handwritten examples are. Although this might be seen as a suggestive result, we caution that this has an important threat to validity: handwritten examples may include scaffolding code that is unnecessary in client methods. For this reason, we advise caution about drawing strong conclusions from the precision of handwritten examples, but we note that this threat does not apply to the recall.

These results suggest that even in very well documented projects with extensive sets of examples, the API usage examples written by developers are still incomplete. While it may not seem surprising that developer-written example directories would be incomplete, recall that we specifically chose our data set to consist only of popular libraries with extensive handwritten examples — indeed, our data set averages 18,000 lines of example code per target API. It is striking that even with projects that are so extensively documented, PAM is still able to infer a list of coverage with substantially greater coverage of the API.

To gain further insight into this issue, we randomly selected three projects from our dataset and looked at the top five API patterns returned by PAM that were not present in any example call sequence. We found that the 15 selected API patterns fell into the following three categories: 7 referred to an API method not in any of the examples, 3 referred to an API class not in any of the examples and 5 referred to an API pattern that was not contained in any API example (although its methods were present in the examples). This provides some support for the hypothesis that the API patterns document part of the API that are used in client code but for which the original developers have not chosen to write specific examples.

Overall these results suggest that the patterns returned by PAM could serve as a useful supplement to code examples written by API developers. Indeed, these results raise the question of whether, in future work, PAM could be used to help detect novel and undocumented API usages and feed them back to library and framework maintainers.

We presented a parameter-free probabilistic API mining algorithm that makes use of a novel probabilistic model to infers the most interesting API call patterns and demonstrated the efficacy of our approach on dataset of several hundred thousand API client files from GitHub. Through our experiments we found suggestions that API calls are not well documented in example code and in future we would like to verify this through a large-scale empirical study.

This work was supported by the Engineering and Physical Sciences Research Council (grant number EP/K024043/1).