A Systematic Evaluation of
Static API-Misuse Detectors

Incorrect usages of an Application Programming Interface (API), or API misuses, are violations of (implicit) usage constraints of the API. An example of a usage constraint is having to check that hasNext() returns true before calling next() on an Iterator, in order to avoid a NoSuchElementException at runtime. Incorrect usage of APIs is a prevalent cause of software bugs, crashes, and vulnerabilities [1, 2, 3, 4, 5, 6, 7]. While high-quality documentation of an API’s usage constraints could help, it is often insufficient, at least in its current form, to solve the problem [8]. For example, a recent empirical study shows that Android developers prefer informal references, such as StackOverflow, over official API documentation, even though the former promotes many insecure API usages [9]. We confirm this tendency for a non-security API as well: Instances of Iterator may not be used after the underlying collection was modified, otherwise they throw a ConcurrentModificationException. Even though this constraint and the consequences of its violation are thoroughly documented, a review of the top-5% of 2,854 threads about ConcurrentModificationException on StackOverflow shows that 57% of them ask for a fix of the above misuse [10].

Ideally, development environments should assist developers in implementing correct usages and in finding and fixing existing misuses. In this paper, we focus on tools that identify misuses in a given codebase, specifically, those that automatically infer API-usage specifications and identify respective violations through static code analysis. We refer to these tools as static API-misuse detectors.

There have been many attempts to address the problem of API misuse. Existing static misuse detectors commonly mine usage patterns, i.e., equivalent API usages that occur frequently, and report any anomaly with respect to these patterns as potential misuse [11, 12, 13, 14, 15, 16, 17, 18, 19, 1, 20]. The approaches differ in how they encode usages and frequency, as well as in the techniques they apply to identify patterns and violations thereof. Despite the vast amount of work on API-misuse detection, API misuses still exist in practice, as recent studies show [21, 9]. To advance the state of the art in API-misuse detection, we need to understand how existing approaches compare to each other, and what their current limitations are. This would allow researchers to improve API-misuse detectors by enhancing current strengths and overcoming weaknesses.

In this work, we propose the API-Misuse Classification (MuC) as a taxonomy for API misuses and a framework to assess the capabilities of static API-misuse detectors. In order to create such a taxonomy, we need a diverse sample of API misuses. In our previous work, we described MuBench, a dataset of 90 API misuses that we collected by reviewing over 1200 reports from existing bug datasets and conducting a developer survey [3]. MuBench provided us with the misuse examples needed to create a taxonomy. To cover the entire problem space of API misuses, for this paper, we add further misuses to this dataset by looking at examples from studies on API-usage directives [8, 22]. Using MuC, we qualitatively compare 12 existing detectors and identify their shortcomings. For example, we find that only few detectors detect misuses related to conditions or exception handling. We confirm this assessment with the detectors’ original authors.

The previous step provides us with a conceptual comparison of existing detectors. We also want to compare these API-misuse detectors empirically, by both their precision and recall. This is a challenging task, due to the different underlying mechanisms and representations used by detectors. To enable this empirical comparison, we build MuBenchPipe, the first automated pipeline to benchmark API-misuse detectors. Our automated benchmark leverages MuBench, and the additional misuses we collect in this work, and creates an infrastructure on top of it to run the detectors and compare their results. We perform three experiments based on 29 real-world projects and 25 hand-crafted examples to empirically evaluate and compare four state-of-the-art detectors. We exclude the other eight detectors since two rely on the discontinued Google Code Search [23], five target C/C++ code, and one targets Dalvik Bytecode, while our benchmark contains Java misuses. In Experiment P, we measure the precision of the detectors in a per-project setup, where they mine patterns and detect violations in individual projects from MuBench. In Experiment RUB, we determine upper bounds to the recall of the detectors with respect to the known misuses in MuBench. We take the possibility of insufficient training data out of the equation, by providing the detectors with crafted examples of correct usages for them to mine required patterns. Finally, in Experiment R, we measure the recall of the detectors against both the MuBench dataset and the detectors’ own confirmed findings from Experiment P using a per-project setup.

Our conceptual analysis shows many previously neglected aspects of API misuse, such as incorrect exception handling and redundant calls. Our quantitative results show that misuse detectors are capable of detecting misuses, when provided with correct usages for pattern mining. However, they suffer from extremely low precision and recall in a realistic setting. We identify four root causes for false negatives and seven root causes for false positives. Most importantly, to improve precision, detectors need to go beyond the naive assumption that a deviation from the most-frequent usage corresponds to a misuse, for example, by building probabilistic models to reason about the likelihood of usages in their respective context. To improve recall, detectors need to obtain more correct usage examples, possibly from different sources, and to consider program semantics, such as type hierarchies and implicit dependencies between API usages. These novel insights are made possible by our automated benchmark. Our empirical results present a wake-up call, unveiling serious practical limitations of tools and evaluation strategies from the field. Foremost, detectors suffer from extremely low recall—which is typically not evaluated. Moreover, we find that the application of detectors to individual projects does not seem to give them sufficient data to learn good models of correct API usage.

In summary, this paper makes the following contributions to the area of API-misuse detection:

• •

  A taxonomy of API misuses, MuC, which provides a conceptual framework to compare the capabilities of API-misuse detectors.

• •

  A survey and qualitative assessment of 12 state-of-the-art misuse detectors, based on MuC.

• •

  A publicly available automated benchmark pipeline for API-misuse detectors, MuBenchPipe, which facilitates systematic and reproducible evaluations of misuse detectors.

• •

  An empirical comparison of both recall and precision of four existing misuse detectors using MuBenchPipe. Our work is the first to compare different detectors on both a conceptual and practical level and, more importantly, the first to measure the recall of detectors, unveiling their poor performance.

• •

  A systematic analysis of the root causes for low precision and recall across detectors, to call researchers to action.

Our benchmarking infrastructure is publicly available [24] and our artifact Web page [10] provides full details on our results.

An API usage (usage, for short) is a piece of code that uses a given API to accomplish some task. It is a combination of basic program elements, such as method calls, exception handling, or arithmetic operations. The combination of such elements in an API usage is subject to constraints, which depend on the nature of the API. We call such constraints usage constraints. For example, two methods may need to be called in a specific order, division may not be used with a divisor of zero, and a file resource needs to be released along all execution paths. When a usage violates one or more such constraints, we call it a misuse, otherwise a correct usage.

The detection of API misuses may be approached through static analyses of source code or binaries and through dynamic analyses, i.e., runtime monitoring or analysis of runtime data, such as traces or logs. In either case, the detection requires either specifications of correct API usage to find violations of or specifications of misuses to find instances of. Such specifications may be crafted manually by experts or inferred automatically by algorithms. Automatic specification inference (or mining) may, again, be approached both statically, e.g., based on code samples or documentation, and dynamically, e.g., based on traces or logs.

Since manually crafting and maintaining specifications is costly, in this work, we focus on automated detectors. We call such tools API-misuse detectors. In the literature, we find static misuse detectors, which statically mine specifications and detect misuses through static analysis, e.g., [13, 15, 1]; dynamic misuse detectors, which dynamically mine specifications and detect misuses through dynamic analysis, e.g., [25, 26]; and hybrid misuse detectors, which, for example, combine dynamic specification mining with static detection [27]. In this work, we focus on static API-misuse detectors.

Static API-misuse detection is often achieved through detecting deviant code [28, 11, 12, 13, 14, 15, 16, 17, 18, 19, 1, 20]. The key idea is that mistakes violate constraints that the code should adhere to and that, given sufficiently many examples of correct usage, such violations appear as anomalies. We call a usage that appears frequently in programs a pattern. The identification of mistakes through the detection of deviant code assumes that patterns correspond to correct usages (specifications) and anomalies with respect to these patterns are, consequently, misuses. Such an approach can detect mistakes in the usage of popular libraries [28, 11, 13, 15, 19, 1].

In our previous work [3], we collected a dataset of Java API misuses by reviewing bug reports of 21 real-world projects and surveying developers about API misuses. We call this dataset MuBench. It contains 90 misuses, 73 misuses from the real-world projects and 17 from the survey (see Table I, Row 1). For each real-world misuse, the dataset identifies the project where the misuse is, the project version that contains the misuse, and the commit that fixed the misuse. For the other misuses, MuBench provides hand-crafted misuse examples and their fixes.

In this section, we introduce the API-Misuse Classification (MuC), our taxonomy for API misuses. We derive MuC from the misuse examples in the MuBench dataset. In Section 4, we use MuC to qualitatively compare the capabilities of existing API-misuse detectors. In Section 7, we use MuC to define our expectations on the detectors’ performance. Before presenting the classification itself, we briefly discuss existing related classifications to motivate the need for MuC.

To advance the state of the art of API-misuse detection, we need to understand the capabilities and short-comings of existing misuse detectors. To identify detectors, we started from the publications about API-misuse detection listed in a survey of automated API-property inference techniques by Robillard et al. [33]. For each publication, we looked at all publications they refer to as related work and all publication that cite them, according to the ACM Digital Library or the IEEE Xplore Digital Library. We recursively repeated this process, until we found no new detectors.

We use MuC to guide the comparison. We provide a conceptual classification of the capabilities of each detector with respect to MuC, summarized in Table III. We use the published description and results of each detector to identify which of MuC categories they can, conceptually, detect. To reduce subjectivity, we confirmed our capability assessment and the detector descriptions with the respective authors, except for PR-Miner and Colibri/ML, whose authors did not respond. We also describe the strategies used to evaluate each detector and summarize those in Table IV.

PR-Miner is a misuse detector for C [11]. It encodes usages as the set of all function names called within the same function and then employs frequent-itemset mining to find patterns with a minimum support of $15$ usages. Violations here are strict subsets of a pattern that occur at least ten times less frequently than the pattern. To prune false positives, PR-Miner applies inter-procedural analysis, i.e., for each occurrence of a violation, it checks whether the missing call occurs within a called method. This analysis follows the call path for up to 3 levels. The reported violations are ranked by the respective pattern’s support. PR-Miner focuses on detecting missing method-calls. The evaluation applied PR-Miner to three target projects individually, thereby finding violations of project-specific patterns. The detector reported 1,601 findings (1,447, 147, and 7 on the individual projects). The authors reviewed the top-60 violations reported across all projects and found 18.1% true positives (26.7%, 10.0%, and 14.3% on the individual projects).

Chronicler is a misuse detector for C [34]. It mines frequent call-precedence relations from an inter-procedural control-flow graph. A relation is considered frequent, if it holds on at least 80% of all execution paths. Paths where such relations do not hold are reported as violations. Chronicler detects missing method calls. Since loops are unrolled exactly once, it cannot detect missing iterations. The evaluation applied Chronicler to five projects, thereby finding violations of project-specific patterns. The authors compared the identified protocols with the documented protocols for one API and discussed a few examples of actual bugs found by their tool.

Colibri/ML is another misuse detector for C [12]. It re-implements PR-Miner using Formal Concept Analysis [35] to strengthen the theoretical foundation of the approach. Consequently, its capabilities are the same as PR-Miner’s. The evaluation applied Colibri/ML to five target projects, thereby finding violations of project-specific patterns. While some detected violations are presented in the paper, no statistics on the quality of the detector’s findings are reported.

Jadet is a misuse detector for Java [13]. It uses Colibri/ML [12], but instead of only method names, it encodes method-call order and call receivers in usages. It builds a directed graph whose nodes represent method calls on a given object and whose edges represent control flows. From this graph, it derives a pair of calls for each call-order relationship. The sets of these pairs form the input to the mining, which identifies patterns, i.e., sets of pairs, with a minimum support of $20$. A violation may miss at most $2$ properties of the violated pattern and needs to occur at least ten times less frequently than the pattern. Detected violations are ranked by $u\times s/v$, where $s$ is the violated pattern’s support, $v$ is the number of violations of the pattern, and $u$ is a uniqueness factor of the pattern. Jadet detects missing method calls. It may detect missing loops as a missing call-order relation from a method call in the loop header to itself. The evaluation applied Jadet to five target projects, thereby finding violations of project-specific patterns. The authors reviewed the top-10 violations reported per project and found 6.5% true positives (0%, 0%, 7.7%, 10.5%, and 13.3% on the individual projects). Other findings were classified as code smells (6.5%) or hints (35.0%).

In a subsequent study, Jadet was applied in a cross-project setting where it was applied to 6,097 projects at once, using a minimum pattern support of 200 [37]. The authors reviewed the top-25% findings from a random sample of 20 projects, a total of 50 findings, and found 8% true positives. Other findings were classified as code smells (14.0%).

RGJ07 is a misuse detector for C [14]. It encodes usages as sets of properties for each variable. Properties are comparisons to literals, argument positions in function calls, and assignments. For each call, it creates a group of the property sets of the call’s arguments. To all groups for a particular function, it applies sequence mining to learn common sequences of control-flow properties and frequent-itemset mining to identify all common sets of all other property types. Subsequently, it identifies violations of the common property sequences and sets. RGJ07 is designed to detect missing conditions. From the properties it encodes, it can detect missing null checks and missing value or state conditions. Since patterns contain preceding calls on arguments, it may also detect missing calls, if the respective call shares an argument with another call in the pattern. The evaluation applied RGJ07 to a single project, thereby finding violations of project-specific patterns. The authors discussed several examples of actual bugs their approach detects, but reported no statistics on the detection performance.

Alattin is a misuse detector for Java [18], specialized in alternative patterns for condition checks. For each target method m, it queries a code-search engine to find example usages. From each example, it extracts a set of rules about pre- and post-condition checks on the receiver, the arguments, and the return value of m, e.g., “boolean check on return of Iterator.hasNext before Iterator.next.” It then applies frequent-itemset mining on these rules to obtain patterns with a minimum support of $40\%$. For each such pattern, it extracts the subset of all groups that do not adhere to the pattern and repeats mining on that subset to obtain infrequent patterns with a minimum support of $20\%$. Finally, it combines all frequent and infrequent patterns for the same method by disjunction. An analyzed method has a violation if the set of rules that hold in it is not a superset of any of the alternative patterns. Violations are ranked by the support of the respective pattern. Alattin, therefore, detects missing null-checks and missing value or state conditions that are ensured by checks and do not involve literals. It may also detect missing method-calls that occur in checks. The evaluation applied Alattin to six projects. Since it queries code-search engines for usage examples, it detects violations of cross-project patterns. The authors manually reviewed all violations of the top-10 patterns per project, a total of 532 findings, and confirmed that 29.5% identify missing condition checks (12.5%, 26.2%, 28.1%, 32.7%, 52.6%, and 100% for the individual projects). Considering frequent alternative patterns reduced false positives by 15.2% on average, which increased precision to 33.3%. Considering both frequent and infrequent alternatives even reduced false positives by 28.1% on average, leading to a precision of 37.8%, but introduced 1.5% additional false negatives, because misuses that occur multiple times are mistaken for infrequent patterns.

AX09 is a misuse detector for C [16], specialized in detecting wrong error handling, realized through returning (and checking for) error codes. It distinguishes normal paths, i.e., execution paths from the beginning of the main function to its end, and error paths, i.e., paths from the beginning of the main function to an exit or return statement in an error-handling block. AX09 uses push-down model checking to generate such paths as sequences of method calls and applies frequent-subsequence mining to find patterns with a minimum support of 80% (but at least 5 usages). It then uses push-down model checking to verify adherence to these patterns and identify respective violations. Finally, it filters false positives by tracking variable values and excluding error cases that cannot occur. It detects missing error-handling as well as missing method calls among error-handling functions. Since it identifies error-handling blocks through a predefined set of checks, it also detects missing null-checks and missing value or state conditions in the case of missing error-handling blocks. The evaluation applied AX09 to three projects individually, thereby finding violations of project-specific patterns. The authors manually reviewed all 292 findings and confirmed $90.4\%$ true positives (50.0%, 90.3%, and 93.5% on the individual projects).

CAR-Miner is a misuse detector for C++ and Java [17], also specialized in detecting wrong error handling. For each analyzed method m in a given code corpus, it queries a code-search engine to find example usages. From the examples, it builds an Exception Flow Graph (EFG), i.e., a control-flow graph with additional edges for exceptional flow. From the EFG, it generates normal call sequences that lead to the currently analyzed call and exception call sequences that lead from the call along exceptional edges. Subsequently, it mines association rules between normal sequences and exception sequences, with a minimum support of $40\%$. To detect violations, CAR-Miner extracts the normal call sequence and the exception call sequence for the target method call. It then uses the learned association rules to determine the expected exception handling and reports a violation if the actual sequence does not include it. CAR-Miner detects missing exception-handling as well as missing method calls among error-handling functions. The evaluation applied CAR-Miner to five projects. Since it queries code-search engines for usage examples, it detects violations of cross-project patterns. The authors manually reviewed all violations of the top-10 association rules for each project, a total of 264 violations, and confirmed that 60.1% identify wrong error handling (41.1%, 54.5%, 68.2%, 68.4%, and 82.3% on the individual projects). Other findings were classified as hints (3.0%).

GROUMiner is a misuse detector for Java [15]. It creates a graph-based object-usage representation (GROUM) for each target method. A GROUM is a directed acyclic graph whose nodes represent method calls, branchings, and loops and whose edges encode control and data flows. GROUMiner performs frequent-subgraph mining on sets of such graphs to detect recurring usage patterns with a minimum support of $6$. When at least 90% of all occurrences of a sub-pattern can be extended to a larger pattern, but some cannot, those rare inextensible occurrences are considered as violations. Note that such violations have always exactly one node less than a pattern. The detection of patterns and violations happens at the same time. Violations are ranked by their rareness, i.e., the support of the pattern over the support of the violation. GROUMiner detects missing method calls. It also detects missing conditions and loops at the granularity of a missing branching or loop node. However, it cannot consider the actual condition. The evaluation applied GROUMiner to nine projects individually, thereby finding violations of project-specific patterns. The authors reviewed the top-10 violations per project, a total of 184 findings, and found $5.4\%$ true positives (three times 0%, five times 6.7%, and once 7.8% on the individual projects). Other findings were classified as code smells (7.6%) or hints (6.0%).

DMMC is a misuse detector for Java [36], specialized in missing method calls. The detection is based on type usages, i.e., sets of methods called on an instance of a given type in a given method. Two usages are exactly similar if their respective sets match and are almost similar if one of them contains exactly one additional method. The detection is based on the assumption that violations should have only few exactly-similar usages, but many almost-similar ones. The likelihood of a usage $x$ being a violation is expressed in the strangeness score $=1-|E(x)|/(|E(x)|+|A(X)|)$, where $E(x)$ is the set of usages that are exactly similar to $x$ and $A(x)$ the set of those that are almost similar. A usage is considered a violation if its strangeness score is above $0.97$. Violations are ranked by the strangeness score. DMMC detects misuses with exactly one missing method-call. The evaluation applied DMMC to a single project, thereby finding project-specific violations. The authors manually reviewed all findings with a strangeness score above 97%, a total of 19 findings, and confirmed 73.7% as true positives. The evaluation was repeated later [1], applying DMMC to three projects individually, thereby finding project-specific violations for a predefined set of APIs. The authors report that they manually reviewed approximately 30 findings, and confirmed 17 ($\approx 56.7\%$) as true positives. Others were classified as workarounds for bugs inside a used API.

Tikanga is a misuse detector for Java [19] that builds on Jadet. It extends the simple call-order properties to general Computation Tree Logic formulae on object usages. Specifically, it uses formulae that require a certain call to occur, formulae that require two calls in order, and formulae that require a certain call to happen after another. It uses model checking to determine all those formulae with a minimum support of 20 in the codebase. Violations are ranked by the conviction measure [38] of the association between the set of present formulae and the set of missing formulae in the violating usage. It then applies Formal Concept Analysis [35] to obtain patterns and violations at the same time. Tikanga’s capabilities are the same as Jadet’s. The evaluation applied Tikanga to six projects individually, finding violations of project-specific patterns. The authors manually reviewed the top-25% of findings per project, a total of 121 findings, and confirmed 9.9% as true positives (0%, 0%, 8.3%, 20.0%, 21.4%, and 33.3% on the individual projects). Other findings were classified as code smells (29.8%).

DroidAssist is a detector for Android Java Bytecode [20]. It generates method-call sequences from source code and learns a Hidden Markov Model from them, to compute the likelihood of a particular call sequence. If the likelihood is too small, the sequence is considered a violation. DroidAssist then explores different modifications of the sequence (adding, replacing, and removing calls) to find a slightly modified, more likely sequence. This allows it to detect missing and redundant method calls and even to suggest solutions for them. An evaluation of this mechanism is not provided in the respective paper.

Table III summarizes the detectors’ capabilities with respect to MuC. Overall, we find that detectors cover only a small subset of all API-misuse categories. While all detectors may, to some degree, identify missing method calls, only four detectors may identify missing null checks and missing value-or-state conditions, only three may identify missing iterations, and only two may identify missing exception handling. None of the detectors targets all of these categories.

Existing detectors use both absolute and relative minimum support thresholds to identify patterns. The exceptions are, again, DroidAssist and DMMC, which use probabilistic approaches. Since many detectors produce a high number of false positives, they use a variety of ranking strategies. Most of these rely mainly on the pattern support, but some use different concepts, such as rareness, strangeness, or conviction. A comparison of different ranking strategies is not reported in any of the publications.

Table IV summarizes the empirical evaluations of the surveyed detectors, as reported in their original papers. Most evaluations apply detectors to target projects individually. In this setting, the detectors learn project-specific patterns and identify respective violations The number of projects ranges from 1 to 20 (average 5.3; median 5). The concrete projects samples are all distinct and mostly even disjunct.

To assess the detection performance, most authors review the top-X findings of their detectors, where X is a fixed number or percentage. They then either present anecdotal evidence of true positives or measure the precision of detectors. Many evaluations also present additional categories of findings, such as code smells, to distinguish false positives from other non-misuse findings that may still be valuable to developers. The definitions of when a finding belongs to which category—if provided—differ between publications, even if they use the same label, e.g., “bug” or “code smell.” No evaluation considers the recall of the respective detector.

Overall, it appears that the detectors that focus on specific violations, such as error handling or missing method calls, have higher precision. However, simply comparing detectors based on their reported empirical results would be unreliable, since the target projects, the review sample sizes, and the criteria to assess detector findings differ between the studies.

In Section 4, we conceptually compared detectors’ capabilities. In this section, we describe the experimental setup we use to empirically compare their capabilities. We design three experiments, to measure both the detectors’ precision and recall. We build these experiments on MuBench as a ground-truth dataset. This enables us to compare all detectors on the same target projects and with respect to the same known misuses.

To systematically assess and compare API-misuse detectors, we built MuBenchPipe, a benchmarking pipeline for API-misuse detectors. MuBenchPipe automates large parts of the experimental setup presented in Section 5 and facilitates the reproduction of our study. It also enables adding new detectors to the comparison, as well as benchmarking with different or extended datasets, in the future. We publish the pipeline [24] for future studies.

We now discuss the results of comparing Jadet, GROUMiner, Tikanga, and DMMC in our experiments. All reviewing data is available on our artifact page [10].

Our study focuses on static misuse detectors. Approaches based on dynamic analyses may perform differently and have unique strengths and weaknesses. To enable dynamic analyses of the project versions in MuBench, we would have to ensure that the respective code is executable (which requires a sufficient run-time environment, in addition to compile-time dependencies) and to provide example inputs for the execution. It is unclear how to do this such that it results in a fair comparison of both static and dynamic techniques, without resorting to comparing apples to oranges. In this work, we focused only on static approaches.

Our experiments focus on detectors that detect misuses in Java code. Therefore, the results may not generalize to detectors for other languages. We decided to focus on this subset of detectors, because the majority of approaches we identified in our survey targets Java. To include detectors that target other languages, we would have to either migrate them to Java or build up additional datasets for the respective languages, both of which is outside the scope of this work.

While we did ask the original authors to confirm our assessment of the conceptual capabilities of their tools, we did not ask them to confirm the empirical results of our experiments. We estimate that, including discussions to resolve disagreements, it required each reviewer on average 2 minutes to verify whether a detector identified one of the known misuses in Experiments RUB and R and 5 minutes to verify whether a detector’s finding identifies an actual misuse in Experiment P, where we needed to understand the respective code, check documentation, and sometimes also look into transitively called methods. This amounts to 24.8 hours of review effort per reviewer, 4 hours for Jadet, 7.2 hours for GROUMiner, 4.7 hours for Tikanga, and 8.9 hours for DMMC. We decided it is unreasonable to expect the original authors to invest this amount of time in verifying our assessments. We do, however, publish all our review data [10] to allow them and others to revisit our decisions.

Our dataset may not be representative of all possible real-world API misuses, especially, because we could only compile 29 (52%) of the 55 project versions and had to exclude the misuses in the other versions from our experiments. Compiling arbitrary versions of projects from the source control history of project is a challenging task. We invested two full weeks work of one of the authors and additional 3 months work of a student, to include as many project versions as possible. Still, loosing the examples for which we could not compile the respective project versions may bias the results of our experiments.

Ideally, our experiments would include thousands of misuses from a large number of projects and in each individual project version, to give us greater confidence in the generalizability of our results. However, currently, there is no such dataset. We invested several months of effort to collect and prepare MuBench in its current state, to make a first step towards a large benchmark. Now that the we have the infrastructure in place, it is straightforward to extend MuBench with misuse examples from different sources.

We publish MuBenchPipe and MuBench [24] and encourage others to extend the dataset and repeat our experiments, also with other detectors and detector configurations.

API-misuse detectors help developers write better software by warning them about potential misuses in their code. Despite the existence of many such detectors, there has been no attempt to systematically study types of API misuses and design detectors accordingly. In this paper, we addressed this gap by creating MuC, based on a dataset of 100 misuses. By evaluating the conceptual capabilities of 12 existing detectors against MuC, we identified shortcomings qualitatively. We then developed an automated benchmark pipeline, MuBenchPipe, to empirically evaluate four existing detectors. Our results reveal that misuse detectors are practically capable of detecting misuses, when explicitly provided with correct usages to mine patterns from. However, they suffer from extremely low precision and recall in a realistic application setting. We identify four root causes for false negatives, seven root causes for false positives, and two general problems with the design of detectors and the commonly-used evaluation setup. These lead us to several observations on how to advance the state-of-the-art in API-misuse detection in future work. We publish all our tooling and our dataset [24] to encourage other researchers to join us along this path.

We thank our students M. Kämmerer and J. Schlitzer for their work on MuBenchPipe and their help preparing MuBench, M. Monperrus for providing DMMC, A. Zeller and A. Wasylkowski for providing Jadet and Tikanga, and M. Pradel for additional examples for MuBench.

This work was partially funded by the German Federal Ministry of Education and Research (BMBF) within the Software Campus project Eko, grant no. 01IS12054, by the DFG as part of CRC 1119 CROSSING, and by the Hessen State Ministry for Higher Education, Research and the Arts (HMWK) within CRISP. The authors assume responsibility for the paper content.