GenTree: Using Decision Trees to Learn Interactions for Configurable Software

GenTree first creates a random 1-way covering array [13, 14] to obtain a set of initial configurations, which contains all possible settings of each individual option. Figure 5 shows the initial configurations and their coverage information for the running example.

For each covered location $l$, GenTree uses a classification algorithm called C5${}_{\textbf{i}}$, developed specifically for this work, (§IV-B) to build a decision tree representing the interaction for $l$. To build the tree for $l$, C5${}_{\textbf{i}}$ uses two sets of data: the hit sets consisting of configurations covering $l$ and the miss set consisting of configurations not covering $l$. For example, for $L8$, GenTree builds the decision tree in Figure 5 from the hit sets $\{c_{2}\}$ and the miss set $\{c_{1},c_{3}\}$.

From the given configurations C5${}_{\textbf{i}}$ determines that the coverage of $L8$ just requires option $s$ being 0 (false). Thus, the interaction for $L8$, represented by the condition of the hit path (a) of the tree in Figure 5, is $\bar{s}$. This interaction is quite different than $\bar{s}\wedge e\equiv 2\wedge((u\wedge\bar{v})\vee(\bar{u}\wedge v))$, the desired interaction for $L8$. However, even with only three initial configurations, the tree is partially correct because configurations having $s$ as true would miss $L8$ and $s$ being false is part of the requirements for hitting $L8$.

GenTree now attempts to create new configurations to refine the tree representing the interaction for location $l$. Observe that if a hit path is precise, then any configuration satisfying its condition would cover $l$ (similarly, any configuration satisfying the condition of a miss path would not cover $l$). Thus, we can validate a path by generating configurations satisfying its condition and checking their coverage. Configurations generated from a hit (or miss) path that do not (or do) cover $l$ are counterexample configurations, which show the imprecision of the path condition and help build a more precise tree in the next iteration.

In the running example, GenTree selects the condition $\bar{s}$ of the hit path (a) of the tree shown in Figure 5 and generates four new configurations shown in Figure 5 with $s=0$ and 1-covering values for the other eight variables. If path (a) is precise, then these configurations would cover $L8$. However, only configuration $c_{7}$ covers $L8$. Thus, $c_{4},c_{5},c_{6}$, which do not cover $L8$, are counterexamples showing that path (a) is imprecise and thus $\bar{s}$ is not the correct interaction for $L8$.

Note that we could also generate new configurations using path (b), which represents the interaction for not covering $L8$. However, GenTree prefers path (a) because the classifier uses one configuration for path (a) and two for path (b), i.e., the condition $\bar{s}$ for covering $l$ is only supported by one configuration and thus is likely more imprecise.

GenTree now repeats the process of building trees and generating new configurations. Continuing with our example on finding the interaction for $L8$, GenTree adds $c_{7}$ to the hit set and $c_{4},c_{5},c_{6}$ to the miss set and builds the new tree for $L8$ in Figure 5. The combination of the hit paths (d) and (e) gives $e\equiv 2\wedge(u\vee(\bar{u}\wedge v))$ as the interaction for $L8$. This interaction contains options $e,u,v$, which appear in the desired interaction $\bar{s}\wedge e\equiv 2\wedge((u\wedge\bar{v})\vee(\bar{u}\wedge v))$.

To validate the new interaction for $L8$, GenTree generates new configurations from paths (c), (d), (e) of the tree in Figure 5, because they have the fewest number of supporting configurations. Figure 5 shows the nine new configurations.

Note that (c) is a miss path and thus $c_{8},c_{9},c_{10}$ are not counterexamples because they do not hit $L8$. Also, in an actual run, GenTree would select only one of these three paths and take two additional iterations to obtain these configurations. For illustration purposes, we combine these iterations and show the generated configurations all together.

In the next iteration, using the new configurations and the previous ones, GenTree builds the decision tree in Figure 5 for $L8$. The interaction obtained from the two hit paths (d) and (e) is $\bar{s}\wedge e\equiv 2\wedge((\bar{v}\wedge u)\vee(v\wedge\bar{u}))$, which is equivalent to the desired one and thus would remain unchanged regardless of any additional configurations GenTree might create.

Finally, GenTree stops when it cannot generate new coverage or refine existing trees for several consecutive iterations. In a postprocessing step, GenTree combines the hit path conditions of the decision tree for each location $l$ into a logical formula representing the interaction for $l$.

GenTree found the correct interactions for all locations in the running example within eight iterations and under a second. The table below shows the number of iterations and configurations used to find the interaction for each location. For example, the desired interaction for $L8$ took 58 configurations and is discovered at iteration 4, and the interaction true of L0 was quickly discovered from the initial configurations.

Overall, GenTree found all of these interactions by analyzing approximately 360 configurations (median over 11 runs) out of 3888 possible ones. The experiments in §VI show that GenTree analyzes an even smaller fraction of the possible configurations on programs with larger configuration spaces.

An interaction for a location $l$ characterizes of the set of configurations covering $l$. For example, we see from Figure 2 that any configuration satisfying $u\wedge v$ (i.e., they have the settings $u=1$ and $v=1$) is guaranteed to cover L3. Although we focus on location coverage, interaction can be associated with more general program behaviors, e.g., we could use an interaction to characterize configurations triggering some undesirable behavior. To obtain coverage, we typically run the program using a configuration and a test suite, which is a set of fixed environment data or options to run the program on, e.g., the test suite for the Unix ls (listing) command might consist of directories to run ls on. In summary, we define program interactions as:

We use a decision tree to represent the interaction for a location $l$. A decision tree consists of a root, leaves, and internal (non-leaf) nodes. Each non-leaf node is labeled with a configuration option and has $k$ outgoing edges, which correspond to the $k$ possible values of the option. Each leaf is labeled with a hit or miss class, which represents the classification of that leaf. The path from the root to a leaf represents a condition leading to the classification of the leaf. This path condition is the conjunction of the settings collected along that path. The union (disjunction) of the hit conditions is the interaction for $l$. Dually, the disjunction of the miss conditions is the condition for not covering $l$. The length of a path is the number of edges in the path.

For illustration purposes, we annotate each leaf with a label $t\;(a)\;k$, where $t$ is either the (h) hit or (m) miss class, $a$ is the path name (so that we can refer to the path), and $k$ is the number of supporting configurations used to classify this path. Intuitively, the more supporting configurations a path has, the higher confidence we have about its classification.

For example, the decision tree in Figure 5 for location $L8$ consists of four internal nodes and seven leaves. The tree has five miss and two hit paths, e.g., path (d), which has length 4 and condition $\bar{s}\wedge e\equiv 2\wedge\bar{v}\wedge u$, is classified as a hit due to one configuration hitting $L8$ ($c_{2}$ in Figure 5), and (g) is a miss path with condition $s$ because seven configurations satisfying this condition miss $L8$. The interaction for $L8$ is $\bar{s}\wedge e\equiv 2\wedge((\bar{v}\wedge u)\vee(v\wedge\bar{u}))$, the disjunction of the two hit conditions.

As mentioned, GenTree is mostly related to iGen, which computes three forms of interactions: purely conjunctive, purely disjunctive, and specific mixtures of the two. In contrast, we use decision trees to represent arbitrary boolean interactions and develop our own classification algorithm C5${}_{\textbf{i}}$ to manipulate decision trees. To illustrate the differences, consider the interaction for location id.c:325, $\overline{\texttt{help}}\land\overline{\texttt{version}}\land\overline{\texttt{Z}}\land\overline{\texttt{g}}\land\overline{\texttt{G}}\land\overline{\texttt{n}}\land(\texttt{u}\lor(\overline{\texttt{r}}\land\overline{\texttt{z}}))$, which can be written as the disjunction of two purely conjunctive interactions: $(\overline{\texttt{help}}\land\overline{\texttt{version}}\land\overline{\texttt{Z}}\land\overline{\texttt{g}}\land\overline{\texttt{G}}\land\overline{\texttt{n}}\land\texttt{u})\vee(\overline{\texttt{help}}\land\overline{\texttt{version}}\land\overline{\texttt{Z}}\land\overline{\texttt{g}}\land\overline{\texttt{G}}\land\overline{\texttt{n}}\land\overline{\texttt{r}}\land\overline{\texttt{z}})$. iGen can infer each of these two purely conjunctions, but it cannot discover their disjunction because iGen does not support this form, e.g., $(a\land b)\lor(a\land c)$. For this example, even when running on all 1024 configurations, iGen only generates $\overline{\texttt{help}}\land\overline{\texttt{version}}\land\overline{\texttt{Z}}\land\overline{\texttt{g}}\land\overline{\texttt{G}}\land\overline{\texttt{n}}\land\texttt{u}$, which misses the relation with r and z. In contrast, GenTree generates this exact disjunctive interaction (and many others) using 609 configurations in under a second (Table II in §VI-B).

Moreover, while both tools rely on the iterative guess-and-check approach, the learning and checking components and their integration in GenTree are completely different from those in iGen, e.g., using heuristics to select likely fragile tree paths to generate counterexamples. Also, while C5${}_{\textbf{i}}$ is a restricted case of C5.0, it is nonetheless a useful case that allows us to generate a tree that is exactly accurate over data instead of a tree that approximates the data. We developed C5${}_{\textbf{i}}$ because existing classification algorithms do not allow easy interaction inference (due to agressive pruning and simplification as explained in §IV-B2).

Researchers have used decision trees and general boolean formulae to represent program preconditions (interactions can be viewed as preconditions over configurable options). The work in [26] uses random SAT solving to generate data and decision trees to learn preconditions, but does not generate counterexample data to refine inferred preconditions, which we find crucial to improve resulting interactions. Similarly, PIE [27] uses PAC (probably approximately correct algorithm) to learn CNF formula over features to represent preconditions, but also does not generate counterexamples to validate or improve inferred results. Only when given the source code and postconditions to infer loop invariants PIE would be able to learn additional data using SMT solving.

GenTree adopts the iterative refinement approach used in several invariant analyses (e.g., [8, 9, 10, 11]). These works (in particular [10, 9] that use decision trees) rely on static analysis and constraint solving to check (and generate counterexamples) that the inferred invariants are correct with respect to the program with a given property/assertion (i.e., the purpose of these works is to prove correct programs correct). In contrast, GenTree is pure dynamic analysis, in both learning and checking, and aims to discover interactions instead of proving certain goals.

GenTree can be considered as a dynamic invariant tool that analyzes coverage trace information. Daikon [28, 29] infers invariants from templates that fit program execution traces. GenTree focuses on inferring interactions represented by arbitrary formulae and combines with iterative refinement. DySy is another invariant generator that uses symbolic execution for invariant inference [30]. The interaction work in [1] also uses the symbolic executor Otter [31] to fully explore the configuration space of a software system, but is limited to purely conjunctive formulae for efficiency. Symbolic execution techniques often have similar limitations as static analysis, e.g., they require mocks or models to represent unknown libraries or frameworks and are language-specific (e.g., Otter only works on C programs). Finally, GenTree aims to discover new locations and learns interactions for all discovered locations. In contrast, invariant generation tools typically consider a few specific locations (e.g., loop entrances and exit points).

The popular BDD data structure [32] can be used to represent boolean formulae, and thus is an alternative to decision trees. Two main advantages of BDDs are that a BDD can compactly represent a large decision tree and equivalent formulae are represented by the same BDD, which is desirable for equivalence checking.

However, our priority is not to compactly represent interactions or check their equivalences, but instead to be able to infer interactions from a small set of data. While C5${}_{\textbf{i}}$ avoids aggressive prunings to improve accuracy, it is inherently a classification algorithm that computes results by generalizing training data (like the original C5.0 algorithm, GenTree performs generalization by using heuristics to decide when to stop splitting nodes to build the tree as described in §IV-B2). To create a BDD representing a desired interaction, we would need many configurations, e.g., $2^{n}+1$ miss or $2^{n}-1$ hit configurations to create a BDD for $a\land(b_{1}\lor b_{2}\lor\dots\lor b_{n})$. In contrast, C5${}_{\textbf{i}}$ identifies and generalizes patterns from training data and thus require much fewer configurations. For instance, the configuration space size of the example in Figure 5 is 3888, and from just 3 configurations $c_{1},c_{2},c_{3}$, C5${}_{\textbf{i}}$ learns the interaction $\bar{s}$ because it sees that whenever $s\equiv 1$, $L8$ is miss, and whenever $s\equiv 0$, $L8$ is hit. BDD would need 1944 configurations to infer the same interaction.

Combinatorial interaction testing (CIT) [14, 13] is often used to find variability bugs in configurable systems. One popular CIT approach is using $t$-way covering arrays to generate a set of configurations containing all $t$-way combinations of option settings at least once. CIT is effective, but is expensive and requires the developers to choose $t$ a priori. Thus developers will often set $t$ to small, causing higher strength interactions to be ignored. GenTree initializes its set of configurations using 1-way covering arrays.

Variability-Aware is another popular type of analysis to find variability bugs [33, 34, 35, 36, 37, 38, 39, 40, 41, 42]. [36] classify problems in software product line research and surveys static analysis to solve them. GenTree’s interactions belong to the feature-based classification, and we propose a new dynamic analysis to analyze them. [40] study feature interactions in a system and their effects, including bug triggering, power consumption, etc. GenTree complements these results by analyzing interactions that affect code coverage.