Vet: Identifying and Avoiding UI Exploration Tarpits

We run Ape  (Gu et al., 2019), a state-of-the-art Android UI testing tool to test a popular app, Microsoft OneNote. The result is illustrated in Figure 1a. We manually set up the account to log in to the app’s main functionalities, and then start Ape. We run Ape without interruptions for one hour and check the test results afterward.

In the one-hour testing period, Ape explores only 12% (9 out of 76) activities. To understand the low testing effectiveness, we investigate the UI trace captured during testing and find the root cause to be exploration tarpits:

1. (1)

   Ape performs exploration around OneNote’s main functionalities for about two minutes, covering 7 (out of 9) of all the activities covered in the entire one-hour test run. We omit this phase in Figure 1a.

2. (2)

   About two minutes after testing starts, Ape arrives at the “Settings” screen (Screen A) and decides to click “Account” (the red-boxed UI element) for further exploration.

3. (3)

   Ape arrives at the “Account” screen (Screen B) and clicks the “Sign Out” button. The click pops up a window (Screen C) asking for confirmation of getting logged out.

4. (4)

   Ape clicks “CANCEL” first, and then goes back to the “Account” screen. However, Ape clicks the “Sign Out” button again, knowing that there is one action not triggered yet in the confirmation dialog. Subsequently, Ape clicks the “OK” button (Screen C) and logs itself out.

5. (5)

   The logout leads to the entry screen (Screen D). From this point, Ape has access to only a small number of functionalities (e.g., logging in). Ape cannot log in due to the difficulty of auto-generating the username/password of the test account. In the remaining 58 minutes, Ape explores two new activities in total.

This example represents Exploration Space Partition described in §1. The essential problem is that Ape does not understand UI semantics—it does not know that the majority of OneNote’s functionalities will be unreachable by clicking the “OK” button at the time of action.

Figure 1b presents another example in which we run Monkey (Google, 2021a), a widely adopted tool, to test another popular industry-quality app, Nike Run Club. In this example, Monkey spends about 22 minutes trying to saturate one of the app’s functionalities. After investigating into the collected UI trace, we find the following behavior when Monkey interacts with the app:

1. (1)

   Monkey explores other functionalities normally before entering Screen E that allows the tool to enter the functionality where the tool later gets trapped. We name the functionality the trapping functionality. Monkey clicks the “START” button and enters the trapping functionality (Screen F).

2. (2)

   Monkey keeps clicking around in the trapping functionality. To escape from the trapping functionality, Monkey first needs to press the Back button, and a confirmation dialog (Screen G) will pop up. Monkey then has to click the “OK” button to finish escaping. However, due to being widget-oblivion, Monkey clicks only randomly on the screen, resulting in constant failures to click “OK” when the confirmation dialog is shown. Furthermore, the dialog disappears when Monkey clicks outside of its boundary, and Monkey needs to press the Back button again to make the confirmation show up one more time. It takes 22 minutes for Monkey to find and execute an effective escaping UI event sequence and finally leave the trapping functionality.

3. (3)

   The aforementioned behaviors are repeatedly observed in the trace (with different amounts of time used for escaping).

This example represents Excessive Local Exploration behavior described in §1. The essential problem is that Monkey is both widget- and state-oblivion, i.e., the tool is unable to locate actionable UI elements efficiently or sense whether it has been trapped and react accordingly (e.g., by restarting the target app).

To prevent such undesirable exploration behaviors, a conceptually simple idea is to de-prioritize exploring the entries to aforementioned trapping states (i.e., the “OK” button in Screen C, and “START” button in Screen F). One potential solution is to develop natural language processing (NLP) or image processing based approaches that can infer the semantics of UI elements (Xi et al., 2019; Xiao et al., 2019; Liu et al., 2018; Packevičius et al., 2018). While solutions based on understanding UI semantics are revolutionary, they are challenging due to fundamental difficulties rooting in NLP and image processing.

In this paper, we explore a more practical and evolutionary solution based on understanding exploration tarpits by mining UI traces. We show that it is feasible to identify the existence and location of such behavior through pattern analysis on interaction history. Given the location of exploration tarpits, we can further identify which UI actions might have led to such behavior. Taking the example of Figure 1, Ape starts to visit a very different set of screens (e.g., the welcome screens in Screen D in Figure 1a) after clicking “OK”, and the number of explored screens dramatically decreases. Therefore, we can look at the screen history and find the time point where the symptom starts to appear. The UI action located at the aforementioned time point is then likely the cause of the symptom. Our Vet system uses a specialized algorithm (§4.2) to effectively locate the starting time point of exploration tarpits similar to the aforementioned instance.

We propose Vet, a general approach and its supporting system that automatically identifies and addresses exploration tarpits for any given Android UI testing tool on any given AUT. Our implementation of Vet is publicly available at (VET Artifacts, 2021).

As illustrated in Figure 2, for a given tool and AUT, Vet works in three stages. First, Vet runs the target tool on the AUT for a certain amount of time and records the interactions between the tool and AUT. With help from our Android framework extension Toller (Wang et al., 2021), Vet collects trace(s) that consist of AUT UIs interleaving with the tool’s actions. Then, Vet analyzes each individual trace with specialized algorithms to identify trace subsequences (termed regions) that manifest the tool’s exploration tarpits. Optionally, one can rank the identified regions based on their time lengths, where longer regions receive higher ranks, to prioritize regions that are likely to exhibit exploration tarpits with higher impacts (see §5.2). Finally, Vet learns from the identified regions and guides the tool in subsequent runs to avoid exploration tarpits, by monitoring the testing progress and taking actions based on findings from the identified regions. With the support from Toller, Vet is currently capable of (1) preventing specified actions by disabling the corresponding UI elements at runtime and (2) assisting the AUT to escape from the specified screens by restarting the AUT. The identified regions additionally support manual investigations of testing efficacy by providing localization help.

We equip Vet with two specialized algorithms targeting two patterns of exploration tarpits: Exploration Space Partition and Excessive Local Exploration. Characteristics of the two algorithms’ targeted patterns are illustrated in Figure 3 and discussed as follows:

• •

  Exploration Space Partition. As shown in Figure 3a, the UI testing tool traverses through a UI subspace (Subspace 2) for a long time after the execution of some action (the red arrow), and the tool is unable to return to the previously visited UI subspace (Subspace 1). Furthermore, the tool visits much fewer distinct screens after the action. The presence of the symptom suggests that the tool has triggered a destructive action (effectively the partition boundary of the entire trace and beginning of the exploration tarpit) that prevents the tool from further exploring the app’s major functionalities. The first motivating example from §2 corresponds to this symptom, where clicking the “OK” button is the destructive action that gets Ape trapped in multiple screens related to logging in (Subspace 2) and prevents Ape from further accessing OneNote’s main functionalities (Subspace 1).

• •

  Excessive Local Exploration. As shown in Figure 3b, the UI testing tool is trapped in a small UI subspace (Subspace 2) for an extended amount of time after the execution of the corresponding destructive action (the red arrow). However, the tool is capable of returning to the previously visited UI subspace (Subspace 1) despite the difficulties. It is also likely that the tool will get trapped again within Subspace 2 after returning to Subspace 1. Consequently, the tool spends an excessive amount of time repetitively testing limited functionalities in this hard-to-escape subspace. The second motivating example from §2 corresponds to this symptom, where clicking the “START” button gets Monkey trapped in Screens F and G (Subspace 2). Clicking “OK” helps Monkey go back to Screen E (within Subspace 1) and other functionalities, but it does not take a long time before the tool gets trapped again within Subspace 2.

As can be seen, Exploration Space Partition targets higher-level irreversible transition of UI exploration space, while Excessive Local Exploration focuses on lower-level difficulties of exercising a specific functionality. Note that it is possible for the regions reported by the two algorithms on the same trace to overlap. For example, Excessive Local Exploration might also capture exploration tarpits within Exploration Space Partition’s trapped UI subspace (corresponding to Subspace 2 in Figure 3). Such overlaps do not prevent us from finding meaningful targeted exploration tarpits: different exploration tarpits revealed by regions identified by both algorithms suggest the existence of different exploration difficulties.

In the remaining of this section, we describe the two algorithms for capturing Exploration Space Partition and Excessive Local Exploration in §4.2 and §4.3, respectively. We show that pattern capturing can be expressed as optimization problems. Table 1 describes the notations used to describe Vet’s algorithms.

According to our introductions of Exploration Space Partition, we need to find a destructive action exerted on screen $S_{n}$ as the partition boundary such that the aforementioned characteristics from §4.1 can be best reflected. For instance, considering our first motivating example in §2, we hope to pick up the screen shown in Figure 1c as $S_{n}$. We optimize the following formula to find the most desirable $S_{n}$ from a trace with $N$ screens:

In the formula, $E_{p}$ is a pre-calculated limit indicating the upper bound of $n$ during optimization, and $\sigma$ denotes the Sigmoid function. Note that $N-n$ can be pulled out of the sum subformula. The intuition of the formula design is as follows:

1. (1)

   As part of the characteristics, the tool should ideally be able to visit few to no screens that have appeared no later than $S_{n}$ after the tool passes $S_{n}$. Correspondingly, in our motivating example, screens shown before Figure 1c (depicting the app’s main functionalities) are dramatically different from the screens afterward (logging in, ToS, etc.). In the formula, the nominator of the first term (intended to be minimized) quantifies the proportion of screens seen before $S_{n}$ within $S_{n+1,N}$.

2. (2)

   As the denominator of the first term, $N-n$ essentially calculates how many (non-distinct) screens the tool visits after $S_{n}$. There are two purposes of this design. First, we hope to normalize the first term in the formula (so that two terms can weigh the same). Given that $\sum_{s\in\{S_{1,n}\}}{|S_{n+1,N}^{s}|}=\sum_{s\in\{S_{1,n}\}\cap\{S_{n+1,N}\}}{|S_{n+1,N}^{s}|}$ $\leqslant\sum_{s\in\{S_{n+1,N}\}}{|S_{n+1,N}^{s}|}=N-n$, the first term is guaranteed to fall within $[0,1]$. Second, we want to push $S_{n}$ backward (note that smaller $n$ makes the first term smaller) because we assume that the design makes $S_{n}$ closer to the exploration tarpit’s root cause, which should appear earlier than other causes.

3. (3)

   As another part of the characteristics, the tool stays within a certain UI subspace for a long time; thus, the tool will go through screens within the subspace very often. If the tool generally uniformly visits most or all distinct screens within the subspace, by observing a small period of exploration (corresponding to $S_{E_{p}+1,N}$ in the formula) we should have a fairly precise estimation (i.e., $\{S_{E_{p}+1,N}\}$) of the subspace boundary, which is characterized by $\{S_{n+1,N}\}$. The second term in the formula corresponds to this intuition, where the closer $\{S_{E_{p}+1,N}\}$ is to $\{S_{n+1,N}\}$ (note that $\{S_{E_{p}+1,N}\}\subseteq\{S_{n+1,N}\}$), the more favorable it becomes during optimization.

4. (4)

   By setting an upper bound $E_{p}$ on $n$ and regularizing the ratio with a Sigmoid function and applying appropriate linear transformations, we can guarantee that the second term in the formula always ranges from 0 to 1, being the same as the first term. In the end, two terms in the formula contribute equally to optimization choices.

To determine $E_{p}$ on each trace, because our optimization scope does not include any interval shorter than $[E_{p},N]$, we choose a value such that $t_{N}-t_{E_{p}}$ is closest to $t_{\min}$.

After obtaining a potentially suitable $S_{n}$ through optimizing the aforementioned formula, we additionally check whether $|\{S_{1,n}\}|>|\{S_{n+1,N}\}|$ is satisfied, essentially enforcing the property that the exploration space should be smaller after the partition. Finally, the reported region is $S_{n+1,N}$.

Based on the characteristics of Excessive Local Exploration from §4.1, we should track the presence of a region showing that the tool is trapped within a small UI subspace for an extended amount of time. For our second motivating example in §2, one valid choice is the 22-minute region starting from the button click in Screen E of Figure 1b. We accordingly optimize the following formula to find the boundaries $S_{l}$ and $S_{r}$ of the most suitable region on a trace:

In the formula, Merge denotes the operation of merging similar screens and returning the groups of merged screens. As the optimization formula suggests, we hope to find a suitable region such that it covers few distinct screen groups despite that the tool tries to explore diligently (by injecting numerous actions quantified by $r-l+1$). Then if $t_{r}-t_{l}\geqslant t_{\min}$, we regard that the exploration tarpit region $S_{l,r}$ can be reported. Accordingly in our motivating example, the choice of $S_{l}$ is Screen F of Figure 1b and $S_{r}$ is the last instance of Screen G of Figure 1b in the 22-minute region. $S_{l-1}$ corresponds to Screen E of Figure 1b, and the destructive action is reported.

Note that there can be more than one region exhibiting Excessive Local Exploration behavior within a single trace, given the possibility for the tool to escape the UI subspaces where Excessive Local Exploration behavior is observed. In order to find all potential regions, we iterate the aforementioned optimization process on remaining region(s) each time after one region is chosen, until no more region can be divided.

Design of Merge. As mentioned in §3, involving screen merging is especially useful for handling Excessive Local Exploration, given that the aforementioned optimization formula is very sensitive to the absolute numbers of distinct screens. Being part of the challenge, an efficient (and mostly effective) screen merging algorithm requires careful design. Given a set of distinct abstract screens (represented by UI hierarchies) to merge, a relatively straightforward (and precise) approach is to first calculate the tree editing distance (Zhang and Shasha, 1989) for each pair of abstract UI hierarchies for similarity check, and then use combinatorial optimization (Wolsey and Nemhauser, 1999) to decide the optimal grouping strategy (e.g., by converting to an Integer Linear Programming (Schrijver, 1986) problem), such that all screens within the same group are mutually similar and the total number of groups is minimal. Unfortunately, such an algorithm requires exponential time in regards to the number of distinct abstract screens to merge. The design will likely fall short on traces collected using industrial-quality apps, from which we can easily capture hundreds to thousands of distinct abstract screens.

Aiming to make the algorithm practically efficient, we relax the definition of similarity and the goal of optimization from the aforementioned merging algorithm based on insights from our observations. Specifically, we find that in many cases, similar screens can be seen as screen variants derived from base screens by inserting a small number of leaf nodes or subtrees into the abstract UI hierarchy. Based on this assumption with some tolerance for inaccuracy, we can (1) design a more efficient tree similarity checker (Algorithm 2), which considers only node insertion distances and has $\Theta(|h_{1}|\cdot|h_{2}|)$ time complexity (compared with $O(|h_{1}|\cdot|h_{2}|\cdot\textsc{Height}(h_{1})\cdot\textsc{Height}(h_{2}))$ for full tree edit distance), and (2) replace the inefficient combinatorial optimization with a highly efficient greedy algorithm (Algorithm 1), which tries to find all the base screens with $O(|H|^{2}\cdot\max_{h\in H}|h|^{2})$ time complexity. In practice, with multiple other optimizations not affecting the level of time complexity, the algorithm needs only several seconds on average to process a trace. Even for a very long trace with 2,000 distinct abstract screens and tens of thousands of concrete screens, the algorithm runs for only several minutes.

Android UI Testing Tools and Android Apps. We use three state-of-the-art/practice Android UI testing tools: Monkey (Google, 2021a), Ape (Gu et al., 2019), and WCTester (Zeng et al., 2016; Zheng et al., 2017). We use 16 popular industry Android apps from the Google Play Store, as shown in Table 2. These 16 apps are from a previous study (Wang et al., 2018), which picks the most popular apps from each of the categories on Google Play and compares multiple testing tools applied on these apps. The apps that we choose need to work properly on our testing infrastructure: (1) they need to provide x86/x64 variants of native libraries (if they have any), (2) they do not constantly crash on our emulators, and (3) Toller is able to obtain UI hierarchies on most of the functionalities. We additionally skip apps that (1) have relatively limited sets of functionalities, or (2) require logging in for access to most features and we are unable to obtain a consistently usable test account (e.g., some apps have disabled our test accounts after some experiments).

Trace Collection. We run each tool on every app for three times to alleviate the potential impacts of non-determinism in testing. Each run takes one hour without interruption, and we restart the tool if it exits before using up the allocated run time. Toller records one UI trace for each test run. While Vet runs separately on each UI trace, results are grouped for each (tool, app) pair. In total, we collect 144 one-hour UI traces from 48 (tool, app) pairs.

Testing Platform. All experiments are conducted on the official Android x64 emulators running Android 6.0 on a server with Xeon E5-2650 v4 processors. Each emulator is allocated with 4 dedicated CPU cores, 2 GiB of RAM, and 2 GiB of internal storage space. Emulators are stored on a RAM disk and backed by discrete graphics cards for minimal mutual influences caused by disk I/O bottlenecks and CPU-intensive graphical rendering. We manually write auto-login scripts for apps with “Login” ticked in Table 2, and each of these scripts is executed only once before the corresponding app starts to be tested in each run. To alleviate the flakiness of these auto-login scripts, we manually check the collected traces afterward and rerun the experiments with failed login attempts.

Overall Statistics. Vet reports 131 regions to exhibit exploration tarpits, averaging 2.7 on each (tool, app) pair. Based on Vet’s reports, the average amount of time involved in exploration tarpits is about 27 minutes per region, with the maximum being 59 minutes and minimum being slightly more than 10 minutes (given that we empirically set $t_{\min}=10$ minutes for all the experiments).

This section shows that the identified exploration tarpit regions by Vet can be used to automatically address tool issues.

We show our analysis of tool issues that are not revealed by any exploration tarpit regions (i.e., false negatives) reported by Vet.