Worst-Case Analysis is Maximum-A-Posteriori Estimation

In this paper, we analyze the worst-case resource usage of a given application by automatically generating inputs—of some given size—that trigger as large resource usage as possible. We assume that the user of our tool provides an interface to collect resource-usage information from multiple executions of the application, with possibly different inputs. In this way, the user can specify a custom resource metric of interest, e.g., the number of executed program statements, jumps (branches), or method calls. Apart from the input specification and the resource-usage information, we require that our tool be black-box, i.e., it does not have any application-specific knowledge for worst-case analysis. Note that in this paper, our goal is not to estimate asymptotic complexities; instead, we focus on maximizing a user-specified concrete metric over inputs of a given size.

To demonstrate our framework, we consider insertion sort as a running example. For an ordinary implementation of insertion sort, it is well known that its worst-case time complexity is $O(n^{2})$ and its best-case complexity is $O(n)$, where $n$ is the length of the array-to-be-sorted. Fig. 1(a) presents an implementation of insertion sort in Java. To make the resource metric explicit, we interface resource accumulation with method calls of the form Resource.tick(int). For the program in Fig. 1(a), the resource metric accounts for the total number of loop iterations. For example, if we want to analyze the worst-case resource usage when the length of the input array is $5$, a worst-case input could be $[5,4,3,2,1]$ with a resource usage of $14$ loop iterations.

To show how the worst-case analysis (WCA) problem is isomorphic to the maximum-a-posteriori (MAP) estimation, we first review Bayesian inference and probabilistic programming.

• •

  Bayesian inference is a method for inferring the posterior distribution of a probabilistic model conditioned on observed data, with applications in artificial intelligence (Ghahramani, 2015), cognitive science (Griffiths et al., 2008), applied statistics (Gelman et al., 2013), etc. At the core of Bayesian inference is Bayes’ law:

  (1)

  $$P(X=x\mid Z=z)=\frac{P(Z=z\mid X=x)\cdot P(X=x)}{P(Z=z)},$$

  where $X$ and $Z$ are random variables standing for parameters and observations, respectively; $P(X=x\mid Z=z)$ is the conditional probability of parameters being $x$ given that observations are $z$, i.e., the posterior; $P(Z=z\mid X=x)$ is the conditional probability of observations being $z$ given that parameters are $x$, i.e., the likelihood; and $P(X=x)$ is the probability of parameters being $x$, i.e., the prior. Because Bayesian inference usually fixes the observations $z$, the denominator $P(Z=z)$ of the right-hand side of Eqn. 1 is usually considered as a constant, and thus we can write Eqn. 1 as

  (2)

  $$P(X=x\mid Z=z)\propto P(Z=z\mid X=x)\cdot P(X=x).$$

  Bayesian inference usually accounts for the posterior distribution $P(X=x\mid Z=z)$ as a function of $x$ given some $z$. People have developed many algorithms to sample from the posterior distribution, e.g., Markov-Chain Monte Carlo (MCMC). In some other scenarios, people have also considered the maximum-a-posteriori (MAP) estimation, i.e., finding an $x^{*}$ that maximizes the posterior probability $P(X=x^{*}\mid Z=z)$.

• •

  Probabilistic programming provides a flexible way of specifying probabilistic models and performing Bayesian inference (Barthe et al., 2020). One way to understand the semantics of a probabilistic program is that it describes the measure that is induced by the product of the likelihood and the prior, i.e., the right-hand side of Eqn. 2. Note that because we ignore the denominator $P(Z=z)$, the product is usually not a probability measure. Consequently, probabilistic programming languages (PPLs)—such as Stan (Carpenter et al., 2017), Pyro (Bingham et al., 2018), Church (Goodman et al., 2008; Goodman and Stuhlmüller, 2014), and Gen (Cusumano-Towner et al., 2019)—have devised many techniques to sample from or approximate unnormalized measures defined by probabilistic programs. A PPL is usually an ordinary programming language with two special extra constructs:

  • –

    sampling, which draws a random value from a prior distribution; and

  • –

    soft conditioning, which records the likelihood of some observation.

  Below gives a simple probabilistic program written as Java pseudocode. The program models a Bayesian-inference task: (i) there is an imprecise scale that responds with a noisy measurement from the Normal distribution $\mathcal{N}(w,0.75^{2})$ when the actual weight is $w$; (ii) we use the scale to weigh an object and read $9.5$ as the measurement; (iii) we have a prior guess that the object’s weight is around $8.5$, modeled with the prior distribution $\mathcal{N}(8.5,1^{2})$; and (iv) we want to know the posterior distribution of the object’s weight.

  ⬇

  1float $w$ = Probability.sample(Distribution.Normal($8.5$, $1$));

  2Probability.observe($9.5$, Distribution.Normal($w$, $0.75$));

  The statement $\mathsf{observe}(\mathit{valu},\mathit{dist})$ is usally a wrapper around the more primitive statement $\mathsf{score}(\ell)$, where $\ell$ is the likelihood of $\mathit{valu}$ being sample from $\mathit{dist}$; in this example, we can compute the likelihood using the probability density function of Normal distributions:

  ⬇

  1float $w$ = Probability.sample(Distribution.Normal($8.5$, $1$));

  2Probability.score( $\frac{1}{\sqrt{2\pi\cdot 0.75^{2}}}e^{-\frac{(9.5-w)^{2}}{2\cdot 0.75^{2}}}$ );

  A probabilistic program can contain multiple soft-conditioning/scoring statements; intuitively, the likelihood of an execution path is the product of scores along the path.

Readers might already notice the resemblance between resource accumulation (via Resource.tick) and likelihood accumulation (via Probability.score). The major difference here is that the resource usage of an execution path is the sum of ticks along the path, whereas the likelihood of an execution path is the product of scores along the path. Observing that likelihoods from probabilistic programming are always positive,¹¹1Some PPLs use zero likelihoods to enforce hard constraints. In this paper, we only consider soft constraints and thus we can assume that all likelihoods are non-zero. we derive a correspondence between WCA and MAP as follows.

• •

  From WCA to MAP. Given a program $M$ and a resource metric $\mathit{tick}:\Sigma\to\mathbb{R}$ that assigns resource usages to program instructions in $\Sigma$, the WCA problem aims to find an input $\theta$ for $M$ such that $\mathit{tick}_{M}(\theta)\mathrel{\overset{\underset{\textnormal{def}}{}}{=}}\sum_{\sigma\in\pi(M,\theta)}\mathit{tick}(\sigma)$ is maximized, where $\pi(M,\theta)$ gives the execution path of $M$ with $\theta$ as its input. Define a likelihood assignment $\mathit{score}:\Sigma\to\mathbb{R}_{>0}$ as $\mathit{score}(\sigma)\mathrel{\overset{\underset{\textnormal{def}}{}}{=}}e^{\mathit{tick}(\sigma)}$ for any $\sigma\in\Sigma$. The map $\mathit{score}_{M}\mathrel{\overset{\underset{\textnormal{def}}{}}{=}}\theta\mapsto\prod_{\sigma\in\pi(M,\theta)}\mathit{score}(\sigma)$ then defines a probabilistic semantics with no prior, where the MAP problem aims to find an input $\theta$ for $M$ such that $\mathit{score}_{M}(\theta)$ is maximized. Because $\mathit{score}_{M}(\theta)=\prod_{\sigma\in\pi(M,\theta)}\mathit{score}(\sigma)=\prod_{\sigma\in\pi(M,\theta)}e^{\mathit{tick}(\sigma)}=e^{\sum_{\sigma\in\pi(M,\theta)}\mathit{tick}(\sigma)}=e^{\mathit{tick}_{M}(\theta)}$ for any $\theta$, a solution to the MAP problem is also a solution to the WCA problem.

• •

  From MAP to WCA. Given a program $M$ and a likelihood assignment $\mathit{score}:\Sigma\to\mathbb{R}_{>0}$ where $\Sigma$ is the set of program instructions. Conceptually, we can model probabilistic sampling with a pre-specified trace, in the sense that each sampling statement reads a value from the trace and records the prior probability accordingly (Borgström et al., 2016; Kozen, 1981). Thus, we can treat the trace as an input and the prior as a map $\mathit{prior}$ such that $\mathit{prior}(\tau)$ gives the prior probability of the trace $\tau$. The MAP problem then aims to find a trace $\tau$ such that $\mathit{score}_{M}(\tau)\mathrel{\overset{\underset{\textnormal{def}}{}}{=}}\mathit{prior}(\tau)\cdot\prod_{\sigma\in\pi(M,\tau)}\mathit{score}(\sigma)$ is maximized, where—similarly to the former case—$\pi(M,\tau)$ gives the execution path of $M$ with $\tau$ as its trace. As discussed above, the probabilities are positive, so we can define a resource metric $\mathit{tick}:\Sigma\to\mathbb{R}$ as $\mathit{tick}(\sigma)\mathrel{\overset{\underset{\textnormal{def}}{}}{=}}\log\mathit{score}(\sigma)$ for any $\sigma\in\Sigma$. The map $\mathit{tick}_{M}\mathrel{\overset{\underset{\textnormal{def}}{}}{=}}\tau\mapsto\log\mathit{prior}(\tau)+\sum_{\sigma\in\pi(M,\tau)}\mathit{tick}(\sigma)$ then defines the resource accumulation of $M$ under the metric $\mathit{tick}$, with the understanding that a resource usage of $\log\mathit{prior}(\tau)$ is triggered at the beginning of a program execution. In this case, the WCA problem aims to find a trace $\tau$ for $M$ such that $\mathit{tick}_{M}(\tau)$ is maximized. Because $\mathit{tick}_{M}(\tau)=\log\mathit{prior}(\tau)+\sum_{\sigma\in\pi(M,\tau)}\mathit{tick}(\sigma)=\log\mathit{prior}(\tau)+\sum_{\sigma\in\pi(M,\tau)}\log\mathit{score}(\sigma)=\log\left(\mathit{prior}(\tau)\cdot\prod_{\sigma\in\pi(M,\tau)}\mathit{score}(\sigma)\right)=\log\mathit{score}_{M}(\tau)$ for any $\tau$, a solution to the WCA problem is also a solution to the MAP problem.

Fig. 1 shows a direct demonstration of the correspondence. In Fig. 1(a), we use Resource.tick(1) to accumulate resource usages, and in Fig. 1(b), we use Probability.score(Math.exp(1)) to accumulate likelihoods. An MAP estimation for the probabilistic program in Fig. 1(b) when the length of the input array is $5$ could be $[5,4,3,2,1]$, which is indeed a worst-case input for the program in Fig. 1(a)—as we discussed at the beginning of section 2.2.

Because worst-case analysis (WCA) is maximum-a-posteriori (MAP) estimation, our goal is then to adapt MAP algorithms from Bayesian inference to carrying out WCA. In this paper, we incorporate sequential Monte Carlo (SMC), evolutionary algorithms, and fuzzing techniques to develop our DSE-SMC framework. We start with a review of SMC and gradually extend it to present DSE-SMC.

Sequential Monte Carlo (SMC) methods form a powerful family of Bayesian-inference algorithms for sampling from a sequence of target distributions (Del Moral et al., 2006). Particle filters are a prominent SMC method for online inference in state-space models, widely applied in tasks such as robot localization (Thrun et al., 2005). An SMC method usually maintains a set of weighted samples as an empirical approximation of the target distribution. During each iteration of the sequential model, SMC uses a proposal distribution to generate new samples from previous ones and reweights the new samples according to their likelihoods on the observed data. When some samples have relatively negligible weights, SMC uses a resampling process in which more promising samples are selected as the basis for future inference. Well-designed proposal distributions can bring significant performance improvements (Gu et al., 2015; Wigren et al., 2018).

Besides sequential modeling, when there is just a single target distribution (e.g., the posterior distribution in the standard setting of Bayesian inference), SMC has been shown to be a promising approach for sampling from distributions with multiple modes (Del Moral et al., 2007; Chopin, 2002; Saad et al., 2023; Zhu et al., 2018). This property renders SMC desirable in our setting of worst-case analysis, because worst-case inputs can usually be fairly separated among the input space and long-tailed in the resource-usage distribution. For example, any decreasing sequence of integers is a worst-case input for the insertion-sort implementation in Fig. 1(a).

In this paper, we adapt a variant of SMC algorithms that is usually called resample-move SMC (Gilks and Berzuini, 2001). Consider a probabilistic semantics $\mathit{score}_{M}:\mathbb{S}\to\mathbb{R}_{>0}$ for some probabilistic program $M$, where $\mathbb{S}$ is its input space. The SMC algorithm is overall iterative: at the $t$th epoch, it maintains a set $\{(\theta_{t}^{\ell},w_{t}^{\ell})\}_{\ell=1}^{L}$ of $L\geq 1$ weighted samples, where the weight $w_{t}^{\ell}>0$ reflects the likelihood of the sample $\theta^{\ell}_{t}\in\mathbb{S}$, i.e., $\mathit{score}_{M}(\theta^{\ell}_{t})$, relative to other samples. Below outlines how the SMC algorithm proceeds. The step (S0) initializes a set of $L$ samples randomly. The $t$th epoch of the algorithm then involves three steps: (S1) reweights each sample by its likelihood with respect to the probabilistic semantics, (S2) resamples the samples based on their weights, and (S3) rejuvenates the samples by running $n\geq 1$ iterations of GenerateNewSample that implements a MCMC transition kernel.

The reweighting step (S1), when considered in a WCA setting, computes $e^{\mathit{tick}_{M}(\theta)}$ as the weight for $\theta$, i.e., this step intuitively prioritizes inputs with large resource usages. In the resampling step (S2), people have been using an adaptive resampling strategy, where resampling is triggered at epoch $t$ if the effective sample size—a commonly used metric for sample diversity—drops under a threshold: $\mathrm{ESS}(w_{t}^{1:L})\mathrel{\overset{\underset{\textnormal{def}}{}}{=}}1/\sum_{\ell=1}^{L}\left(\frac{w_{t}^{\ell}}{\sum_{k=1}^{L}w_{t}^{k}}\right)^{2}$. The rejuvenating step (S3) makes use of arbitrary MCMC kernels to update the samples and thus renders the SMC algorithm generic because the implementation of GenerateNewSample can be application-specific. The soundness of the SMC algorithm arises from its convergence properties, i.e., when the number of samples $L$ approaches infinity, the empirical approximation $\{(\theta_{t}^{\ell},w_{t}^{\ell})\}_{\ell=1}^{L}$ asymptotically converges to the target distribution given by $\mathit{score}_{M}$. For example, if the normalizing constant for the measure given by $\mathit{score}_{M}$ is $Z_{M}$, the SMC algorithm guarantees that $\frac{1}{L}\sum_{\ell=1}^{L}\frac{w_{t}^{\ell}}{\sum_{k=1}^{L}w_{t}^{k}}\phi(\theta_{t}^{\ell})\xlongrightarrow{L\to\infty}\sum_{\theta\in\mathbb{S}}\frac{\mathit{score}_{M}(\theta)}{Z_{M}}\phi(\theta)$ almost surely, for any $\phi:\mathbb{S}\to\mathbb{R}$ that satisfies a few conditions (Gilks and Berzuini, 2001; Del Moral et al., 2006).

So far we have reviewed SMC for approximating posterior distributions, but our goal is to use SMC for MAP estimation, which is an optimization problem. We now consider integrating SMC with Evolutionary Algorithms (EA), a popular family of optimization algorithms that take inspiration from the biological evolution process. A powerful class of EA is genetic algorithms (GA) (Goldberg, 1989), which resemble the natural selection process, and proceed by maintaining a population of candidate solutions and evolving them via genetic operations, such as (i) selection, which selects candidates from the population based on their fitness, i.e., how well they optimize the objective, (ii) crossover, which simulates reproduction by taking two selected candidates and mixing them to create offspring, and (iii) mutation, which allows random changes in the candidate solutions to maintain the diversity of the population.

It is worth noting that SMC algorithms and genetic algorithms have many similarities (Del Moral et al., 2001). Recall the resample-move SMC algorithm we just reviewed: it maintains a set of weighted samples, as GA maintains a population of candidates; its reweighting step computes the likelihoods for samples, as GA computes the fitness scores for candidates; its resampling step randomly picks samples with higher likelihoods, as GA’s selection prioritizes candidates with higher fitness scores; and its rejuvenating step randomly generate new samples from existing ones, as GA’s crossover and mutation evolve the population. There have been studies on the integration of SMC and GA (or EA) (Kwok et al., 2005; Zhu et al., 2018; Dufays, 2016), which show that such integration is a promising approach for both posterior approximation and MAP estimation.

In this paper, we develop DSE-SMC upon the generic resample-move SMC algorithm and integrate techniques from EA in the resampling and rejuvenating steps. DSE-SMC incorporates two ideas:

• •

  Dual Strategy. Optimization algorithms usually need to take care of the balance between exploration and exploitation of the solution space. DSE-SMC uses a dual-strategy approach inspired from $K$-strategy and $R$-strategy (Andrews and Harris, 1986), which are usually referred to two poles that describe the growth and reproduction strategies of organisms. Intuitively, $K$-strategy and $R$-strategy correspond to crowded and uncrowded environments, respectively. In terms of optimization, $K$-strategy tries to produce a few offspring with high fitness scores, whereas $R$-strategy tries to produce a lot of offspring to increase diversity. DSE-SMC splits the weighted samples into multiple groups, each of which uses $K$-strategy or $R$-strategy for rejuvenation. In each epoch, DSE-SMC allows migration among the groups.

• •

  Evolutionary MCMC. DSE-SMC adapts evolutionary MCMC (Drugan and Thierens, 2003) as an adaptive and generic approach for implementing rejuvenation, i.e., the GenerateNewSample routine. MCMC is a general framework to sample from unnormalized measures such as the one induced by $\mathit{score}_{M}$. The overall idea is to construct a Markov chain with the target distribution as the chain’s stationary distribution; thus, simulating the chain generates correctly-distributed samples. We consider the Metropolis-Hastings (MH) algorithms for MCMC: let $\mathit{proposal}(\theta_{\mathsf{new}};\theta_{\mathsf{old}})$ be a proposal distribution that generates a candidate new sample from an old one, then the new sample is accepted with probability $\min(1,\frac{\mathit{score}_{M}(\theta_{\mathsf{new}})\cdot\mathit{proposal}(\theta_{\mathsf{old}};\theta_{\mathsf{new}})}{\mathit{score}_{M}(\theta_{\mathsf{old}})\cdot\mathit{proposal}(\theta_{\mathsf{new}};\theta_{\mathsf{old}})})$. It has been shown that both mutation and crossover can be recasted in terms of MH (Drugan and Thierens, 2003; Liang and Wong, 2000; Jasra et al., 2007). For mutation, the proposal distribution simply implements how the random changes are applied to existing candidates. For crossover, one can have a crossover proposal distribution $\mathit{proposal}(\langle\theta_{\mathsf{new},1},\theta_{\mathsf{new},2}\rangle;\langle\theta_{\mathsf{old},1},\theta_{\mathsf{old},2}\rangle)$ and then the acceptance ratio is computed as $\min(1,\frac{\mathit{score}_{M}(\theta_{\mathsf{new}_{1}})\cdot\mathit{score}_{M}(\theta_{\mathsf{new},2})\cdot\mathit{proposal}(\langle\theta_{\mathsf{old},1},\theta_{\mathsf{old},2}\rangle;\langle\theta_{\mathsf{new},1},\theta_{\mathsf{new},2}\rangle)}{\mathit{score}_{M}(\theta_{\mathsf{old}_{1}})\cdot\mathit{score}_{M}(\theta_{\mathsf{old},2})\cdot\mathit{proposal}(\langle\theta_{\mathsf{new},1},\theta_{\mathsf{new},2}\rangle;\langle\theta_{\mathsf{old},1},\theta_{\mathsf{old},2}\rangle)})$. It is also possible for the population size to change along epochs (Drugan and Thierens, 2003); thus, DSE-SMC allows the set of weighted samples to have a dynamic size.

We have shown the key components of DSE-SMC, a generic, adaptive, and sound framework for MAP estimation. The final step of our development is then instantiating the framework to carry out worst-case analysis (WCA) of resource usage. As discussed above, the implementation of the rejuvenation step can be application-specific; thus, in the WCA setting, we apply fuzzing techniques (Zhu et al., 2022) to implement the GenerateNewSample routine. In particular, we adapt crossover and mutation operations from evolutionary fuzzers such as AFL (Zalewski, 2023) and LibFuzzer (LLVM Project, 2023) to manipulate program inputs of different types, e.g., strings, integers, and arrays. In our implementation, we recast those genetic operations as MCMC kernels.

To present a versatile evaluation of how DSE-SMC may have an impact on both the basic algorithms and complex industrial applications, we select eight evaluation subjects, summarized in Tab. 1. Some of the subjects are adapted from recent work on WCA, e.g., SlowFuzz (Petsios et al., 2017) and Mayhem (Cha et al., 2012). The subjects on the left column of Tab. 1 are basic algorithms that may serve to create a general sense of how DSE-SMC works in terms of input mutation and worst-case navigating. On the other hand, the subjects on the right column of Tab. 1 are believed to be of great importance to the software industry. Note that we use different resource metrics for different subjects. The subjects on the left column are to be evaluated by observing the steps that the algorithm takes to halt, whereas those on the right column via counting the jumps that the application takes to complete the task.

For all evaluation subjects, we ran four WCA tools: (1) Kelinci, (2) KelinciWCA, (3) Locally-optimal DSE-SMC, and (4) DSE-SMC. Kelinci provides an interface for running AFL (Zalewski, 2023) on Java programs (Kersten et al., 2017) and KelinciWCA extends Kelinci to prioritize execution paths with high resource consumption (Noller et al., 2018). We include in the comparison a locally-optimal variant of DSE-SMC, which replaces the MCMC-based rejuvenation step with a local search that only keeps a locally optimal new candidate. It is of vital importance to control the basic setup and the initial value as they can greatly affect the performance of each tool. Therefore, we use the same meaningless inputs and default parameters to setup our evaluation. We ran each tool on each subject for at most 100 epochs or 100 minutes, which we will elaborate in the setup of each evaluation subject.

Note that there are also recently proposed WCA tools, such as SlowFuzz (Petsios et al., 2017), that execute evolutionary WCA on binary codes. Because we setup our experiments on Java programs, we cannot compare DSE-SMC with them directly. Nevertheless, we find that Badger (Noller et al., 2018), a WCA tool that integrates fuzzing and symbolic execution, can serve as an indirect indicator for comparison. The authors of Badger claim that black-box fuzzing-based WCA tools like SlowFuzz are in spirit similar to KelinciWCA. Thus, our comparison with KelinciWCA should reflect how our DSE-SMC framework performs against SlowFuzz, etc. A more systematic comparison (e.g., reimplement SlowFuzz and Mayhem’s algorithms for analyzing Java programs) is left for future work.

All of our experiments were conducted on an x86-64 architecture, Intel Core i7-1365UE 4.9GHz Linux machine with 32 GB of memory. We used OpenJDK 1.9.0_132 and configured the Java VM to use at most 10 GB of memory.

We embark on the very basic algorithm in computer science: comparing and ordering. The first experiment we conducted is on pairing procedures to collect pairs that satisfy specific conditions, or in the case of our setup, ordered adjacent pairs. Because the structure of the experiment is relatively less sophisticated, the result shows that our approach behaves significantly better than Kelinci and KelinciWCA.

Fig. 3 displays the result of generating ordered pairs on a given array, where $N$ is the length of the array. The exact maximum score in this task is $269$, and DSE-SMC reaches $95\%$ worst-case performance in less than $40$ minutes. All of the evaluated tools reach scores no less than $65\%$ of the maximum score in $100$ epochs, and converge no greater than $80$ epochs.

We then evaluate how our framework reacts to textbook algorithms, which are also evaluated in other WCA tools. In this section, we consider three sorting algorithms. Insertion sort and quicksort vary in terms of time complexity and the number of jumps. Both of them are easy to implement and execute, yet the worst-case behavior may be significantly influenced by the size of the input. In line with the execution time limit, we conducted our experiment on an implementation of insertion sort under input size of $20$ and present our results in Fig. 4, where $N$ is the input size.

The exact maximum score (i.e., the number of swaps) possible for insertion sort is $289$. From the result shown in Fig. 4, we observe that the score increases continuously when running DSE-SMC. The use of genetic mutation and crossover generally reduce the running time consumed to converge by $60$ epochs. Last but not least, KelinciWCA utilizes the customized cost function and indeed performed better at first $30$ epochs, yet failed to mutate towards larger scores in a restricted time frame.

The results for quicksort are shown in Fig. 5. After executing for 100 epochs, DSE-SMC behaves better than Kelinci by $20\%$. On the other hand, we also notice that DSE-SMC behaves quite unsteady on the quicksort task. The minor drawback may be explained by the probabilistic mutation process and its inherited randomness. Overall, or in a slightly longer period, DSE-SMC converges 8 times quicker than KelinciWCA.

The last sorting task we investigated is the process we called tree sort. It is used to demonstrate the performance of DSE-SMC on subjects that use complex data structures. In this subject, we implemented a red-black tree for inserting integers and set the resource metric to be the number of red-black tree operations. The tendency and convergence rate of DSE-SMC in comparison with other tools are obvious, as shown in Fig. 6: our tool achieved a significantly faster speed to discover worst-case inputs.

To secure an idea of what our approach behaves in extreme cases which are generally considered important in detecting vulnerabilities, we conduct the experiment on regular expression Denial-of-Service (ReDos) vulnerabilities. In accordance with prior work (e.g., (Noller et al., 2018)), we used the java.util.regex JDK package. It is generally used for vulnerability revelation and not exactly suitable to test worst-case generating. Yet the results may be of help to understand the cons of applying black-box techniques to those type of problems.

In this experiment, we fixed the regular expression and attempted to mutate the text for worst-case inputs. The ultimate goal for this example is that the WCA tools generate a password that matches the regular expression ((?=.*\d)(?=.*[a - z])(?=.*[A - Z])(?=.*[@#$%]).{6, 20}).

Fig. 7 shows the experimental result. DSE-SMC behaved slightly worse than the locally-optimal variant of DSE-SMC, and got caught up by Kelinci at approximately $110$ minutes. The results can be interpreted as lack of detailed domain knowledge with regard to the program structure. White- or grey-box tools in general have components for condition analysis and dynamic instrumentation. While black-box tools (like ours) execute faster in a single epoch, there is a choice between sacrificing computational speed and extracted information in one epoch when it comes to structural information within the program. Interestingly, the performance of DSE-SMC varies greatly across different runs. Inherited from probabilistic computation, DSE-SMC displays specific characters of randomness.

The hash-table subject is also implemented with considerations similar to RegEx. We modified the input part of the hash-table algorithm in accordance with prior work. Based on the outcome, we observe that there are several noises that seem to show that our tool generates worst-case inputs significantly better. By the one-hour time limit, our full version of DSE-SMC has already generated input cases that can trigger more than $6,300$ jumps, while KelinciWCA and locally-optimal DSE-SMC got stuck at near $5,500$ jumps. That displays a significant advancement, as shown in Fig. 8.

Another observation is that our tool continues to make progress at the end of the evaluation process. This indicates that probabilistic implementation of the genetic operations possess higher potential of exploration and the dual-strategy approach does make sense with regard to biological investigation. In addition, unlike the previous subjects of experiments, the performance of our tool on this subject is relatively stable in comparison, indicating the particular posterior distribution is a relatively smooth territory.

A normal application of WCA is to analyze the performance of compression algorithms. Compression algorithms are a hot topic in both the industry and the academia. We used the org.apache.commons.compress JDK package to conduct our experiment. The subject to be tested is the performance of compression algorithms on $100$ bytes of input data.

Fig. 9 shows the experimental result. Despite the difference in initial performances, DSE-SMC reaches 1680 jumps at least 3 times faster than KelinciWCA, and the locally optimal DSE-SMC makes steady progress and performs slightly better than KelinciWCA. The locally-optimal variant of DSE-SMC generated initial inputs that already lead to relatively high resource costs, yet slowed down quickly and got caught up by KelinciWCA at 23 minutes. DSE-SMC, on the other hand, reveals great potential in exploration that it boosts quickly, for example, in the $11$th and $21$th minutes. This can be explained by the fact that we use genetic mutation loops, which adds more steps relevant to exploration. It is also worth mentioning that we observe a slowdown after the boosts, indicating that our framework probably reaches a local optimum and bit-level mutation does no longer lead to better exploration. Although DSE-SMC continues to make progress after 100 minutes, at that point, one should decrease the rate of annealing the population and the migration rate of candidates applying the $R$-strategy to address the issue and look for stabler balance between exploration and exploitation.

One of the most interesting potentials for worst-case estimation is the potential of gas fuzzing in modern blockchains. Generally, the gas is a deterministic indicator that determines the cost of a smart contract and the resource consumption in executing it. We setup the environment in a simple smart contract in a Java-based form. Details can be found in the replication package.

Results for this subject are displayed in Fig. 10. As the cost is user-defined, we have to manually set the cost in order to feed Kelinci and KelinciWCA. We conducted the experiment for multiple times and got results with huge variation. Our DSE-SMC tool does not perform as good as one would expect. An observation from the smart-contract experiment is that DSE-SMC generally got stuck at 10 minutes and behaved worse than KelinciWCA. In addition, time consumed in one epoch is relatively greater than other tools due to the implementation of different strategies and relatively complex rejuvenation routines. Note that the smart-contract experiment we conduct is based on its gas consumption, whose calculation mainly replies on memory allocation and pointers’ inner relationships. Without inner relationship or fine-grained program analysis, black-box based tools are easy to miss target branches which consume a lot of gas, and therefore, fail to observe overall worse-case gas-usage behavior. It would be interesting future research to integrate our black-box technique with white- or grey-box approaches that can provide more application-specific information.