Accelerated Parameter Estimation with DALE$\chi$

MCMC uses chains of samples to evaluate the posterior probability of the data having a good fit to a point in parameter space, calculated through a likelihood function and priors on the parameter distribution. In the commonly used Metropolis-Hastings algorithm, a point is evaluated to check if it has a higher probability than the previous point, and accepted or rejected according to a probability criterion. The Markov chain part of the algorithm indicates that only the preceding point affects the next point, while the Monte Carlo part denotes the random selection of candidate points and the integration of the probability over the parameter space to determine the confidence contours. Note that MCMC per se is only trying to find the maximum; points defining the confidence contours are in some sense incidental.

DALE$\chi$ as discussed below is designed to guide the selection of desirable points by using all the information already collected, rather than only the previous point, taking a more global view, and also to accelerate the robust estimation of the confidence intervals by focusing on points informing this result and tracing contours rather than concentrating points near the probability maximum.

Nested sampling (Skilling, 2004) is a Monte Carlo technique for sweeping through the parameter space by comparing a new point with the likelihood of all previous points, and accepting it if its likelihood is higher than some of them (the lowest of which is then abandoned). Thus the sample moves through nested shells of probability. MultiNest breaks the volume of sampling points into ellipsoidal subvolumes and carries out the nested sampling within their union, but samples within one ellipsoid at a time. This concentrates the evaluation in a more likely viable region of parameter space while allowing for multi-modal distributions.

MultiNest exhibits accelerated convergence relative to MCMC, and better treatment of multimodal probabilities. However, MultiNest is still designed to integrate the entire Bayesian posterior over all of parameter space. Credible limit contours on this posterior are derived as a side-effect, rather than being intentionally mapped out in detail. One result of this, which we present in Section 5.2, is that MultiNest spends considerable amount of time evaluating $\chi^{2}$ at points in parameter space well outside of the credible limit contour. Such evaluations are important when trying to find the full Bayesian evidence of a model, but are not very helpful if all one is interested in are the multi-dimensional credible limit contours. We designed DALE$\chi$ for those situations where we want to accurately map out the confidence intervals of parameters, rather than characterize the full Bayesian posterior across all of parameter space.

MCMC and MultiNest are both sampling algorithms. They attempt to find Bayesian credible limits by drawing random samples from parameter space in such a way that the distribution of the samples accurately characterizes the Bayesian posterior probability distribution on the parameter space. This strategy directly carries out the optimize step described in the introduction – drawing samples with a frequency determined by their posterior Bayesian probability ensures that the maximum likelihood point in parameter space has a high probability of being found – and only incidentally carries out the characterize step.

The APS algorithm (Bryan, 2007; Bryan et al., 2007; Daniel et al., 2014), was designed to separately optimize and characterize the $\chi^{2}$ function on parameter space. Separate searches are implemented to find local minima in $\chi^{2}$ – this is the optimization – and then to locate points about those minima such that $\chi^{2}=\chi^{2}_{\rm lim}$, where $\chi^{2}_{\rm lim}$ defines the desired frequentist confidence limit (for example in two dimensions the 95% CL is $\chi^{2}_{\rm lim}=\chi^{2}_{\text{min}}+6.0$). Daniel et al. (2014) present a way of translating these into Bayesian credible limits (see their Section 2.4). These latter searches represent the characterization.

Though APS improved over MCMC in accurately characterizing complex probability surfaces, and its convergence properties were comparable, it remains desirable to speed up parameter estimation. Indeed, as mentioned in the previous subsection, nested sampling techniques such as MultiNest can also deal well with multimodal distributions and run faster than MCMC. Motivated by the desire for speed, while retaining and improving APS’ ability to efficiently characterize confidence intervals, even for complex probability surfaces, we present an algorithm (DALE$\chi$ ) with accelerated convergence in mapping the likelihood.

The optimization carried out by DALE$\chi$ is based on the simplex searching algorithm of Nelder and Mead (1965). It is well known that the simplex search is not robust against functions with multiple disconnected local minima. We therefore augment the simplex with a Monte Carlo component designed to allow DALE$\chi$ to dig itself out of false $\chi^{2}$ minima.

Qualitatively, the algorithm described below involves exploring the parameter space along diverse directions, detecting local minima, finding absolute minima, and checking for multimodality. In detail, for a parameter space of dimension $D$, initialize $2\times\left(D+1\right)+D/2$ points on an ellipsoid whose axis in each direction is the span of the full parameter space to be searched. As will be seen below, this initial search culminates by using two independent Nelder-Mead simplexes to try to find the global $\chi^{2}_{\rm min}$. The number of points initialized is chosen to be enough to initialize these simplexes (the Nelder-Mead simplex is driven by the motion of $D+1$ ‘particles’ in parameter space) plus a little extra. Each of these initial $2\times\left(D+1\right)+D/2$ particles will take $100\times D$ steps driven by a Metropolis-Hastings MCMC algorithm with Gibbs sampling (Casella and George, 1992) and simulated annealing (Kirkpatrick et al., 1983).

That is to say: at each step for each particle, randomly choose a basis vector (i.e. direction in basis parameter space) along which to step. Select a value $r$ from a normal distribution with mean zero and standard deviation unity. Step from the particle’s current position along the chosen basis vector a distance $r\times n_{i}$ where $n_{i}$ is the difference between the maximum and minimum particle coordinate value along the chosen basis vector. Accept the step with probability $\exp\left[-0.5\times(\chi^{2}_{\text{new}}-\chi^{2}_{\text{old}})/T\right]$ where the “temperature” $T$ is initialized at unity and adjusted every $10\times D$ steps. When adjusted, $T$ is set to that value which would have caused half of the steps taken since the last $T$-adjustment to be accepted. Every $4\times D$ steps, select a new set of random basis vectors and adjust the $n_{i}$ values accordingly. Keep track of both the absolute $\chi^{2}$ minimum and the recent $\chi^{2}$ minimum (to be defined momentarily) encountered by each particle. This is not necessarily the particle’s current position, as the $T$ term in the acceptance probability is designed to cause the particles to jump out of local minima in search of other local minima. If ever a particle goes $10\times D$ steps without updating its recent minimum, assume that it has settled into a local minimum from which it will not escape. Move the particle back onto the ellipsoid where the particles where initialized. Set the particle’s recent $\chi^{2}$ minimum (but not its absolute) to the value at its new location (which will almost certainly not be a local minimum) and resume walking.

After each particle has taken its $100\times D$ steps, choose the $D+1$ smallest absolute minima encountered by the particles, and use them to seed a simplex minimization search. After that search, choose the $D+1$ smallest absolute minima which are not connected to the absolute minimum with the lowest $\chi^{2}$ value and use them to seed another simplex search, in case the first simplex search converged to a false minimum. The connectedness of two absolute minima is judged by evaluating $\chi^{2}$ at the point directly between the two minima. If $\chi^{2}$ at this midpoint is less than the larger of the two $\chi^{2}$ absolute minima, then it assumed that there is a valley in $\chi^{2}$ connecting the two, and they are deemed “connected.” If not, it is assumed that there is a barrier of high $\chi^{2}$ cutting the two off from each other, and they are deemed “disconnected.”

This is a fairly weighty algorithm, requiring a number of $\chi^{2}$ evaluations that scales as $D^{2}$. However, in the case of highly complex likelihood surfaces, we find that this is necessary for accuracy in the search for the absolute minimum as well as multimodality. In addition it increases efficiency in that it keeps DALE$\chi$ from wasting time characterizing the $\chi^{2}$ contour around a false minimum. We have tested this algorithm on several multi-dimensional toy $\chi^{2}$ functions, running the algorithm many times on each function, each time with a different random number seed. We find that the current algorithm with its values tuned ($100\times D$ steps per particle, $4\times D$ steps before resetting the bases, $10\times D$ steps before adjusting $T$, etc.) consistently finds the global $\chi^{2}_{\rm min}$ or a point reasonably close to it. In fact we find in Sec. 5 that it frequently finds a lower $\chi^{2}_{\rm min}$ than MultiNest. That being said, we are open to the possibility that a more efficient means of robustly optimizing $\chi^{2}$ exists and would welcome its incorporation into DALE$\chi$ . The algorithm described above is implemented in the files include/dalex_initializer.h and src/dalex/dalex_initializer.cpp in our codebase, for anyone wishing to inspect it further.

Once DALE$\chi$ has found $\chi^{2}_{\rm min}$, it can begin to characterize the shape of the $\chi^{2}\leq\chi^{2}_{\rm lim}$ contour surrounding that minimum. This is achieved via successive minimizations of the cost function introduced in equation (1). Equation (1) is designed to drive DALE$\chi$ to find points that are both inside the desired contour and far from points previously discovered. The neighbor function $N(\vec{\theta})$ increases as DALE$\chi$ gets farther from previously known points. The exterior function $E(\chi^{2})$ exponentially decays with $\chi^{2}$ once DALE$\chi$ leaves the confines of $\chi^{2}\leq\chi^{2}_{\rm lim}$. The parameter $\ell$ in $E(\chi^{2})$ can be tuned to allow DALE$\chi$ to take circuitous routes through parameter space, swerving in and out of $\chi^{2}\leq\chi^{2}_{\rm lim}$ in search of the minimum of equation (1). We implement equation (1) as a C++ class cost_fn defined in the files include/cost_fn.h and src/dalex/cost_fn.cpp in our codebase. We describe that class below.

Before proceeding, we have to define exactly what contours we are searching for. (The reader more interested in a direct comparison of results from different algorithms can skip to Section 5.) A significant philosophical difference between the sampling algorithms (MCMC and MultiNest) and the search algorithms (APS and DALE$\chi$ ) is in how each defines the desired boundary in parameter space (the “confidence limit” in frequentist terminology; the “credible limit” for Bayesians). Sampling algorithms learn the definition boundary as they go. They draw samples from the posterior probability distribution and, after convergence has been achieved, declare the limit to be the region of parameter space containing the desired fraction of that total distribution. Search algorithms require the user to specify some $\chi^{2}_{\rm lim}$ defining the desired limit a priori. This can be done either by setting an absolute value of $\chi^{2}_{\rm lim}$ or by demanding that $\chi^{2}$ be within some $\Delta\chi^{2}$ of the discovered $\chi^{2}_{\rm min}$, i.e. $\chi^{2}_{\rm lim}=\chi^{2}_{\rm min}+\Delta\chi^{2}$. These two approaches are not wholly irreconcilable. Wilks’ Theorem (Wilks, 1938) states that, for approximately Gaussian likelihood functions, the Bayesian credible limit (i.e. the contour bounding $(1-\alpha)\%$ of the total posterior probability) in a $D$-dimensional parameter space is equivalent to the contour containing all points corresponding to $\chi^{2}\leq\chi^{2}_{\text{min}}+\chi^{2}_{(1-\alpha)\%}$ where $\chi^{2}_{(1-\alpha)\%}$ is the value of $\chi^{2}$ bounding the desired probability for a $\chi^{2}$ probability distribution with $D$ degrees of freedom. Even given this result, there remains some ambiguity in how one sets $\chi^{2}_{(1-\alpha)\%}$. Namely: one must choose the correct value of $D$ for the degrees of freedom. Traditionally, credible and confidence limits are plotted in 2-dimensional sub-spaces of the full parameter space. The limits that are plotted are either marginalized or projected down to two dimensions. In the case of marginalization, the Wilks’-theorem equivalence between confidence and credible limits holds for $D=2$ degrees of freedom and $\Delta\chi^{2}=6.0$ for the 95% contour. In the case of projections, $D$ is equal to the full dimensionality of the parameter space. We argue and demonstrate below that the preferred choice, with the most robust results, is projection with $D$ equal to the full dimensionality of the parameter space being considered. We illustrate this with a toy example.

In order to compare different definitions of confidence and credible limits, we construct a cartoon likelihood function on 4 parameters. Choosing such a low-dimensional parameter space allows us to grid the full space and directly integrate the Bayesian credible limits without requiring unreasonable computational resources. Figure 1 shows the marginalized likelihood of our 4-dimensional cartoon likelihood function in each of the 2-dimensional parameter sub-spaces. As one can see, we have constructed our cartoon likelihood function to have a severely curved degeneracy in the $\{\theta_{0},\theta_{1}\}$ parameter sub-space and significantly non-Gaussian contours in the other five sub-spaces.

In Figures 2 and 3, we show the confidence and credible limits of our 4-dimensional toy likelihood function computed in several ways. The red contours show the marginalized Bayesian credible limits. These are the contours enclosing 95% (68%) of the marginalized likelihood shown in Figure 1. The green contours are the projections of the 4-dimensional contours enclosing $\chi^{2}\leq\chi^{2}_{\text{min}}+6.0\ (2.28)$, i.e. they were generated by finding all of the points in 4-dimensional space satisfying $\chi^{2}\leq\chi^{2}_{\text{min}}+6.0\ (2.28)$, converting those points into a scatter-plot in each of the 2-dimensional sub-spaces, and then drawing a contour around those scatter plots. Naively, one would expect these contours to be equivalent to the marginalized Bayesian contours, since 6.0 (2.28) is the 95% (68%) limit for a $\chi^{2}$ distribution with two degrees of freedom. (Note that the 95.4% and 68.3% limits, i.e. $2\sigma$ and $1\sigma$ in Gaussian probability, would have $\Delta\chi^{2}=6.17$ and 2.30.) This, however, is clearly not the case. The non-Gaussian nature of our toy likelihood prevents Wilks’ theorem from applying in the marginalized case. While you can integrate away $d$ dimensions of a $D$-dimensional multivariate Gaussian and come up with a $D-d$ multivariate Gaussian, no such guarantee exists for arbitrary, non-Gaussian functions.

The blue contours in Figure 2 and 3 show the projected Bayesian credible limits of our cartoon function. These were constructed by assembling all of the points in 4-dimensional space that contained 95% (68%) of the total likelihood, converting those points to a scatter plot in the 2-dimensional sub-spaces, and drawing a contour around the scatter plots. The purple contours show the similarly projected contours containing all of the points for which $\chi^{2}\leq\chi^{2}_{\text{min}}+9.49\ (4.70)$, with 9.49 (4.70) being the 95% (68%) limit for a $\chi^{2}$ distribution with four degrees of freedom. We refer to these contours as the “full-dimensional LRT (likelihood ratio test)” contours. We see remarkably good agreement between the projected Bayesian and full-dimensional LRT contours in both the 95% and the 68% limit, in accordance with Wilks’ theorem (since we have not marginalized away any of the 4 parameters in our parameter space, Wilks’ theorem still appears to hold).

In the rest of this paper, we will examine likelihood functions based on their projected Bayesian and full-dimensional LRT contours. Though it is traditional to consider marginalized contours in 2-dimensional sub-spaces, we note concerns that this may discard important information about the actual extent of parameter space considered reasonable by the likelihood function. In all cases in Figures 2 and 3, the projected Bayesian and full-dimensional LRT contours were larger than their 2-dimensional marginalized counterparts. This should not be surprising. The projected Bayesian and full-dimensional LRT contours contain all of the pixels in the full 4-dimensional space that fall within the desired credible limit. The marginalized counterparts take those full 4-dimensional contours and then weight them by the amount of marginalized parameter space volume they contain. The final marginalized contour thus represents an amalgam of the raw likelihood function and its extent in parameter space. It is not obvious that this is necessarily what one wants when inferring parameter constraints from a data set.

If a point in parameter space falls within the projected Bayesian or full-dimensional LRT contour, it means that the fit to the data provided by that combination of parameters is, by some metric, reasonable. This is somewhat a matter of taste, but if one is really interested in all of the parameter combinations that give reasonable fits to the data, the conservative approach would be to keep all such combinations, rather than throwing out some, simply because they include rare – but fully accepted by the data – combinations of, say, $\{\theta_{1},\theta_{3}\}$ (in the case of the gap between the marginalized and projected Bayesian contours in the $\{\theta_{0},\theta_{2}\}$ plot in Figure 2). For this reason, we proceed¹¹1 We do not consider useful the strictly frequentist perspective that, since our cartoon likelihood function “measures” $\vec{y}(x)$ at 100 values of $x$, the confidence limit limit should therefore be set according to the $\chi^{2}$ distribution with 100 degrees of freedom. This would set the 95% confidence limit at $\chi^{2}=124.35$. Note that since $\chi^{2}_{\text{min}}$ for our toy likelihood function ended up being 83.12, such frequentist confidence limit contours would be significantly larger than anything shown in Figures 2 and 3. Such large contours seem unlikely to be useful. , comparing the performance of DALE$\chi$ with MultiNest by examining the projected Bayesian and full-dimensional LRT contours.

We acknowledge that there are some applications for which the approximate equivalence of $\chi^{2}\leq\chi^{2}_{\rm lim}$ with a Bayesian credible limit for which we have argued above is insufficient. If a likelihood function is sufficiently non-Gaussian or a data set is sufficiently noisy, the two will not be equivalent. Indeed, in Figures 2 and 3, the blue and purple contours are not identical, but merely approximately the same. In cases where only the Bayesian credible limit is acceptable, we still believe that there is a use for DALE$\chi$ . Bayesian sampling algorithms generally get bogged down during “burn-in”, the period of time at the beginning of the sampling run during which the algorithm is simply learning where the minimum $\chi^{2}$ point is and what the degeneracy directions of the contour are without efficiently sampling from the posterior. We show below that DALE$\chi$ converges to its confidence limits, with the same $\chi^{2}_{\rm min}$ and degeneracy directions as the Bayesian credible limit, irrespective of one’s opinion of Bayesian versus frequentist statistical interpretations, after an order of magnitude fewer calls to $\chi^{2}$ than required by MultiNest. If one does not feel inclined to trust the full confidence limits returned by DALE$\chi$ , one should be able to achieve a significant speed-up in Bayesian sampling algorithms by using DALE$\chi$ as pre-burner: defining the region of parameter space to be sampled before sampling commences.

We showed above that DALE$\chi$ found similar confidence limits, but took much longer to converge than did MultiNest on a 4-dimensional likelihood function. Now we compare DALE$\chi$ to MultiNest on a 12-dimensional cartoon likelihood function. Here we find that DALE$\chi$ converges an order of magnitude faster than MultiNest.

Before we show the direct comparison between DALE$\chi$ and MultiNest, we must be careful to define what we are comparing. MultiNest involves several user-determined parameters that control how the sampling is done. One in particular, the number of ‘live’ points, which is somewhat analogous to the number of independent chains run in an MCMC, has a profound effect both on how fast MultiNest converges and what limit it finds when it does converge. We created two 12-dimensional instantiations of our cartoon likelihood function, one with a very curved parameter space degeneracy, one without, and ran several instantiations of MultiNest with different numbers of ‘live’ points on those functions. Figures 5 and 6 show the projected Bayesian 95% credible limits found by MultiNest in two 2-dimensional sub-spaces for each of those likelihood functions as a function of the number of ‘live’ points. The figures also note how many evaluations of the likelihood function were necessary before MultiNest declared convergence. Not only does increasing the number of ‘live’ points increase the number of likelihood evaluations necessary for MultiNest to converge, it also expands the discovered credible limits. We interpret this to mean that running MultiNest with a small number of ‘live’ points results in convergence to an incomplete credible limit. When comparing DALE$\chi$ to MultiNest, we must therefore be careful which instantiation of MultiNest we compare against. Selecting a MultiNest run with too few ‘live’ points could mean that MultiNest appears as fast as DALE$\chi$ , but converges to an incorrect credible limit. Selecting a MultiNest run with too many ‘live’ points ensures that MultiNest has converged to the correct credible limit, but in an artificially large number of likelihood evaluations. In Figures 7 and 8, and Figures 10 and 11 below, we compare DALE$\chi$ both to a MultiNest instantiation that we believe has converged to the correct credible limit and a MultiNest instantiation that converges in a time comparable to DALE$\chi$ . This is to illustrate the relationship between convergence time and accuracy in MultiNest.

In Figures 7 and 8 we compare the convergence of DALE$\chi$ with that of MultiNest as a function of time. The likelihood function considered is a 12-dimensional cartoon constructed as in Section 4, but without the highly curved degeneracy direction. Each panel in the figures shows the state of the 95% confidence limits discovered by DALE$\chi$ after a specified number of evaluations of the likelihood function. The limits are plotted as scatter plots of the $\chi^{2}\leq\chi^{2}_{\rm lim}$ points discovered by DALE$\chi$ at that point in time. Because MultiNest does not produce chains in the same way that MCMC does, we cannot display the evolution of the credible limit found by MultiNest as a function of time. Instead, we plot two credible limits: one found by an instantiation of MultiNest that converges after $210,000$ evaluations of the likelihood function and one that converges after $2,700,000$ evaluations of the likelihood function. As explained above, this difference in convergence time was achieved by tuning the number of ‘live’ points in the MultiNest run. There are two important features to note here.

The first feature is that DALE$\chi$ has converged to its final credible limit after about 125,000-150,000 calls to $\chi^{2}$. The second feature is the difference between the credible limits discovered by the two instantiations of MultiNest. True, one of the MultiNest instantiation converges in roughly (within a factor of two) as many likelihood evaluations as DALE$\chi$ . This instantiation of MultiNest, however, fails to explore the full non-Gaussian wings of the credible limit. See Figure 9 for a visualization of the non-Gaussianity. The slower MultiNest instantiation does find the same limits as DALE$\chi$ . However, it requires more than an order of magnitude more likelihood function evaluations to do so. This is the origin of our claim that DALE$\chi$ converges an order of magnitude faster than MultiNest on high-dimensional likelihood functions. DALE$\chi$ does not do as well as MultiNest characterizing the full width of the contour in the narrow degeneracy direction in Figure 8. However, by the end of the full 250,000 $\chi^{2}$-evaluation run, one can see the wings of the contour beginning to trace out the full boundary. (Alternately, if one adopted the strategy suggested at the end of Section 4.1, using DALE$\chi$ to set the proposal distribution for a sampling algorithm, one could imagine filling in these contours more thoroughly than with just MultiNest alone while still only requiring $\sim$ 10% more evaluations of the likelihood function.)

Figures 10 and 11 show a similar comparison in the case of a 12-dimensional likelihood function with a highly curved parameter space degeneracy. Here again we see that DALE$\chi$ finds essentially the same limits as MultiNest, but more than an order of magnitude faster. We have done preliminary tests with a 16-dimensional likelihood function. These show almost two orders of magnitude speed-up of DALE$\chi$ vis-á-vis MultiNest. In the 16-dimensional case, MultiNest requires 75 million calls to $\chi^{2}$ to converge. DALE$\chi$ finds the same credible limit after only 1 million calls to $\chi^{2}$.

Regarding some computational specifics, note we have been measuring the speed of DALE$\chi$ and MultiNest in terms of how many calls to $\chi^{2}$ are required to reach convergence. This is based on the assumption that, for any real physical use of these algorithms, the evaluation of $\chi^{2}(\vec{\theta})$ will be the slowest part of the computation. For MultiNest, this is a fair assumption. As a Markovian process, the present behavior of MultiNest does not depend on its history. The same statement is not true of DALE$\chi$ . Because DALE$\chi$ is attempting to evaluate $\chi^{2}$ at parameter-space points as far as possible away from previous $\chi^{2}$ evaluations, DALE$\chi$ must always be aware of the full history of its search. Thus, the DALE$\chi$ algorithm implies some computational overhead in addition to the expense of simple evaluating $\chi^{2}$. For our 12-dimensional test cases, this overhead averages out to between 0.005 and 0.015 seconds per $\chi^{2}$ evaluation on a personal laptop with a 2.9 GHz processor. For the 16-dimensional test case referenced above, this overhead varied between 0.01 and 0.03 seconds per $\chi^{2}$ evaluation on the same machine. It is possible that future work with an eye towards computational optimization could reduce these figures. It is also possible that the steady march of hardware innovation could render such concerns irrelevant going forward. The memory footprint of DALE$\chi$ appears to be well within the capabilities of modern personal computers.

Note that both MultiNest and DALE$\chi$ depend on random number generators: MultiNest for driving sampling of the posterior, DALE$\chi$ for driving the Metropolis-Hastings-like algorithms described in Sections 3.1 and 3.2.2, and for selecting seeds for the simplex minimizations described in Sections 3.2.3 and 3.2.4. We have found that the convergence performance of DALE$\chi$ can depend on how that random number generator is seeded. Figure 12 plots six different runs of DALE$\chi$ , each with a different random number seed. The runs plotted were specifically selected to represent instances of DALE$\chi$ that did not find the full $(\theta_{0},\theta_{1})$ confidence limit after 400,000 $\chi^{2}$ evaluations (all six successfully found the $(\theta_{6},\theta_{9})$ confidence limit in 400,000 samples). In each panel, we show the confidence limit as found by DALE$\chi$ after 400,000, 600,000, 800,000 $\chi^{2}$ evaluations plotted against the true credible limit as discovered by a run of MultiNest with 50,000 live points, taking 35 million evaluations. In each case, the 400,000 evaluation confidence limit traces out the general shape of the true contour while the subsequent evaluations allow DALE$\chi$ to fill in the gaps in that first discovered confidence limit. Referring back to Figure 6, we see that, even after 4.3 million $\chi^{2}$ evaluations, MultiNest has not found its final credible limit. Therefore, even these instances of sub-optimal runs of DALE$\chi$ demonstrate a significant speed-up vis-à-vis MultiNest. The difficulty is in knowing how many $\chi^{2}$ evaluations is enough for DALE$\chi$ . This is the danger in not having a clear global convergence criterion for DALE$\chi$ and a subject worthy of further study.

For those interested, and in the interest of giving credit where credit is due, DALE$\chi$ uses the random number generator proposed by Marsaglia (2003) with initialization parameters proposed by Press et al. (2007). This is implemented in the class Ran defined in include/goto_tools.h in our code base.

Figure 13 shows the distribution points evaluated by DALE$\chi$ and MultiNest as a function of $\chi^{2}$ when run on the highly curved likelihood function from Figures 6, 10, and 11. Note how DALE$\chi$ places a much larger fraction of its points on the $\chi^{2}=\chi^{2}_{\rm lim}$ contour than does MultiNest. This should not be surprising as DALE$\chi$ was designed with limit-finding in mind. MultiNest is a code designed to calculate the integrated Bayesian evidence of a model. The fact that one can draw confidence limit contours from the samples produced by MultiNest is basically a happy side-effect of this evidence integration. Because it focuses on finding the contours directly, rather than integrating the posterior over parameter space, DALE$\chi$ is highly efficient at both the optimization and characterization tasks, i.e. the key elements of determining the best fit and its error estimation, as designed. This tradeoff between obtaining the two critical elements for parameter estimation versus mapping the posterior more broadly (and sometimes less accurately) is the root cause of the vastly improved speed demonstrated by DALE$\chi$ .