Verification of Neural Reachable Tubes via Scenario Optimization and Conformal Prediction

It is important to design provably safe controllers for autonomous systems. Hamilton-Jacobi (HJ) reachability analysis provides a powerful framework to design such controllers for general nonlinear dynamical systems (Lygeros, 2004; Mitchell et al., 2005). In reachability analysis, safety is characterized by the system’s Backward Reachable Tube (BRT). This is the set of states from which trajectories will eventually reach a given target set despite the best control effort. Thus, if the target set represents undesirable states, the BRT represents unsafe states and should be avoided. Along with the BRT, reachability analysis provides a safety controller to keep the system outside the BRT.

Traditionally, the BRT computation in HJ reachability is formulated as an optimal control problem. The BRT can then be obtained as a sub-zero level solution of the corresponding value function. Obtaining the value function requires solving a partial differential equation (PDE) over a state-space grid, resulting in an exponentially scaling computation complexity with the number of states (Bansal et al., 2017). To overcome this challenge, a variety of solutions have been proposed that trade off between the class of dynamics they can handle, the approximation quality of the BRT, and the required computation. These include specialized methods for linear and affine dynamics (Greenstreet and Mitchell, 1998; Frehse et al., 2011; Kurzhanski and Varaiya, 2000, 2002; Maidens et al., 2013; Girard, 2005; Althoff et al., 2010; Bak et al., 2019; Nilsson and Ozay, 2016), polynomial dynamics (Majumdar et al., 2014; Majumdar and Tedrake, 2017; Dreossi et al., 2016; Henrion and Korda, 2014), monotonic dynamics (Coogan and Arcak, 2015), and convex dynamics (Chow et al., 2017) (see Bansal et al. (2017); Bansal and Tomlin (2021) for a survey).

Owing to the success of deep learning, there has also been a surge of interest in approximating high-dimensional BRTs (Rubies-Royo et al., 2019; Fisac et al., 2019; Djeridane and Lygeros, 2006; Niarchos and Lygeros, 2006; Darbon et al., 2020) and optimal controllers (Onken et al., 2022) through deep neural networks (DNNs). Building upon this line of work, Bansal and Tomlin (2021) have proposed DeepReach – a toolbox that leverages recent advances in neural implicit representations and neural PDE solvers to compute a value function and a safety controller for high-dimensional systems. Compared to the aforementioned methods, DeepReach can handle general nonlinear dynamics, the presence of exogenous disturbances, as well as state and input constraints during the BRT computation. Consequently, methods for verifying neural reachable tubes have been proposed. For example, Lin and Bansal (2023) propose an iterative scenario-based method (Campi et al., 2009) to recover probabilistically safe reachable tubes from DeepReach solutions up to a desired confidence level and bound on violation rate. Unfortunately, the method does not allow an after-the-fact risk-return trade-off, and as a result, it is highly sensitive to outlier errors in the learned solutions. This can lead to highly conservative reachable tubes and a severe loss of recovery in the case of stringent safety requirements, as we demonstrate in our case studies.

In this work, we propose two different verification methods, one based on robust scenario optimization and the other based on conformal prediction, to provide probabilistic safety guarantees for neural reachable tubes. Both methods are resilient to the outlier errors in neural reachable tubes and automatically trade-off the strength of the probabilistic safety guarantees based on the outlier rate. The proposed methods can evaluate any candidate tube and are not restricted to a specific class of system dynamics or value functions. We further prove that these seemingly different verification methods naturally reduce to one another, providing a unifying viewpoint for uncertainty quantification (typical use case of conformal prediction) and error optimization (typical use case of scenario optimization) in neural reachable tubes. Based on these insights, we propose an outlier-adjusted verification approach that can recover a greater safe volume from a neural reachable tube by harnessing information about the distribution of error in the learned solution. To summarize, the key contributions of this paper are:

• •

  probabilistic safety verification methods for neural reachable tubes that enable a direct trade-off between resilience and the probabilistic strength of safety,

• •

  a proof that split conformal prediction reduces to a scenario-based approach in general, demonstrating a strong relationship between the two highly related but disparate fields,

• •

  an outlier-adjusted verification approach that recovers greater safe volumes from tubes, and

• •

  a demonstration of the proposed approaches for the high-dimensional problems of multi-vehicle collision avoidance and rocket landing with no-go zones.

Consider a dynamical system with state $x\in X\subseteq\mathbb{R}^{n}$, control $u\in\mathcal{U}$, and dynamics $\dot{x}=f(x,u)$ governing how $x$ evolves over time until a final time $T$. Let $\xi_{x,t}^{u}(\tau)$ denote the state achieved at time $\tau\in[t,T]$ by starting at initial state $x$ and time $t$ and applying control $u(\cdot)$ over $[t,\tau]$. Let $\mathcal{L}$ represent a target set that the agent wants to either reach (e.g. goal states) or avoid (e.g. obstacles).

In this setting, we are interested in computing the system’s initial-time Backward Reachable Tube, which we denote as BRT. We define BRT as the set of all initial states in $X$ from which the agent will eventually reach $\mathcal{L}$ within the time horizon $[0,T]$, despite best control efforts: $\text{BRT}=\{x:x\in X,\forall u(\cdot),\exists\tau\in[0,T],\xi_{x,0}^{u}(\tau)\in\mathcal{L}\}$. When $\mathcal{L}$ represents unsafe states for the system, as it does in our running example, staying outside of BRT is desirable. When $\mathcal{L}$ instead represents the states that the agent wants to reach, BRT is defined as the set of all initial states in $X$ from which the agent, acting optimally, can eventually reach $\mathcal{L}$ within $[0,T]$. Thus, staying within BRT is desirable.

The above 9D system is intractable for traditional grid-based methods, motivating the use of DeepReach to learn a neural BRT. Our goal in this work is to recover an approximation of the safe set with probabilistic guarantees. Specifically, we want to find $\mathcal{S}$ such that $\underset{x\in\mathcal{S}}{\mathbb{P}}(x\in\text{BRT})\leq\epsilon$ for some violation parameter $\epsilon\in(0,1)$. When $\mathcal{L}$ represents goal states, we want $\underset{x\in\mathcal{S}}{\mathbb{P}}(x\in\text{BRT}^{C})\leq\epsilon$.

Here, we provide a quick overview of Hamilton-Jacobi reachability analysis, a specific toolbox, DeepReach, to compute high-dimensional neural reachable tubes, and an iterative scenario-based method for recovering probabilistically safe tubes from learning-based methods like DeepReach.

Here, we propose a robust scenario-based probabilistic safety verification method for neural reachable tubes. The new method is a straightforward application of a scenario-based sampling-and-discarding approach to chance-constrained optimization problems, which quantifies the trade-off between feasibility and performance of the optimal solution based on finite samples (Campi and Garatti, 2011). First, we explain the method when $\mathcal{L}$ represents undesirable states. In the end, we comment on when $\mathcal{L}$ represents desirable states.

All proofs can be found in the Appendix of the extended version of this article¹¹1See \urlhttps://sia-lab-git.github.io/Verification_of_Neural_Reachable_Tubes.pdf. Disregarding the confidence parameter $\beta$ for a moment, \theoremreftheorem:scenario-based_method states that the fraction of $\mathcal{S}$ that is unsafe is bounded above by the violation parameter $\epsilon$, where $\epsilon$ is computed empirically using Equation (2) based on the outlier rate $k$ encountered within $N$ samples. $\epsilon$ is thus a reflection of the safety quality of $\mathcal{S}$, which degrades with the increase in the number of outliers $k$, as expected. This can also be seen for the running example in \figurereffig:fixed_N (the red curve). Overall, \theoremreftheorem:scenario-based_method allows us to compute probabilistic safety guarantees for any neural set $\mathcal{S}$ based on a finite number of samples. Subsequently, this result can be used to find some $\mathcal{S}$ for which $\epsilon$ is smaller than a desired threshold, as we discuss later in this section.

To interpret $\beta$, note that $k$ is a random variable that depends on the randomly sampled $x_{1:N}$. It may be the case that we just happen to draw an unrepresentative sample, in which case the $\epsilon$ bound does not hold. $\beta$ controls the probability of this adverse event happening, which regards the correctness of the probabilistic safety guarantee in Equation (3). Fortunately, $\beta$ goes to $0$ exponentially with $N$, so $\beta$ can be chosen to be an extremely small value, such as $10^{-16}$, when we sample large $N$. $1-\beta$ will then be so close to $1$ that it does not have any practical importance.

We have just explained the robust scenario-based probabilistic safety verification method in the case where $\mathcal{L}$ represents undesirable states. When the system instead wants to reach $\mathcal{L}$, $\mathcal{S}$ will be a sublevel set instead of a superlevel set of the learned value function. The cost inequality should be flipped when computing $k$, and the value inequality should be flipped in Equation (3).

We now propose a conformal probabilistic safety verification method for neural reachable tubes which is intended to be the direct analogue of the robust scenario-based method in \sectionrefsec:scenario-based_method. The method is a straightforward application of split conformal prediction, a widely used method in the machine learning community for uncertainty quantification (Angelopoulos and Bates, 2023).

Using the same procedures as described in \sectionrefsec:scenario-based_method, split conformal prediction can be used instead of robust scenario optimization to provide a probabilistic guarantee on the safety of the neural reachable tube and its complement, the neural safe set $\mathcal{S}$:

theorem:conformal_method can be established via a straightforward application of conformal prediction with $-J_{\tilde{\pi}}(x,0)$ as the scoring function. The proof is in the Appendix of the extended version of this article¹. The above theorem states that the fraction of $\mathcal{S}$ that is safe is distributed according to the Beta distribution with shape parameters $N-k$ and $k+1$. Intuitively, the mass in the distribution shifts towards $0$ as $k$ increases for a fixed $N$, implying that it is more likely that a smaller fraction of $\mathcal{S}$ is safe, as expected. For a fixed ratio $N:k$, $N$ controls how concentrated the mass is around the mean; i.e., for larger sample sizes $N$, we can more confidently determine the fraction of $\mathcal{S}$ that is safe.

To better understand \theoremreftheorem:conformal_method, we show in \figurereffig:beta the Beta distribution of $\underset{x\in\mathcal{S}}{\mathbb{P}}\left(J_{\tilde{\pi}}(x,0)>0\right)$ for a solution learned by DeepReach on the multi-vehicle collision avoidance running example in \sectionrefsec:problem_setup, for which $k=731$ outliers are found from $N=3684118$ samples.

Even though \theoremreftheorem:conformal_method provides the distribution of the safety level, when we compute safety assurances in practice, it is often desirable to know a lower-bound on the safety level with at least some desired confidence. This corresponds to choosing a lower-bound whose accumulated probability mass is smaller than some confidence parameter $\beta$ (shaded red in \figurereffig:beta). The following lemma formalizes this by using the CDF of the Beta distribution in \theoremreftheorem:conformal_method.

lemma:conformal_method is, in fact, precisely the same result as obtained by \theoremreftheorem:scenario-based_method using robust scenario optimization. This is no coincidence, as one can show that split conform prediction more generally reduces to a robust scenario-optimization problem.

Due to the equivalence between conformal method and robust scenario-based methods, the analysis in \sectionrefsec:scenario-based_method holds here as well. More generally, we hope that this insight will lead to future research into further investigating the close relationship between the two methods.

The verification methods in \sectionrefsec:scenario-based_method,sec:conformal_method are limited by the quality of the neural reachable tube. Although they can account for outliers, the computed safety level can be low if the outlier rate is high. This can lead to significant losses in the safe volume, as demonstrated in \sectionrefsec:rocketlanding,sec:reachavoidrocketlanding.

To address this issue, we propose an outlier-adjusted approach that can recover a larger safe volume for any desired $\epsilon$. Note that in the verification methods, the key quantity which determines $\epsilon$ is the number of safety violations $k$. This corresponds to the number of samples $x_{i}$ which are marked safe by membership in $\mathcal{S}$, i.e., $\tilde{V}(x_{i},0)\geq\delta$, but are not guaranteed to be safe, i.e., $J_{\tilde{\pi}}\left(x_{i},0\right)\leq 0$. It is easy to see that the best we can do to simultaneously minimize $k$ and maximize volume is to compute $\mathcal{S}$ as the super-$\delta$ level set of the induced cost function $J_{\tilde{\pi}}(x,0)$. For example, the largest possible $\mathcal{S}$ that is guaranteed to be violation-free is precisely the super-zero level set of $J_{\tilde{\pi}}(x,0)$. Thus, our overall approach will be to refine $\tilde{V}(x,0)$ so that it more accurately reflects $J_{\tilde{\pi}}(x,0)$.

Modeling $J_{\tilde{\pi}}(x,0)$ can be formulated as a supervised learning problem, since we can sample a state $x_{i}$ and compute its cost $J_{\tilde{\pi}}(x_{i},0)$ in simulation. We learn an approximation $\tilde{J}_{\tilde{\pi}}(x,0)$ by retraining $\tilde{V}(x,0)$ on a training dataset $\mathcal{T}$ of $n$ samples, $\mathcal{T}=(x_{1},J_{\tilde{\pi}}(x_{1},0)),...,(x_{n},J_{\tilde{\pi}}(x_{n},0))$. Specifically, we use the weighted MSE loss $\frac{1}{n}\sum_{i=1}^{n}w_{i}(\tilde{V}(x_{i},0)-J_{\tilde{\pi}}(x_{i},0))^{2}$, where $w_{i}=w$ if the error is conservative $\left(\tilde{V}(x_{i},0)<J_{\tilde{\pi}}(x_{i},0)\right)$, otherwise $w_{i}=1$. We introduce $w$ as a hyperparameter to underweight conservative errors because in the end, we are concerned with recovering larger safe volumes. Thus, selecting a small $w$ allows us to focus on reducing optimistic errors $\left(\tilde{V}(x_{i},0)>J_{\tilde{\pi}}(x_{i},0)\right)$ which are more safety-critical and correspond to outlier safety violations.

To avoid overfitting, we select the training checkpoint that performs best on a validation dataset $\mathcal{V}$. The validation metric we use is the maximum learned cost of an empirically unsafe state: $\max_{x\in\mathcal{V}}\{\tilde{J}_{\tilde{\pi}}(x,0):J_{\tilde{\pi}}(x,0)\leq 0\}$, which one can think of as a proxy for the recoverable safe volume. We demonstrate the efficacy of the proposed outlier-adjusted approach for the high-dimensional systems of multi-vehicle collision avoidance and rocket landing with no-go zones. For all case studies, we set $w=10^{-3}$ during retraining, fix the confidence parameter $\beta=10^{-16}$ and find a safe volume that satisfies $\epsilon\leq 10^{-4}$ ($99.990\%$ safety) using the robust method in \sectionrefsec:scenario-based_method.

In this work, we propose two different verification methods, based on robust scenario optimization and conformal prediction, to provide probabilistic safety guarantees for neural reachable tubes. Our methods allow a direct trade-off between resilience to outlier errors in the neural tube, which are inevitable in a learning-based approach, and the strength of the probabilistic safety guarantee. Furthermore, we show that split conformal prediction, a widely used method in the machine learning community for uncertainty quantification, reduces to a scenario-based approach, making the two methods equivalent not only for verification of neural reachable tubes but also more generally. We hope that our proof will lead to future insights into the close relationship between the highly related but disparate fields of conformal prediction and scenario optimization. Finally, we propose an outlier-adjusted verification approach that harnesses information about the error distribution in neural reachable tubes to recover greater safe volumes. We demonstrate the efficacy of the proposed approaches for the high-dimensional problems of multi-vehicle collision avoidance and rocket landing with no-go zones. Altogether, these are important steps toward using learning-based reachability methods to compute safety assurances for high-dimensional systems in the real world.

In the future, we will explore how the key idea of the outlier-adjusted verification approach, using cost labels as a supervised learning signal, can be used to enhance the accuracy of learning-based reachability methods like DeepReach. Other directions include providing safety assurances in the presence of worst-case disturbances and in real-time for tubes that are generated online.

This work is supported in part by a NASA Space Technology Graduate Research Opportunity, the NVIDIA Academic Hardware Grant Program, the NSF CAREER Program under award 2240163, and the DARPA ANSR program.