Learning Contraction Policies from Offline Data

While learning-based controllers have achieved significant success, they still lack safety guarantees. For instance, in general, the temporal evolution of a robot’s trajectories under a learned policy cannot be certified. On the other hand, when a system’s dynamics are known, control-theoretic properties, such as stability and contraction, directly examine the temporal progression of a system’s states to verify whether a system remains within a safe set, and whether the system’s trajectories converge. In this paper, we seek to enforce the desired temporal evolution of the closed-loop system’s states while learning the policy from an offline set of data, i.e. we seek to learn control policies such that under the learned policy, the convergence of a robot’s trajectories is achieved.

To achieve such trajectory convergence, our design approach leverages Contraction theory [1]. Contraction theory provides a framework for identifying the class of nonlinear dynamic systems that have asymptotic convergent trajectories. Intuitively, a region of the state space is a contraction space if the distance between any two close neighboring trajectories decays over time. This notion of convergence is relevant to many robotic tasks such as tracking controllers where we want a robot to either reach a goal or track a reference trajectory. In this paper, we want to learn policies from offline data such that they achieve convergence of a robot’s trajectories in closed loop. While contraction theory provides a simple and intuitive characterization of convergent trajectories, finding the distance metric with respect to which a robot’s trajectories exhibit contraction – which is called the contraction metric – is often non-trivial. To address this challenge, we propose to jointly learn the robot policy and its corresponding contraction metric.

We learn the robot dynamics model from an offline data set consisting of the robot’s state and input trajectories. This setting is similar to the setting of offline model-based reinforcement learning (RL) where a dynamics model and a policy are learned from a set of robot trajectories that are collected offline. Learning from offline data is appropriate for safety-critical applications where online data collection is dangerous [2]. We learn a dynamics model of the system from the data and propose a data augmentation algorithm for learning contraction policies. Randomly sampled states are propagated forward in time through the learned dynamics model to generate auxiliary sample trajectories. We then learn both our policy and our contraction metric such that the distance between the robot trajectories from the data set and the auxiliary sample trajectories decreases over time. Learning contraction policies is particularly relevant to offline RL as it allows us to regard the errors in the learned dynamics model as external disturbances and obtain a tracking error bound in regions where the learning errors of the dynamics model are bounded [3, 4].

We evaluate the performance of our proposed framework on a set of simulated robotic goal-reaching tasks. The performance of our proposed framework is compared with a number of control algorithms. We demonstrate that as a result of enforcing contraction, the robot’s trajectories converge faster to the goal position with a higher degree of accuracy. It is further shown that learning contraction policies increases the robustness of the learned policy with respect to learned dynamics model mismatch, i.e. enforcing contraction increases the robustness of the learned policies. In summary, our contributions are the following:

• •

  We propose a framework for learning convergent robot policies from an offline data set using Contraction theory.

• •

  We develop a data augmentation algorithm for learning contraction policies from the offline data set.

• •

  We provide a formal analysis for bounding contraction policy performance as a function of dynamics model mismatch.

• •

  We perform numerical evaluations of our proposed policy learning framework and demonstrate that enforcing contraction results in favorable convergence and robustness performance.

The organization of this paper is as follows. In Section 2, we provide an overview of the related and prior work. We provide an overview of Contraction theory in Section 3 and present our problem formulation in Section 4. We then discuss our proposed framework in Section 5. Section 6 provides a discussion and analysis of the robustness of learned contraction policies. In Section 7, we evaluate and compare the performance of our policy learning algorithm. Finally, we will conclude the paper in Section 8.

For systems with unknown dynamics, several offline RL algorithms have been developed recently which either directly learn a policy using an offline data-set [5, 6, 7, 8] or learn a surrogate dynamics model from the offline data to learn an appropriate policy [9, 10]. However, the majority of such RL algorithms lack formal safety guarantees, and the convergent behavior of the learned policies is not certified [11, 12].

When the system dynamics are known, robust and certifiable control policy design can be achieved through various control-theoretic methods such as reachability analysis [13], Funnels [14, 15], and Hamilton-Jacobi analysis [16, 17]. Lyapunov stability criteria, Contraction Theory, and Control Barrier Functions have been extensively utilized for providing strong convergence guarantees for nonlinear dynamical systems [18, 19, 20, 21, 1]. However, even when the dynamics are known, finding a proper Lyapunov function or a control barrier function is itself a challenging task. To address these challenges, learning algorithms have been utilized for learning the Lyapunov and Control Barrier Functions [22, 23, 24].  [11] and [25] propose to learn contraction metrics to find contraction control policies for known systems.

Various recent works have considered combining control-theoretic tools with learning algorithms to enable learning safe policies even when dynamics are unknown.  [26, 27] consider learning stable dynamics models. In [28], Contraction theory is used to learn stabilizable dynamics models of unknown systems. In [12, 29], Lyapunov functions are used for ensuring the stability of the learned policies. [30] proposed to learn the system dynamics and its corresponding Lyapunov function jointly to ensure the stability of the learned dynamics model.

In this work, we consider learning contraction policies from offline data for systems with unknown dynamics. Our work is closely related to [11] and [25], where Contraction theory has been used for certifying convergent trajectories. The current work is different in that, unlike these approaches where dynamics are explicitly known and assumed to be control-affine, we consider access to only an offline data set. We assume that we can learn an implicit model of system dynamics, in the form of a neural network function approximator, and provide robustness guarantees with respect to the errors of the learned dynamics model.

Contraction theory assesses the stability properties of dynamical systems by studying the convergence behavior of neighboring trajectories [1]. The convergence is established by directly examining the evolution of the weighted Euclidean distance of close neighboring trajectories. Formally, consider a differentiable autonomous discrete-time dynamical system $g(\boldsymbol{\mathbf{x}}):^{n}\to^{n}$ defined as

with Jacobian

Now, consider a differential displacement $\delta\boldsymbol{\mathbf{x}}_{t}$. The differential displacement dynamics at $\boldsymbol{\mathbf{x}}_{t}$ are governed by

The system dynamics $g(\boldsymbol{\mathbf{x}}_{t})$ are contractive if there exists a full rank state dependant metric $\boldsymbol{\mathbf{\Theta}}(\boldsymbol{\mathbf{x}})\in^{n}\times^{n}$ such that the system trajectories satisfy

Equation (4) indicates that the weighted distance between any two infinitesimally close states decreases as the dynamics evolve [31]. When $\boldsymbol{\mathbf{\Theta}}(\boldsymbol{\mathbf{x}})=\boldsymbol{\mathbf{I}}$ the distances between trajectories are measured in the Euclidean norm sense. Figure 1 illustrates the behavior of two trajectories of a contractive system when $\boldsymbol{\mathbf{\Theta}}(\boldsymbol{\mathbf{x}})=\boldsymbol{\mathbf{I}}$.

For a small finite displacement $\Delta\boldsymbol{\mathbf{x}}_{t}$, as an approximation of infinitesimal small displacement $\delta\boldsymbol{\mathbf{x}}_{t}$, the first-order Taylor expansion of the system dynamics allows us to locally approximate the forward evolution of the displacement

Thus, we may approximate the contraction condition (4) as

Establishing a system as contractive allows for several useful stability properties to be deduced. We state motivating results from [1] in the following definition and proposition.

It is shown in [1] that (7) is equivalent to condition (4) holding for all $\boldsymbol{\mathbf{x}}_{t}$ in the contraction region. Thus, by Proposition 1, we may conclude that a unique equilibrium exists within a convex region if (4) holds everywhere inside the region. Therefore, (6) represents a useful numerical analog that can be enforced in order to drive a region towards being contractive. By choosing a set of $\Delta\boldsymbol{\mathbf{x}}_{t}$, we will use condition (6) to enforce contracting behavior of the closed-loop system. Going beyond autonomous systems, when a system is subject to control input $\boldsymbol{\mathbf{u}}_{t}$, i.e., $\boldsymbol{\mathbf{x}}_{t+1}=g(\boldsymbol{\mathbf{x}}_{t})=f(\boldsymbol{\mathbf{x}}_{t},\boldsymbol{\mathbf{u}}_{t})$, contraction theory can be used to design state feedback policies $\boldsymbol{\mathbf{u}}_{t}=\boldsymbol{\mathbf{u}}(\boldsymbol{\mathbf{x}}_{t})$ such that the closed-loop system trajectories converge to a given reference state. This may be done by determining $\boldsymbol{\mathbf{u}}(\boldsymbol{\mathbf{x}}_{t})$ such that the convex region of interest is contractive and the unique equilibrium is the desired reference state. Such a control design process is outlined in the following sections.

We consider the problem of control policy design for a robot with unknown discrete-time dynamics model $f(\boldsymbol{\mathbf{x}},\boldsymbol{\mathbf{u}}):\mathcal{X}\times\mathcal{U}\to\mathcal{X}$, where $\mathcal{X}\in^{n}$ is convex, $\mathcal{U}\in^{m}$. We assume that we can use an offline data set $\mathcal{D}$ consisting of tuples of state transitions and control inputs $(\boldsymbol{\mathbf{x}}_{t},\boldsymbol{\mathbf{x}}_{t+1},\boldsymbol{\mathbf{u}}_{t})$ satisfying unknown system dynamics

Our objective is to obtain a data-driven state-feedback control policy $\boldsymbol{\mathbf{u}}_{t}=\boldsymbol{\mathbf{u}}(\boldsymbol{\mathbf{x}}_{t})$ to steer the system (8) towards a desired reference state $\boldsymbol{\mathbf{x}}^{r}\in^{n}$, i.e. $\boldsymbol{\mathbf{x}}_{t}\to\boldsymbol{\mathbf{x}}^{r}$ as $t\to\infty$. To compensate for the lack of knowledge of the true system dynamics, we propose using a model of the system dynamics that we learn from the offline data $\mathcal{D}$. Note that this indicates that our learned dynamics model may still not be available explicitly and may only be available as implicit dynamics such as neural networks approximators. More specifically, we aim to design a control policy $\boldsymbol{\mathbf{u}}(\boldsymbol{\mathbf{x}}_{t})$ that leverages the learned dynamics model

to drive the system asymptotically to $\boldsymbol{\mathbf{x}}^{r}$.

To develop a policy that results in contractive behavior, we seek to enforce the approximate condition in (6), requiring the weighted distances of close neighboring trajectories to decrease over time. To enforce this condition, we need to ensure that we have sufficiently close neighboring points for each point within our training set. However, our training data set may not include such neighboring trajectories. We augment our data set with auxiliary trajectories that enable us to enforce this condition at each data point. That is, for each $\boldsymbol{\mathbf{x}}_{t}\in\mathcal{D}$, we augment our data set with a $\Delta\boldsymbol{\mathbf{x}}_{t}$ sampled from

where the parameter $\lambda$ is set in the training process. We sample points from $\Delta_{t}$ to ensure that $\Delta\boldsymbol{\mathbf{x}}_{t}$ is a small displacement with respect to the training data set. Then, for each data point $\boldsymbol{\mathbf{x}}_{t}$, we create the auxiliary state $\tilde{\boldsymbol{\mathbf{x}}}_{t}=\boldsymbol{\mathbf{x}}_{t}+\Delta\boldsymbol{\mathbf{x}}_{t}$. Both of these points are propagated through our learned dynamics model to calculate the states at the next time step: ${\boldsymbol{\mathbf{x}}}^{{}^{\prime}}_{t+1}={f}^{{}^{\prime}}(\boldsymbol{\mathbf{x}}_{t},\boldsymbol{\mathbf{u}}(\boldsymbol{\mathbf{x}}_{t}))$ and $\tilde{\boldsymbol{\mathbf{x}}}_{t+1}^{{}^{\prime}}={f}^{{}^{\prime}}(\tilde{\boldsymbol{\mathbf{x}}}_{t},\boldsymbol{\mathbf{u}}(\tilde{\boldsymbol{\mathbf{x}}}_{t}))$. The initial state, the auxiliary state, and the predicted evolution of these two states are then combined into a tuple $(\boldsymbol{\mathbf{x}}_{t},\tilde{\boldsymbol{\mathbf{x}}}_{t},{\boldsymbol{\mathbf{x}}}^{{}^{\prime}}_{t+1},\tilde{\boldsymbol{\mathbf{x}}}_{t+1}^{{}^{\prime}})$. The collection of all such tuples over each $\boldsymbol{\mathbf{x}}_{t}\in\mathcal{D}$ form the augmented data set $\mathcal{D}^{\prime}$.

Now, we want to evaluate the contracting behavior of the controller $\boldsymbol{\mathbf{u}}(\boldsymbol{\mathbf{x}}_{t})$ through the learned model on the augmented data set $\mathcal{D}^{\prime}$. Thus, we seek to enforce condition (6) for the elements of $\mathcal{D}^{\prime}$

with respect to a contraction metric $\boldsymbol{\mathbf{\Theta}}(\boldsymbol{\mathbf{x}}_{t})$. Contractive behavior is illustrated in Figure 2, showing the weighted distance between $\tilde{\boldsymbol{\mathbf{x}}}_{t}$ and $\boldsymbol{\mathbf{x}}_{t}$ decays as the system evolves to $\tilde{\boldsymbol{\mathbf{x}}}_{t+1}^{{}^{\prime}}$ and ${\boldsymbol{\mathbf{x}}}^{{}^{\prime}}_{t+1}$. We evaluate the approximate contraction condition only at the states $\boldsymbol{\mathbf{x}}_{t}$ that exist in the data set $\mathcal{D}$. This is due to the fact that the dynamics model is learned from $\mathcal{D}$ and hence ${f}^{{}^{\prime}}$ is expected to behave the most accurately at these points, which in turn will increase the quality of the learned policy. This will enforce contractive behavior with respect to the learned dynamics model $f^{\prime}$. Later we will discuss how we can ensure contractive behavior of the closed-loop behavior of the true dynamics model $f$.

Since in general, the contraction metric $\boldsymbol{\mathbf{\Theta}}(\boldsymbol{\mathbf{x}})$ is not known, and it is directly coupled to the control policy, we propose a learning-based approach to jointly learn both the control policy and the metric with respect to which the policy exhibits contraction. We refer to such a policy as a deep contraction policy. To this end, let us start by assuming that we know a control policy $\boldsymbol{\mathbf{u}}(\boldsymbol{\mathbf{x}})$ that makes ${f}^{{}^{\prime}}(\boldsymbol{\mathbf{x}},\boldsymbol{\mathbf{u}}(\boldsymbol{\mathbf{x}}))$ contractive. Consider now that we want to learn a corresponding contraction metric. Let this contraction metric be represented by a model $\hat{\boldsymbol{\mathbf{\Theta}}}(\boldsymbol{\mathbf{x}};\boldsymbol{\mathbf{w}}_{\Theta})$ which is parameterized by weights $\boldsymbol{\mathbf{w}}_{\Theta}$. We then obtain the best parameters of this contraction metric, denoted by $\boldsymbol{\mathbf{w}}_{\Theta}^{\star}$, from

where

The term $\|\hat{\boldsymbol{\mathbf{\Theta}}}({\boldsymbol{\mathbf{x}}}^{{}^{\prime}}_{t+1};\boldsymbol{\mathbf{w}}_{\Theta})(\tilde{\boldsymbol{\mathbf{x}}}_{t+1}^{{}^{\prime}}-{\boldsymbol{\mathbf{x}}}^{{}^{\prime}}_{t+1})\|-\|\hat{\boldsymbol{\mathbf{\Theta}}}(\boldsymbol{\mathbf{x}}_{t};\boldsymbol{\mathbf{w}}_{\Theta})\Delta\boldsymbol{\mathbf{x}}_{t}\|$ is an approximate measure of the contraction condition (11) which ideally should be negative for all elements of $\mathcal{D}^{\prime}$. Since enforcing (11) directly results in a non-differentiable optimization, we minimize (5) as a proxy for (11). Note that $L_{\boldsymbol{\mathbf{\Theta}}}$ is computed over all data points in $\mathcal{D}^{\prime}$. When paired with differentiable contraction metric $\hat{\boldsymbol{\mathbf{\Theta}}}(\boldsymbol{\mathbf{x}};\boldsymbol{\mathbf{w}}_{\Theta})$, the choice of loss function (5) is differentiable and is amenable to gradient decent optimization.

Now, let’s consider the more general case where both the policy and its contraction metric are unknown. We want to learn both the state-feedback policy and the contraction metric together. We want to learn a control policy represented by a function approximator $\hat{\boldsymbol{\mathbf{u}}}(\boldsymbol{\mathbf{x}};\boldsymbol{\mathbf{w}}_{\boldsymbol{\mathbf{u}}})$, parameterized by weights $\boldsymbol{\mathbf{w}}_{\boldsymbol{\mathbf{u}}}$, such that the closed-loop system is contractive with respect to the metric model $\hat{\boldsymbol{\mathbf{\Theta}}}$. To achieve this, we propagate the initial data points in $\mathcal{D}^{\prime}$ with the control policy model $\hat{\boldsymbol{\mathbf{u}}}(\boldsymbol{\mathbf{x}};\boldsymbol{\mathbf{w}}_{\boldsymbol{\mathbf{u}}})$ as ${\boldsymbol{\mathbf{x}}}^{{}^{\prime}}_{t+1}={f}^{{}^{\prime}}(\boldsymbol{\mathbf{x}}_{t},\hat{\boldsymbol{\mathbf{u}}}(\boldsymbol{\mathbf{x}}_{t};\boldsymbol{\mathbf{w}}_{\boldsymbol{\mathbf{u}}}))$ and $\tilde{\boldsymbol{\mathbf{x}}}_{t+1}^{{}^{\prime}}={f}^{{}^{\prime}}(\tilde{\boldsymbol{\mathbf{x}}}_{t},\hat{\boldsymbol{\mathbf{u}}}(\tilde{\boldsymbol{\mathbf{x}}}_{t};\boldsymbol{\mathbf{w}}_{\boldsymbol{\mathbf{u}}}))$.

We obtain the parameters of the contraction metric $\boldsymbol{\mathbf{w}}_{\Theta}$, denoted by $\boldsymbol{\mathbf{w}}_{\Theta}^{\star}$, and the parameters of the control policy $\boldsymbol{\mathbf{w}}_{\boldsymbol{\mathbf{u}}}$, denoted by $\boldsymbol{\mathbf{w}}_{\boldsymbol{\mathbf{u}}}^{\star}$, by minimizing a loss function $L_{u}$ over the data set $\mathcal{D}^{\prime}$

where

Loss function (5) ensures that the region of interest $\mathcal{X}$ is contractive with respect to $\hat{\boldsymbol{\mathbf{\Theta}}}$ and the learned dynamics model $f^{\prime}$. However, so far there has been no mechanism to ensure that the unique equilibrium of the contractive system is indeed the desired reference value $\boldsymbol{\mathbf{x}}^{r}$. To alleviate this, we need the learning process to be aware of the desired reference value, which we would like to be the equilibrium of the contraction region. The measure of awareness that we introduce is based on the ability of the controller $\hat{\boldsymbol{\mathbf{u}}}(\boldsymbol{\mathbf{x}};\boldsymbol{\mathbf{w}}_{\boldsymbol{\mathbf{u}}})$ to steer the system from an initial state $\boldsymbol{\mathbf{x}}_{0}\in\mathcal{X}$ to the desired state value $\boldsymbol{\mathbf{x}}^{r}$ in $k$ time steps, i.e. how close ${\boldsymbol{\mathbf{x}}}^{{}^{\prime}}_{k}$ gets to $\boldsymbol{\mathbf{x}}^{r}$. Therefore, to enforce the system’s states to contract to $\boldsymbol{\mathbf{x}}^{r}$, we add another penalty term to our loss function to obtain the final loss function utilized for learning the policy and contraction metric:

where

is the tracking loss with $\alpha\in_{>0}$ as the penalty factor. Here, ${\boldsymbol{\mathbf{x}}}^{{}^{\prime}}_{k}(\boldsymbol{\mathbf{x}}_{0})$ is the $k^{\text{th}}$ state value of the process ${\boldsymbol{\mathbf{x}}}^{{}^{\prime}}_{t+1}={f}^{{}^{\prime}}({\boldsymbol{\mathbf{x}}}^{{}^{\prime}}_{t},\hat{\boldsymbol{\mathbf{u}}}({\boldsymbol{\mathbf{x}}}^{{}^{\prime}}_{t};\boldsymbol{\mathbf{w}}_{\boldsymbol{\mathbf{u}}}))$, initialized at ${\boldsymbol{\mathbf{x}}}^{{}^{\prime}}_{0}=\boldsymbol{\mathbf{x}}_{0}$ where $\boldsymbol{\mathbf{x}}_{0}$ is drawn from a countable set $\mathcal{Y}\in\mathcal{X}$. The number of time steps $k$ is set by the designer and, as the reader may infer, affects the transient behavior of the closed-loop system.

We evaluate the performance of the contraction policies in a set of goal-reaching robotic tasks by comparing our method against a number of offline control methods suitable for systems with learned dynamics models. In particular, we compare our framework with the following algorithms:

1. 1.

   MPC: An iterative Linear Quadratic Controller (iLQR) as described in [34] ran in a receding horizon fashion.

2. 2.

   Learning without contraction: To evaluate the effectiveness of the contraction penalty, we further evaluate the robot’s performance in the absence of any contraction terms in the loss function.

3. 3.

   Reinforcement Learning: We also use the state-of-the-art offline RL method Conservative Q-Learning (CQL) [35] for further comparisons.

We evaluate the performance of our approach on two different robotic settings involving nonlinear dynamical systems of varying complexity represented by neural networks. The dynamics of these systems have closed-form expressions, but it is assumed that we do not have access to such expressions. We assume that we only have access to a set of system trajectories and learn a dynamics model from the state-action trajectories. The learned dynamics are represented as neural networks to the model-based control methods: deep contraction policy, MPC controller, and contraction-free learning. The RL implementation develops the policy directly from the same offline data set that is used to train the dynamics model in a model-free fashion. This allows us to implement our algorithm on the learned systems while having an analytical baseline to compare against to quantify robustness. Additionally, we consider state and control sets $\mathcal{X},\mathcal{U}$ defined by box constraints in order to constrain the size of the training data. Clearly for such constraints, $\mathcal{X}$ is convex. The dynamical systems we have chosen for our performance evaluation are as follows:

1. 1.

   2D Planar Car: A planar vehicle that is capable of controlling its acceleration, $\alpha$, and angular velocity, $\omega$. Here $\boldsymbol{\mathbf{x}}:=[p_{x},p_{y},\theta,v]$ and $\boldsymbol{\mathbf{u}}:=[\alpha,\omega]$ where $p_{x},p_{y}$ are the planar positions, $v$ is the velocity, and $\theta$ is the heading angle. The system dynamics are governed by: $\dot{\boldsymbol{\mathbf{x}}}=[v\cos(\theta),v\sin(\theta),\omega,\alpha]^{\top}$.

2. 2.

   3D Drone: An adaptation of a drone model that is given by [11] and [36]. This model describes an aerial vehicle capable of directly controlling the rate of change of it’s normalized thrust $\dot{F}$, and Euler Angles, $\dot{\phi},\dot{\theta},\dot{\psi}$. Here $\boldsymbol{\mathbf{x}}:=[p_{x},p_{y},p_{z},v_{x},v_{y},v_{z},F,\phi,\theta,\psi]$ and $\boldsymbol{\mathbf{u}}:=[\dot{\phi},\dot{\theta},\dot{\psi}]$ where $p_{i},v_{i}$ are the translational positions and velocities along the $i_{\text{th}}$ axis, respectively. Omitting the first order integrators in $p_{i},F,\phi,\theta,\psi$ for brevity, the dynamics can then be expressed as $[\dot{v}_{x},\dot{v}_{y},\dot{v}_{z}]=[-F\sin(\theta),F\cos(\theta)\sin(\phi),g-\cos(\theta)\cos(\phi)]$, where $g$ is the acceleration due to gravity.

For both systems we assume a timestep of $\Delta t=0.1$s and a final time of $T=10$s.

In this paper, we established a framework for learning a converging control policy for an unknown system from offline data. We leveraged Contraction theory and proposed a data augmentation method for encoding the contraction conditions directly into the loss function. We jointly learned the control policy and its corresponding contraction metric. We compared our method with several state-of-the-art control algorithms and showed that our method provides faster convergence, a smaller tracking error, lower variance of trajectories. For our future work, we would like to extend the current work to develop the stochastic confidence bound of our control design approach design.

• Lemma 1

  We ground our error analysis on the training error of the tuples $(\boldsymbol{\mathbf{x}}_{t},\boldsymbol{\mathbf{u}}_{t})\in\mathcal{D}$ and propagate the error to the general state and control tuples $(\boldsymbol{\mathbf{x}},\boldsymbol{\mathbf{u}})\in\mathcal{X}\times\mathcal{U}$.

  	$\displaystyle\|f(\boldsymbol{\mathbf{x}},\boldsymbol{\mathbf{u}})-{f}^{{}^{\prime}}(\boldsymbol{\mathbf{x}},\boldsymbol{\mathbf{u}})\|\leq\|f(\boldsymbol{\mathbf{x}}_{t},\boldsymbol{\mathbf{u}}_{t})-{f}^{{}^{\prime}}(\boldsymbol{\mathbf{x}}_{t},\boldsymbol{\mathbf{u}}_{t})\|+$
  	$\displaystyle\|(f(\boldsymbol{\mathbf{x}},\boldsymbol{\mathbf{u}})-{f}^{{}^{\prime}}(\boldsymbol{\mathbf{x}},\boldsymbol{\mathbf{u}}))-(f(\boldsymbol{\mathbf{x}}_{t},\boldsymbol{\mathbf{u}}_{t})-{f}^{{}^{\prime}}(\boldsymbol{\mathbf{x}}_{t},\boldsymbol{\mathbf{u}}_{t}))\|\leq$
  	$\displaystyle\|f(\boldsymbol{\mathbf{x}}_{t},\boldsymbol{\mathbf{u}}_{t})-{f}^{{}^{\prime}}(\boldsymbol{\mathbf{x}}_{t},\boldsymbol{\mathbf{u}}_{t})\|+\mathsf{L}_{f-{f}^{{}^{\prime}}}\left\|\begin{bmatrix}\boldsymbol{\mathbf{x}}\\ \boldsymbol{\mathbf{u}}\end{bmatrix}-\begin{bmatrix}\boldsymbol{\mathbf{x}}_{t}\\ \boldsymbol{\mathbf{u}}_{t}\end{bmatrix}\right\|.$

  The first and the second inequalities are obtained by adding and subtracting the terms $f(\boldsymbol{\mathbf{x}}_{t},\boldsymbol{\mathbf{u}}_{t})$ and ${f}^{{}^{\prime}}(\boldsymbol{\mathbf{x}}_{t},\boldsymbol{\mathbf{u}}_{t})$, and also using the norm and Lipschitz constant properties. If we define $E(\boldsymbol{\mathbf{x}}_{t},\boldsymbol{\mathbf{u}}_{t},\boldsymbol{\mathbf{x}},\boldsymbol{\mathbf{u}})$ as the right hand side of the second inequality, then $\underset{(\boldsymbol{\mathbf{x}},\boldsymbol{\mathbf{u}})\in\mathcal{X}\times\mathcal{U}}{\textit{max}}\,\,\underset{(\boldsymbol{\mathbf{x}}_{t},\boldsymbol{\mathbf{u}}_{t})\in\mathcal{D}}{\textit{min}}\,\,E(\boldsymbol{\mathbf{x}}_{t},\boldsymbol{\mathbf{u}}_{t},\boldsymbol{\mathbf{x}},\boldsymbol{\mathbf{u}})\leq\varepsilon$ where

  $$\varepsilon=\underset{(\boldsymbol{\mathbf{x}}_{t},\boldsymbol{\mathbf{x}}_{t+1},\boldsymbol{\mathbf{u}}_{t})\in\mathcal{D}}{\textit{max}}\left\|\boldsymbol{\mathbf{x}}_{t+1}-{f}^{{}^{\prime}}(\boldsymbol{\mathbf{x}}_{t},\boldsymbol{\mathbf{u}}_{t})\right\|+\mathsf{L}_{f-{f}^{{}^{\prime}}}\mathsf{D},$$

  which concludes the proof. $\Box$

• Proposition 2

  We want to derive a sufficient condition which ensures that contraction condition (11) holds for the true dynamics model. Using the learned dynamics model, the left-hand side of (11) for $\boldsymbol{\mathbf{x}}_{t}\in\mathcal{X}$ can be bounded for the true dynamics as

  	$\displaystyle\|\hat{\boldsymbol{\mathbf{\Theta}}}(\boldsymbol{\mathbf{x}}_{t+1})(\tilde{\boldsymbol{\mathbf{x}}}_{t+1}-\boldsymbol{\mathbf{x}}_{t+1})\|-\|\hat{\boldsymbol{\mathbf{\Theta}}}(\boldsymbol{\mathbf{x}}_{t})\Delta\boldsymbol{\mathbf{x}}_{t}\|\leq$
  	$\displaystyle\|\hat{\boldsymbol{\mathbf{\Theta}}}({\boldsymbol{\mathbf{x}}}^{{}^{\prime}}_{t+1})(\tilde{\boldsymbol{\mathbf{x}}}_{t+1}^{{}^{\prime}}-{\boldsymbol{\mathbf{x}}}^{{}^{\prime}}_{t+1})\|-\|\hat{\boldsymbol{\mathbf{\Theta}}}(\boldsymbol{\mathbf{x}}_{t})\Delta\boldsymbol{\mathbf{x}}_{t}\|+$
  	$\displaystyle\|(\hat{\boldsymbol{\mathbf{\Theta}}}(\boldsymbol{\mathbf{x}}_{t+1})-\hat{\boldsymbol{\mathbf{\Theta}}}({\boldsymbol{\mathbf{x}}}^{{}^{\prime}}_{t+1}))(\tilde{\boldsymbol{\mathbf{x}}}_{t+1}^{{}^{\prime}}-{\boldsymbol{\mathbf{x}}}^{{}^{\prime}}_{t+1})\|+$
  	$\displaystyle\|\hat{\boldsymbol{\mathbf{\Theta}}}({\boldsymbol{\mathbf{x}}}^{{}^{\prime}}_{t+1})((\tilde{\boldsymbol{\mathbf{x}}}_{t+1}-\tilde{\boldsymbol{\mathbf{x}}}_{t+1}^{{}^{\prime}})-(\boldsymbol{\mathbf{x}}_{t+1}-{\boldsymbol{\mathbf{x}}}^{{}^{\prime}}_{t+1}))\|+$
  	$\displaystyle\|(\hat{\boldsymbol{\mathbf{\Theta}}}(\boldsymbol{\mathbf{x}}_{t+1})-\hat{\boldsymbol{\mathbf{\Theta}}}({\boldsymbol{\mathbf{x}}}^{{}^{\prime}}_{t+1}))((\tilde{\boldsymbol{\mathbf{x}}}_{t+1}-\tilde{\boldsymbol{\mathbf{x}}}_{t+1}^{{}^{\prime}})-(\boldsymbol{\mathbf{x}}_{t+1}-{\boldsymbol{\mathbf{x}}}^{{}^{\prime}}_{t+1}))\|,$

  where $\boldsymbol{\mathbf{x}}_{t+1}=f(\boldsymbol{\mathbf{x}}_{t},\hat{\boldsymbol{\mathbf{u}}}(\boldsymbol{\mathbf{x}}_{t}))$, $\tilde{\boldsymbol{\mathbf{x}}}_{t+1}=f(\tilde{\boldsymbol{\mathbf{x}}}_{t},\hat{\boldsymbol{\mathbf{u}}}(\tilde{\boldsymbol{\mathbf{x}}}_{t}))$, ${\boldsymbol{\mathbf{x}}}^{{}^{\prime}}_{t+1}={f}^{{}^{\prime}}(\boldsymbol{\mathbf{x}}_{t},\hat{\boldsymbol{\mathbf{u}}}(\boldsymbol{\mathbf{x}}_{t}))$, and $\tilde{\boldsymbol{\mathbf{x}}}_{t+1}^{{}^{\prime}}={f}^{{}^{\prime}}(\tilde{\boldsymbol{\mathbf{x}}}_{t},\hat{\boldsymbol{\mathbf{u}}}(\tilde{\boldsymbol{\mathbf{x}}}_{t}))$. The inequality holds due to addition and subtraction of proper terms and norm properties. The inequality can be further simplified using the Frobenius norm of the contraction metric $\hat{\boldsymbol{\mathbf{\Theta}}}$. Since, by assumption, the entries of the contraction metric are bounded by $\gamma$, we have $\|\hat{\boldsymbol{\mathbf{\Theta}}}(\boldsymbol{\mathbf{x}})\|_{F}\leq n\gamma$. Having an upper bound estimate of the Lipschitz constant of entries of the contraction metric $\mathsf{L}_{\boldsymbol{\mathbf{\Theta}}_{ij}}$ and recalling that $\|(\tilde{\boldsymbol{\mathbf{x}}}_{t+1}^{{}^{\prime}}-{\boldsymbol{\mathbf{x}}}^{{}^{\prime}}_{t+1})\|\leq\varepsilon$ from Lemma 1, leads to the result $\|(\hat{\boldsymbol{\mathbf{\Theta}}}(\boldsymbol{\mathbf{x}}_{t+1})-\hat{\boldsymbol{\mathbf{\Theta}}}({\boldsymbol{\mathbf{x}}}^{{}^{\prime}}_{t+1}))\|_{F}\leq\varepsilon\sqrt{\underset{ij}{\sum}\mathsf{L}_{\boldsymbol{\mathbf{\Theta}}_{ij}}^{2}}$. In addition, using the estimated Lipschitz constant $\mathsf{L}_{f-{f}^{{}^{\prime}}}$, we have that $\|((\tilde{\boldsymbol{\mathbf{x}}}_{t+1}-\tilde{\boldsymbol{\mathbf{x}}}_{t+1}^{{}^{\prime}})-(\boldsymbol{\mathbf{x}}_{t+1}-{\boldsymbol{\mathbf{x}}}^{{}^{\prime}}_{t+1}))\|\leq\mathsf{L}_{f-{f}^{{}^{\prime}}}\left\|\begin{bmatrix}\tilde{\boldsymbol{\mathbf{x}}}\\ \boldsymbol{\mathbf{u}}(\tilde{\boldsymbol{\mathbf{x}}})\end{bmatrix}-\begin{bmatrix}\boldsymbol{\mathbf{x}}\\ \boldsymbol{\mathbf{u}}(\boldsymbol{\mathbf{x}})\end{bmatrix}\right\|$. Now, using the Lipschitz constant of $\boldsymbol{\mathbf{u}}(\boldsymbol{\mathbf{x}})$ as $\mathsf{L}_{\boldsymbol{\mathbf{u}}}$, we have that $\|((\tilde{\boldsymbol{\mathbf{x}}}_{t+1}-\tilde{\boldsymbol{\mathbf{x}}}_{t+1}^{{}^{\prime}})-(\boldsymbol{\mathbf{x}}_{t+1}-{\boldsymbol{\mathbf{x}}}^{{}^{\prime}}_{t+1}))\|\leq\mathsf{L}_{f-{f}^{{}^{\prime}}}\lambda(1+\mathsf{L}_{\boldsymbol{\mathbf{u}}})$. Finally, we can write the following inequality:

  		$\displaystyle\|\hat{\boldsymbol{\mathbf{\Theta}}}(\boldsymbol{\mathbf{x}}_{t+1})(\tilde{\boldsymbol{\mathbf{x}}}_{t+1}-\boldsymbol{\mathbf{x}}_{t+1})\|-\|\hat{\boldsymbol{\mathbf{\Theta}}}(\boldsymbol{\mathbf{x}}_{t})\Delta\boldsymbol{\mathbf{x}}_{t}\|\leq$
  		$\displaystyle\|\hat{\boldsymbol{\mathbf{\Theta}}}({\boldsymbol{\mathbf{x}}}^{{}^{\prime}}_{t+1})(\tilde{\boldsymbol{\mathbf{x}}}_{t+1}^{{}^{\prime}}-{\boldsymbol{\mathbf{x}}}^{{}^{\prime}}_{t+1})\|-\|\hat{\boldsymbol{\mathbf{\Theta}}}(\boldsymbol{\mathbf{x}}_{t})\Delta\boldsymbol{\mathbf{x}}_{t}\|+$
  		$\displaystyle\lambda(\varepsilon\tau\mathsf{L}_{f^{{}^{\prime}}_{u}}+(\varepsilon\tau+n\gamma)\mathsf{L}_{f-{f}^{{}^{\prime}}}(1+\mathsf{L}_{\boldsymbol{\mathbf{u}}})),$		(18)

  where $\tau=\sqrt{\underset{ij}{\sum}\mathsf{L}_{\boldsymbol{\mathbf{\Theta}}_{ij}}^{2}}$. With Lipschitz constant $\mathsf{L}_{C}$, we can derive an upper bound for $C_{{f}^{{}^{\prime}}(\boldsymbol{\mathbf{x}}_{t},\hat{\boldsymbol{\mathbf{u}}}(\boldsymbol{\mathbf{x}}_{t}))}(\boldsymbol{\mathbf{x}},\Delta\boldsymbol{\mathbf{x}}_{t})$, $\boldsymbol{\mathbf{x}}_{t}\in\mathcal{X}$ and $\Delta\boldsymbol{\mathbf{x}}_{t}\in\Delta_{t}$, such that $C_{{f}^{{}^{\prime}}(\boldsymbol{\mathbf{x}},\hat{\boldsymbol{\mathbf{u}}}(\boldsymbol{\mathbf{x}}))}(\boldsymbol{\mathbf{x}}_{t},\Delta\boldsymbol{\mathbf{x}}_{t})<\zeta$ where

  $$\zeta=\underset{\boldsymbol{\mathbf{x}}\in\mathcal{X}}{\textit{max}}\,\,\underset{\boldsymbol{\mathbf{x}}_{t}\in\mathcal{D}}{\textit{min}}\,\,C_{{f}^{{}^{\prime}}(\boldsymbol{\mathbf{x}}_{t},\hat{\boldsymbol{\mathbf{u}}}(\boldsymbol{\mathbf{x}}_{t}))}(\boldsymbol{\mathbf{x}}_{t},\Delta\boldsymbol{\mathbf{x}}_{t})+\mathsf{L}_{C}\|\boldsymbol{\mathbf{x}}_{t}-\boldsymbol{\mathbf{x}}\|$$

  . Finally by taking the expectation on Equation (Proposition 2), we get

  	$\displaystyle\mathbb{E}_{\Delta\boldsymbol{\mathbf{x}}_{t}}\big{(}\|\hat{\boldsymbol{\mathbf{\Theta}}}(\boldsymbol{\mathbf{x}}_{t+1})(\tilde{\boldsymbol{\mathbf{x}}}_{t+1}-\boldsymbol{\mathbf{x}}_{t+1})\|-\|\hat{\boldsymbol{\mathbf{\Theta}}}(\boldsymbol{\mathbf{x}}_{t})\Delta\boldsymbol{\mathbf{x}}_{t}\|\big{)}\leq$
  	$\displaystyle\zeta+\lambda(\varepsilon\tau\mathsf{L}_{f^{{}^{\prime}}_{u}}+(\varepsilon\tau+n\gamma)\mathsf{L}_{f-{f}^{{}^{\prime}}}(1+\mathsf{L}_{\boldsymbol{\mathbf{u}}}))$

  which concludes the proof. $\Box$