Optimizing Closed-Loop Performance with Data from Similar Systems:
A Bayesian Meta-Learning Approach

Performance-driven optimization of closed-loop control systems often involves optimizing system-specific performance functions. Some performance functions lack a complete mathematical representation, or exhibit such complex models to represent accurately, that standard model-based techniques cannot be deployed without oversimplifying assumptions or introducing conservativeness in the design. As argued in [1, 2], Bayesian optimization (BO) possesses some clear advantages that have led to its adoption for closed-loop performance optimization [3, 4, 5, 6]. Concretely, BO is a gradient-free, global optimization method that works well in data-limited settings while concurrently learning a probabilistic approximation of the underlying performance function from data. This has led BO to be implemented in a range of engineering applications, including energy systems [7, 8, 9], vehicles [10], robotics [11], and aerospace [12].

In the performance-driven BO literature, a standard assumption is that the target system to be optimized is available for closed-loop experimentation. Therefore, at design time, one can evaluate the performance of the target system by changing some parameters such as controller gains or setpoints, and observing the system response. Often, at the start of the BO tuning procedure, the amount of data available from the target system is very limited. Approximating a performance function using very limited data generally results in an inaccurate surrogate model, which needs to be improved in subsequent BO iterations by exploring the search space [13]. Though data available from the target system under consideration may be scarce, in many industrial applications there is often data collected from previous performance optimization tasks on ‘similar systems’ [14]. We posit that this previously collected data, combined with the limited target system data, can warm-start BO by estimating a good initial surrogate model, potentially improving the convergence of BO for the target task.

To this end, we propose the use of few-shot meta-learning (or just meta-learning for brevity), wherein a machine learning model is trained on a variety of related ‘source’ optimization tasks so that it can make accurate predictions for a new target task, in spite of the paucity of target task data. Modern meta-learning algorithms typically require solving a bi-level optimization problem, where the outer level is dedicated to extracting task-independent features across a set of source tasks, and the inner level is devoted to adapting to a specific optimization task with few iterations and limited data. This bi-level training loop is the basis of model-agnostic meta-learning (MAML) [15]. However, MAML is not always easy to train and often exhibits numerical instabilities [16]. In addition, MAML does not generate uncertainty estimates around the predictions since it is a deterministic algorithm. Consequently, we adopt a Bayesian meta-learning approach in this work, which builds on the methodology proposed in [17, 18, 19]. To the best of our knowledge, this paper is the first to propose a Bayesian meta-learning algorithm for closed-loop system optimization capable of leveraging data from similar systems.

Our main contribution in this work is to propose a meta-learned Bayesian optimization methodology for rapidly optimizing closed-loop performance of systems with dynamics and performance functions whose mathematical representations are unknown, but can be evaluated by experiment/simulation. Specifically, we employ Bayesian meta-learning to systematically learn using data collected from a variety of systems that are different to a target system to be optimized, and leverage this disparate dataset to generate a good surrogate model for the target system performance function, despite having very few data samples from the target system. Our Bayesian meta-learning framework is implemented using deep kernel networks (DKNs) which have the advantage of requiring single-level optimization for training as the inner training loop is replaced by a Gaussian process (GP) base kernel. Additionally, our approach is simple to implement using standard deep learning toolkits such as PyTorch. The choice of GPs as the base learner lends itself to classical BO, where the default choice of probabilistic surrogate model is a GP. We demonstrate via simulation experiments that this performance-driven meta-learned BO can generate near-optimal solutions with fewer online experiments than classical BO for systems with unmodeled dynamics, without requiring explicit parameter estimation. While meta-learning has recently been explored in the context of adaptive control in [20, 21, 22], our work differs from these works in a few key aspects: (i) we propose a framework for meta-learning to optimize closed-loop performance rather than controller gain adaptation for tracking/regulation, and (ii) we adopt Bayesian deep learning for meta-learning and instead of adaptive controllers; and, (iii) we do not assume any modeling knowledge on the closed-loop dynamics, controller structure, or performance function.

The rest of the paper is organized as follows. The closed-loop performance optimization problem is formalized in Section 2 and the meta-learning setup is explained therein. In Section 3, we provide a brief overview of BO and deep kernel networks, and describe how they are used for meta-learned BO in Section 4. We demonstrate the effectiveness of the proposed approach using a numerical example in Section 5, and conclude in Section 6.

We consider a class of stable closed-loop systems of the form

where $x,x_{+}\in\mathbb{R}^{n_{x}}$ denote the current and updated state of the system, respectively, $r\in\mathcal{R}\subset\mathbb{R}^{n_{r}}$ is a control parameter to be tuned (e.g., a setpoint or controller gain), and $\theta\in\Theta\subset\mathbb{R}^{n_{\theta}}$ denotes unknown system parameters. The admissible sets of control parameters and system parameters: $\mathcal{R}$ and $\Theta$, respectively, are known. Additionally, we assume that for each $\theta\in\Theta$ and each $r\in\mathcal{R}$, the closed-loop system (1) is globally asymptotically stable to a parameter-dependent equilibrium state $x^{\infty}(r,\theta)$, and that the map $x^{\infty}(\cdot,\cdot)$ is continuous on $\mathcal{R}\times\Theta$.

To determine how the closed-loop system performs, we define a continuous function $J:\mathcal{R}\times\Theta\to\mathbb{R}$. The performance output $J(r,\theta)$ is available for measurement. Note that for our purposes, it suffices that $f$ and $J$ are smooth and can be evaluated either by simulation or experiments—however, we do not assume access to a mathematical representation of these functions, nor can we compute their gradients analytically. In other words, no knowledge of $f$ or $J$ is used in the algorithm. However, $J$ is assumed to lie in the reproducing kernel Hilbert space (RKHS) of common (e.g., Gaussian or Matérn) kernels so that it can be approximated reasonably by kernel methods.

For a target system of the form (1) with unknown system parameter $\theta$, our target optimization task is to compute the control parameter

that optimizes the performance of the closed-loop system (1) directly from data. We will do this without estimating $\theta$, which would prove difficult since $f$ is unknown, and $x$ is not measured.

In practical applications that involve optimization of closed-loop performance, it is often the case that the optimization task has been performed before, most likely for similar, but not necessarily identical, systems. While solving these source optimization tasks, one has likely generated optimization-relevant data that can prove valuable to the target optimization task, even if the target system is not exactly the same as one of the systems encountered before. In BO-based tuning, such source data (data obtained while solving source optimization tasks) is typically ignored, and the target system is optimized from scratch [1]. This can lead to slow convergence of BO since the algorithm’s initial estimate of the target performance function, constructed with very few target data points, is often inaccurate or highly uncertain. In this work, we propose meta-learning from similar systems’ source data and integrating the information extracted from the source data to the target optimization task. In the sequel, we will show that meta-learning from disparate sources of data can enable optimization of closed-loop performance in a few-shot manner (i.e., with very limited target system data) and thereby lead to significant acceleration of the target optimization procedure.

Concretely, we assume that a dataset $\mathcal{D}^{S}\triangleq\{\mathcal{D}^{S}_{k}\}_{k=1}^{N_{S}}$ is available, comprising data collected from $N_{S}\in\mathbb{N}$ source optimization tasks performed in the past. Here,

is a data sequence of $(r,J)$ pairs collected over $T_{k}\in\mathbb{N}$ optimization iterations from the $k$-th source task. Each $k$-th source data pair consists of the control parameter $r_{t,k}$, and the corresponding performance output $J(r_{t,k},\theta_{k})$ evaluated on a source system with dynamics modeled by (1) with system parameter $\theta_{k}\in\Theta$, where each $\theta_{k}$ is unique. We reiterate that we do not assume access to the source systems or the system parameters $\theta_{k}$. For the target task, our objective is to compute (2) by learning a variety of performance functions from the source dataset $\mathcal{D}^{S}$ and generating a good estimate of the performance function for the target optimization task, despite having access to a very limited target dataset

where $T\ll\min_{k}T_{k}$.

To this end, we propose the use of Bayesian meta-learning to learn a distribution of performance functions from the source data, and predict the target performance function by conditioning on the target data. With successful meta-learning, we expect that the estimated target task performance function will be significantly more accurate than an initial estimate that uses only the limited target dataset, and in turn, expect this to improve the convergence of BO. Our meta-learning framework comprises a deep kernel network (DKN). Deep kernel networks contain a latent encoder and a Gaussian process (GP) output layer. The latent encoder takes an input $r$ and transforms it to a latent variable of user-defined size, thereby lending itself to target systems with a large number of control parameters. Since the final layer is a GP regressor, it integrates seamlessly into classical BO procedures and well-studied acquisition functions can be used with the DKN without modification.

In this section, we explain how classical Bayesian optimization is typically used to solve (2) for the closed-loop system (1) for a fixed $\theta$, using only target task data. We also describe deep kernel networks that will be used in the next section for meta-learning.

Recall that $\mathcal{D}^{S}$ is a collection of data obtained from source tasks, where the data of the $k$-th source task is described in (3). Most recent Bayesian meta-learning paradigms [18, 19] work under the assumption that all the source tasks (and the target task) share a set of task-independent hidden parameters $\omega$ and $\gamma$ (the reasons for using two symbols here will be explained shortly), and a set of task-specific parameters $\{\rho_{k}\}_{k=1}^{N_{S}}$. Given an input $r_{*}$ from an unseen task, the Bayesian treatment for inference requires estimating the task-independent parameters $\omega$ and $\gamma$ from the source task dataset, and then estimating two distributions: the posterior of the task-specific parameters $\pi(\rho_{k}|r_{*},\mathcal{D}^{S},\omega,\gamma)$ and the posterior for the output $\pi(J_{*}|r_{*},\omega,\gamma)$. However, as argued in [17], handling these two distributions requires sampling or amortized distributions, resulting in significantly complex network architectures, and bi-level learning has been shown to require higher-order derivative information, in spite of which the training might suffer from instabilities. By using deep kernel networks, one can marginalize out the task-specific parameters with kernel functions defined in the latent space, which leads to a simple architecture (see Fig. 1), allows for quantification of uncertainty during inference, all the while avoiding the numerical conditioning issues that are reported to occur [16] during training via bi-level optimization (e.g., in MAML).

We consider the following nonlinear system, previously studied in [28, 29]:

where $\kappa(x,r)=-6x_{1}+(\theta_{1}-5)(x_{2}-\theta_{1}x_{1})+\theta_{2}x_{1}^{2}+r$, is a control policy that renders the closed-loop system stable. For the target system, the true parameters are $\theta=[2,5]$ and the optimal control parameter is $r^{\star}=-1.2$ at which the system attains its maximum $J^{\star}=1.2$; this is not known to the designer. We choose $\mathcal{R}=[-10,10]$ and $\Theta=[1,6]^{2}$.

In order to generate the source dataset, we extract $N_{S}=20$ unique system parameters from $\Theta$ using a 2D uniform random distribution. We evaluate the performance output $J$ for a given $r$ by forward simulating each system realized by sampling from $\Theta$ for $t_{f}=10~{}s$, which provides enough time for the transient response to fade. For each sampled system parameter, we solve the source optimization task of maximizing the performance function given in (9c) within $T_{k}\equiv 50$ iterations, which depends on $\theta$ and therefore has a variety of optimizers and optimal values. The source data collected is illustrated in Fig. 2[A] using colored circles joined by thin continuous lines. We observe that indeed the source performance functions are not identical to the target task performance function which is shown with the continuous thick black line. Our proposed approach is agnostic to the optimization strategy used for optimizing each source task; in fact, a mix of random search and BO are used for source task optimization and source data collection. The target task data is collected by evaluating the performance of the target system at $T=5$ randomly selected $r$ values.

Our DKN architecture involves a latent encoder with 100 neurons per hidden layer, with 4 hidden layers following the input layer. We select a latent dimension of 10, and the activation functions are rectified linear units (ReLUs). The latent output is passed to a GP with a scaled Matern-3/2 kernel as the covariance and a constant mean kernel. The entire pipeline is implemented using PyTorch [30] and GPyTorch [31]. The DKN is trained for 10000 iterations with the learning rate scheduled at: $\beta_{\omega}=10^{-3}$ and $\beta_{\gamma}=10^{-2}$ for the first 2000 iterations, and $\beta_{\omega}=10^{-4}$ and $\beta_{\gamma}=10^{-3}$ thereafter. We train using the Adam algorithm and fix the batch-size at 8. The weights corresponding to the highest log marginal likelihood was saved as $\omega^{\star}$ and $\gamma^{\star}$.

Fig. 2[B] illustrates the effectiveness of meta-learning for predicting the target performance function. With the same limited target data, the trained DKN generates a more accurate mean estimate of the true performance function, as compared to a GP trained for 1000 iterations with a squared-exponential kernel: see the blue (DKN) and orange (GP) continuous lines. Furthermore, it is encouraging that the uncertainty bands for the DKN are less pronounced than the GP in the areas that are most relevant for optimization.

Fig. 3 demonstrates an instance of DKN-BO for optimizing control performance. We start the target system at the origin and simulate the system forward in time, collecting performance outputs every 10 s for $r$ value candidates generated by the DKN-BO procedure. The state variation $x_{1}$ and $x_{2}$ shows that we do not assume that we can reset the experiment each time a new $r$ candidate is selected, and the closed-loop performance is optimized in an online manner. The corresponding variation of the control parameter $r$. The bottom right subplot, which shows the performance output, illustrates clearly that DKN-BO finds the optimal value within 100 s, which is comparable to the performance reported in [29], even though we do not have access to any model information and do not estimate $\theta$.

We also repeated our experiments 100 times with varying target datasets, and report the statistics of the simple regret decay in Fig. 4. We observe that DKN-BO performs well in a few-shot setting, and converges to a median regret that is two orders of magnitude below classical GP-BO within 10 iterations. Furthermore, the DKN-BO algorithm is robust to the target dataset as is evident from its tighter confidence intervals compared with GP-BO.

This paper provides a highly generalizable and computationally simple framework for leveraging prior data for optimizing unseen closed-loop performance optimization tasks, even if the prior data is not from the task to be optimized. This is done in a few-shot manner using meta-learned Bayesian optimization and deep kernel networks as surrogate models. The latent encoding of the DKN enables a task-independent representation of the controlled inputs to the closed-loop system, and the base kernel allows conditioning for task-specific predictions.