Non-Myopic Multi-Objective Bayesian Optimization

Many engineering design and scientific applications involve performing a sequence of experiments under a small-resource budget to optimize multiple expensive-to-evaluate objective functions. Consider the example of searching nanoporous materials optimized for hydrogen-powered vehicles. In this real-world problem, we need to perform physical lab experiments to make a candidate material and evaluate its volumetric and gravimetric hydrogen uptake properties. Our goal is to approximate the optimal Pareto front of solutions under a given small resource budget for experiments.

We consider the problem of finite-horizon sequential experimental design (SED) to solve multi-objective optimization (MOO) problems over expensive-to-evaluate objectives to find high-quality Pareto solutions. We solve this problem within the framework of Bayesian optimization (BO) Shahriari et al. [2015], which has been shown to be sample-efficient in practice. The key idea behind BO is to train a surrogate model (e.g., a Gaussian process) from past experimental data, and intelligently select the future experiments guided by an acquisition function. Most of the prior acquisition functions for BO are myopic in nature (i.e., focused on the immediate advancement of the goal). There are a few non-myopic acquisition functions for BO in the single-objective setting Jiang et al. [2020a], Lam and Willcox [2016]. These non-myopic strategies rely on formulating the optimal policy for SED as a dynamic program using the recursive Bellman optimality principle, and then solving for the optimal policy using methods with varying approximations (e.g., rollout strategies).

This work studies the design of non-myopic acquisition functions for MOO problems for the first time. The Bellman optimality principle does not hold in most MOO problems and requires the reward function of the underlying Markov decision process (MDP) to satisfy some conditions Roijers et al. [2013], Van Moffaert and Nowé [2014] 1) only scalar rewards; 2) monotonically increasing rewards with respect to better actions; and 3) additivity condition—for the optimality to hold. These conditions are challenging to satisfy in MOO problems and require a careful design of the reward function. We address this challenge by using hypervolume improvement (HVI) as our scalarization approach to define the reward function. Unlike the hypervolume function, hypervolume improvement satisfies the additivity condition. A key factor for these conditions to hold is to define optimality as Pareto optimality with respect to HVI rather than individual optimality for each objective function. This reward function allows us to use a lower-bound on the Bellman equation to approximately solve the finite-horizon SED problem using a batch utility function for MOO (e.g., qEHVI Daulton et al. [2020]). Our approach generalizes to other scalarization approaches that satisfy the above conditions.

We derive three non-myopic multi-objective (NMMO) acquisition functions (AFs) for MOO with varying speed and accuracy trade-offs: 1) the NMMO-Nested AF, based on exact computation of the lower bound and computationally intractable even for small horizons, 2) the NMMO-Joint AF, a scalable lower bound on the nested AF, and 3) the BINOM AF, a fast and approximate version of EHVI. Our experiments on real-world MOO problems demonstrate that the proposed non-myopic acquisition functions substantially improve performance over the baseline BO methods for MOO.

Contributions. The key contribution of our work is the development and evaluation of principled non-myopic acquisition functions to solve MOO problems. In particular, we

This section formally defines the problem setting and provides the background necessary for our solution approach.

Multi-objective optimization (MOO). Let $\mathfrak{X}\subset\mathbb{R}^{d}$ be the input space of $d$ design variables, where each candidate input $\mathbf{x}\in\mathfrak{X}$ is a $d$-dimensional vector. Let $\{f_{1},\cdots,f_{K}\}$ with $K\geq 2$ represent the black-box objective functions defined over the input space $\mathfrak{X}$, where $f_{1}(\mathbf{x}),\cdots,f_{K}(\mathbf{x}):\mathfrak{X}\rightarrow\mathbb{R}$. We denote the function evaluation at $\mathbf{x}$ as $\mathbf{y}=[y_{1},\cdots,y_{K}]$, where $y_{i}=f_{i}(\mathbf{x})$ for all $i\in\{1,\cdots,K\}$. For example, to optimize nanoporous materials for hydrogen-powered vehicles, $\mathbf{x}$ represents a candidate material, $f_{1}(\mathbf{x})$ and $f_{2}(\mathbf{x})$ corresponds to a physical lab experiment to evaluate the volumetric and gravimetric hydrogen uptake of the material. WLOG, we assume maximization for all $K$ objective functions.

An input $\mathbf{x}$ Pareto-dominates another input $\mathbf{x}^{\prime}$ if and only if $\forall i:f_{i}(\mathbf{x})\geq f_{i}(\mathbf{x}^{\prime})$ and $\exists j:f_{j}(\mathbf{x})>f_{j}(\mathbf{x}^{\prime})$. The optimal solution of the MOO problem is a set of inputs $\mathcal{X}^{*}\subset\mathfrak{X}$ such that no input $\mathbf{x}\in\mathfrak{X}\setminus\mathcal{X}^{*}$ Pareto-dominates another input $\mathbf{x}^{\prime}\in\mathcal{X}^{*}$. The set of input solutions $\mathcal{X}^{*}$ is called the optimal Pareto set, and the corresponding set of function values $\mathcal{Y}^{*}$ is called the optimal Pareto front. In SED, we select one input for evaluation in each iteration, and our goal is to uncover a high-quality Pareto front while minimizing the total number of expensive function evaluations. A typical measure to evaluate the quality of a Pareto front is the Pareto hypervolume (PHV) indicator Zitzler and Thiele [1999] of the Pareto front with respect to a reference point $\mathbf{r}$.

Non-myopic MOO problem. Our goal is to approximate the optimal Pareto front for a MOO problem within a maximum experimental budget $T$, which is typically small (i.e., finite-horizon SED). This problem can be formulated as:

where $P(\mathfrak{X})$ denotes the power set of $\mathfrak{X}$ and $X=\{\mathbf{x}_{1}\dots\mathbf{x}_{T}\}$ is the set of inputs selected for function evaluation. We solve this problem using a non-myopic decision policy, where, at each iteration, the algorithm selects one input for evaluation while reasoning about candidate experimental design plans within the budget $T$.

Overview of Bayesian optimization (BO). BO is an effective framework for solving expensive black-box optimization problems. BO has three main steps: (i) conducting an expensive experiment with an input selected by BO, (ii) updating the surrogate model by including the new evaluations, and (iii) selecting a new candidate input for the next function evaluation. The two key components of BO are the surrogate model and an acquisition function. Surrogate models are probabilistic models (typically Gaussian processes Williams and Rasmussen [2006]) that are trained on data from past function evaluations. A surrogate model is able to predict the output of unknown inputs without requiring an expensive experiment and to quantify the uncertainty of its predictions. The acquisition function (e.g., expected improvement (EI) Mockus [1989]) is used to decide which input to select for the next function evaluation. It uses the surrogate model to score the utility of selecting inputs for the next experiment by trading off exploitation and exploration. The goal is to quickly direct the search toward high-quality solutions with the least number of iterations. BO in a multi-objective setting requires particular consideration for modeling and acquisition function design. For MOO problems, we model the objective functions $f_{1},\cdots,f_{K}$ using $K$ independent GP models $\mathcal{GP}_{1},\cdots,\mathcal{GP}_{K}$ and discuss acquisition functions in Section 3.

Non-myopic decision policies for BO. We review important facts for optimal sequential decision-making Osborne and Osborne [2010]. Consider collecting $t$ observations $D_{t}$, and let $u$ denote a utility function defining a quality metric over the data $D_{t}$. The marginal gain in the utility of query $\mathbf{x}$ w.r.t. $D_{t}$ is expressed as

Consider the case where $T-t$ steps are remaining. The expected marginal gain for $T-t$-steps is expressed through the Bellman recursion Jiang et al. [2020a] as

The optimal expected policy to select the $T-t$ lookahead steps can be obtained by sequentially maximizing Equation 3 at each step

Being a sequence of $T-t$ nested integrals, solving the optimization problem in equation 4 to optimality is intractable for even a small value of $T-t$ Lam et al. [2016], Jiang et al. [2020b].

Recent work made a connection between non-myopic input selection and batch input selection Jiang et al. [2020a]. Assuming parallel evaluation, the expected marginal utility of a new batch of evaluations $Y=\{\mathbf{y}_{t},\cdots,\mathbf{y}_{T}\}$ given their associated batch of inputs $X=\{\mathbf{x}_{t},\cdots,\mathbf{x}_{T}\}$ is defined as:

Respectively, the maximum expected marginal utility of a batch of evaluations at input locations $X^{\prime}$ of size $|X^{\prime}|=T-t-1$ conditioned on the addition of an observation $(\mathbf{x},\mathbf{y})$ to the data is defined as

Comparing equations 3 and 5, the second term in equation 3 has an adaptive utility with a nested maximization while equation 5 has a joint batch utility with an outer maximization. This leads to a lower-bound connection between the two:

Consequently, we obtain a lower bound on the expected marginal gain defined by the Bellman recursion in equation 3

The lower bound on the Bellman equation (Equation 9) provides an efficient alternative for optimizing the sequential decision-making process in non-myopic single-objective BO Jiang et al. [2020a], where the utility function $u(D_{t})$ is defined as the incumbent, and the marginal utility $u(y|\mathbf{x},D_{t})$ as the incumbent improvement. However, applying the same strategy in the multi-objective BO (MOBO) setting is non-trivial. The Bellman equation is inherently single-dimensional with respect to the output of the utility function and, therefore, presents challenges when faced with the multi-output nature of MOBO, where conflicting objectives require simultaneous consideration. This necessitates an adapted formulation capable of capturing the trade-offs across multiple objectives and guiding the exploration and exploitation balance. The methodological challenges to address the extension of the Bellman equation for MOBO and the proposed solutions are described in Section 4.

Most of the prior work on acquisition functions in BO falls under the myopic category. There is less work on non-myopic acquisition functions with a focus on single-objective BO.

Myopic single-objective BO. Recent advances in single-objective BO have predominantly focused on refining and extending traditional acquisition functions. The most notable techniques Garnett [2023], Wang and Jegelka [2017], Hernández-Lobato et al. [2014], Ament et al. [2024] use Gaussian processes as surrogate models.

Non-myopic single-objective BO. Non-myopic approaches in single-objective BO Jiang et al. [2020a], Lam and Willcox [2016], González et al. [2016], Ginsbourger and Le Riche [2009], Osborne and Osborne [2010], Osborne et al. [2009] represent an emerging direction aimed at overcoming the limitations of myopic acquisition functions, which typically optimize for immediate gains. These lookahead methods consider the future impact of current decisions Wu and Frazier [2019], Lee et al. [2021], thereby enhancing long-term optimization outcomes when the experimental resource budget is small.

Multi-objective BO. When compared to single-objective BO, multi-objective BO (MOBO) has received relatively less attention. The predominant methods for designing MOBO strategies include information-theoretic methods Tu et al. [2022], Hernández-Lobato et al. [2016], Belakaria et al. [2019], Suzuki et al. [2020], Garrido-Merchán and Hernández-Lobato [2021], uncertainty based methods Belakaria et al. [2020b], hypervolume-based methods Konakovic Lukovic et al. [2020], Daulton et al. [2020], Ahmadianshalchi et al. [2024a], and scalarization-based methods Daulton et al. [2022], Knowles [2006]. The joint entropy search for MOBO (JESMO) algorithm Tu et al. [2022] aims to maximize information gain about the optimal Pareto front by evaluating inputs that maximally reduce its uncertainty. The expected hypervolume improvement (EHVI) method Daulton et al. [2020] extends the concept of EI from a single-objective to a MOO problem by assessing the potential improvement in hypervolume from new input evaluations. Several efforts to extend these approaches to handle constraints and preferences have resulted in more general frameworks that accommodate complex real-world decision-making scenarios Garrido-Merchán and Hernández-Lobato [2019], Belakaria et al. [2020a], Ahmadianshalchi et al. [2024b].

Non-myopic MOBO. Despite advances in myopic strategies for MOBO, a notable knowledge gap remains. There is no prior work on non-myopic acquisition functions for MOBO problems with a finite horizon. A recent approach proposed for settings with decoupled function evaluation Daulton et al. [2023] considers a one-step lookahead only and thus lacks strategic foresight. Our goal is to precisely fill this critical gap in the current state of knowledge.

In this section, we provide details of our proposed approach for non-myopic MOBO. We first discuss the challenge of maintaining the validity of the Bellman optimality principle and, by consequence, the ability to use the Bellman equation and its lower-bound to solve the SED problem in the multi-objective setting. We then discuss design choices for the marginal utility function that enables the effective use of the Bellman equation in our setting. We finally provide the details of the non-myopic acquisition functions we propose, their computational challenges, and practical algorithms.

This section describes our experimental evaluation comparing the proposed non-myopic methods with baselines on several real-world and synthetic MOO problems.

Benchmark MOO problems: We provide a brief description of our real-world MOO benchmarks, including the number of input dimensions ($d$) and the number of objectives ($K$).

We also include the ZDT-3 ($d$ = 9, $K$ = 2) Zitzler et al. [2000] problem as a synthetic MOO benchmark. We provide additional results in the Appendix (Section 7.4) and an overview of all our MOO benchmarks(Appendix Table LABEL:tab:benchmark_details).

Baselines: We proposed three non-myopic acquisition functions (NMMO-Joint, NMMO-Nested, and BINOM) with varying trade-offs between computational complexity and approximation accuracy. In an ideal scenario with a perfect surrogate model, evaluating the entire lookahead horizon $H=T-t$ at each iteration $t$ would yield the closest results to the optimal policy. However, previous research on non-myopic Bayesian optimization [Jiang et al., 2020a, b, Lee et al., 2021, 2020] has shown that models often struggle with long-term predictions due to compounding uncertainty at each lookahead step. This is because the model’s predictions at each step are based on the uncertain predictions from the previous steps. As a result, querying an extended horizon can hinder the optimization process by including suboptimal or misleading inputs in the non-myopic horizon, potentially guiding the search in less promising directions and incurring higher computational costs. To address this issue, we adopt the same strategy from previous work and introduce a maximum horizon length to limit the size of the horizon.

The NMMO-Nested method stands out with its potential for superior optimization performance due to its comprehensive approach to handling nested optimizations. However, this method has practical challenges. Notably, this method requires a pre-defined grid for optimization, as internal maximization prevents using gradient-based optimizers. Furthermore, Its computational complexity constrains the size of the grid that can be feasibly used, which in turn limits the method’s operational speed and accuracy. Hence, the nested approach does not perform as well as expected compared to the other proposed non-myopic approaches. We provide results including NMMO-Joint with lookahead horizon $H\in\{4,8\}$ and BINOM with $H$ = 4. We report the results for NMMO-Nested with $H=2$ in the Appendix Section 7.5.

We consider two state-of-the-art MOBO baselines to benchmark the performance of our non-myopic methods. First, we use the multi-objective variant of the joint entropy search for multi-objective Bayesian optimization (JESMO) Tu et al. [2022], Balandat et al. [2020]. Second, we use the expected hypervolume improvement (EHVI) method Emmerich et al. [2006], Balandat et al. [2020] as a hypervolume-based baseline. Another baseline is the hypervolume knowledge gradient (HVKG), a one-step lookahead method for cost-aware MOBO problems Daulton et al. [2023]. While the HVKG algorithm is designed for cost-aware settings with decoupled evaluations, we use a modified version of this method where the objective function costs are set to zero, and we calculate all objectives for a true evaluation of the hypervolume. We report the results in the Appendix Section 7.5.

The baseline methods were chosen due to their distinct but complementary approaches to handling multi-objective trade-offs, providing a comprehensive comparative perspective against our proposed NMMO method. The effectiveness of each method is measured using the hypervolume indicator.

Evaluation metric. The HVI is a widely-used metric in MOO that measures the volume of the space dominated by the Pareto front relative to a predefined reference point: a larger hypervolume indicates better Pareto solutions. We will compare the hypervolume metric after each BO iteration.

Experimental details: All experiments were averaged among 15 runs, initialized with five points randomly generated from Sobol sequences, and run for 65 iterations. We emphasize that non-myopic solutions are well-suited for the smaller experimental budgets. We model each of the multiple objectives using an independent GP with a Matérn 5/2 ARD kernel. We use implementations from the BoTorch Python package Balandat et al. [2020] for all baselines, including JESMO, EHVI, and HVKG. All methods were evaluated under identical initial conditions. This includes consistent initialization, identical computational budgets, and the same evaluation criteria across all experiments.

This paper introduced a novel set of non-myopic methods for multi-objective Bayesian optimization (MOBO), to specifically address the problem of finite-horizon sequential experimental design (SED) for MOO problems. By addressing the difficulties related to scalarization, monotonicity, and additivity of the reward function typically encountered in MOO scenarios, we successfully adapted the Bellman optimality principle for MOBO by adopting a new approach to scalarization using hypervolume improvement (HVI). Our proposed methods, including the Nested, Joint, and BINOM acquisition functions, represent the first non-myopic acquisition functions tailored for MOBO problems. Our experiments on multiple real-world problems, show that our methods show significant improvements by consistently outperforming the state-of-the-art myopic approaches in limited-budget settings.

Let $\mathcal{Y}_{0}$ be the initial Pareto front, and let $(\mathbf{x}_{t},\mathbf{y}_{t})$ be the new data point added at time step $t$, where $t\in[1,T]$. We denote the updated Pareto front at time step $t$ as $\mathcal{Y}_{t}$.

The hypervolume improvement at time step $t$ is given by:

Now, let’s consider the final hypervolume improvement after $T$ time steps, we denote the finaly hypervolumen improvemennt as $HVI_{Total}$

We want to prove that:

By expanding the summation, we get a telescoping series:

The intermediate terms of the series canceling out lead to:

Therefore,

This proves that the final hypervolume improvement is indeed the summation of hypervolume improvements occurring at individual time steps.

The intuition behind this proof is that the hypervolume improvement at each time step represents the additional volume dominated by the updated Pareto front compared to the previous Pareto front. By summing up these individual improvements, we account for the total volume gained from the initial Pareto front to the final Pareto front.