Bi-fidelity Modeling of Uncertain and Partially Unknown Systems Using DeepONets

The ubiquitous presence of uncertainty often affects the modeling of a physical system using differential equations. For example, in engineering systems, the sources of uncertainty can include material properties, geometry, or loading conditions. This type of uncertainty is known as parametric uncertainty and differs from one realization to another. In contrast, during modeling, some physical phenomena are often ignored or simplified, which leads to the so called structural uncertainty [1]. Both types of uncertainties can be present in a real-life engineering system, and should be addressed when developing digital twins of the system. Models developed using polynomial chaos expansion [2, 3], Gaussian process regression [4, 5], and other response surfaces [6, 7, 8] address parametric uncertainty, which defines possible realizations of the system. However, the cost of developing these models increases significantly with the dimension of the uncertain variables [9].

Neural networks, commonly used in computer vision [10] and pattern recognition problems [11], have been used to model physical systems in the presence of uncertainty [12, 13]. To model physical systems, Raissi et al. [14] augmented the standard data discrepancy loss with the residual of the corresponding governing equations. This approach was termed "Physics-informed neural networks." Subsequently, a significant amount of work employing this approach to address basic computational mechanics problems came into prominence in the literature, some notable ones being Karniadakis et al. [15], Cai et al. [16], and Viana and Subramaniyan [17]. Among these several works address the parameteric uncertainty in the system [18, 19, 20, 21, 22, 23]. However, machine learning studies that address structural uncertainty in modeling are scarce. Among them, Blasketh et al. [24] used neural networks combined with governing equations to address modeling and discretization errors. Zhang et al. [25] used the physics-informed approach to quantify both the parametric and approximation uncertainty of the neural network. Duraisamy and his colleagues further trained neural networks to comprehend the discrepancies between turbulence model prediction and the available data [26, 27, 28].

Recently, several neural network architectures capable of approximating mesh-independent solution operators for the governing equations of physical systems have been proposed [29, 30, 31, 32, 33, 34, 35], drawing inspiration from the universal approximation theorem for operators in Chen and Chen [36]. These architectures overcome two shortcomings of the physics-informed use of neural networks to learn a system’s behavior: firstly, they do not depend on the mesh used to generate the training dataset. Secondly, they are solely data-driven and do not require the knowledge of the governing equations. Among these network architectures, the deep operator network (DeepONet) proposed in Lu et al. [29, 34] uses two separate networks — one with the external source term as input and the other with the coordinates where the system response is sought as input. The former network is known as the branch network while the latter is known as the trunk network. Outputs from these two networks are then used in a linear regression model to produce the system response. Lu et al. [37] showed how the computational domain can be decomposed into many small subdomains to learn the governing equation with only one solution of the equation. While Cai et al. [38] used DeepONets for a multiphysics problem involving modeling of field variables across multiple scales., Ranade et al. [39] and Priyadarshini et al. [40] employed DeepONets for approximating joint subgrid probability density function in turbulent combustion problems and to model non-equilibrium flow physics respectively. Additionally, Wang et al. [41, 42] used the error in satisfying the governing equations to train the DeepONets, with a variational formulation of the governing equations being applied by Goswami et al. [43] to train DeepONets to model brittle fracture. Further comparisons of DeepONet with the iterative neural operator architecture proposed in Li et al. [32] were performed in Kovachki et al. [44] and Lu et al. [45]. The approximation errors in the latter neural operator architectures were subsequently discussed in Kovachki et al. [46], Lanthaler et al. [47], Deng et al. [48], and Marcati and Schwab [49].

The training datasets required to construct the DeepONets for a complex physical system can often be limited due to the computational burden associated with simulating such systems. Similarly, repeating experiments to obtain measurement data can become infeasible. In the presence of a small dataset from the true high-fidelity system, we propose herein a modeling strategy with DeepONets that uses a similar but computationally inexpensive model, known as low-fidelity, that captures many aspects of the true system’s behavior. The discrepancy between the response from the true physical system and the low-fidelity model is estimated using DeepONets. The assumption used here is that the modeling of the discrepancy requires only a small training dataset of the true system’s response, and as a result, bi-fidelity approaches are often advantageous with scarce training data from the high-fidelity simulations [50, 51, 52]. In principle, we train a DeepONet that uses the low-fidelity response as the input of the branch network to predict the discrepancy between the low- and high-fidelity data. However, we do not incorporate physics loss or error in satisfying the governing equations (as previously employed in physics informed neural networks) to circumvent potential bias incorporation in partially known engineering systems subject to the presence of structural uncertainty.

In this paper, we employ three numerical examples to show the efficacy of the proposed bi-fidelity approach to modeling the response of an uncertain and partially unknown physical system. The first two examples consider a nonlinear oscillator and a heat transfer problem, respectively, where uncertainty is present in some of the parameters of the governing equations. The low-fidelity model used in these examples ignores the nonlinearity or some of the uncertain components of the governing equations. In the third example, we model power generated in a wind farm, , with the two and three dimensional simulations respectively serving as high and low-fidelity model systems.

The rest of the paper is organized as follows: first, we provide a brief background on DeepONets in Section 2. Then, we present our proposed approach for modeling the discrepancy between a low-fidelity system response and the true response for uncertain and partially unknown systems in Section 3. Three numerical examples involving a dynamical system, a heat transfer problem, and a wind farm application are subsequently used in Section 4 to showcase the efficacy of the proposed approach. Finally, we conclude the paper with a brief discussion on future applications of the proposed approach.

Let us assume the governing equation of a physical system in a domain $\Omega$ is given by

where $\mathcal{L}$ is a nonlinear differential operator; $\mathbf{x}$ is the location; $u$ is the response of the system; and $f$ is the external source. For a boundary value problem, Dirichlet boundary condition $u(\mathbf{x})={u}_{D}(\mathbf{x})$ on $\mathbf{x}\in\partial\Omega_{D}$ and/or Neumann boundary condition $\mathcal{L}_{N}(u)(\mathbf{x})={u}_{N}(\mathbf{x})$ on $\mathbf{x}\in\partial\Omega_{N}$ are also present, where $\mathcal{L}_{N}$ is the differential operator for the Neumann boundary condition; $\partial\Omega_{D}\cup\partial\Omega_{N}=\partial\Omega$ with the boundary of the domain being $\partial\Omega$; and $\partial\Omega_{D}\cap\partial\Omega_{N}=\emptyset$. For an initial value problem, an initial condition $u(\mathbf{x}_{0})=u_{0}$ is used instead, where $\mathbf{x}_{0}$ is the initial instant. The solution of (1) is given by $G(f)(\mathbf{x})$, where $G:\mathcal{X}\rightarrow\mathcal{Y}$ denotes the solution operator. For $\mathcal{X}$ and $\mathcal{Y}$, there exist embeddings $\mathcal{X}\hookrightarrow L^{2}(D)$ and $\mathcal{Y}\hookrightarrow L^{2}(\Omega)$.

The DeepONet architecture proposed in Lu et al. [29] approximates the solution operator $G(f)(\mathbf{x})$ as

where $c_{0}$ is a constant bias parameter; $c_{i}(\cdot),~i=1,\dots,p,$ are obtained from a neural network, known as the branch network, that has the input function (e.g., external source term) $f(\cdot)$ measured at $\{\mathbf{x}_{i}\}_{i=1}^{m}$ locations as input; $\psi_{i}(\cdot),~i=1,\dots,p,$ are obtained from a neural network, known as the trunk network, that has the location $\mathbf{x}$ as the input; and $\boldsymbol{\theta}=[\boldsymbol{\theta}_{\mathrm{branch}}^{T}\quad\boldsymbol{\theta}_{\mathrm{trunk}}^{T}]^{T}$ is the parameter vector containing weights and biases $\boldsymbol{\theta}_{\mathrm{branch}}$ and $\boldsymbol{\theta}_{\mathrm{trunk}}$ from the branch and trunk networks, respectively, and $c_{0}$. This approximation is based on the universal approximation theorem in Chen and Chen [36] for operators. A schematic of the DeepONet architecture used in this study is shown in Figure 1. The main advantage of using a DeepONet architecture over other standard ones, such as feedforward, recurrent, etc., is that DeepONet provides a mesh-independent approximation of the solution operator since DeepONet uses a set of parameters $\boldsymbol{\theta}$ trained only once with different mesh discretizations [32, 29].

Lanthaler et al. [47] showed that the DeepONet can be thought of as a composition of three components. The first component is an encoder $\mathcal{E}$ that maps the input function space to $m$ sensor inputs $\{f(\mathbf{x}_{i})\}_{i=1}^{m}$ for $\{\mathbf{x}_{i}\}_{i=1}^{m}\subset D$. The second component, the approximator $\mathcal{A}$, models the relation between $N_{f}$ sensor inputs $\{f(\mathbf{x}_{i})\}_{i=1}^{m}$ and the coefficients $\{c_{k}\}_{k=1}^{p}$. The encoder and the approximator together comprise the branch network. The third or the final component of DeepONet is the reconstructor, which maps the output of the branch network to the response $u(\mathbf{x})$, thereby showing dependence on the trunk network outputs $\{\psi_{k}\}_{k=1}^{p}$. With approximate inverses of the encoder and reconstructor defined as decoder $\mathcal{D}$ and projector $\mathcal{P}$, respectively, three errors associated with DeepONet approximation can be defined as encoding error $\mathcal{E}_{e}=\lVert\mathcal{D}\circ\mathcal{E}-\mathbb{I}\rVert_{L^{2}(\mu)}$, approximation error $\mathcal{E}_{a}=\lVert\mathcal{A}-\mathcal{P}\circ G\circ\mathcal{D}\rVert_{L^{2}(\mathcal{E}_{\#}\mu)}$, and reconstruction error $\mathcal{E}_{r}=\lVert\mathcal{R}\circ\mathcal{P}-\mathbb{I}\rVert_{L^{2}(G_{\#}\mu)}$, where $\mathbb{I}$ is an identity matrix; $\mu$ is the measure associated with $f$; $\mathcal{E}_{\#}\mu$ and $G_{\#}\mu$ are push-forward measures associated with the encoder and operator $G$, respectively. Figure 2 shows the three components of a DeepONet and associated errors as defined here. The total approximation error associated with a DeepONet depends on these three errors as outlined in Theorem 3.2 in Lanthaler et al. [47].

The parameter vector $\boldsymbol{\theta}$ is estimated by solving the optimization problem

where in the training dataset for each of $N_{f}$ realizations of the external source term $\{f_{j}(\mathbf{x})\}_{j=1}^{N_{f}}$, $N_{d}$ measurements of the response at locations $\{\mathbf{x}_{i}\}_{i=1}^{N_{d}}$, which might not be the same as the locations where the external source term is measured, are used. These sensor locations can be random or equispaced inside the domain and kept fixed during training. The optimization problem in (3) can be solved using stochastic gradient descent [53, 54, 55] to obtain optimal parameter values of the DeepONets. In this paper, we use a popular variant of stochastic gradient descent, known as the Adam algorithm [56], to train DeepONets. Note that in Lu et al. [29], the input function $f(\mathbf{x})$ is sampled from function spaces with the locations $\mathbf{x}$ as domain and the external source $f$ as range, such as Gaussian random field and Chebyshev polynomials. However, for practical reasons, in this paper, we use probability distribution of the uncertain variables associated with the problems to sample the input function. However, the number of training data for DeepONets can be limited if the computational cost of the simulation of the physical system or the cost of repeating the experiments is significant. In this study, to address the challenge of training DeepONets in the presence of limited data, we propose a bi-fidelity training approach utilizing a computationally inexpensive low-fidelity model of the physical system. In the next section, we detail this proposed approach and discuss two applications to address parametric and structural uncertainties present in the system.

In the present work, we assume a low-fidelity model of the physical system in (1) is given by

where $u_{l}(\mathbf{x})$ is the low-fidelity response; $\mathcal{L}_{l}$ is the differential operator related to the low-fidelity model; and $f_{l}(\mathbf{x})$ is the external source in the low-fidelity model. Next, we denote the discrepancy between the solution of the low-fidelity model $u_{l}(\mathbf{x})$ and that of the physical system $u(\mathbf{x})$ as

As it is often the case that the low-fidelity model will capture many important aspects of the physical system’s response, we assume that the training of a DeepONet for modeling $u_{d}(\mathbf{x})$ is easier than modeling the true response $u$ and that generating the low-fidelity response $u_{l}(\mathbf{x})$ is computationally inexpensive. We present two applications of the proposed approach next. Note that $u_{d}(\mathbf{x})$ is estimated from the DeepONet and $u_{l}(\mathbf{x})$ from the low-fidelity model for any new $\mathbf{x}\in\Omega$ to predict the response $u(\mathbf{x})$. Further, this new location $\mathbf{x}$ need not be the same locations used during training or the places where the external source is measured due to the use of DeepONet to model $u_{d}(\mathbf{x})$. We also do not use another DeepONet or neural network to model the low-fidelity response as the simulation of the low-fidelity model is computationally inexpensive, and this avoids any approximation error introduced through a low-fidelity trained network.

In this section, we illustrate the proposed approach in approximating the discrepancy between the low-fidelity response and the physical system’s response with DeepONets using three numerical examples. The first example uses nonlinear ordinary differential equation for an initial value problem. In the second example, we model a nonlinear heat transfer in a thin plate, where a partial differential equation for the boundary problem is used. We consider both parametric and structural uncertainties to illustrate the applications I and II of the proposed approach from Section 3 in these two examples. The third example considers flow through a wind farm with a combination of parametric and structural uncertainties, where two-dimensional and three-dimensional simulations are used as the low- and high-fidelity model of the system, respectively. In these examples, we use two separate datasets, namely a training dataset $\mathcal{D}_{\mathrm{tr}}=\left\{(\mathbf{x}_{i},\boldsymbol{\xi}_{i}),u(\mathbf{x}_{i};\boldsymbol{\xi}_{i}),\{f^{(i)}(\mathbf{x}_{j})\}_{j=1}^{m}\right\}_{i=1}^{{N}_{\mathrm{tr}}}$ and a validation dataset $\mathcal{D}_{\mathrm{val}}=\left\{(\mathbf{x}_{i},\boldsymbol{\xi}_{i}),u(\mathbf{x}_{i};\boldsymbol{\xi}_{i}),\{f^{(i)}(\mathbf{x}_{j})\}_{j=1}^{m}\right\}_{i=1}^{{N}_{\mathrm{val}}}$. We use the training dataset $\mathcal{D}_{\mathrm{tr}}$ to train the DeepONets, and the accuracy of the trained DeepONets is measured using the validation dataset $\mathcal{D}_{\mathrm{val}}$, with the relative validation root mean squared error (RMSE), $\varepsilon_{\mathrm{val}}$, defined as follows

where $\lVert\cdot\rVert_{2}$ is the $\ell_{2}$-norm of its argument, $\mathbf{u}_{\mathrm{pred}}$ is the vector of predicted response using DeepONets, and $\mathbf{u}_{\mathrm{val}}$ is from the validation dataset $\mathcal{D}_{\mathrm{val}}$. Note that we select the number of hidden layers in the trunk and branch networks and the number of neurons per hidden layer using an iterative procedure, where we increase the number of neurons per hidden layer gradually up to an upper limit (100 in this study) and then add hidden layers to the network until we observe no reduction of $\varepsilon_{\mathrm{val}}$. For the number of terms $P$ in (8) and (12), we increase $P$ in multiples of 10 to a maximum, which is the number of neurons in the hidden layers, to observe any reduction of $\varepsilon_{\mathrm{val}}$.

In this paper, we propose a bi-fidelity approach for modeling uncertain and partially unknown system responses using DeepONets. In this approach, we utilize a similar and computationally less expensive model of the physical system as a low-fidelity model. The difference between the response from the low-fidelity model and the true system is subsequently modeled using DeepONets. As is often the case, with low-fidelity model we can capture the important characteristics of the system’s behavior. Therefore, we make a reasonable assumption in this study that the modeling of the discrepancy between the low-fidelity model and the system’s response can be done with a smaller high-fidelity dataset than a standard DeepONet to model the system’s response in full to achieve similar validation error. In particular, we use DeepONets with low-fidelity model response, the locations, and the realizations of the uncertain variables as inputs. We apply the proposed approach to problems with parametric as well as with structural uncertainty, where some of the physics is unknown or not modeled. First, a nonlinear oscillator’s response is modeled using this proposed approach, followed by the modeling of nonlinear heat transfer in a thin plate. Both examples show that the proposed approach provides an improvement in the validation error by almost an order of magnitude. Next, we model a wind farm with six wind turbines, where power generated from two-dimensional simulations is used as the low-fidelity response and the true system’s response is obtained using three-dimensional simulations. Again, the proposed approach provides significant improvement in validation error, showcasing the efficacy of the proposed approach. These three case studies demonstrate that the bi-fidelity DeepONet approach is a powerful tool in situations with limited high-fidelity data. In the future, we plan to develop a multi-fidelity extension of this approach to determine the effect of regularization in the loss function and modeling of complex multiphysics phenomena.

This work was authored in part by the National Renewable Energy Laboratory, operated by Alliance for Sustainable Energy, LLC, for the U.S. Department of Energy (DOE) under Contract No. DE-AC36-08GO28308. This work was supported by funding from DOE’s Advanced Scientific Computing Research (ASCR) program under agreement DE-AC36-08GO28308. The research was performed using computational resources sponsored by the Department of Energy’s Office of Energy Efficiency and Renewable Energy and located at the National Renewable Energy Laboratory. The views expressed in the article do not necessarily represent the views of the DOE or the U.S. Government. The U.S. Government retains and the publisher, by accepting the article for publication, acknowledges that the U.S. Government retains a nonexclusive, paid-up, irrevocable, worldwide license to publish or reproduce the published form of this work, or allow others to do so, for U.S. Government purposes. The work of AD was partially supported by the AFOSR grant FA9550-20-1-0138.