Physics-informed active learning with simultaneous weak-form latent space dynamics identification

The low-dimensional latent variable ($\mathbf{z}$) discovered by the autoencoder inherits the time dependency from the high-dimensional FOM variable ($\mathbf{u}$). The dynamics of the latent variable in the latent space can be described by an equation of the following form,

where $\mathbf{z}(t;\boldsymbol{\mu})=\boldsymbol{\phi}_{e}(\mathbf{u}(t;\boldsymbol{\mu}))$ and $\boldsymbol{\psi}_{DI}$ represents a Dynamics Identification (DI) function governing the latent space dynamics. $\boldsymbol{\psi}_{DI}$ can be approximated by a system of ODEs and identified by the sparse identification of nonlinear dynamics (SINDy) method [39].

where $\boldsymbol{\Theta}(\mathbf{z}_{n}^{(i)})\in\mathbb{R}^{N_{l}}$ is a user-defined library that contains ${N_{l}}$ linear and nonlinear terms [39, 23, 24]; $\boldsymbol{\Xi}^{(i)}\in\mathbb{R}^{N_{l}\times N_{z}}$ is a coefficient matrix associated with the parameter $\boldsymbol{\mu}^{(i)}\in\mathcal{D}$. The time derivative $\dot{\mathbf{z}}_{n}^{(i)}$ can be obtained by using the chain rule:

The coefficients of all parameters $\boldsymbol{\Xi}=\big{[}\boldsymbol{\Xi}^{(1)},...,\boldsymbol{\Xi}^{(N_{\mu})}\big{]}\in\mathbb{R}^{N_{\mu}\times N_{l}\times N_{z}}$ can be obtained by minimizing the following function:

This loss function $\mathcal{L}_{\dot{\mathbf{z}}}$ provides constraints to the encoder-predicted latent space dynamics ($\dot{\mathbf{Z}}$) such that it is consistent with that from the DI model ($\dot{\hat{\mathbf{Z}}}$), which is critical for discovery of simple and smooth latent dynamics and therefore model performance [24].

To further enhance the accuracy of the physical dynamics predicted by the decoder, the following loss function is constructed to ensure the consistency between the predicted dynamics gradients ($\dot{\hat{\mathbf{u}}}_{n}^{(i)}$) and the gradients of the solution data ($\dot{\mathbf{u}}_{n}^{(i)}$):

where $\dot{\hat{\mathbf{u}}}_{n}^{(i)}$ can be obtained by applying the chain rule:

Therefore, the loss function of gLaSDI is defined as

where $\beta_{1}$ and $\beta_{2}$ denote the regularization parameters to balance the scale and contributions from the loss terms.

The latent space dynamics identification described in Section 3.2.1 is based on the strong-form ODE. If the high-dimensional input data contains noise and errors, the latent representation could be nonlinear and oscillatory, which poses challenges to the strong-form dynamics identification. To enhance the robustness against noise, we leverage the variance-reduction nature of the weak form and integrate it into gLaSDI.

We begin by multiplying Eqs. (7) and (8) with a continuous and compactly supported test function $\varphi(t)\in\mathcal{C}_{c}(\mathbb{R},\mathbb{R})$ and integrating them over a time domain:

When a piece-wise linear activation function, such as ReLU, is employed in the encoder network, $\partial\mathbf{z}_{n}^{(i)}/\partial\mathbf{u}_{n}^{(i)}$ is piece-wise constant, which is independent of time and can be taken out of the time integral in Eq. (14):

Due to the compact support of $\varphi$, we know that $\varphi(t_{b})=\varphi(t_{a})=0$, and applying integration by parts yields:

We apply the trapezoidal rule to discretize the time integrals. Thus, the weak-form transformation allows $\int_{t_{a}}^{t_{b}}\dot{\mathbf{u}}_{n}^{(i)}\varphi(t)dt$ to be computed without using the time derivative of the data ($\dot{\mathbf{u}}_{n}^{(i)}$), which is challenging to obtain in the presence of noise. It contributes to the enhanced accuracy and stability of latent space dynamics identification, as will be demonstrated in Section 4. We note that alternatively, $\int_{t_{a}}^{t_{b}}\dot{\mathbf{z}}_{n}^{(i)}\varphi(t)dt$ can be computed by directly applying integration by parts:

The weak-form loss function $\tilde{\mathcal{L}}_{\dot{\mathbf{z}}}$ corresponding to the strong-form counterpart $\mathcal{L}_{\dot{\mathbf{z}}}$ that enforces $\dot{\mathbf{z}}_{n}^{(i)}=\dot{\hat{\mathbf{z}}}_{n}^{(i)}$, as defined in Eq. (9), becomes

where $\int_{t_{a}}^{t_{b}}\dot{\hat{\mathbf{z}}}_{n}^{(i)}\varphi(t)dt$ is obtained by Eq. (13) and $\int_{t_{a}}^{t_{b}}\dot{\mathbf{z}}_{n}^{(i)}\varphi(t)dt$ can be computed by either Eq. (16) or (17). We will compare the effects of these two approaches on the model performance in Section 4.

Similarly, we can derive the weak-form loss function $\tilde{\mathcal{L}}_{\dot{\mathbf{u}}}$ corresponding to the strong-form counterpart $\mathcal{L}_{\dot{\mathbf{u}}}$ defined in Eq. (10) that enforces

Multiplying Eq. (19) with the test function $\varphi(t)$ and integrating it over a time domain gives:

The right-hand side can be obtained from multiplying Eq. (11) with the test function $\varphi(t)$ and integrating it over a time domain:

where $\partial\hat{\mathbf{u}}_{n}^{(i)}/\partial\mathbf{z}_{n}^{(i)}$ can be taken out of the time integral when a piece-wise linear activation function is employed in the decoder network. Alternatively, $\int_{t_{a}}^{t_{b}}\dot{\hat{\mathbf{u}}}_{n}^{(i)}\varphi(t)dt$ can be obtained by directly applying integration by parts:

The weak-form loss function $\tilde{\mathcal{L}}_{\dot{\mathbf{u}}}$ is defined as:

where $\int_{t_{a}}^{t_{b}}\dot{\mathbf{u}}_{n}^{(i)}\varphi(t)dt=-\int_{t_{a}}^{t_{b}}\mathbf{u}_{n}^{(i)}\varphi^{\prime}(t)dt$ and $\int_{t_{a}}^{t_{b}}\dot{\hat{\mathbf{u}}}_{n}^{(i)}\varphi(t)dt$ can be obtained by either Eq. (21) or (22). We will compare the effects of these two approaches on the model performance in Section 4.

The loss function of weak-form gLaSDI is defined as

where the last term is a sparsity constraint of the ODE coefficients to achieve better-conditioned ODEs for the latent space. Fig. 1 depicts the interactive training of the autoencoder and WENDy in the proposed WgLaSDI framework to counteract noisy data. In this study, piece-wise polynomials are employed for the test function. For further details about the selection of test functions, please refer to [53, 50, 47].

In the first test, we demonstrate the effects of the weak form on prediction accuracy when dealing with noisy training data. Both gLaSDI and WgLaSDI are trained by 16 predefined training samples uniformly distributed on the parameter space $\mathcal{D}=[0.7,0.9]\times[0.9,1.1]$. We employ an architecture of 1,001-100-5 ($N_{z}=5$) with ReLU activation for the encoder and a symmetric architecture for the decoder. Linear polynomials are adopted as basis functions for latent space dynamics identification. The hyperparameters in the WgLaSDI loss function (Eq. (24)) are defined as $\beta_{1}=1,\beta_{2}=1$, and $\beta_{3}=10^{-5}$, while those in the gLaSDI loss function (Eq. (12)) are defined as $\beta_{1}=1$ and $\beta_{2}=1$. We investigated two WgLaSDI loss functions, i.e., Type I based on Eqs. (16) and (21) and Type II based on Eqs. (17) and (22). The training is performed for 20,000 epochs. For $k$-NN interpolation of latent space dynamics, $k=4$ is employed for both gLaSDI and WgLaSDI.

Fig. 3 compares the latent space dynamics $(a=0.7,w=0.9)$ and maximum relative errors from gLaSDI and WgLaSDI (based on the Type-I loss function) over the parameter space. Fig. 3(a-b) shows that the DI model (SINDy) in gLaSDI struggles to learn the latent space dynamics predicted by the encoder, while the DI model (WENDy) in WgLaSDI successfully identifies the latent space dynamics. The consistency between the encoder-predicted and the DI-predicted latent space dynamics is critical to the prediction accuracy of the physical dynamics as reflected in Fig. 3(c-d). WgLaSDI outperforms gLaSDI in the whole parameter space, achieving at most 1.5$\%$ error vs. 285$\%$ of gLaSDI. It is worth mentioning that the loss function of gLaSDI, as defined in Eq. (10) and (12), requires the gradient of the dynamics data, which is very oscillatory when the data contains noise, as shown in Fig. 4, leading to difficulties in training. In contrast, the weak-form loss function of WgLaSDI, as defined in Eq. (18), (23), and (24), avoids the need for the data gradient and takes advantage of the gradient of the test function, which is noise invariant, leading to significantly improved learning performance. The results of this test demonstrate that the weak form enhances the accuracy and robustness of latent space dynamics identification in the presence of noise.

Fig. 5(a) and (c) show that WgLaSDI trained by the Type-II loss function also successfully identifies simple latent space dynamics and achieved a maximum error of 2.8$\%$, slightly higher than that (1.5$\%$) of WgLaSDI based on the Type-I loss function (Fig. 3(d)). The enhanced prediction accuracy achieved by the Type-I loss function is contributed by the involvement of the gradients of the encoder and the decoder networks, leading to stronger constraints on the autoencoder.

In the second test, we compare the effects of fitting the surrogate DI model simultaneously while training the autoencoder versus a sequential training approach. Specifically, we compare the weak-form gLaSDI with the weak-form LaSDI [26]. For a fair comparison, hyperparameters such as the autoencoder architecture and the DI basis function, as well as the data are kept the same between the two methods. Fig. 5(b) and (d) show the results of WLaSDI, with a maximum error of $4\%$ across the entire parameter space, slightly larger than those from WgLaSDI for both types of losses. The error from WLaSDI includes both the projection error introduced by the autoencoder and the DI error from WENDy, with the projection error being the predominant factor. As discussed in Section 4.1.1, it is critical to ensure the consistency between the encoder-predicted and the DI-predicted latent space dynamics for satisfactory prediction accuracy of the physical dynamics. Due to the lack of interaction between the autoencoder and WENDy of WLaSDI, there is no constraint and regularization from the DI model on the autoencoder, leading to oscillatory latent dynamics from the encoder, as shown in the latent dynamics gradient ($dz/dt$) in Fig. 5(b). Although WENDy’s variance reduction capability enables it to learn the oscillatory latent dynamics from the encoder, the inconsistency between their predicted latent dynamics still leads to large projection errors of the physical dynamics. This highlights the impact of using simultaneous training versus the sequential approach.

In the previous tests, the training samples are predefined on a uniform grid in the parameter space, which may not be optimal, especially for highly nonlinear physical systems. The greedy physics-informed active learning strategy in WgLaSDI allows maximum exploration of the parameter space and optimal samples to be selected on the fly during the training process. To demonstrate that, we train WgLaSDI using the same autoencoder and DI basis functions as defined in Section 4.1.1 together with active learning.

WgLaSDI without active learning already achieves excellent performance, with at most 1.5$\%$ maximum relative error over the parameter space, as shown in Fig. 3(d). WgLaSDI with active learning achieves the same maximum error (1.5$\%$) over the parameter space, as shown in Fig. 6, with an average error of 0.75$\%$, slightly lower than that (1.0$\%$) of WgLaSDI without active learning. It is observed that active learning tends to guide WgLaSDI to select more samples in the higher range of $a$. The test results show that physics-informed active learning could potentially guide WgLaSDI to select samples for enhanced prediction accuracy over the parameter space. Compared with the high-fidelity simulation (in-house Python code) that has an around $2\%$ maximum relative error with respect to the high-fidelity data used for WgLaSDI training, WgLaSDI achieves 121$\times$ speed-up.

In the first test, we demonstrate the effects of the weak form on prediction accuracy when dealing with noisy training data. Both gLaSDI and WgLaSDI are trained by 25 predefined training samples uniformly distributed on the parameter space $\mathcal{D}=[0.7,0.9]\times[0.9,1.1]$. We employ an architecture of 7200-100-5 ($N_{z}=5$) with ReLU activation for the encoder and a symmetric architecture for the decoder. Quadratic polynomials are adopted as basis functions for latent space dynamics identification. The hyperparameters in the loss functions (Eq. (24)) are defined as $\beta_{1}=1,\beta_{2}=1$, and $\beta_{3}=10^{-5}$, while those in the gLaSDI loss function (Eq. (12)) are defined as $\beta_{1}=0$ and $\beta_{2}=1$. We investigated two WgLaSDI loss functions, i.e., Type-I based on Eqs. (16) and (21) and Type-II based on Eqs. (17) and (22). The training is performed for 100,000 epochs. For $k$-NN interpolation of latent space dynamics, $k=4$ is employed for both gLaSDI and WgLaSDI.

Fig. 8 shows that WgLaSDI (based on the Type-I loss function) outperforms gLaSDI in learning latent space dynamics in the presence of noise, achieving at most 7.2$\%$ error vs. 3,051$\%$ of gLaSDI across the parameter space. It further demonstrates the enhanced robustness and performance against noise due to the weak-form latent space dynamics identification.

Fig. 9(a) shows the latent space dynamics identified by WgLaSDI based on the Type-II loss function, which is more oscillatory than that from the Type-I loss (Fig. 8(b)). Fig. 9(c) shows that WgLaSDI based on the Type-II loss achieves a maximum error of 10$\%$, higher than that (7.2$\%$) from the Type-I loss (Fig. 8(d)). It further demonstrates the enhanced performance achieved by the Type-I loss function. We will adopt the Type-I loss function for WgLaSDI in the following examples.

In the second test, we further demonstrate the effects of simultaneous training of the autoencoder and the DI model by comparing WgLaSDI with WLaSDI. WLaSDI is trained using the same autoencoder, DI basis functions, and data as those employed for WgLaSDI in Section 4.2.1. As shown in Fig. 9(d), the WLaSDI error ranges from 6$\%$ to 20$\%$, significantly larger than that of WgLaSDI for both types of losses (Fig. 8(d) and Fig. 9(c)). Comparing Fig. 8(b) and Fig. 9(a) with Fig. 9(b) shows that simultaneous training enforces constraints on the latent space dynamics discovered by the autoencoder through the interaction with the DI model, enabling simpler latent space dynamics, reduced projection and DI errors, and therefore better generalization performance.

To demonstrate the effects of the greedy physics-informed active learning strategy in WgLaSDI on the model performance, we train WgLaSDI using the same autoencoder and DI basis functions defined in Section 4.2.1 together with active learning. Fig. 10 shows that WgLaSDI with active learning achieves at most 5.5$\%$ error over the parameter space, lower than that (7.2$\%$) of WgLaSDI without active learning, as shown in Fig. 8(d). Interestingly, active learning tends to guide WgLaSDI to select more samples in the higher range of $w$ to achieve better performance in the parameter space. Compared with the high-fidelity simulation (in-house Python code) that has an around $6\%$ maximum relative error with respect to the high-fidelity data used for WgLaSDI training, WgLaSDI achieves 1,374$\times$ speed-up.

To examine the effects of the weak form on model performance when dealing with noisy data, we train gLaSDI and WgLaSDI using 16 predefined samples uniformly distributed on the parameter space $\mathcal{D}=[1.5,2.0]\times[2.0,2.5]$. Both models adopt an architecture of 9,216-100-3 ($N_{z}=3$) with ReLU activation for the encoder and a symmetric architecture for the decoder, with linear polynomials as basis functions for latent space dynamics identification. The hyperparameters in the WgLaSDI loss function (Eq. (24)) are defined as $\beta_{1}=1,\beta_{2}=1$, and $\beta_{3}=10^{-4}$, while those in the gLaSDI loss function (Eq. (12)) are defined as $\beta_{1}=1$ and $\beta_{2}=1$. The training is performed for 100,000 epochs. For $k$-NN interpolation of latent space dynamics, $k=4$ is employed for both gLaSDI and WgLaSDI.

Fig. 12 shows that WgLaSDI outperforms gLaSDI in learning latent space dynamics in the presence of noise, achieving at most 3.6$\%$ error vs. 12.6$\%$ of gLaSDI across the parameter space. WgLaSDI demonstrates enhanced robustness and performance against noise due to the weak-form dynamics identification.

The effect of simultaneous training is further illustrated in this example of radial advection. Both WgLaSDI and WLaSDI employ the same autoencoder, DI basis functions, and data described in Section 4.3.1. According to Fig. 13(c), WLaSDI achieves a maximum error of 3.8$\%$ across the entire parameter space, slightly higher than that (3.6$\%$) of WgLaSDI (Fig. 12(d)). It further demonstrates the effectiveness and advantages of simultaneous training.

To further demonstrate the effects of the greedy physics-informed active learning on model performance, we train WgLaSDI using the same autoencoder and DI basis functions defined in Section 4.2.1 together with active learning. Fig. 13 shows that physics-informed active learning helps WgLaSDI to further reduce the maximum error from 3.6$\%$ to 3.5$\%$ and the average error from 2.7$\%$ to 2.2$\%$ in the parameter space. Compared with the high-fidelity simulation (MFEM [59]) that has an around $3\%$ maximum relative error with respect to the high-fidelity data used for WgLaSDI training, the WgLaSDI model achieves 178$\times$ speed-up.

To examine the effects of the weak form on model performance when dealing with noisy training data, we train gLaSDI and WgLaSDI using 16 predefined samples uniformly distributed on the parameter space $\mathcal{D}=[0.9,1.1]\times[1.0,1.2]$. Both models adopt an architecture of 4,096-1000-100-50-50-50-3 ($N_{z}=3$) with ReLU activation for the encoder and a symmetric architecture for the decoder, with linear polynomials as basis functions for latent space dynamics identification. The hyperparameters in the WgLaSDI loss function (Eq. (24)) are defined as $\beta_{1}=10,\beta_{2}=100$, and $\beta_{3}=10^{-5}$, while those in the gLaSDI loss function (Eq. (12)) are defined as $\beta_{1}=0$ and $\beta_{2}=10$. The training is performed for 400,000 epochs. For $k$-NN interpolation of latent space dynamics, $k=4$ is employed for both gLaSDI and WgLaSDI.

Fig. 15 shows that WgLaSDI achieves at most 7.1$\%$ error in the parameter space, significantly lower than 953$\%$ of gLaSDI. WgLaSDI again demonstrates enhanced robustness and performance against noise. Compared with the high-fidelity simulation (HyPar [60]), the WgLaSDI model achieves 1,779$\times$ speed-up.

We trained a WLaSDI with the same autoencoder, DI basis functions, and data described in Section 4.4.1. The large discrepancy between the encoder-predicted and the DI-predicted latent space dynamics from WLaSDI, as shown in Fig. 16(a), leads to larger errors (20$\%$ to 26$\%$) across the parameter space (Fig. 16(b)) than those (3.5$\%$ - 7.1$\%$) of WgLaSDI (Fig. 15(d)). It again highlights that simultaneous training enhances latent space dynamics learning and contributes to simpler latent space dynamics with a stronger connection and consistency between the autoencoder and the DI model, leading to better performance.