The Effect of Sensor Fusion on Data-Driven Learning of Koopman Operators

The rapid evolution in sensor technology enables us to acquire a wealth of information about the governing dynamics of nonlinear systems. Novel sensors are being designed and fabricated at an rapid rate to enhance the data acquisition process for systems in various domains like biology [1, 2, 3, 4, 5], mechanics [6, 7, 8, 9, 10], and transportation [11, 12, 13, 14, 15]. The tools, techniques, and theories that integrate all the data to augment our knowledge of the system is the broad area of sensor fusion [16]. Sensor fusion plays a pivotal role in many applications. In the human gait system, senor fusion methods are deployed to study gait dynamics [17, 18, 19, 20, 21], detect gait anomalies [22, 23, 24] and control prosthetics [25, 26, 27, 28, 29, 30]. Numerous sensors are utilized in inertial navigation systems with data fusion architectures for accurate state estimation to aid in navigation and control [31, 32, 33, 34, 35, 36, 37, 38]. Maximizing the throughput of manufacturing processes [39, 40, 41, 42, 43, 44, 45, 46, 47, 48], minimizing traffic congestion [12, 13, 14], and monitoring structural health [49, 50, 51] are some of the other areas where sensor fusion plays a critical role.

In recent years, there has been significant interest in developing sensor fusion techniques for biological systems, spurred in part by decreasing costs of next-generation omics measurements [52, 53, 54, 55]. While some techniques like transcriptomics [56] and proteomics [57] inform the genetic activity within the cell, other instruments like flow cytometers [58], plate readers [59], and microscopes [60] inform the phenotypic characteristics outside the cell. Substantial progress towards fusing a combination of these datasets with prior knowledge of the system has resulted in static models that provide insights about the underlying network topology of the interaction between the genes and the proteins [52, 53, 61, 62, 54, 55, 63, 64].

An area of research emerging in the last decade is the problem of sensor fusion for biological networks, that couple distinct streams of omics measurements with fluorescence data. Fluorescence data is often used to parameterize or identify dynamical system models describing governing dynamics and network topology, while omics measurements are used to deduce whole-cell statistics and steady-state phenotypes of cellular metabolism, stress, and fitness. To that end, our goal in this paper is to develop sensor fusion techniques that integrate various types of time-series data to construct dynamic genotype-to-phenotype models.

Koopman operator methods have recently demonstrated great promise in simultaneously discovering A) the governing dynamics and B) a spectral decomposition of a complex physical system represented purely by data [65, 66]. The key premise of Koopman operator theory is that a collection of state functions, or observables, can be constructed, discovered, or estimated, to embed nonlinear dynamics of a physical system in a high dimensional space. The Koopman operator then acts as a linear operator on the function space of observables, governing time-evolution of the system dynamics, as a linear system.

In reality, the Koopman operator is infinite dimensional and must be approximated numerically. The problem of finding a Koopman operator and a collection of observable functions is known as the Koopman operator learning problem. The classical approach to solve this problem is to use dynamic mode decomposition (DMD) [67]. More recently, variations on DMD involve approximating observable functions with a broad set of dictionary functions (E-DMD) [68, 69], which can be generated using deep learning [70, 71, 72, 73], by casting the learning problem, as a robust optimization problem to handle sparse data [74, 75] or to treat heterogeneously sampled data [76, 77]. The power of Koopman operators lies in their ability to capture the underlying modes that drive the system [78, 79, 80], directly from data. Koopman operators also enable the construction of observers [81, 82, 83, 84] and controllers [85, 86, 87, 88, 89] for nonlinear systems in a linear framework.

Koopman operators can fuse sensor measurements into a single dataset to learn the underlying dynamics, as seen in systems like traffic dynamics [90, 91], human gait [92, 93, 94], and robotics [95]. The work of Williams et al. [96] and Mezic [97] use Koopman operator theory to elucidate the fusion of the dynamics evolving on two different state-spaces, provided that there is a function map between the state-spaces. Williams et al. [96] consider two datasets that are rich enough to reconstruct the system state and develop an algorithm to map the eigenfunctions of the Koopman operator identified from each dataset. Mezic [97] proves a relational mapping between eigenfunctions of both spaces, when exact conjugacy is not possible. For dynamics that evolve on two different state-spaces, they define the factor conjugacy of the dynamics for the function that maps the two spaces.

This paper builds on existing Koopman operator fusion theory [96, 97] to examine a special case: we consider learning a sensor fusion model for a physical system represented by direct state data and a series of output measurements. We consider the scenario where both the Koopman operator, observables, and the relational map between Koopman observable and output measurements are unknown. In a standard Koopman operator learning problem, the state measurement data is sufficient to approximate the Koopman operator. However, we examine the effect of adding output measurements now as a series of behavioral constraints on the dynamics of the Koopman operator — we aim to know the effect of incorporating output measurements (sensor fusion) on the solution of the Koopman operator learning problem. Specifically, we seek to understand how Koopman eigenfunctions, spectra, and modes change as a consequence of sensor fusion.

The formulation of an output-constrained Koopman operator is not novel. In the literature, output-constrained Koopman operators are used for various applications like observability analysis [98, 83], observer synthesis [81, 82, 84], and sensor placement [77, 76, 99] for nonlinear systems. In this paper, we prove that output-constrained Koopman operators satisfy the following properties:

1. (i)

   The output dynamics of the nonlinear system always span a subspace of observable functions for the output-constrained Koopman operator

2. (ii)

   The observables of the output-constrained Koopman operators can capture the dynamics of both states and outputs

3. (iii)

   State-inclusive output-constrained Koopman operators exist in the region of convergence of the Taylor series expansion of the dynamics and output functions of any nonlinear system.

Heretofore, there have been few algorithms that identify output-constrained Koopman operators, to the best of our knowledge. To identify output-constrained Koopman operators (OC-KO) from data, we pose the output-constrained DMD (OC-DMD) problem as a special extension of the DMD problem to incorporate output constraints. We propose two variants of the problem: the direct OC-DMD solves for the state and output dynamics concurrently, while the sequential OC-DMD solves for them sequentially. Sequential OC-DMD explicitly reveals the effect of having outputs in the KOR learning problem. To implement OC-DMD in practice, we build on the deepDMD algorithm developed by Yeung et al. [71], where neural networks represent vector valued observables of the Koopman operator. We then study the effect of affine transformations on the output-constrained Koopman operator learning problem to take into account— how data preprocessing methods like normalization or standardization modify the output-constrained Koopman operator. We use simulation examples to investigate the performance of OC-DMD algorithms. Our findings include:

1. (i)

   The solution space of the direct OC-DMD and sequential OC-DMD optimization problems are equivalent

2. (ii)

   Affine state transformations yield OC-KOs with an eigenvalue on the unit circle

3. (iii)

   OC-KOs optimized for multi-step predictions are required to capture the dynamics with limit cycles.

The paper is organized as follows. In section 2, we formulate the OC-KO representation. In section 3, we briefly discuss Koopman operator theory, Koopman operator sensor fusion and the necessary DMD algorithms. We discuss the properties of OC-KOs in Section 4 and the OC-DMD algorithms in Section 5. We show the simulation results in Section 6 and conclude our analysis in Section 7.

Our goal in this paper will be to consider physical systems represented by sampled time-series data, even their underlying governing dynamics are continuous. For methods that estimate the Koopman generator (the continuous-time extension of the discrete-time Koopman operator), we refer the reader to [100]. Suppose we have an autonomous discrete-time nonlinear dynamical system with output

where $x\in\mathcal{M}\subseteq\mathcal{R}^{n}$ is the state, $y\in\mathbb{R}^{p}$ is the output, $f:\mathcal{M}\rightarrow\mathcal{M}$ and $h:\mathcal{M}\rightarrow\mathbb{R}^{p}$ are analytic functions and $k$ is the discrete time index indicating the time point $kT_{s}$ with $T_{s}$ being the sampling time. Let

be the OC-KO of (1) where $\psi:\mathcal{M}\rightarrow\mathbb{R}^{n_{L}}$ is a vector function of state-dependent scalar observable functions ( $n_{L}\leq\infty$), $K\in\mathbb{R}^{n_{L}\times n_{L}}$ is the Koopman operator and $W_{h}\in\mathbb{R}^{p\times n_{L}}$ is a projection matrix that projects observables to the space of output functions. We record state and output measurements ($x$ and $y$, respectively) from (1) as

where $X_{P}\in\mathbb{R}^{n\times N}$ is the state data collected from time points $0,...,N-1$, and $X_{F}\in\mathbb{R}^{n\times N}$ and $Y_{P}\in\mathbb{R}^{p\times N}$ are the collection of state and output measurements propagated one step forward from the elements in $X_{P}$, respectively. Here, we use lower-case notation for the state $x$ and output $y$ variables and upper-case notation for variables containing sampled time-series snapshots involving the state variable $x$ with $X_{P},X_{F}$ and time-series with output variable $y$ with $Y_{P}$, respectively.

We now briefly introduce the formal mathematical elements of Koopman operator theory, its modal decomposition, Koopman operator sensor fusion and relevant DMD algorithms.

We consider the fusion of state and output measurements from the nonlinear system (1) using the Koopman operator sensor fusion method as delineated in Section 3.3. To do so, we need to establish a factor conjugate map between the state dynamics and the output dynamics.

We suppose the state dynamics is captured by the KO $\psi(x_{k+1})=K\psi(x_{k})$. To integrate the state dynamics with the outputs (that are nonlinear functions of the state), we consider the OC-KO (2).

The model structure of OC-KO is such that its dictionary of observables $\psi(x)$ span the nonlinear output functions $h(x)$ by augmenting the KO with a linear output equation $y_{k}=h(x_{k})=W_{h}\psi(x_{k})$. To establish a conjugacy between the state and output dynamics, all we need is to do is construct the dynamics of the output. The output dynamics can be given by

where $\psi(x_{k})=W_{\psi}y_{k}$. We begin by considering the simplest case of inverting the $W_{h}$ matrix in $y_{k}=W_{h}\psi(x_{k})$ to identify the $W_{\psi}$ matrix.

Case (1): Since $W_{h}$ has full column rank, an exact left inverse exists such that

and hence $\psi_{i}\in span\{h_{1},h_{2},...,h_{p}\}$ $\forall$ $i\in\{1,2,...,n_{L}\}$.
Case (2): Suppose $rank(W_{h})=r<n_{L}$. The singular value decomposition of $W_{h}$ yields

$W_{h\psi}$ has a full column rank $r>p$ and hence

Therefore, under a similarity transformation $T=\begin{bmatrix}V_{r}&V_{n_{L}-r}\end{bmatrix}$, the model (2) takes the form of (12) with $T^{-1}=T^{T}$ where the first $r$ lifting functions of $\tilde{\psi}$ given by $\psi_{h}$ lie in the span of $h$.  ∎

We see that if $W_{h}$ is full column rank, the outputs can be determined completely by the OC-KO observables and the $W_{\psi}$ in (11) exists. In the event that $n_{L}<p$, the map $y=W_{h}\psi(x)$ is a map from a low dimensional space $\psi(x)\in\mathbb{R}^{n_{L}}$ to a high dimensional space $y\in\mathbb{R}^{p}$ and factor conjugacy is not defined for that case (in Section 3.3). Hence, we project the outputs to a lower dimensional space $z=W_{h}^{T}y\in\mathbb{R}^{n_{L}}$ to find a conjugate map. The following two corollaries illustrate how output functions can be used to identify all or a subset of Koopman observables (in Section 3.3).

The dynamics of the output can be constructed when $W_{h}$ is full column rank. The dynamics of the entire output or the projected output has a diffeomorphic conjugacy with the dynamics of the OC-KO observables $\psi(x)$ depending on whether $p=n_{L}$ or $p>n_{L}$ respectively under the map $y=W_{h}\psi(x)$. The solution to the dictionary of Koopman observables that capture the state dynamics are generally nonunique; they have infinite KOs for a given system (1a). The fusion of the states and the outputs impose a constraint on the observables of the KOs. We explore this in the following corollary.

We explicitly see that the output equations place a constraint on the OC-KO observables and that the dynamics of the outputs can be constructed when $rank(W_{h})=n_{L}$. In the case (ii) of Theorem 1 where $rank(W_{h})<n_{L}$, using (4) to construct the output dynamics, we see that

where $\bar{\psi}_{h}(x_{k})$ is a leakage term that cannot be represented in the output space. Hence, the output cannot be constructed. To capture more information on $\bar{\psi}_{h}(x_{k})$, we need to construct the time-delay embedded outputs as the lifting functions [111, 112, 113]. This is a typical approach used to construct KORs when outputs partially measure the states [111, 112, 113, 114, 94, 90]. In the case of pure output measurements, the time-delay embedded outputs capture the maximum dynamics that is observed by the outputs. But, when we fuse it with the state dynamics, there is no guarantee that that the output dynamics capture the entire state dynamics. Hence, we consider a more general case where the outputs capture a portion of the state dynamics. Since the OC-KO has the structure of a linear time invariant system, we invoke the observable decomposition theorem from [115] to separate out the dynamics of the observable lifted states that can be fused with the output dynamics.

The Koopman operator representation (2) is a linear time invariant model. Using the observable decomposition theorem (Theorem 16.2 in [115]), there exists a similarity transformation $T$ that takes the system (2) to the form (16) such that the subsystem

is completely observable; $\psi_{o}(x)\in\mathbb{R}^{n_{o}}$ can be uniquely reconstructed from the outputs. Note that the system being observable is different from the observable functions in the context of Koopman. Then, there exists a $N\in\mathbb{Z}_{\geq 0}$ such that $Np\geq n_{o}$ and

where $\mathcal{O}\in\mathbb{R}^{Np\times n_{o}}$ has full column rank $n_{o}$. Then we can define a projection $W_{\psi}=\mathcal{O}^{T}$ to get (17) such that

where $\mathcal{O}^{T}\mathcal{O}\in\mathbb{R}^{n_{o}\times n_{o}}$ is square invertible. The dynamics of $z$ is given by

For the map $H:\mathbb{R}^{n_{o}}\rightarrow\mathbb{R}^{n_{o}}$, there exists an inverse map $H^{-1}:\mathbb{R}^{n_{o}}\rightarrow\mathbb{R}^{n_{o}},H^{-1}(z)=(\mathcal{O}^{T}\mathcal{O})^{-1}z_{k}$ since $\mathcal{O}^{T}\mathcal{O}$ is full rank and invertible. Using the above, we can see that

Hence, the dynamics of $z$ and $\psi_{o}(x)$ are diffeomorphically conjugate.  ∎

When outputs partially measure the states, we see that time-delay embedded outputs have a diffeomorphic map with a subspace of the lifting functions under a similarity transform and the dynamics evolving in the two spaces are diffeomorphically conjugate.

We see that the OC-KO architecture can fuse the state and output dynamics even if the outputs do not capture the entire state dynamics. The above analysis shows that the OC-KO structure is such that the lifting functions capture the dynamics of the time-delay embedded outputs. This is an implicit constraint in the model structure of OC-KO.

State-inclusive observables (5) are useful since we can recover the dynamics by simply dropping the nonlinear observables. We show a sufficient condition for the existence of state-inclusive OC-KOs using a similar argument developed in [83]. We prove the following lemma which plays a crucial role in showing the invariance of the basis in the series expansions of analytic functions.

Consider two functions $\psi_{\alpha}(x),\psi_{\beta}(x)\in\mathcal{D}$. Their product is given by

Since the product of two functions lies in $\mathcal{D}$, the product of any number of functions from $\mathcal{D}$ also lies in $\mathcal{D}$ as can be seen by taking repeated product of functions.  ∎

If the monomial basis in the Taylor series expansion are propagated by one time step, we encounter a product of these monomials and Lemma 10 defines a set that captures all these monomial functions and their products. We use the this lemma to build on the result from [83], which shows the existence of state-inclusive KO for (1a), to find state-inclusive OC-KOs for (1).

Let us consider a dictionary of polynomial lifting functions $\mathcal{D}=\{\psi_{1}(x),\psi_{2}(x),\ldots\},\quad\psi_{r}(x)=\displaystyle\prod_{i=1}^{n}x_{i}^{p_{r,i}}$ where $p_{r,i}\in\mathbb{Z}^{+}$. Since $f$ is real analytic in $\mathcal{M}$, for any $x_{0}\in\mathcal{M}$, there exists a Taylor series expansion centered about $x_{0}$ that converges to $f(x)$ for any neighborhood of $x_{0}$. Suppose $f(x)=\begin{bmatrix}f_{1}(x)&\cdots f_{n}(x)\end{bmatrix}^{T}$, the Taylor series expansion of $f$ about $x_{0}$ yields

where each term lies in $\mathcal{D}$. Suppose $x_{k}=\begin{bmatrix}x_{k,1}&\cdots x_{k,n}\end{bmatrix}^{T}$ where $x_{k,i}$ indicates the $i^{th}$ state at discrete time index $k$. $x_{k}$ propagated by one time step yields a linear combination of functions in $\mathcal{D}$ as

To construct a linear system, we propagate each function on the right hand side $\psi_{j}(x_{k})$ by one time step

Since Lemma 10 states that the product of any number of functions in $\mathcal{D}$ lies in $\mathcal{D}$, $\psi_{j}(x_{k+1})=\sum_{r=1}^{\infty}k_{ir}\psi_{r}(x_{k})$. Concatenating the expressions of $x_{k+1,j}$ and $\psi_{j}(x_{k+1})$ for all $j$, we get the Koopman representation $\psi(x_{k+1})=K\psi(x_{k})$.

Similarly, for the output equation, we can expand each function $h_{i}$ in $h(x)=\begin{bmatrix}h_{1}(x)&h_{2}(x)&\cdots h_{m}(x)\end{bmatrix}^{T}$ using the Taylor series expansion about $x=x_{0}$ to yield

Hence, if $f$ and $h$ of the nonlinear system (1) are analytic in the open set $\mathcal{M}$, a state inclusive OC-KO of the form (2) exists for that system.  ∎

We see that state-inclusive OC-KOs exist for nonlinear systems whose dynamics ($f$) and output ($h$) functions are real analytic. This is a sufficient condition and not a necessary one. In the next section, we explore how to identify OC-KOs from data using the DMD formulation.

The OC-KO identification involves fusing two datasets (states and outputs) to learn the dynamics of the nonlinear system (1). The DMD algorithms typically identify KOs on one dataset. We introduce the more general OC-DMD formulation that incorporates the output constraint in the DMD problem to identify the OC-KOs. The OC-DMD formulation is

where $||.||_{F}$ is the Frobenius norm. The equality constraint is very stringent as the presence of output measurement noise could result in overfit models. Hence, we pose the weaker problem

which concurrently solves for $\psi(x),K$ and $W_{h}$. We refer to this as the direct OC-DMD problem formulation. To explicitly show the effect of the output dynamics in learning OC-KOs, we also propose the sequential OC-DMD problem formulation where the following sub-problems are solved sequentially:

This problem is called sequential OC-DMD, because it obtains a solution for the output-constrained Koopman learining problem as a sequence of optimization problems. The solution generated by sequential OC-DMD, yields an OC-KO of the form:

where