Stabilizing Linear Systems under Partial Observability:
Sample Complexity and Fundamental Limits

Learning-based control of unknown dynamical systems is of critical importance for many autonomous control systems [Beard et al., 1997; Li et al., 2022; Bradtke et al., 1994; Krauth et al., 2019; Dean S. Mania, 2020]. However, existing methods make strong assumptions such as open-loop stability, availability of initial stabilizing controller, and fully observable systems [Fattahi, 2021; Sarkar et al., 2019]. However, such assumptions may not hold in many autonomous control systems. Motivated by this gap, this paper focuses on the problem of stabilizing an unknown, partially-observable, unstable system without an initial stabilizing controller. Specifically, we consider the following linear-time-invariant (LTI) system:

where $x_{t}\in\mathbb{R}^{n}$ and $u_{t}\in\mathbb{R}^{d_{u}},y_{t}\in\mathbb{R}^{d_{y}}$ are the state, control input, and observed output at time step $t\in\{0,\dots,T-1\}$, respectively. The system also has additive observation noise $v_{t}\sim\mathcal{N}(0,\sigma_{v}^{2}I)$.

While there are works studying system identification for partially observable LTI systems [Fattahi, 2021; Sun et al., 2020; Sarkar et al., 2019; Oymak and Ozay, 2018b; Zheng and Li, 2020], none of these works address the subsequent stabilization problem; in fact, many assume open-loop stability [Fattahi, 2021; Sarkar et al., 2019; Oymak and Ozay, 2018b]. Other adaptive control approaches can solve the learn-to-stabilize problem in (1) and achieve asymptotic stability guarantees [Pasik-Duncan, 1996; Petros A. Ioannou, 2001], but the results therein are asymptotic in nature and do not study the transient performance.

Recently, in the special case when $C=I$, i.e. fully-ovservable LTI system, Chen and Hazan [2020] reveals that the transient performance during the learn-to-stabilize process suffers from exponential blow-up, i.e. the system state can blow up exponentially in the state dimension. This is mainly due to that in order to stabilize the system, system identification for the full system is needed, which takes at least $n$ samples on a single trajectory, during which the system could blow up exponentially for $n$ steps.

To relieve this problem, Hu et al. [2022] proposed a framework of separating the unstable component of the system and stabilizing the entire system by stabilizing the unstable subsystem. This reduces the sample complexity to only grow with the number of the unstable eigenvalues (as opposed to the full state dimension $n$). This result was later extended to noisy setting in Zhang et al. [2024], and such a dependence on the number of unstable eigenvalues is the best sample complexity so far for the learn-to-stabilize problem in the fully observable case. Compared with this line of work, our work focuses on partially observable systems, which is fundamentally more difficult than the fully observable case. First, partially observable systems typically need a dynamic controller, which renders stable feedback controllers in the existing approach not applicable [Hu et al., 2022]. Second, the construction in “unstable subspace” in Hu et al. [2022] is not uniquely defined in the partially observable case. This is because the state and the $A,B,C$ matrices are not uniquely identifiable from the learner’s perspective, as any similarity transformation of the matrices can yield the same input-output behavior. Given the above discussions, we ask the following question:

Is it possible to stabilize a partially observable LTI system by only identifying an unstable component (to be properly defined) with a sample complexity independent of the overall state dimension?

Contribution. In this paper, we answer the above question in the affirmative. Firstly, we propose a novel definition of “unstable component” to be a low-rank version of the transfer function of the original system that only retains the unstable eigenvalues. Based on this unstable component, we propose LTS-P, which leverages compressed singular value decomposition (SVD) on a “lifted” Hankel matrix to estimate the unstable component. Then, we design a robust stabilizing controller for the unstable component, and show that it stabilizes the full system under a small-gain-type assumption on the $H_{\infty}$ norm of the stable component. Importantly, our theoretical analysis shows that the sample complexity of the proposed approach only scales with the dimension of unstable component, i.e. the number of unstable eigenvalues, as opposed to the dimension of the full state space. We also conduct simulations to validate the effectiveness of LTS-P, showcasing its ability to efficiently stabilize partially observable LTI systems.

Moreover, the novel technical ideas behind our approach are also of independent interest. We show that by using compressed singular value decomposition (SVD) on a properly defined lifted Hankel matrix, we can estimate the unstable component of the system. This is related to the classical model reduction [Dullerud and Paganini, 2000, Chapter 4.6], but our dynamics contain unstable modes with a blowing up Hankel matrix as opposed to the stable Hankel matrix in Dullerud and Paganini [2000] and the system identification results in Fattahi [2021]; Sarkar et al. [2019]; Oymak and Ozay [2018b]. Interestingly, the $H_{\infty}$ norm condition on the stable component, derived from the small gain theorem, is a necessary and sufficient condition for stabilization, revealing the required subspace to be estimated in order to stabilize the unknown systems. Such a result not only suggests the optimality of LTS-P but also informs the fundamental limit on stabilizability.

Related Work. Our work is mostly related to adaptive control, learn-to-control with known stabilizing controllers, learn-to-stabilize on multiple trajectories, and learn-to-stabilize on a single trajectory. In addition, we will also briefly cover system identification.

Adaptive control. Adaptive control enjoys a long history of study [Pasik-Duncan, 1996; Petros A. Ioannou, 2001; Chen and Astolfi, 2021]. Most classical adaptive control methods focus on asymptotic stability and do not provide finite sample analysis. The more recent work has started to study non-asymptotic sample complexity of adaptive control when a stabilizing controller is unknown [Chen and Hazan, 2020; Faradonbeh, 2017; Lee et al., 2023; Tsiamis and Pappas, 2021; Tu and Recht, 2018]. Specifically, the most typical strategy to stabilize an unknown dynamic system is to use past trajectory to estimate the system dynamics and then design the controller [Berberich et al., 2020; De Persis and Tesi, 2020; Liu et al., 2023]. Compared with those works, we can stabilize the system with fewer samples by identifying and stabilizing only the unstable component.

Learn to control with known stabilizing controller. Extensive research has been conducted on stabilizing LTI under stochastic noise [Bouazza et al., 2021; Jiang and Wang, 2002; Kusii, 2018; Li et al., 2022]. One branch of research uses the model-free approach to learn the optimal controller [Fazel et al., 2019; Jansch-Porto et al., 2020; Li et al., 2022; Wang et al., 2022; Zhang et al., 2020]. Those algorithms typically require a known stabilization controller as an initialization policy for the learning process. Another line of research utilizes the model-based approach, which requires an initial stabilizing controller to learn the system dynamics for controller design [Cohen et al., 2019; Mania et al., 2019; Plevrakis and Hazan, 2020; Zheng et al., 2020]. On the other hand, we focus on learn-to-stabilize. Our method can be used as the initial policy in these methods to remove their requirement on initial stabilizing controllers.

Learn-to-stabilize on multiple trajectories. There are also works that do not assume open-loop stability and learn the full system dynamics before designing a stabilizing controller while requiring $\widetilde{\Theta}(\text{poly}(n))$ complexity [Dean S. Mania, 2020; Tu and Recht, 2018; Zheng and Li, 2020]. Recently, a model-free approach via the policy gradient method offers a novel perspective with the same complexity [Perdomo et al., 2021]. Compared with those works, the proposed algorithm requires fewer samples to design a stabilizing controller.

Learn-to-stabilize on a single trajectory. Learning to stabilize a linear system in an infinite time horizon is a classic problem in control [Lai, 1986; Chen and Zhang, 1989; Lai and Ying, 1991]. There have been algorithms incurring regret of $2^{O(n)}O(\sqrt{T})$ which relies on assumptions of observability and strictly stable transition matrices [Abbasi-Yadkori and Szepesvári, 2011; Ibrahimi et al., 2012]. Some studies have improved the regret to $2^{\tilde{O}(n)}+\tilde{O}(\text{poly}(n)\sqrt{T})$ [Chen and Hazan, 2020; Lale et al., 2020]. Recently, Hu et al. [2022] proposed an algorithm that requires $\tilde{O}(k)$ samples. While these techniques are developed for fully observable systems, LTS-P works for partially observable unstable systems, which requires significantly greater technical challenges as detailed above.

System identification. Our work is also related to works in system identification, which focuses on determining the parameters of the system [Oymak and Ozay, 2018a; Sarkar and Rakhlin, 2018; Simchowitz et al., 2018; Xing et al., 2022]. Our work utilizes a similar approach to partially determine the system parameters before constructing the stabilizing controller. While these works focus on identifying the system dynamics, we close the loop and provide state-of-the-art sample complexity for stabilization.

Notations. We use $\left\|\cdot\right\|$ to denote the $L^{2}$-norm for vectors and the spectral norm for matrices. We use $M^{*}$ to represent the conjugate transpose of $M$. We use $\sigma_{\min}(\cdot)$ and $\sigma_{\max}(\cdot)$ to denote the smallest and largest singular value of a matrix, and $\kappa(\cdot)$ to denote the condition number of a matrix. We use the standard big $O(\cdot)$, $\Omega(\cdot)$, $\Theta(\cdot)$ notation to highlight dependence on a certain parameter, hiding all other parameters. We use $f\lesssim g$, $f\gtrsim g$, $f\asymp g$ to mean $f=O(g),f=\Omega(g),f=\Theta(g)$ respectively while only hiding numeric constants.

For simplicity, we primarily deal with the system where $D=0$. For the case where $D\neq 0$, we can easily estimate $D$ in the process and subtract $Du_{t}$ to obtain a new observation measure not involving control input. We briefly introduce the method for estimating $D$ and how to apply the proposed algorithm in the case when $D\neq 0$ in Appendix D.

Learn-to-stabilize. As the unknown system as defined in (1) can be unstable, the goal of the learn-to-stabilize problem is to return a controller that stabilizes the system using data collected from interacting with the system on $M$ rollouts. More specifically, in each rollout, the learner can determine $u_{t}$ and observe $y_{t}$ for a rollout of length $T$ starting from $x_{0}$, which we assume $x_{0}=0$ for simplicity of proof.

Goal. The sample complexity of stabilization is the number of samples, $MT$, needed for the learner to return a stabilizing controller. Standard system identification and certainty equivalence controller design need at least $\Theta(n)$ samples for stabilization, as $\Theta(n)$ is the number of samples needed to learn the full dynamical system. In this paper, our goal is to study whether it is possible to stabilize the system with sample complexity independent from $n$.

In this paper, we assume the matrix $A$ does not have marginally stable eigenvalues, and the eigenvalues are ordered as follows:

The high-level idea of the paper is to first decompose the system dynamics into the unstable and stable components (Section 3.1), estimate the unstable component via a low-rank approximation of the lifted Hankel matrix (Section 3.2), and design a robust stabilizing for the unstable component that stabilizes the whole system (Section 3.3).

Based on the ideas developed in Section 3, we now design an algorithm that learns to stabilize from data. The pseudocode is provided in Algorithm 1.

Step 1: Approximate Low-Rank Lifted Hankel Estimation. Section 3 shows that if $\tilde{H}$ defined in (25) is known, then we can design a controller satisfying (28) to stabilize the system. In this section, we discuss a method to estimate $\tilde{H}$ with singular value decomposition (SVD) of the lifted Hankel metrix.

Data collection and notation. Consider we sample $M$ trajectories, each with length $T$. To simplify notation, for each trajectory $i\in\{1,\dots,M\}$, we organize input and output data as

where each $u_{j}^{(i)}\sim\mathcal{N}(0,\sigma_{u}^{2})$ are independently selected. We also define the a new matrix for the observation noise as

respectively. To substitute the above into (1), we obtain

We further define an upper-triangular Toeplitz matrix

and we define the following notations:

Note that the lifted Hankel matrix $H$ in (20) can be recovered from (34) as they contain the same block submatrices.

Estimation low rank lifted Hankel from measurement. The measurement data (32) in each rollout can be written as

Therefore, we can estimate $\Phi$ by the following ordinary least square problem:

We then estimate the lifted Hankel matrix as follows:

Let $\hat{H}=\hat{U}_{H}\hat{\Sigma}_{H}\hat{V}_{H}^{*}$ denote the singular value decomposition of $\hat{H}$, and we define the $k$-th order estimation of $\hat{H}$:

where $\hat{U}$ ($\hat{V}$) is the matrix of the top $k$ left (right) singular vector in $\hat{U}_{H}$ ($\hat{V}_{H}$), and $\hat{\Sigma}$ is the matrix of the top $k$ singular values in $\hat{\Sigma}_{H}$.

Step 2: Esitmating unstable transfer function F. With the SVD $\hat{\tilde{H}}=\hat{U}\hat{\Sigma}\hat{V}^{*}$, we further do the following factorization:

With the above, we can estimate $\bar{N}_{1},\bar{C},\bar{B}$ similar to the procedure introduced in Section 3.2:

To reduce the error of estimating $CQ_{1}$ and $R_{1}B$ and avoid compounding error caused by raising $\hat{N}_{1}$ to the $m/2$’th power, we also directly estimate $\bar{N}_{1}^{\frac{m}{2}}$ from both $\hat{\mathbf{\mathcal{O}}}$ and $\hat{\mathbf{\mathcal{C}}}$.

In practice, the estimation obtained from (41) and (42) are very similar, and we can use either one to estimate $\bar{N}_{1}^{\frac{m}{2}}$. Lastly, we estimate $\bar{C}$ and $\bar{B}$ as follows:

where for simplicity, we use $\widehat{N_{1,O}^{-\frac{m}{2}}},\widehat{N_{1,C}^{-\frac{m}{2}}}$ to denote the invserse of $\widehat{N_{1,O}^{\frac{m}{2}}},\widehat{N_{1,C}^{\frac{m}{2}}}$. We will provide accuracy bounds for those estimations in Section 6. With the above estimations, we are ready to design a controller with the following transfer function.

Step 3: Designing robust controller. After estimating the system dynamics and obtain the estimated transfer function in (44) as discussed in Section 3.3, we can design a stabilizing controller by existing $H_{\infty}$ control methods to minimize $\|K(z)F_{K}(z)\|_{H_{\infty}}$. The details on how to design the robust controller can be found in robust control documentations, e.g. Chapter 7 of Dullerud and Paganini [2000].

What if $k$ is not known a priori? The proposed method requires knowledge of $k$, the number of unstable eigenvalues. If $k$ is not known, we show in Lemma 6.1 and Lemma A.3 that the first $k$ singular values of $\hat{H}$ and $H$ increase exponentially with $m$, and the remaining singular values decrease exponentially with $m$. Therefore, with a reasonably sized $m$ and if $p,q$ are chosen to be larger than $k$, there will be a large spectral gap among the singular values of $\hat{H}$. The learner can use the location of the spectral gap to determine the value of $k$.

In this section, we provide the sample complexity needed for the proposed algorithm to return a stabilizing controller. We first introduce a standard assumption on controllabilty and observability.

We also need an assumption on the existence of a controller that meets the small gain theorem’s criterion (28).

To state our main result, we introduce some system theoretic quantity.

Controllability/observability Gramian for unstable subsystem. For the unstable subsystem $N_{1},R_{1}B,CQ_{1}$, its $\alpha$-observability Gramian is

and its $\alpha$-controllability Gramian is $\mathcal{G}_{\mathrm{con}}=[R_{1}B,N_{1}R_{1}B,\ldots,N_{1}^{\alpha-1}R_{1}B]$. Per Lemma F.1 and 5.1, we know the $N_{1},R_{1}B,CQ_{1}$ subsystem is both controllable and observable, and hence we can select $\alpha$ to be the smallest integer such that both $\mathcal{G}_{\mathrm{ob}},\mathcal{G}_{\mathrm{con}}$ are rank-$k$. Note that we always have $\alpha\leq k$. Our main theorem will use the following system-theoretic parameters: $\alpha$, $\kappa(\mathcal{G}_{\mathrm{ob}}),\kappa(\mathcal{G}_{\mathrm{con}})$ (the condition numbers of $\mathcal{G}_{\mathrm{ob}},\mathcal{G}_{\mathrm{con}}$ respectively), and $\sigma_{\min}(\mathcal{G}_{\mathrm{ob}}),\sigma_{\min}(\mathcal{G}_{\mathrm{con}})$ (the smallest singular values of $\mathcal{G}_{\mathrm{ob}},\mathcal{G}_{\mathrm{con}}$ respectively).

Umbrella upper bound $L$. We use a constant $L$ to upper bound the norms $\|A\|,\|B\|,\|C\|,\|N_{1}\|,$ $\|R_{1}B\|,\|CQ_{1}\|$. We will use the Jordan form decomposition for $N_{1}=P_{1}\Lambda P_{1}^{-1}$, and let $L$ upper bound $\|P_{1}\|\|P_{1}^{-1}\|=\kappa(P_{1})$. We also use $L$ in $\sup_{|z|=1}\|(zI-N_{1})^{-1}\|\leq\frac{L}{|\lambda_{k}|-1}$. Lastly, we use $L$ to upper bound the constant in Gelfand’s formula (Lemma F.2) for $N_{2}$ and $N_{1}^{-1}$. Specifically we will use $\|N_{2}^{t}\|\leq L(\frac{|\lambda_{k+1}|+1}{2})^{t}$, $\|(N_{1}^{-1})^{t}\|\leq L(\frac{\frac{1}{|\lambda_{k}|}+1}{2})^{t}$, $\forall t\geq 0$.

With the above preparations, we are ready to state the main theorem.

The total number of samples needed for the algorithm is $MT=(3m+2)M$. In the bound in Theorem 5.3, the only constant that explicitly grows with $k$ is the controllability/observability index $\alpha=O(k)$ which appears in the bound for $m$. Therefore, the sample complexity $MT=O(m^{2})=O(\alpha^{2})=O(k^{2})$ grows quadratically with $k$ and independent from system state dimension $n$.In the regime $k\ll n$, this significantly reduces the sample complexity of stabilization compared to methods that estimate the full system dynamics. To the best of our knowledge, this is the first result that achieves stabilization of unknown partially observable LTI system with sample complexity independent from the system state dimension $n$.

Dependence on system theoretic parameters. As the system becomes less observable and controllable, $\kappa(\mathcal{G}_{\mathrm{con}})$,$\kappa(\mathcal{G}_{\mathrm{ob}})$ increases and $\sigma_{\min}(\mathcal{G}_{\mathrm{con}}),\sigma_{\min}(\mathcal{G}_{\mathrm{ob}})$ decreases, increasing $m$ and the sample complexity $MT$ grows in the order of $(\log\frac{\kappa(\mathcal{G}_{\mathrm{ob}})\kappa(\mathcal{G}_{\mathrm{con}})}{\min(\sigma_{\min}(\mathcal{G}_{\mathrm{ob}}),\sigma_{\min}(\mathcal{G}_{\mathrm{con}}))})^{2}$. Moreover, when $|\lambda_{k+1}|,|\lambda_{k}|$ are closer to $1$, the sample complexity also increases in the order of $(\max(\frac{1}{1-|\lambda_{k+1}|},\frac{1}{|\lambda_{k}|-1}\log\frac{1}{|\lambda_{k}|-1}))^{2}$. This makes sense as the unstable and stable components of the system would become closer to marginal stability and harder to distinguish apart. Lastly, if the $\epsilon_{*}$ in 5.2 becomes smaller, we have a smaller margin for stabilization, which also increases the sample complexity in the order of $(\log\frac{1}{\epsilon_{*}})^{2}$.

Non-diagonalization case. Theorem 5.3 assumes $N_{1}$ is diagonalizable, which is only used in a single place in the proof in Lemma B.1. More specifically, we need to upper bound $J^{t}(J^{*})^{-t}$ for some Jordan block $J$ and integer $t$, and in the diagonalizable case an upper bound is $1$ as the Jordan block is size $1$. In the case that $N_{1}$ is not diagonalizable, a similar bound can still be proven with dependence on the size of the largest Jordan block of $N_{1}$, denoted as $\gamma$. Eventually, this will lead to an additional multiplicative factor $\gamma$ in the sample complexity on $m$ (cf. (141)), whereas the requirements for $p,q,M$ is the same. The derivations can be found in Appendix E. In the worst case that $\gamma=k$ ($N_{1}$ consists of a single Jordan block), the sample complexity will be worsend to $\tilde{O}(k^{4})$ but still independent from $n$.

Necessity of Assumption 5.2. 5.2 is needed if one only estimates the unstable component of the system, treating the stable component as unknown. This is because the small gain theorem is an if-and-only-if condition. If 5.2 is not true, then no matter what controller $K(z)$ one designs, there must exist an adversarially chosen stable component $\Delta(z)$ (not known to the learner) causing the system to be unstable, even if $F(z)$ is perfectly learned. For a specific construction of such a stability-breaking $\Delta(z)$, see the proof of the small gain theorem, e.g. Theorem 8.1 of Zhou and Doyle [1998]. Although Hu et al. [2022]; Zhang et al. [2024] do not explictly impose this assumption, they impose similar assumptions on the eigenvalues, e.g. the $|\lambda_{1}||\lambda_{k+1}|<1$ in Theorem 4.2 of Zhang et al. [2024], and we believe the small gain theorem and 5.2 is the intrinsic reason underlying those eigenvalue assumptions in Zhang et al. [2024]. We believe this requirement of 5.2 is the fundamental limit of the unstable + stable decomposition approach. One way to break this fundamental limit is that instead of doing unstable + stable decomposition, we learn a larger component corresponding to eigenvalues $\lambda_{1},\ldots,\lambda_{\tilde{k}}$ for some $\tilde{k}>k$, which effectively means we learn the unstable component and part of the stable component of the system. Doing so, 5.2 will be easier to satisfy as when $\tilde{k}$ approaches $n$, $\|\Delta(z)\|_{H_{\infty}}$ will approach $0$. We leave this as a future direction.

Relation to hardness of control results. Recently there has been related work on the hardness of the learn-to-stabilize problem [Chen and Hazan, 2020; Zeng et al., 2023]. Our setting does not conflict these hardness results for the following reasons. Firstly, our result focuses on the $k\ll n$ regime. In the regime $k=n$, our result does not improve over other approaches. Secondly, our 5.2 is related to the co-stabilizability concept in Zeng et al. [2023], as 5.2 effectively means the existence of a controller that stabilizes the system for all possible $\Delta(z)$. In some sense, our results complements these results showing when is learn-to-stabilize easy under certain assumptions.

In this section, we will give a high-level overview of the key proof ideas for the main theorem. The full proof details can be found in Appendix A (for Step 1), Appendix B (for Step 2), and Appendix C (for Step 3).

Proof Structure. The proof is largely divided into three steps following the idea developed in Section 3. In the first step, we examine how accurately the learner can estimate the low-rank lifted Hankel matrix $\tilde{H}$. In the second step, we examine how accurately the learner estimates the transfer function of the unstable component from $\tilde{H}$. In the last step, we provide a stability guarantee.

Step 1. We show that $\hat{\tilde{H}}$ obtained in (38) is an accurate estimate of $\tilde{H}$.

We later (in Step 2, 3) will show that for a sufficiently small $\epsilon$, the robust controller produced by the algorithm will stabilize the system in (1). We will invoke Lemma 6.1 at the end of the proof in Step 3 to pick $m,p,q,M$ values that achieve the required $\epsilon$ (in Appendix C).

Step 2. In this step, we will show that $N_{1},R_{1}B,CQ_{1}$ can be estimated up to an error of $O(\epsilon)$ and up to a similarity transformation, given that $\left\|\hat{\tilde{H}}-\tilde{H}\right\|<\epsilon$.

Step 3. We show that, under sufficiently accurate estimation, the controller $\hat{K}(z)$ returned by the algorithm stabilizes plant $F(z)$ and hence $\|\hat{K}(z)F_{\hat{K}}(z)\|_{H_{\infty}}$ is well defined. This is done via the following helper proposition.