Topology Identification under Spatially Correlated Noise

Networks and graphical models provide convenient tools for effective representations of complex high dimensional multi-agent systems. Such a representation is useful in applications including power grids [32, 47], meteorology [19], neuroscience [4], and finance [5]. Knowledge of the network interaction structure (also known as network topology) helps in understanding, predicting, and in many applications, controlling/identifying the system behavior [16, 24, 28, 35, 39, 45]. In applications such as power grid, finance, and meteorological system, it is either difficult or impossible to intervene and affect the system. Hence, inferring the network properties by passive means such as time-series measurements is of great interest in these applications. An example is unveiling the correlation structure between the stocks in the stock market from daily share prices [5], which is useful in predicting the market behavior.

Learning the conditional independence relation between variables from time-series measurements is an active research field among machine learning (ML), probabilistic graphical model (PGM), and statistics communities [6, 8, 29, 30, 33, 38], where the system modules are considered as random variables. However, such studies fail to capture dynamic dependencies between the entities in a system, which are prevalent in most of the aforementioned applications. For dynamical systems, autoregressive (AR) models that are excited by exogenous Gaussian noise sources, which are independent across time and variables, are explored in [1, 2, 3, 40]. Here, the graph topology captured the sparsity pattern of the regressor coefficient matrices, which characterized the conditional independence between the variables. It was shown that the sparsity pattern of the inverse power spectral density matrix (IPSDM), also known as concentration matrix, identifies the conditional independence relation between the variables (see [3, 40]). As shown in [27], the conditional independence structure between the variables is equivalent to the moral-graph structure of the underlying directed graph. For multi-agent systems with linear dynamical interaction, excited by wide sense stationary (WSS) noise sources that are mutually uncorrelated across the agents, moral graph reconstruction using Wiener projection has gained popularity in the last decade [21, 25, 27]. Here, the moral-graph of the underlying linear dynamic influence model (LDIM) is recovered from the magnitude response of Wiener coefficients, obtained from time-series measurements. For a wide range of applications, the spurious connections in the moral-graph can be identified–returning the true topology–by observing the phase response of the Wiener coefficients [43]. Furthermore, for systems with strictly causal dynamical dependencies, Granger causality based algorithms can unveil the exact cause-effect nature of the interactions; thus recovering the exact parent-child relationship of the underlying graph [14, 15, 27].

In many networks, time-series at only a subset of nodes is available. The nodes where the time-series measurements are not available form the latent/confound/hidden nodes. Topology reconstruction is more challenging in the presence of latent nodes as additional spurious connections due to confounded effects are formed when applying the aforementioned techniques. AR model identification of the independent Gaussian time-series discussed above, in the presence of latent nodes, was studied in [9, 10, 11, 13, 17, 22, 48, 49]. Here, the primary goal is to eliminate the spurious connections due to latent nodes and retrieve the original conditional independence structure from the observed time-series. In applications such as power grid, there is a need to retrieve the complete topology of the network, including that of the latent nodes. Such problems are studied for bidirected tree [42], poly-forest [36], and poly-tree [26, 37] networks excited by WSS noise sources that are uncorrelated across the agents. Recently, an approach to reconstruct the complete topology of a general linear dynamical network with WSS noise was provided in [46]. The work in [46] can be considered as a generalization of AR model identification with latent nodes, in asymptotic time-series regime. However, one major caveat of the aforementioned literature on graphical models is that the results fail if the exogenous noise sources are spatially correlated, i.e. if the noise is correlated across the agents/variables. Prior works have studied the systems with spatially correlated noise sources [18, 28, 35]; however, these studies assume the knowledge of topology of the network. In a related work, [34] studied topology estimation under spatially correlated noise sources and used this estimated topology in local module identification.

This article studies the problem of topology identification of the LDIMs that allow spatially correlated noise sources, similar to the problem in [34]. However, this article provides an alternate treatment, where the noise correlations are transformed to latent nodes. This transformation enables one to gain additional insights and apply techniques from topology identification with latent nodes to solve the problem.

The first major result of this article is to provide a transformation that converts an LDIM without latent nodes, but excited by spatially correlated exogenous noise sources, to LDIMs with latent nodes. Here, the latent nodes are characterized by the maximal cliques in the correlation graph, the undirected graph that represents the spatial correlation structure. It is shown that, under affine correlation assumption, that is, when the correlated noise sources are related in affine way (Assumption 1), there exist transformed LDIMs with latent nodes, where all the nodes are excited by spatially uncorrelated noise sources. A key feature of the transformation is that the correlations are completely captured using the latent nodes, while the original topology remains unaltered. The original moral-graph/topology is the same as the moral-graph/topology among the observed nodes in the altered graph. Thus, this transformed problem is shown to be equivalent to topology identification of networks with latent nodes. Consequently, any of the aforementioned techniques for the networks with latent node can be applied on the transformed LDIM to reconstruct the original moral graph/topology.

Next, relaxing the affine correlation assumption, polynomial correlation is considered (Assumption 2); here, the focus is on noise sources with distributions that are symmetric around the mean. It is shown that, in this scenario, the transformed dynamical model can be excited by clusters of noise sources, where the clusters are uncorrelated. However, the noise sources inside the clusters can be correlated. Using the sparse$+$low-rank decomposition of IPSDM from [46], the true topology of the network along with the correlation structure is reconstructed, from the IPSDM of the original LDIM, without any additional information, if the network satisfies a necessary and sufficient condition. Notice that, the results discussed here are applicable to the networks with static random variables and AR models also, when the exogenous noise sources are spatially correlated.

The article is organized as follows. Section 2 introduces the system model, including LDIM, and the essential definitions. Section 3 discusses IPSDM based topology reconstruction. Section 4 describes the correlation graph to latent nodes transformation and some major results. Section 5 discusses transformation of LDIMs with polynomial correlation to LDIMs with latent nodes. Section 6 explains the sparse$+$low-rank decomposition technique and how the topology can be reconstructed without the knowledge of correlation graph. Simulation results are provided in Section 7. Finally, Section 8 concludes the article.

Notations: Bold capital letters, ${\mathbf{A}}$, denotes matrices; $A_{ij}$ and $({\mathbf{A}})_{ij}$ represents $(i,j)^{\text{th}}$ entry of ${\mathbf{A}}$; Bold small letters, ${\mathbf{v}}$, denotes vectors; $v_{i}$ indicates $i^{\text{th}}$ entry of ${\mathbf{v}}$; $[n]:=\{1,\dots,n\}$; The subscript $o$ denotes the observed nodes index set $[n]$ and the subscript $h$ indicates the latent nodes index set $\{n+1,\dots,n+L\}$, For example, ${\mathbf{x}}_{o}:=\{x_{1},\dots,x_{n}\}$ and ${\mathbf{x}}_{h}:=\{x_{n+1},\dots,x_{n+L}\}$; $\Phi_{\mathbf{x}}(z):=[(\Phi_{\mathbf{x}}(z))_{ij}],~{}z\in\mathbb{C},~{}|z|=1$ denote power spectral density matrix, where $(\Phi_{\mathbf{x}})_{ij}$ is cross power spectral density between the time-series at nodes $i$ and $j$; ${\mathcal{F}}$ denotes the set of real rational single input single output transfer functions that are analytic on unit circle; ${\mathcal{A}}$ denotes the set of rationally related zero mean jointly wide sense stationary (JWSS) scalar stochastic process; ${\mathcal{Z}}(.)$ denotes bilateral $z$-transform; ${\mathbb{S}}^{n}$ denotes the space of all skew symmetric $n\times n$ matrices; $\mathbb{N}$ denotes the set of natural numbers, $\{0,1,2,\dots\}$ ; For a set ${\mathcal{S}}$, $|{\mathcal{S}}|$ denotes the cardinality of the set. $\|{\mathbf{A}}\|_{*}$ denotes the nuclear norm and $\|{\mathbf{A}}\|_{1}$ denote sum of absolute values of entries of ${\mathbf{A}}$. $\phase{a}$ denotes the phase of the complex number $a$.

Consider a network of $n$ interconnected nodes, where node $i$ is equipped with time-series measurements, $(\widetilde{x}_{i}(t))_{t\in\mathbb{Z}}$, $i\in[n]$. The network interaction is described by,

where ${\mathbf{x}}(z)=[x_{1}(z),x_{2}(z),\dots,x_{n}(z)]$, ${\mathbf{x}}_{i}(z)=\mathcal{Z}[{{\widetilde{x}}_{i}}]$. ${\mathbf{e}}(z)=[e_{1}(z),e_{2}(z),\dots,e_{n}(z)]\in{\mathcal{A}}^{n}$ are the exogenous noise sources with $\Phi_{e}(z)$ is non-singular, but possibly non-diagonal. ${\mathbf{H}}(z)$ is the weighted adjacency matrix with $H_{ii}=0$, for all $i\in[n]$. An LDIM is defined as the pair $({\mathbf{H}},{\mathbf{e}})$, whose output process is given by (1). The LDIM is well-posed if every entry of $({\mathbf{I}}_{n}-{\mathbf{H}}(z))^{-1}$ is analytic on the unit circle, $|z|=1$ and topologically detectable if $\Phi_{\mathbf{e}}$ is positive definite.

Every LDIM has two associated graphs, viz: 1) a directed graph, the linear dynamic influence graph (LDIG), ${\mathcal{G}}({\mathcal{V}},{\mathcal{E}})$, where ${\mathcal{V}}=[n]$ and ${\mathcal{E}}:=\{(i,j):H_{ji}\neq 0\}$ and 2) an undirected graph, the correlation graph, ${\mathcal{G}}_{c}({\mathcal{V}},{\mathcal{E}}_{c})$, where ${\mathcal{E}}_{c}:=\{(i,j):(\Phi_{e})_{ij}\neq 0,\ i\neq j\}$. Notice that, in a directed graph, if $(i,j)\in\mathcal{E}$ then there is a directed arrow from $i$ to $j$ in the graphical representation of $\mathcal{E}$ (see Fig. 1(a) for example). Concisely, “LDIG $({\mathbf{H}},{\mathbf{e}})$” is used to denote the LDIG ${\mathcal{G}}({\mathcal{V}},{\mathcal{E}})$ corresponding to the LDIM $({\mathbf{H}},{\mathbf{e}})$. For a directed graph ${\mathcal{G}}({\mathcal{V}},{\mathcal{E}})$, the parent, the children, and the spouse sets of node $i$ are $Pa(i):=\{j:(j,i)\in{\mathcal{E}}\}$, $Ch(i):=\{j:(i,j)\in{\mathcal{E}}\}$, and $Sp(i):=\{j:j\in Pa(Ch(i))\}$ respectively. The topology of a directed graph ${\mathcal{G}}({\mathcal{V}},{\mathcal{E}})$, denoted ${\mathcal{T}}({\mathcal{V}},{\mathcal{E}})$, is the undirected graph obtained by removing the directions from every edge $(i,j)\in{\mathcal{E}}$. Moral-graph (also called kin-graph) of ${\mathcal{G}}({\mathcal{V}},{\mathcal{E}})$, denoted $kin({\mathcal{G}})$ is an undirected graph, $kin({\mathcal{G}}):={\mathcal{G}}({\mathcal{V}},{\mathcal{E}}^{\prime})$, where ${\mathcal{E}}^{\prime}:=\{(i,j):i\neq j,\ i\in\left(Sp(j)\bigcup Pa(j)\bigcup Ch(j)\right)\}$. A clique is a sub-graph of a given undirected graph where every pair of nodes in the sub-graph are adjacent. A maximal clique is a clique that is not a subset of a larger clique. For a correlation graph, ${\mathcal{G}}_{c}({\mathcal{V}},{\mathcal{E}}_{c})$, $q$ denotes the number of maximal cliques with clique size $>1$.

The problem we address is described as follows.

In this section, the IPSDM based topology reconstruction is presented. In [27], the authors showed that, for any LDIM characterized by $\eqref{eq:Corr_TF_model}$, the availability of IPSDM, which can be written as

is sufficient for reconstructing the moral-graph of $({\mathbf{H}},{\mathbf{e}})$. That is, if $(\Phi_{{\mathbf{x}}}^{-1})_{ij}\neq 0$, then $i$ and $j$ are kins. However, an important assumption for the result to hold true is that $\Phi_{e}^{-1}$ is diagonal. If $\Phi_{e}^{-1}$ is non-diagonal, then the result does not hold in general. For $i\neq j$,

and any of the four terms can cause $(\Phi_{{\mathbf{x}}}^{-1})_{ij}\neq 0$, depending on $\Phi_{\mathbf{e}}^{-1}$. Hence, this technique cannot be applied directly to solve Problem 1. For example, consider the network ${\mathcal{G}}$ given in Fig. 1(a) with $(\Phi_{\mathbf{e}}^{-1})_{13}\neq 0$. Then, $(\Phi_{{\mathbf{x}}}^{-1})_{23}\neq 0$ since ${\mathbf{H}}^{*}_{21}(\Phi_{\mathbf{e}}^{-1})_{13}\neq 0$, which implies that the estimated topology has $(2,3)$ present, while $(2,3)\notin kin({\mathcal{G}})$.

In the next section, the correlation graph, ${\mathcal{G}}_{c}({\mathcal{V}},{\mathcal{E}}_{c})$, is scrutinized and properties of ${\mathcal{G}}_{c}({\mathcal{V}},{\mathcal{E}}_{c})$ are evaluated as the first step in unveiling the true topology when noise sources admit correlations.

In this section, a transformation of the LDIM, $({\mathbf{H}},{\mathbf{e}})$, to an LDIM with latent nodes by exploiting the structural properties of the noise correlation is obtained. The transformation converts an LDIM without latent nodes and driven by spatially correlated exogenous noise sources to an LDIM with latent nodes that are excited by spatially uncorrelated exogenous noise sources. Assuming perfect knowledge of the noise correlation structure, the latent nodes and their children in the transformed LDIM are characterized. It is shown that although the transformation is not unique, the topology of the transformation is unique under affine correlation (Assumption 1).

For $\mathbf{{\widetilde{e}}}=[\mathbf{{\widetilde{e}}}_{o};\mathbf{{\widetilde{e}}}_{h}]$ and $\mathbf{{\widetilde{H}}}:=$ $\left[\begin{matrix}{\mathbf{H}}&{\mathbf{F}}\\ {\mathbf{0}}&{\mathbf{0}}\end{matrix}\right]$, ${\mathbf{H}}\in{\mathcal{F}}^{n\times n},\ {\mathbf{F}}\in{\mathcal{F}}^{n\times L},~{}L\in{\mathbb{N}}$, $(\mathbf{{\widetilde{H}}},\mathbf{{\widetilde{e}}})$ (or with a slight abuse of notation, $([{\mathbf{H}},{\mathbf{F}}],\mathbf{{\widetilde{e}}})$) denotes an LDIM with $L$ latent nodes, where ${\mathbf{F}}_{ik}$ denotes the directed edge weight from latent node $k$ to observed node $i$ and ${\mathbf{H}}_{ik}$ denotes the directed edge weight from observed node $k$ to observed node $i$. Notice that the latent nodes considered in this article are strict parents and they do not have incoming edges.

The results in the previous section assumed that the correlations between the exogenous noise sources are affine in nature. In this section, a generalization of the affine correlation is addressed. It is shown that the noise sources that are correlated via non-affine, but a polynomial, interaction can be characterized using latent nodes with non-linear interaction dynamics. By lifting the processes to a higher dimension, the non-linearity is converted to linear interactions.

The following definitions are useful for the presentation in this section.

In the previous sections, frameworks that convert a network with spatially correlated noise to a network with latent nodes were studied. In this section, a technique is provided to reconstruct the topology of the transformed LDIMs using the sparse plus low-rank matrix decomposition of the IPSDM obtained from the observed time-series. Note that this result does not require any extra information other than the IPSDM obtained from the true LDIM.