Kalman Filter from the Mutual Information Perspective

Kalman filter has been widely in various fields as an effective state estimator for integrated navigation [1], robotics [2], etc. The classical Kalman filter can be derived as a best linear unbiased estimate [1] and it is easy understand it from the probabilistic perspective [2]. Recently, Kalman filter has also been presented using the methods of maximum relative entropy [3] and the temporal derivative of the Rényi entropy [4], which go beyond the general Bayesian filter. More and more evidences show that Kalman filter can be regarded as a direct extension of information theory. This note gives a new perspective of the Kalman filter from the mutual information, which further bridges the gap between the optimal state estimation and the information theory. The main contribution of this note is to derive the Kalman filter from the perspective of mutual information and extend it to the Rényi mutual information case.

For following discrete-time state-space model

$$X_{k}={\Phi_{k|k-1}}X_{k-1}+{\Gamma_{k|k-1}}W_{k-1}$$

(1)

$$Z_{k}=H_{k}X_{k}+V_{k}$$

(2)

where $X_{k}$ is n-dimensional state vector; $Z_{k}$ is m-dimensional measurement vector; ${\Phi_{k|k-1}}$, ${\Gamma_{k|k-1}}$ and $H_{k}$ are the known system structure parameters, which are called the $n\times n$ dimensional one-step state update matrix, the $n\times l$ dimensional system noise distribution matrix, and the $m\times n$ dimensional measurement matrix, respectively; $W_{k-1}$ is the $l$-dimensional system noise vector, and $V_{k}$ is the m-dimensional measurement noise vectors. Both of them are Gaussian noise vector sequences with zero mean value, and independent to each other:

$$\mathbb{E}[W_{k}]=0,\mathbb{E}[W_{k}W_{j}^{T}]=Q_{k}\delta_{kj}$$

(3)

$$\mathbb{E}[V_{k}]=0,\mathbb{E}[V_{k}V_{j}^{T}]=R_{k}\delta_{kj}$$

(4)

$$\mathbb{E}[W_{k}V_{j}^{T}]=0$$

(5)

The one-step prediction covariance matrix is denoted as $\Sigma_{k|k-1}$. The state estimation at $t_{k}$ is denoted as $\mathcal{N}(\hat{X}_{k},\Sigma_{k})$, where $\hat{X}_{k}$ is the mean of estimated state and $\Sigma_{k}$ is the covariance matrix of the estimated covariance matrix. Assuming the optimal estimation of the state can be calculated as follows:

$$\hat{X}_{k}={X_{k|k-1}^{-}}+K_{k}{\tilde{Z}_{k|k-1}}$$

(6)

where $K_{k}$ is the undetermined correction factor matrix, $X_{k|k-1}^{-}=X_{k|k-1}-\hat{X}_{k-1}$ is the state estimation error, $\tilde{Z}_{k|k-1}=Z_{k}-H_{k}X_{k|k-1}^{-}$ is the measurement one-step prediction error.

Then, the mean square error matrix of state estimation $\hat{X}_{k}$ is given by [1]

$${\Sigma}_{k}=({I}-K_{k}H_{k}){{\Sigma}_{k|k-1}}({I}-K_{k}H_{k})^{T}+K_{k}R_{k}K_{k}^{T}$$

(7)

the mean square error matrix $\Sigma_{k}$ is positive definite as $({I}-K_{k}H_{k}){{\Sigma}_{k|k-1}}({I}-K_{k}H_{k})^{T}$ is positive definite and $K_{k}R_{k}K_{k}^{T}$ is positive definite.

The joint Gaussian distribution can be expressed

$$p(X,Y)\sim\left(\begin{bmatrix}\hat{X}\\ \hat{Y}\end{bmatrix},\begin{bmatrix}\Sigma_{xx}&\Sigma_{xy}\\ \Sigma_{yx}&\Sigma_{yy}\end{bmatrix}\right)$$

(8)

where $X\sim\mathcal{N}(\hat{X},\Sigma_{xx})$,$Y\sim\mathcal{N}(\hat{Y},\Sigma_{yy})$.

The mutual information for a joint Gaussian PDF can be represented by

	$\displaystyle I(X,Y)$	$\displaystyle=H(X)+H(Y)-H(X,Y)=H(X)-H(X|Y)$		(9)
		$\displaystyle=\frac{1}{2}\ln((2\pi e)^{N}\det\Sigma_{xx})+\frac{1}{2}\ln((2\pi e)^{M}\det\Sigma_{yy})-\frac{1}{2}\ln((2\pi e)^{M+N}\det\Sigma)$
		$\displaystyle=-\frac{1}{2}\ln\left(\frac{\det\Sigma}{\det\Sigma_{xx}\det\Sigma_{yy}}\right)$

where $H(X)=\frac{1}{2}\ln((2\pi e)^{N}\det\Sigma_{xx})$ is the entropy of a Gaussian random variable, and

$$\det\Sigma=\det\Sigma_{xx}\det\left(\Sigma_{yy}-\Sigma_{yx}\Sigma_{xx}^{-1}\Sigma_{xy}\right)=\det\Sigma_{yy}\det\left(\Sigma_{xx}-\Sigma_{xy}\Sigma_{yy}^{-1}\Sigma_{yx}\right)$$

(10)

Therefore, the mutual information describes the reduction of the uncertainty in variable X due to gaining knowledge of variable Y.

Similarly, the mutual information at time step $t_{k+1}$ can be easily computed by the a priori, a posteriori PDF and the Kalman gain $K_{k}$ as

$$I(\hat{X}_{k|k-1},Z_{k})=\frac{1}{2}\ln\left(\frac{\det\Sigma_{k|k-1}}{\det\Sigma_{k}}\right)=\frac{1}{2}\ln\left(\frac{\det\Sigma_{k|k-1}}{\det\left(({I}-K_{k}H_{k}){{\Sigma}_{k|k-1}}({I}-K_{k}H_{k})^{T}+K_{k}R_{k}K_{k}^{T}\right)}\right)$$

(11)

It describes the reduction of the uncertainty of the state due to gaining knowledge from the measurement $Z_{k}$. Consequently, we want to maximize the mutual information $I(\hat{X}_{k|k-1},Z_{k})$. It is obvious that the maximum of the mutual information is equivalent to the minimum of $\ln\det\Sigma_{k}$. We can note that equation (11) is the function of the unknown factor matrix and thereby the maximization of it can be calculated by taking derivative of it with respect to $K_{k}$ and setting it equal to zero,

$$\frac{dI(\hat{X}_{k|k-1},Z_{k})}{dK_{k}}=-\Sigma_{k}^{-T}\frac{d\Sigma_{k}}{dK_{k}}=-\Sigma_{k}^{-T}\left(-2(I-K_{k}H_{k})\Sigma_{k|k-1}H_{k}^{T}+2K_{k}R_{k}\right)=0$$

(12)

where $\frac{\partial\ln\det X}{\partial X}=X^{-T}$ [5] has been used and then solving for $K_{k}$ gives

$$K_{k}=\Sigma_{k|k-1}H_{k}^{T}(H_{k}\Sigma_{k|k-1}H_{k}^{T}+R_{k})^{-1}$$

(13)

A subsequent derivative of equation (12) must be performed to check for a maximum, that is

$$\frac{d}{dK_{k}}\left(-\Sigma_{k}^{-T}\left(-2(I-K_{k}H_{k})\Sigma_{k|k-1}H_{k}^{T}+2K_{k}R_{k}\right)\right)$$

(14)

After substituting the equation (12) into above equation results in

$$\frac{d}{dK_{k}}\left(-\Sigma_{k}^{-T}\left(-2(I-K_{k}H_{k})\Sigma_{k|k-1}H_{k}^{T}+2K_{k}R_{k}\right)\right)=-\Sigma_{k}^{-T}(H_{k}\Sigma_{k|k-1}H_{k}^{T}+R_{k})$$

(15)

which is always negative definite by the definition of the covariance matrices $R_{k}$ and $H_{k}\Sigma_{k|k-1}H_{k}^{T}$, ensuring the solution for $K_{k}$ is a maximum.

Moreover, the Rényi mutual information of a joint Gaussian PDF can be calculated similar to equation (9):

	$\displaystyle I_{R}^{\alpha}(X,Y)$	$\displaystyle=H_{R}^{\alpha}(X)+H_{R}^{\alpha}(Y)-H_{R}^{\alpha}(X,Y)$		(16)
		$\displaystyle=\frac{1}{2}\ln|(2\pi)^{N}\alpha^{\frac{N}{\alpha-1}}\det\Sigma_{xx}|+\frac{1}{2}\ln|(2\pi)^{M}\alpha^{\frac{M}{\alpha-1}}\det\Sigma_{yy}|-\frac{1}{2}\ln|(2\pi)^{N+M}\alpha^{\frac{N+M}{\alpha-1}}\det\Sigma|$
		$\displaystyle=\frac{1}{2}\ln\left(\frac{\det\Sigma_{xx}\det\Sigma_{yy}}{\det\Sigma}\right)$
		$\displaystyle=I(X,Y)$

where $H_{R}^{\alpha}(X)=\frac{1}{2}\ln|(2\pi)^{N}\alpha^{\frac{N}{\alpha-1}}\det\Sigma_{xx}|$ is the Rényi entropy of order $\alpha$ for a continuous Gaussian random variable with a multivariate Gaussian PDF. Consequently, we can know that the mutual information is the same as the Rényi mutual information for the joint Gaussian PDF. Similarly, we can get the same result as equation (13) from the Rényi mutual information.

In this paper, Kalman filter is derived from the perspective of mutual information and extended to the Rényi mutual information case. We show that the measurement update of the Kalman filter can minimize the uncertainty of the state by formulating it as the mutual information between the evolving state and the measurement and maximizing the mutual information. Furthermore, we can think of Kalman filter a little more radically as an extension of the information theory.