A Simple Non-Stationary Mean Ergodic Theorem, with Bonus Weak Law of Large Numbers

I assume some familiarity with basic probability and stochastic processes, along the lines of Grimmett and Stirzaker (1992), but re-prove a number of basic results, to show that everything really is elementary.

Let $X_{1},X_{2},\ldots X_{t},\ldots$ be a sequence of real-valued random variables. Assume that $\mu_{t}\equiv\mathbb{E}\left[X_{t}\right]$ and $\mathrm{Cov}\left[X_{t},X_{s}\right]$ exist and are finite for all $t,s$. The time average of the $X_{t}$s,

converges on the average of the expectation values,

under a condition on the sum of the covariances,

The condition will be obvious after the following lemma, which will also be useful for extensions.

Proof: Since, for any $Z$, $\mathbb{E}\left[Z^{2}\right]=(\mathbb{E}\left[Z\right])^{2}+\mathrm{Var}\left[Z\right]$, we have

By linearity of expectation,

On the other hand,

through repeated application of the identity $\mathrm{Var}\left[B+C\right]=\mathrm{Var}\left[B\right]+\mathrm{Var}\left[C\right]+2\mathrm{Cov}\left[B,C\right]$, and the definition of $V_{n}$. Substituting into Eq. 5,

as was to be shown. $\Box$

Proof: Convergence in $L_{2}$ means that $\mathbb{E}\left[(A_{n}-m_{n})^{2}\right]\rightarrow 0$, and, by the lemma, $\mathbb{E}\left[(A_{n}-m_{n})^{2}\right]=V_{n}/n^{2}$. By the assumption of the theorem, $V_{n}=o(n^{2})$, hence $V_{n}/n^{2}=o(1)$, so we have

Thus $A_{n}-m_{n}\rightarrow 0$ in $L_{2}$; this is a mean (or mean-square) ergodic theorem. $\Box$

Since $V_{n}$ is a sum of $n^{2}$ terms, if the sum is to be $o(n^{2})$, most of those terms must be shrinking rapidly to zero as $n$ grows. That is, $\mathrm{Cov}\left[X_{t},X_{t+h}\right]$ must go to zero as $h\rightarrow\infty$. Stationarity (see immediately below) is not required.

If one does assume weak stationarity, a sufficient condition for $V_{n}=o(n^{2})$ is that $\sum_{h=-\infty}^{\infty}{\gamma(h)}=\tau\gamma(0)<\infty$, since in that case $V_{n}=O(n)$. $\tau$ has names like “correlation time”, “autocorrelation time”, “integrated autocovariance time”, etc.¹¹1Some authors use these names for $\sum_{h=0}^{\infty}{\gamma(h)}$, or even for $\sum_{h=1}^{\infty}{\gamma(h)}$. Any one of these sums is finite if and only if the others are.. From Lemma 4, for uncorrelated variables, $\mathrm{Var}\left[A_{n}\right]=\gamma(0)/n$, but when $0<\tau<\infty$, $n\mathrm{Var}\left[A_{n}\right]\rightarrow\tau\gamma(0)$, so the “effective sample size” is reduced from $n$ to $n/\tau$.

It is unnecessary for the theorem to assume that $m_{n}$ converges, i.e., that the instantaneous expectations $\mu_{n}$ are Cèsaro-convergent. Still less is it necessary to assume that the $\mu_{n}$ have a limit. $\mathrm{Var}\left[X_{t}\right]$ need not be constant or tending to a limit either, though it cannot grow too fast.

$L_{2}$ convergence implies convergence in probability, or what is usually known as the weak law of large numbers.

Proof: Use Chebyshev’s inequality (re-proved below as Proposition 1): for any random variable $Z$,

Applied to $A_{n}$, this gives, for each fixed $\epsilon$,

which is the definition of convergence to 0 in probability. $\Box$

The convergence results carry over easily to $d$-dimensional vectors, at least if $d$ is fixed as $n$ grows.

Proof: Apply the theorem and the previous corollary to each coordinate of $Y$ separately to get convergence along each coordinate, and hence convergence of the Euclidean distance to zero. $\Box$

If $V_{n}$ grows too fast, then the convergence to a deterministic limit fails.

Proof: $V_{n}=\Omega(n^{2})$ means that $\liminf{V_{n}/n^{2}}=v>0$. From the lemma, we know that $\mathrm{Var}\left[A_{n}\right]=V_{n}/n^{2}$, which does not go to zero, so convergence in $L_{2}$ must fail. $\Box$

Convergence in probability is slightly more delicate.

Proof: Begin with the previous corollary, and apply the Paley-Zygmund inequality (Proposition 3) to the non-negative random variable $(A_{n}-\mu_{n})^{2}$, whose expected value is (again, from Lemma 4) $V_{n}/n^{2}$. By the inequality, for any $\epsilon\leq V_{n}/n^{2}$,

Restrict ourselves to $\epsilon<v/2$. Then, for all sufficiently large $n$,

so $A_{n}-\mu_{n}\not\rightarrow 0$ in probability. $\Box$

Remark 3: I suspect the extra condition needed to force non-convergence in probability can be weakened, because the underlying Paley-Zygmund inequality used in the proof isn’t necessarily sharp. But an example helps show that some condition is necessary. Suppose that for each $t$, $X_{t}=\pm t^{3/2}$ with probability $\frac{1}{2}t^{-2}$, otherwise $X_{t}=0$, and that the $X_{t}$ are all mutually independent. Then $m_{n}=0$ for all $n$. Moreover, $\mathbb{P}\left(X_{t}\neq 0\right)=1/t^{2}$. Since those probabilities are summable, by the Borel-Cantelli lemma, $\mathbb{P}\left(X_{t}\neq 0~{}\text{infinitely often}\right)=0$. But then $X_{t}=0$ for all but finitely many $t$ almost surely, hence $A_{n}\rightarrow 0$ in probability. On the other hand, $\mathrm{Var}\left[X_{t}\right]=t$, so $V_{n}=n(n+1)/2$, $\lim{V_{n}/n^{2}}=1/2$, and $A_{n}\not\rightarrow 0$ in $L_{2}$. Verifying that the second condition of the corollary does not hold involves some straightforward but detailed algebra, given in Appendix C. While this example is deliberately stylized, it does get at what’s needed to have convergence in probability without convergence in $L_{2}$: the probability that $|A_{n}-m_{n}|\geq\epsilon$ has to be going to zero, no matter how small we set $\epsilon$, but when there are fluctuations in $A_{n}$ away from $m_{n}$, they need to be getting larger and larger. Whether this is a realistic concern or a paranoid fear will depend on the application.

Remark 4: Convergence of the $A_{n}$ not to the deterministic $m_{n}$ but to a random limit, as in the full mean-square ergodic theorem for weakly stationary sequences (as given by, e.g., Grimmett and Stirzaker (1992, §9.5, theorem 3)) would seem to require more advanced tools.