Asymptotic analysis of ML-covariance parameter estimators based on covariance approximations

Maximum likelihood (ML) estimators for covariance parameters are highly popular in inference for random fields. Aiming towards asymptotic properties of such estimators, one needs to specify how the observation points and the associated sampling domain behave as the number of observation points increases. Two well-studied asymptotic frameworks are referred to as infill-domain asymptotics (also termed fixed-domain asymptotics) and increasing-domain asymptotics (see [13], p. $100$ for an introduction of terms). In infill-domain asymptotics, observation points are sampled within a bounded sampling domain, whereas in increasing-domain asymptotics, the sampling domain grows as the number of observation points increases. When referring to infill- and increasing-domain asymptotics, one often places additional assumptions on the minimum distance between any two distinct observation points. In increasing-domain asymptotics, the latter distance is often assumed to be bounded away from zero, while in infill-domain asymptotics, one frequently assumes that distinct observation points can be sampled arbitrarily close to each other (see for example [37]). There is a fair amount of literature which demonstrates that asymptotic properties of ML estimators for covariance parameters can be quite different under the two mentioned asymptotic frameworks (see [37] or more lately [6]). For example, it is known that some covariance parameters can not be estimated consistently under an infill-domain asymptotic framework ([34], [36]), whereas they can be estimated consistently, under given regularity conditions, within an increasing-domain asymptotic framework ([25], [4]). It is worth noting that in infill-domain asymptotics, these results can depend on the dimension $d$ of the Euclidean space $\mathbb{R}^{d}$, where the random field is assumed to be observed. For example, when the true covariance function belongs to the Matérn family ([26]), and smoothness parameters are given, it is shown in [36], that for $d=1,2,3$, the scale and variance parameters can not be estimated consistently via an ML approach in an infill-domain asymptotic framework. The case where $d=4$ is still open, but for $d\geq 5$, it is shown in [2] that under infill-domain asymptotics, all covariance parameters of the Matérn family can be estimated consistently using an ML approach.

In recent years, the dataset sizes have steadily increased such that statistical analyses on random fields can become quite expensive in terms of computational resources (see for example [15] for a recent discussion). One prominent issue with large datasets is the large size of covariance matrices, constructed upon applying an underlying covariance function to given data. However, in certain fields of application, observed correlations are assumed to vanish beyond a certain cut-off distance (see [18] p. $750-751$, and references therein, for an example in meteorology or also [10] and [19]). On the other hand, in the context of real valued random fields, it is common practice to multiply a presumed covariance function with a known positive-definite and compactly supported covariance function, called the covariance taper. The resulting compactly supported covariance function is referred to as the tapered covariance function. For an introduction to covariance tapering we refer to [17]. The use of compactly supported covariance functions can thus be of great importance for some fields of application. Not only do they potentially reflect the nature of the underlying covariance structure, but also, their application can lead to sparse covariance matrices. The latter are helpful in terms of the high computational costs in the context of large datasets. An excellent introduction to the construction of compactly supported covariance functions, associated to stationary and isotropic Gaussian random fields, is given in [21]. Additional results are available in [35], [28] and [11].

The parametric family of generalized Wendland covariance functions represents one example of a family of compactly supported covariance functions which allows, similar to the Matérn family, for a continuous parametrization of smoothness (in the mean square sense) of the underlying random field. Its origin is due to Wendland ([32]) and an early adaptation for statistical applications was given by Gneiting ([20]). In its general form (see [21] and [28] for special cases) the generalized Wendland covariance function with smoothness parameters $\nu$ and $\kappa$, variance parameter $\sigma^{2}$ and range parameter $\beta$ is given by

$$\phi(t)\coloneqq\frac{\sigma^{2}}{\operatorname{B}(2\kappa,\nu+1)\beta^{2\kappa+\nu}}\int_{t}^{\beta}w(w^{2}-t^{2})^{\kappa-1}(\beta-w)^{\nu}dw,$$

(1)

if $t\in[0,\beta)$ and is zero otherwise. In the above display, $\operatorname{B}$ is the beta function. For technical details about valid parameter values, we refer to [9] or Section 6 of the present article. Clearly, in comparison with closed-form covariance functions, computing (1) is cumbersome, as it involves numerical integration. Depending on the support $\beta$ and a set of locations $s_{1},\dotsc,s_{n}\in\mathbb{R}^{d}$, the $n\times n$ covariance matrix $\Sigma_{i,j}=\phi(\lVert s_{i}-s_{j}\rVert)$ requires at most $n(n-1)/2$ calculations of (1). One strategy, which facilitates computing $\Sigma$, is to reduce the number of times (1) must be calculated. As an illustration, we give three examples which involve approximations $\tilde{\phi_{i}}$, $i=1,2,3$, of $\phi$ (respectively approximations $\widetilde{\Sigma}_{i}$ of $\Sigma$):

1. ($\tilde{\phi_{1}}$)

   Truncation of the support

2. ($\tilde{\phi_{2}}$)

   linear interpolation

3. ($\tilde{\phi_{3}}$)

   addition of a nugget effect

For $\tilde{\phi_{1}}$, we truncate $\phi$ to obtain $\tilde{\phi_{1}}$ which has a smaller support compared to $\phi$. This becomes especially interesting, when the original function $\phi$ tails off slowly (high degree of differentiability at the origin). As a result, $\widetilde{\Sigma}_{1}$ will be more sparse compared to $\Sigma$. Example $\tilde{\phi_{2}}$ is to predefine the numbers at which (1) is calculated. This is achieved by introducing a partition $0<t_{1}<\dotsc<t_{N}=\beta$ of the support of $\phi$. Then, $\tilde{\phi_{2}}$ results in $N$ calculations of $\phi$. This defines a closed form approximation of $\phi$. Notice that $t_{1},\dotsc,t_{N}$ do not need to be equispaced. Finally, $\tilde{\phi_{3}}$ can be interpreted as a tuning option for a given approximation $\tilde{\phi}_{*}$ of $\phi$:

$$\tilde{\phi_{3}}(t)\coloneqq\begin{cases}\tilde{\phi}_{*}(t)+\delta,&t=0,\\ \tilde{\phi}_{*}(t),&t\neq 0,\end{cases}\quad\delta\geq 0.$$

With regard to practical usage, this form of approximation increases numerical stability. Further, it allows for more flexibility in practice, where the number of observations $n$ is given and $\widetilde{\Sigma}_{*}$ based on $\tilde{\phi}_{*}$ might not be positive-definite.

Following up the above examples, we picture an approximation $\tilde{\phi}$ of $\phi$ (respectively approximation $\widetilde{\Sigma}$ of $\Sigma$). Several questions arise:

• •

  What are conditions on $\tilde{\phi}$ to ensure that $\widetilde{\Sigma}$ is asymptotically (as $n\xrightarrow[]{}\infty$) equivalent to $\Sigma$ and eventually (for $n$ large enough) remains positive-definite?

• •

  In terms of ML estimators for covariance parameters, how shall a log-likelihood approximation based on $\tilde{\phi}$ be defined?

• •

  Under which conditions on $\tilde{\phi}$ are ML estimators based on $\tilde{\phi}$ consistent and asymptotically normal?

In the more general setting of a given parametric family of covariance functions, the present study gives a concrete context, where the latter questions are answered by introducing the notion of truncated-ML estimators.

Truncated-ML estimators for covariance parameters are based on truncated-likelihood functions. The latter are defined upon parametric families of sequences of functions, which approximate a presumed family of covariance functions on a common domain. Colloquially we will call these parametric sequences of functions covariance approximations. The respective matrices, constructed upon applying covariance approximations to a given collection of observation points, will be termed covariance matrix approximations. We will allow for covariance matrix approximations that are not necessarily positive semi-definite. Therefore, truncated-likelihood functions are more general than existing likelihood approximations methods such as low-rank, Vecchia, or covariance tapering approaches (see [22] for a summary of commonly used methods).

We work in an increasing-domain asymptotic framework, where collections of observation points are realizations of finite collections of a randomly perturbed regular grid (see also [4]). We consider a stationary Gaussian random field, with a zero-mean function and a true unknown covariance function that belongs to a given parametric family of covariance functions. If the presumed family of covariance functions is compactly supported, we provide sufficient conditions under which truncated-ML estimators and (regular) ML estimators for covariance parameters are consistent and asymptotically normal. Some conditions imposed on families of covariance functions are identical to the conditions that were already considered in [4]. The main difference is that we work with compactly supported covariance functions. Therefore, it is possible to simplify some of the conditions that were set up in [4]. As for statistical applications, we apply these results to the family of generalized Wendland covariance functions. In contrast to the infill-domain asymptotic framework considered in [9], we show that under the studied increasing-domain asymptotic framework, under some conditions on the parameter space, (regular) ML estimators for variance and range parameters are consistent and asymptotically normal. Further, we show that the same asymptotic results are recovered for truncated-ML estimators, based on various generalized Wendland approximations, such as truncations, linear interpolations and added nugget effects.

Additionally, we provide an extension to families of covariance functions which are not compactly supported. We combine our results with the covariance tapering approach. That is, we study covariance taper approximations and their asymptotic influence on the conditional Kullback-Leibler divergence of the misspecified distribution from the true distribution (see also [5]). We show that the latter divergence is minimized by truncated-tapered ML estimators.

The rest of the article is organized as follows. Section 2 establishes the context. We introduce some primary notation, define the sampling domain and the random field itself. In Section 3 we introduce regularity conditions on covariance functions and approximations. In Section 4 we present intermediate asymptotic results on covariance matrices and approximations. Section 5 contains our main results: We introduce truncated-ML estimators and present results on consistency and asymptotic normality. In Section 6, we apply our results to the family of generalized Wendland covariance functions and discuss several examples of generalized Wendland approximations. Then, in the context of non-compactly supported covariance functions, Section 7 contains results on the asymptotic influence of taper approximations on the Kullback-Leibler divergence. Section 8 gives an outlook and some final comments. The Appendix is split into three parts. Covariance approximations for isotropic random fields are discussed in Appendix A. Appendix B contains additional supporting results, whereas all the proofs are left for Appendix C.

The set $\mathbb{N}_{+}$ and $\mathbb{R}_{+}$ shall represent the set of positive integers and non-negative real numbers, respectively. For $d\in\mathbb{N}_{+}$, we use the notation $\mathrm{B}(x;r)$ ($\mathrm{B}[x;r]$) for the open (closed) ball of radius $r>0$ with center $x\in\mathbb{R}^{d}$. Given $n\in\mathbb{N}_{+}$, for some set $A\subset\mathbb{R}^{n}$, we write $\mathfrak{B}(A)$ for the Borel $\sigma$-algebra on $A$.

For a vector $(w_{1},\dotsc,w_{d})=w\in\mathbb{R}^{d}$, we write $\left\lVert w\right\rVert=\big{(}w_{1}^{2}+\cdots+w_{d}^{2}\big{)}^{1/2}$ for the Euclidean norm of $w$ on $\mathbb{R}^{d}$. In the case of $d=1$ we use the notation $\left\lvert\cdot\right\rvert$ for the Euclidean norm. For two vectors $w,w^{\prime}\in\mathbb{R}^{d}$, $\langle w,w^{\prime}\rangle=w^{\mathrm{t}}w^{\prime}=\sum_{i=1}^{d}w_{i}w^{\prime}_{i}$ represents the inner product that induces $\left\lVert\cdot\right\rVert$ on $\mathbb{R}^{d}$. Given $D\subset\mathbb{R}^{d}$, we write $\mathcal{B}_{\text{C}}(D;S)$ for the space of real valued, uniformly bounded functions on $D$, having compact support $S\subset D$. If $f\in\mathcal{B}_{\text{C}}(D;S)$ and $f$ is also continuous, we use the notation $\mathcal{C}_{\text{C}}(D;S)$ instead of $\mathcal{B}_{\text{C}}(D;S)$. For $f\in\mathcal{C}_{\text{C}}(D;S)$ we write ${\left\lVert f\right\rVert}_{\infty}=\sup\{\left\lvert f(h)\right\rvert\colon h\in D\}$ for the uniform norm on $\mathcal{C}_{\text{C}}(D;S)$. For vectors $w\in\mathbb{R}^{d}$, $\left\lvert w\right\rvert_{\infty}=\max_{i=1,\dotsc,d}\left\lvert w_{i}\right\rvert$ denotes the uniform norm on $\mathbb{R}^{d}$.

For a real $n\times n$ matrix $A$, ${\left\lVert A\right\rVert}_{2}=\max_{\{z\colon z^{\mathrm{t}}z=1\}}\langle z,A^{\mathrm{t}}Az\rangle^{1/2}$ denotes the spectral norm of $A$. We write $A\succ 0$ ($A\prec 0$) to indicate that $A$ is positive-definite (negative-definite). Further, $\lambda_{1}(A)\geq\cdots\geq\lambda_{n}(A)$ denote the $n$ real eigenvalues of a matrix $A\in\mathrm{S}_{n\times n}(\mathbb{R})$, where $\mathrm{S}_{n\times n}(\mathbb{R})$ represents the space of real symmetric $n\times n$ matrices.

We use the notation $\nabla f(x)=\big{(}\frac{\partial f}{\partial x_{1}}(x),\dotsc,\frac{\partial f}{\partial x_{p}}(x)\big{)}$ for the gradient of $f$ at $x$, where $x\mapsto f(x)$ is any differentiable, real valued function, defined on some $E\subset\mathbb{R}^{p}$. Further, for a vector valued, differentiable function $g(x)=(g_{1}(x),\dotsc,g_{m}(x))$, with values in $\mathbb{R}^{m}$, defined on some $U\subset\mathbb{R}^{p}$, we write $J_{g}(x)_{l,k}=\frac{\partial g_{l}}{\partial x_{k}}(x)$, $1\leq l\leq m$, $1\leq k\leq p$, for the Jacobi-matrix of $g$ at $x$.

A mapping $Y$ from a probability space $(\Omega,\mathcal{F},\mathbb{P})$ to a measure space $(E,\mathcal{A})$ will be called a random element if it is $\mathcal{F}/\mathcal{A}$ measurable. If we write that $Y\colon(\Omega,\mathcal{F})\to(E,\mathcal{A})$ is measurable, we mean that it is $\mathcal{F}/\mathcal{A}$ measurable. If $(Y_{n}){}_{n\in\mathbb{N}_{+}}$ denotes a sequence of random elements, where for any $n\in\mathbb{N}_{+}$, $Y_{n}$ is a mapping form a probability space $(\Omega,\mathcal{F},\mathbb{P})$ to a measure space $(E,\mathcal{A})$, we use the notation

$$\displaystyle Y_{n}\xrightarrow[n\to\infty]{\mathbb{P}}Y\text{ and }Y_{n}\xrightarrow[n\to\infty]{\mathrm{d}}\mathcal{L},$$

to indicate convergence of $(Y_{n}){}_{n\in\mathbb{N}_{+}}$ to a random element $Y$ in probability and in distribution, respectively. Note that for convergence in distribution, the introduced notation indicates that the limit $Y$ has law $\mathcal{L}$ on $(E,\mathcal{A})$. A sequence of estimators $(\hat{\theta}_{n})_{n\in\mathbb{N}_{+}}$ for $\theta_{0}\in\mathbb{R}^{p}$ will be referred to as consistent if it converges in probability to $\theta_{0}$. Finally, $\mathcal{N}(\mu,\Sigma)$ indicates a multivariate normal distribution with mean vector $\mu$ and covariance matrix $\Sigma$.

On a probability space $(\Omega,\mathcal{F},\mathbb{P})$, we consider a real valued Gaussian random function $Z$, which has sample functions on $\mathbb{R}^{d}$. We assume that $Z$ is stationary (homogeneous) with zero-mean function and covariance function $c_{\theta_{0}}(s)$, $s\in\mathbb{R}^{d}$, where $\theta_{0}\in\Theta$, with $\Theta\subset\mathbb{R}^{p}$, compact and convex. Thus, we consider a real valued random field $\{Z_{s}\colon s\in\mathbb{R}^{d}\}$, which has true and unknown covariance function $c_{\theta_{0}}$ that belongs to a family of covariance functions $\{c_{\theta}\colon\theta\in\Theta\}$.

Let $\mathcal{Q}\coloneqq[-1,1]^{d}$ and $X\colon\Omega\to\mathcal{Q}^{\mathbb{N}_{+}}$ be a stochastic process, defined on the same probability space $(\Omega,\mathcal{F},\mathbb{P})$, but independent of $Z$. We assume that the sequence $(X_{i})_{i\in\mathbb{N}_{+}}$ is a sequence of independent random vectors with common law on $\mathcal{Q}$, which has a strictly positive probability density function on $\mathcal{Q}$ (see also Remark 2.1). Given $\tau\in[0,1/2)$ and a sequence of deterministic points $(v_{i}){}_{i\in\mathbb{N}_{+}}$, with $v_{i}\in\mathbb{N}_{+}^{d}$, we define a randomly perturbed regular grid $S$ as the process

$$\big{\{}S_{i}\coloneqq v_{i}+\tau X_{i}\colon i\in\mathbb{N}_{+}\big{\}},$$

(2)

where we assume that for all $I\in\mathbb{N}_{+}$, $\big{\{}v_{i},1\leq i\leq I^{d}\big{\}}=\big{\{}1,\dotsc,I\big{\}}^{d}$. Therefore, for any $\omega\in\Omega$, $S(\omega)$ is a sequence on $\mathbb{N}_{+}$, with image $S[\mathbb{N}_{+}](\omega)\subset\prod_{i=1}^{\infty}\big{(}v_{i}+\tau\mathcal{Q}\big{)}\eqqcolon\mathcal{G}$ and any first $I^{d}$ coordinates are in $\{1,\dotsc,I\}^{d}+\tau\mathcal{Q}$ (see also Figure 1). At this point we remark that if nothing is mentioned, the parameter $\tau\in[0,1/2)$ and the sequence $(v_{i}){}_{i\in\mathbb{N}_{+}}$ shall be fixed. Let $X_{(n)}\coloneqq(X_{1},\dotsc,X_{n})$ and $S_{(n)}\coloneqq(S_{1},\dotsc,S_{n})$ denote finite collections of $X$ and $S$, respectively. We use the notation $x_{(n)}\coloneqq(x_{1},\dotsc,x_{n})$ for a vector that contains the first $n$ entries of a given sequence in $\mathcal{Q}^{\mathbb{N}_{+}}$. Correspondingly, given $\tau\in[0,1/2)$, $v_{1},\dotsc,v_{n}$ and $x_{(n)}\in\mathcal{Q}^{n}$, we write $s_{(n)}\coloneqq(s_{1},\dotsc,s_{n})$, $s_{i}=v_{i}+\tau x_{i}$, for $n$ perturbed grid locations in $\mathcal{G}_{n}\coloneqq\prod_{i=1}^{n}(v_{i}+\tau\mathcal{Q})$.

On $(\Omega,\mathcal{F},\mathbb{P})$, we define the random vector

$\displaystyle\omega\mapsto Z_{(n)}(\omega)\coloneqq\big{(}Z_{S_{1}(\omega)}(\omega),\dotsc,Z_{S_{n}(\omega)}(\omega)\big{)}=(z_{s_{1}},\dotsc,z_{s_{n}})\eqqcolon z_{(n)},$

(3)

which denotes $Z$ observed at a finite collection of $S$. The situation, where a Gaussian random field is assumed to be observed at a randomly perturbed regular grid, with parameter $\tau$ and deterministic points $(v_{i}){}_{i\in\mathbb{N}_{+}}$, as introduced above, is also considered in [4].

Given $\theta\in\Theta$, we let $\Sigma_{\theta}(s_{(n)})\coloneqq[c_{\theta}(s_{i}-s_{j})]_{1\leq i,j\leq n}$ denote the non-random $n\times n$ covariance matrix based on an arbitrary $s_{(n)}\in\mathcal{G}_{n}$. On $(\Omega,\mathcal{F},\mathbb{P})$, we write

$$\displaystyle\omega\mapsto\Sigma_{n,\theta}(\omega)\coloneqq\Sigma_{\theta}(S_{(n)}(\omega)),\quad\theta\in\Theta,$$

for the $n\times n$ random covariance matrix based on a finite collection $S_{(n)}$ of $S$.

Given $\theta\in\Theta$, we let $(\tilde{c}_{m,\theta})_{m\in\mathbb{N}_{+}}$ denote a sequence of real valued functions defined on $\mathbb{R}^{d}$. The families $\big{\{}(\tilde{c}_{m,\theta})_{m\in\mathbb{N}_{+}}\colon\theta\in\Theta\big{\}}$ can be put under the following assumption.

To make the notation easier, we write $(\tilde{c}_{m,\theta})\coloneqq(\tilde{c}_{m,\theta}){}_{m\in\mathbb{N}_{+}}$. In the following, we formally introduce covariance matrix approximations (random and non-random versions). To do so, let $r\colon\mathbb{N}_{+}\to\mathbb{N}_{+}$ be such that $r(n)\xrightarrow{}\infty$ as $n\xrightarrow{}\infty$. Given $s_{(n)}\in\mathcal{G}_{n}$, we let $\widetilde{\Sigma}_{\theta}(s_{(n)})\coloneqq[\tilde{c}_{r(n),\theta}(s_{i}-s_{j})]_{1\leq i,j\leq n}$ denote the non-random $n\times n$ matrix based on a given family $\big{\{}(\tilde{c}_{m,\theta})\colon\theta\in\Theta\big{\}}$. Then, on $(\Omega,\mathcal{F},\mathbb{P})$, if $\big{\{}(\tilde{c}_{m,\theta})\colon\theta\in\Theta\big{\}}$ is a family of Borel measurable sequences of functions, we write

$$\displaystyle\omega\mapsto\widetilde{\Sigma}_{n,\theta}(\omega)\coloneqq\widetilde{\Sigma}_{\theta}(S_{(n)}(\omega)),$$

for the $n\times n$ random matrix based on a finite collection $S_{(n)}$ of $S$. Colloquially we will use the term covariance approximation when we refer to a given family $\big{\{}(\tilde{c}_{m,\theta})\colon\theta\in\Theta\big{\}}$, which can approximate a family of covariance functions $\{c_{\theta}\colon\theta\in\Theta\}$ in the sense of Assumption 3.2. In these terms $\{c_{\theta}\colon\theta\in\Theta\}$ itself is a covariance approximation. The expression covariance matrix approximation will be used for both, $\widetilde{\Sigma}_{\theta}(s_{(n)})$ and its random version $\widetilde{\Sigma}_{n,\theta}$. Similar, we use the expression covariance matrix for both, $\Sigma_{\theta}(s_{(n)})$ and $\Sigma_{n,\theta}$.

The main results of this section are that under suitable conditions on the families of covariance functions and approximations, truncated-ML estimators for covariance parameters are not only consistent (Theorem 5.2 and Corollary 5.3) but also asymptotically normal (Theorem 5.4 and Corollary 5.5). In particular, we will make use of the conditions presented in Assumptions 3.1 and 3.2. However, in the context of random fields that are observed at randomly perturbed regular grid locations as defined in (2), we will further make use of the following two technical conditions that were also imposed in [4]. Associated to the common range $\mathcal{Q}$, of the process $X$, we define the set $D_{\tau}\coloneqq\bigcup_{z\in\mathbb{Z}^{d}\setminus\{0\}}(z+\tau U_{\mathcal{Q}})$, where $U_{\mathcal{Q}}\coloneqq\{u_{1}-u_{2}\colon u_{1}\in\mathcal{Q},u_{2}\in\mathcal{Q}\}$ denotes the set of differences between two points in $\mathcal{Q}$.

The following corollary is immediate.

Before we present the results about asymptotic normality, it is helpful to consider some additional notation. Let $K\in\mathbb{N}_{+}$, such that for any $\omega\in\Omega$, the sequences of functions

$$\big{(}l_{n,K}\!(\theta)(\omega)\big{)}_{n\in\mathbb{N}_{+}}\coloneqq\big{(}l_{n+K-1}(\theta)(\omega)\big{)}_{n\in\mathbb{N}_{+}}$$

(8)

and

$$\big{(}\tilde{l}_{n,K}\!(\theta)(\omega)\big{)}_{n\in\mathbb{N}_{+}}\coloneqq\big{(}\tilde{l}_{n+K-1}(\theta)(\omega)\big{)}_{n\in\mathbb{N}_{+}}$$

(9)

are differentiable with respect to $\theta$. Note that if $\{c_{\theta}\colon\theta\in\Theta\}$ satisfies Assumption 3.1 and the collection $\big{\{}(\tilde{c}_{m,\theta})\colon\theta\in\Theta\big{\}}$ satisfies Assumption 3.2, then we know about the existence of such a $K$ under application of Proposition 5.1. For the given $K\in\mathbb{N}_{+}$, on $(\Omega,\mathcal{F},\mathbb{P})$, we introduce the sequence of random functions

$$\displaystyle\big{\{}\big{(}\underbrace{\omega\mapsto G_{n,K}(\omega,\theta)}_{\eqqcolon G_{n,K}(\theta)}\big{)}_{n\in\mathbb{N}_{+}}\colon\theta\in\Theta\big{\}},$$

where for $n\in\mathbb{N}_{+}$ and $\theta\in\Theta$, the random vector $G_{n,K}(\theta)$ has components $G_{j,n,K}(\theta)$, $j=1,\dotsc,p$, with

$$\displaystyle G_{j,n,K}(\theta)=\frac{\partial l_{n,K}}{\partial\theta_{j}}(\theta)-\mathbb{E}\bigg{[}\frac{\partial l_{n,K}}{\partial\theta_{j}}(\theta)\;\bigg{|}\;S_{(n)}\bigg{]},$$

and thus

$$G_{n,K}(\theta)=\nabla l_{n,K}(\theta)-\mathbb{E}\big{[}\nabla l_{n,K}(\theta)\;\big{|}\;S_{(n)}\big{]}.$$

(10)

Similarly, on $(\Omega,\mathcal{F},\mathbb{P})$, we introduce the sequence of random functions

$$\displaystyle\big{\{}\big{(}\underbrace{\omega\mapsto\widetilde{G}_{n,K}(\omega,\theta)}_{\eqqcolon\widetilde{G}_{n,K}(\theta)}\big{)}_{n\in\mathbb{N}_{+}}\colon\theta\in\Theta\big{\}},$$

where for any $n\in\mathbb{N}_{+}$ and $\theta\in\Theta$, the components of $\widetilde{G}_{n,K}(\theta)$ are given by

$$\displaystyle\widetilde{G}_{j,n,K}(\theta)=\frac{\partial\tilde{l}_{n,K}}{\partial\theta_{j}}(\theta)-\mathbb{E}\bigg{[}\frac{\partial\tilde{l}_{n,K}}{\partial\theta_{j}}(\theta)\;\bigg{|}\;S_{(n)}\bigg{]},\quad j=1,\dotsc,p,$$

and thus

$$\widetilde{G}_{n,K}(\theta)=\nabla\tilde{l}_{n,K}(\theta)-\mathbb{E}\big{[}\nabla\tilde{l}_{n,K}(\theta)\;\big{|}\;S_{(n)}\big{]}.$$

(11)

If the collection $\{c_{\theta}\colon\theta\in\Theta\}$ satisfies Assumption 3.1, we simply write, for any $n\in\mathbb{N}_{+}$,

$$\displaystyle J_{G_{n}}(\theta_{0})\coloneqq J_{G_{n,1}}(\theta_{0}),$$

for the random Jacobi-matrix of $\theta\mapsto G_{n,1}(\theta)$ evaluated at $\theta_{0}$.