The Equivariance Criterion in a Linear Model for Fixed-X Cases

Consider a general linear model with a response $Y\in\mathbb{R}$ and a covariate vector ${\bf X}\in\mathbb{R}^{p}$. The aims of the linear model are to build a linear functional relationship between $Y$ and ${\bf X}$ from paired observations, $({\bf X}_{1},Y_{1}),\dots,({\bf X}_{n},Y_{n})$, and predict a future response, $Y_{0}$, given a covariate, ${\bf X}_{0}$. Predominately in literature, a fixed-$X$ assumption has been used with the following understandings:

(i) The covariate values are fixed before sampling while the only randomness comes from the responses $Y_{1},\dots,Y_{n}$;

(ii) There is no other relationship among those covariate values.

Meanwhile, random-$X$ cases have been investigated, e.g., (Breiman and Spector (1992)), where the assumption is to set the covariate values sampled from a random vector.

In this article, we will start with a fixed-$X$ as a combination of $p$ linearly independent rows each repeated $n_{i}\geq 1$ times, i.e.,

where $\sum n_{i}=n$. One can argue that this is the general form for a fixed-$X$ case as the rank of the design matrix $X$ is $p$ allowing only $p$ distinct rows. The difference between a fixed-$X$ and a random-$X$ lays on the fact whether $X$ is fully known before sampling. Naturally, one should set the number of parameters as the rank of $X$, $p$. Otherwise if there are more parameters than $p$, then the model carries redundant parameters, some of which may not be identifiable and thus not estimable. On the other hand, if there are fewer parameters than $p$, then more parameters will be needed or some of $X$ may be considered as unknown before sampling and hence $X$ is random and out of scope of this article. To derive the optimal solution, we will introduce equivariance criterion, which has been widely considered in the linear model, e.g., Rao (1965); Eaton (1989) and the distinctions between fixed-$X$ and random-$X$ cases are clearly noticed (Rao (1973), Rosset and Tibshirani (2020)).

As a principle of symmetry, equivariance criterion is proposed in literature as a logic constraint on the solutions to derive the optimal one. Compared to the unbiasedness, it is more focused on the symmetry of the problem and self-consistency of the solutions. In the era of machine learning, it has received rising attention due to widely existing symmetry in the applications. Lehmann and Casella (1998, Chap. 3) has given a detailed discussion of equivariance criterion in the location-scale family while Berger (1985, Chap. 6) has presented the theory in a decision theoretic framework. Besides those two classical textbooks, other important references are as follows: Hora and Buehler (1966); Eaton (1989); Wijsman (1990).

Equivariance criterion consists of two principles: (i) functional equivariance, which states the action in a decision problem should be consistence across different measures and (ii) formal equivariance, which requires the decision rule to be of the same form for two problems with the identical structure.

Formally, a decision problem is described by $({\cal X},{\cal P},\Theta,{\cal D},L)$, where ${\cal X}$ is a sample space, $\cal P$ is a family of distributions with $\theta$ as the parameter or the true state of nature, $\Theta$ is the parameter space, $\cal D$ is a decision space and $L$ is a loss function on ${\cal D}\otimes\Theta$.

Without loss of generalities, one starts with the identifiability on both the distribution family and loss function, which, for linear model, suggests that there is no redundant $\boldsymbol{\beta}\in\mathbb{R}^{p}$ and $d\in\mathbb{R}^{p}$. The principle of functional equivariance criterion requires preservance of the model and invariance of loss function under a group of 1-1 and onto transformations, which, are defined as follows.

Under the structure of an invariant decision problem, we can induce equivariance criterion.

In this way, equivariance criterion imposes a constraint on the possible decision rules that one can use reasonably to derive the optimal decision rule. In this article, we will pursue the best decision rule (minimum risk equivariant rule, MRE rule) in the sense of minimizing the risk function as the expectation of the loss function over ${\bf X}$, which is a function of the parameter and the decision rule.

Lehmann and Casella (1998) has applied equivariance criterion implicitly in linear model for fixed-$X$ cases, where the most important feature is to transform the problem to a canonical form and derive the best equivariant estimators for the coefficient vector and common variance. Wu and Yang (2002) has discussed the existence of best equivariant estimators for the coefficient vector and the covariance matrix (in three forms) in normal linear models with fixed-$X$s and derived the forms when they exist. Kurata and Matsuura (2016) has derived the best equivariant estimator of regression coefficients in a seemingly unrelated regression (SUR) model with a known correlation matrix. Further, Matsuura and Kurata (2020) derived the best equivariant estimator of the variance matrix in a SUR model. It is noted that most literature works assume a single population for the linear model and thus apply a common invariant transformation on the responses to derive the optimal equivariant estimators. However, as in experimental design, one usually views each distinct covariate vector as an independent population from any other and a SUR model uses a natural multivariate extension of the common linear model. Therefore, a multivariate extension of the common linear model combined with equivariance criterion warrants further investigation, especially from the fundamental logic of equivariance criterion.

This article focuses on applying the logic of equivariance criterion to the fixed-$X$ linear model, where the linear model is extended to allow multiple populations instead of a single one and the invariant transformation is distinct for each population. In Section 2, we derive the best equivariant estimators for the coefficient vector and the diagonal covariance matrix in a normal linear model specially tuned for equivariance criterion. Section 3 is devoted to some concluding remarks & future work.

Consider the linear regression model ${\bf y}=X{\bf\boldsymbol{\beta}}+{\bf\epsilon}$, where ${\bf y}$ is an $n\times 1$ vector, $X$ is an $n\times p$ matrix and ${\bf\epsilon}$ is the noise vector. To derive the best equivariant estimators in the model, we will walk through the basic elements of equivariance criterion first and then provide a linear model tuned for equivariance criterion starting from the basic concept of a population.

Preservance of the Model: Without loss of generalities, we assume that the design matrix $X$ is predetermined and of full rank with $n\geq p+1$, where only the response ${\bf y}$ is random and thus transformations only on ${\bf y}$ will be considered. For a fixed-$X$ linear model, it can be viewed that the samples actually come from different populations as in the experimental design, which imposes a restriction over the choice of the transformation group, in addition to the specification of the model. Equivariance criterion implicitly requires the transformations over the samples from the same population to be identical while being distinct for samples from different populations. Therefore, it is of essence to determine the number of populations inside a linear model.

For the fixed-$X$ cases, one can argue that there can only be $p$ populations inside the linear model as the rank of the design matrix is $p$. In this regard, we will assume that $\epsilon_{i_{j}}$’s are independently distributed as a normal distribution with mean $0$ and unknown variance $\sigma_{i}^{2}$, $i=1,\ldots,p$. Thus, we have the normal linear model tuned for equivariance criterion as follows,

Note that here, it is natural to assume independent variances for each population in contrast to the traditional linear model where all the variances for ${\bf y}$ are assumed to be identical. In this case, we will have the usual identifiability of the model with respect to ${\bf\boldsymbol{\beta}}$ and $\Sigma$. Since $\Sigma=Diag(\sigma_{1}^{2},\ldots,\sigma_{1}^{2},\sigma_{2}^{2},\ldots,\sigma_{2}^{2},\ldots,\sigma_{p}^{2},\ldots,\sigma_{p}^{2})$ contains only $p$ parameters, one can simplify the problem to consider $\Sigma_{p}=Diag(\sigma_{1}^{2},\ldots,\sigma_{p}^{2})$.

The transformation group $G$ keeping the model invariant is of the form $g({\bf y})=C{\bf y}+{\bf a}$, with $C=Diag(c_{1},\ldots,c_{1},c_{2},\ldots,c_{2},\ldots,c_{p},\ldots,c_{p})$, and ${\bf a}=(a_{1},\ldots,a_{1},a_{2},\ldots,a_{2},\ldots,a_{p},\ldots,a_{p})^{\prime}$, $c_{i}>0$, $a_{i}\in\mathbb{R},i=1,\ldots,p$. (Both $c_{i}$ and $a_{i}$ are repeated $n_{i}$ times.) Here we use independent transformations for those populations, compared to the literature where most are using one single common transformation, $g_{c}({\bf y})=c{\bf y}+a,c>0,a\in\mathbb{R}$. To facilitate discussion, we denote that $X_{p}=({\bf X_{1},X_{2}},\ldots,{\bf X_{p}})^{\prime}$, $C_{p}=Diag(c_{1},\ldots,c_{p})$, ${\bf a}_{p}=(a_{1},\ldots,a_{p})^{\prime}$ and thus $X=KX_{p},{\bf a}=K{\bf a}_{p},$ $C=\left[\begin{matrix}c_{1}I_{n_{1}}&0&\cdots&0\\ 0&c_{2}I_{n_{2}}&\ddots&\vdots\\ \vdots&\ddots&\ddots&0\\ 0&\cdots&0&c_{p}I_{n_{p}}\\ \end{matrix}\right]$ with $\ K=\left[\begin{matrix}\vec{1}_{n_{1}}&\vec{0}_{n_{1}}&\vec{0}_{n_{1}}&\vec{0}_{n_{1}}&\vec{0}_{n_{1}}\\ \vec{0}_{n_{2}}&\vec{1}_{n_{2}}&\vec{0}_{n_{2}}&\vec{0}_{n_{2}}&\vec{0}_{n_{2}}\\ \vec{0}_{n_{i}}&\vec{0}_{n_{i}}&\vec{1}_{n_{i}}&\vec{0}_{n_{i}}&\vec{0}_{n_{i}}\\ &&&\ddots&&\\ \vec{0}_{n_{p}}&\vec{0}_{n_{p}}&\vec{0}_{n_{p}}&\vec{0}_{n_{p}}&\vec{1}_{n_{p}}\\ \end{matrix}\right]_{n\times p}$. The corresponding transformation group $\bar{G}$ on the parameter space is of the form

It can be shown that the parameter space is transitive under $\bar{G}$ as $n\geq p+1$.

In the linear model, the usual targets of interest are the coefficient vector $\boldsymbol{\beta}$ and the covariance matrix $\Sigma$. We will discuss those two separately in the context of Invariance of the Loss Function.

To derive an MRE decision rule, Lehmann and Casella (1998, Chap. 3) have used maximal invariants to characterize all the equivariant estimators and then minimized the constant risk when $\bar{G}$ is transitive for location-scale families. We will follow such an approach to derive the best estimators as follows.

This paper has served as an initial effort to explore the usage of equivariance criterion in the field of machine learning, where we are interested at which method yields the optimal solution and what properties the optimal solution carries, especially in the context of equivariance criterion. We start from least square in the linear model, the simplest and foundational method in machine learning, to derive the optimal solution.

In this paper, we have established that MRE estimators for the coefficient vector and the condensed covariance matrix is the least square and the vector of the sample variance within each population, respectively. In addition, we have demonstrated that in our setting 2.1, the least square estimator is essentially the vector of the sample mean within each population. Such a finding has further solidified their optimality from the perspective of multivariate normal distribution theory.

The linear model with a full rank design matrix can be of different forms across literature via the number of populations. The commonly used one is of a single population, whose characteristic is to assume a common distribution for all the noises. Naturally, in this setup, one would use a single univariate location-scale transformation to apply equivariance criterion. In this paper, we relax such an assumption and allow $p$ populations to accommodate the usage of a larger transformation group, which is consisted of multivariate location-scale transformations. From the perspective of experimental design, such a relaxation is quite intuitive: the design matrix is chosen carefully before sampling and in a sense, those sampling points are independent and each rank consist of a population. Meanwhile one can argue that $p$ populations should be the maximum number allowed in a linear model.

In terms of estimating the coefficient vector, the form of $X$ doesn’t have much impact on the MRE solution and thus 2.1 gives a simple path to the MRE solution. However, to estimate the condensed covariance matrix, it is essential to choose the form and thus decide the size of each population. In an experimental design setting, one would usually use $n=kp$, where $k$ is an integer, and $n_{i}=k$. Such a form is recommended both for its symmetry and computation convenience, which also is a special case of a seemingly unrelated regression (SUR) problem with a known correlation matrix. However, one can convert a SUR problem back into the setting $n_{i}=k$ and functional equivariance guarantees that the MRE solution is of a corresponding form. In this sense, our model is more general than the SUR model used in Kurata and Matsuura (2016); Matsuura and Kurata (2020).

We have discussed the best equivariant estimators for the coefficient vector and the condensed covariance matrix for a normal linear model specially tuned for equivariance criterion, where the commonly used one is a special example. Such a model requires a bigger transformation group, which, in turn, will result in a smaller set of equivariant decision rules, where the MRE estimator exists. Interestingly, Wu and Yang (2002) has shown that the commonly used single location-scale transformation induces too many equivariant decision rules under $L_{q}$ that there is no MRE estimator for the covariance matrix. Meanwhile, each population inside the normal model is the linear model commonly used in literature and the resulted estimators for each population are the traditional MRE ones, which are equivalent to the least square solutions. Kurata and Matsuura (2016); Matsuura and Kurata (2020) used the same transformation group for the SUR model, where a $p$-dimensional distribution family was considered and samples comes from such a single multivariate population.

The choice of the invariant transformation group is an important topic in equivariance literature. Wu and Yang (2002) presented a case where the group is too large to allow an optimal solution. Usually, the group is chosen to be isomorphic to sample space/parameter space, especially when considering the Haar Prior. There are some interesting cases that all invariant transformation groups pose a nesting relationship and the largest one admits only the optimal solution in the smaller ones.

Likelihood loss is a multivariate extension of the Stein loss, which is preferred in literature (Brown (1968)). It can be seen that it induces an MRE & UMVU estimator, that is always larger than the one under the quadratic one. Meanwhile, likelihood loss is more evenhanded over the range as the covariance matrix is set to be positive definite. Such a form is quite similar to the logistic transformation in a generalized linear model.

An extension to prediction will be a future direction. However, existing frameworks (e.g., Zhou and Nayak (2015)) on the equivariance criterion can’t handle the prediction problem well in a linear model. In the literature, the predicted response is assumed to be unobservable, which is not the case in the linear model. One could also notice that overfitting (Stone (1974)) arises for a prediction problem in a linear model, which usually doesn’t occur in estimation as in deriving the least square solution, sample prediction error is used, which will converge to a univariate form of $L_{\bf\boldsymbol{\beta}}$, $({\bf d}-{\bf\boldsymbol{\beta}})^{T}X_{p}^{T}X_{p}({\bf d}-{\bf\boldsymbol{\beta}})/\sigma^{2}$ with $\sigma^{2}=1$, with an extra term constant to $d$.

Linear model with fixed-$X$ cases though predominantly used in literature, is of limited usage in practice, especially in our settings, where the experimental design is the ideal scenario. Linear model with random-$X$ cases and mixture cases are more interesting and challenging. The concept of the randomness of linear model has drawn wide attention and numerous efforts have been spent to clarify the differences between fixedness and randomness. Little (2019) has given a straightforward definition of randomness as being unknown from a Bayesian view. In his argument, the treatment indicator from a clinical trial can be considered both fixed and random. It is true that those semi-controlled covariates pose challenges to the definition of randomness. Individually, it is unknown and thus can be considered random. Population-wise, its distribution is usually under control and thus can be treated as a fixed effect in analyses.

The Gauss-Markov Theorem is the fundamental result for the linear model, where the optimality of the least square solution has been established. In most textbooks, its proof is based on the predominantly assumed fixed-$X$ cases. Shaffer (1991) has shown some interesting results where the Gauss-Markov Theorem no longer holds for some random-$X$ cases. We will investigate such a phenomenon in the context of equivariance for the random-$X$ cases.

In terms of randomness, it is noticed that the setup before sampling is crucial and one can classify $X$ before sampling into following categories: known values, from a known distribution, from a distribution family with unknown parameters, totally unknown. In a typical experimental design setting, design factors/parameters are of known values, which, in our settings, refer to fixed-$X$ cases. In a typical clinical trial setting, the treatment indicator is from a known distribution. In the classical parametric inference setting, we may assume $X$ from a distribution family with unknown parameters. For the non-parametric setting, X is usually seen as totally unknown. The latter three scenarios refer to random-$X$ cases, which will be another future topic.

For a linear model with a non-normal distribution, we will refer to extensions of the current results to the generalized linear model, where the challenges start from the invariant transformations. In a normal linear model, one can easily find an invariant location-scale transformation group that leave the parameter space transitive, which facilitates the derivation of the MRE solutions. This may not be the case in a non-normal linear model.