Estimating overidentified linear models with heteroskedasticity and outliers

The model concerns a typical endogeneity problem:

	$\displaystyle y_{i}$	$\displaystyle=X_{i}^{*}\beta^{*}+W_{i}\gamma^{*}+\epsilon_{i}$		(1)
	$\displaystyle X_{i}$	$\displaystyle=Z_{i}^{*}\pi^{*}+W_{i}\delta^{*}+\eta_{i}$		(2)

Eq.(1) contains the parameter of interest $\beta^{*}$. $y_{i}$ is the response/outcome/dependent variable and $X_{i}^{*}$ is an $L_{1}$-dimensional row vector of endogenous covariate/explanatory/independent variables. Exogenous control variables, denoted as $W_{i}$, is an $L_{2}$-dimensional vector. Eq.(2) relates all the endogenous explanatory variables $X_{i}$ to instrumental variables, $Z_{i}$, and included exogenous control variables $W_{i}$ from Eq.(1). $Z_{i}$ is a $K_{1}$-dimensional row vector, where $K_{1}\geq L_{1}$. Since this paper focuses on overidentified cases, I will assume that $K_{1}>L_{1}$ throughout the rest of the paper. As Hausman et al. (2012), Bekker and Crudu (2015), this paper treats $Z$ as nonrandom (Alternatively, $Z$ can be assumed to be random, but conditioned on like in Chao et al. (2012)). $X_{i}$ is endogenous, ${\rm Cov}(\epsilon_{i},X_{i})={\rm Cov}(\epsilon_{i},Z_{i}\pi+\eta_{i})={\rm Cov}(\epsilon_{i},\eta_{i})=\sigma_{\epsilon\eta}\neq 0$. I further assume that each pair of $(\epsilon_{i},\eta_{i})$ are independently and identically distributed with mean zero and covariance matrix $\begin{pmatrix}\sigma_{\epsilon}^{2}&\sigma_{\epsilon\eta}\\ \sigma_{\eta\epsilon}&\sigma_{\eta}^{2}\end{pmatrix}$. $\sigma_{\epsilon}$ is a scalar. $\sigma_{\epsilon\eta}$ is a $L$-dimensional vector. $\sigma_{\eta}$ is a $L\times L$ matrix. I also impose a relevance constraint that $\pi\neq 0$. In matrix notation, I have Eqs.(1) and (2) as:

	$\displaystyle y=$	$\displaystyle X^{*}\beta^{*}+W\gamma^{*}+\epsilon$		(3)
	$\displaystyle X=$	$\displaystyle Z^{*}\pi^{*}+W\delta^{*}+\eta$		(4)

where $y$ and $\epsilon$ are $(N\times 1)$ column vector; $X$ and $\eta$ are $(N\times L_{1})$ matrices; $W$ is $(N\times L_{2})$ and $Z$ is a $(N\times K_{1})$ matrix, where $N$ is the number of observations. I also define the following notations for convenience:

	$\displaystyle X$	$\displaystyle=[X^{*}\quad W]\qquad\beta$		$\displaystyle=\begin{pmatrix}\beta^{*}\\ \gamma^{*}\end{pmatrix}$
	$\displaystyle Z$	$\displaystyle=[Z^{*}\quad W]\qquad\pi$		$\displaystyle=\begin{pmatrix}\pi^{*}&0_{K_{1}\times L_{2}}\\ \delta^{*}&I_{L_{2}}\end{pmatrix}$

Then, we have the following equivalent expressions for Eqs. (3) and (4):

	$\displaystyle y=X\beta+\epsilon$		(5)
	$\displaystyle X=Z\pi+\eta$		(6)

It is also useful to define the following partialled out version of variables:

	$\displaystyle\tilde{y}=y-W(W^{\prime}W)^{-1}W^{\prime}y$
	$\displaystyle\tilde{X}=X^{*}-W(W^{\prime}W)^{-1}W^{\prime}X^{*}$
	$\displaystyle\tilde{Z}=Z^{*}-W(W^{\prime}W)^{-1}W^{\prime}Z^{*}$

IV estimator is often used to solve this simultaneous equation problem. I tabulate in Table 1 some of the existing IV estimators this paper repeatedly refers to. They all have the matrix expression $(X^{\prime}C^{\prime}X)^{-1}(X^{\prime}C^{\prime}y)$. The caption of Table 1 summarizes how these estimators are linked to each other.

The following definition of approximate bias has been used by AIK 1995 and AD 2009 to motivate their development of new estimators whose $C$ matrix satisfies property $CZ=Z$ (and hence $CX=Z\pi+C\eta$):

For readers’ convenience, I justify why this definition can be used as a reasonable heuristic rule for motivating new estimators in Appendix A. Note that both AIK 1995 and AD 2009 assume that $CZ=Z$ which restricts the application of Definition 1. The condition that $CZ=Z$ is satisfied by OLS, TSLS, all k-class estimators, JIVE1, IJIVE1, and UIJIVE1, but not many other estimators such as JIVE2, HLIM, and HFUL from Hausman et al. (2012), $\lambda_{2}$- and $\omega_{2}$-class estimators which I will introduce in later sections of the paper. In Appendix B, I show that the condition $CZ=Z$ is not necessary, Definition 1 is a reasonable heuristic rule for the estimators as mentioned earlier which do not satisfy the condition $CZ=Z$.

Definition 1 yields Corollary 1 and Definitions 2 and 3. I will use them to analyze the existing IV estimators in Table 1.

I compute the approximate bias by showing the term $tr(C)-\mathcal{L}-1$ for the following estimators. Larger magnitude of $tr(C)-\mathcal{L}-1$ means a larger approximate bias. Ideally, $tr(C)-\mathcal{L}-1=0$, so that we can claim the estimator to be approximately unbiased.

The approximate bias computation for OLS, TSLS, and k-class estimators is straightforward: OLS’s approximate bias is proportional to $N-L-1$, TSLS’s approximate bias is proportional to $K-L-1$, k-class estimator’s approximate bias is proportional to $kK+(1-k)N-L-1$, where $k$ is a user-defined parameter. Setting $k=\frac{N-L-1}{N-K}$ gives us an approximately unbiased estimator. I call this estimator AUK (Approximately unbiased k-class estimator).

As explained earlier, TSLS’ approximate bias is proportional to $K-L-1$ which is large when the degree of overidentification is large. AIK 1995 proposes a jackknife version of TSLS to reduce the approximate bias of TSLS when the degree of overidentification is large. The authors call the proposed estimators JIVE1 and JIVE2 (Jackknife IV Estimator). Evaluating the approximate bias of JIVE2 is straight forward:

$$tr(P_{Z}-D)-\mathcal{L}-1=-L-1$$

We obtain the approximate bias of JIVE1:

$$tr((I-D)^{-1}(P_{Z}-D))-\mathcal{L}-1=\sum_{i=1}^{N}\frac{0}{1-D_{i}}-L-1=-L-1$$

AD 2009 reduces the approximate bias of the two JIVEs by paritialling out $W$ from $(y,X^{*},Z^{*})$. We have the approximate bias of IJIVE1 proportional to

$$tr((I-\tilde{D})^{-1}(P_{\tilde{Z}}-\tilde{D}))-\mathcal{L}-1=\sum_{i=1}^{N}\frac{0}{1-\tilde{D}_{i}}-L_{1}-1=-L_{1}-1$$

Note that $L_{1}<L$, so IJIVE1 potentially has a smaller approximate bias than JIVEs.³³3The comparison requires knowledge on $\lim_{N\to\infty}\pi^{\prime}\frac{\tilde{Z}^{\prime}\tilde{Z}}{N}\pi$. However, since approximate bias is used only as a heuristic rule to motivate new estimators, evaluating $tr(C)-tr(P_{Z\pi})-1$ is arguably sufficient for motivating purposes. AD 2009 also further reduces approximate bias by adding a constant ($\frac{L_{1}+1}{N}$) to the diagonal of the $C$ matrix of IJIVE1. The new estimator is called UIJIVE1.

To evaluate the approximate bias of UIJIVE1, I make the following assumption about the leverage of $Z$, ${\{{D}_{i}\}}_{i=1}^{N}$:

I state the assumption in terms of $Z$ as I will repeatedly use it for $Z$ in Section 6. Here, to compute the approximate bias of IJIVE1, I make assumption BA, but for ${\{\tilde{D}_{i}\}}_{i=1}^{N}$ instead of for ${\{{D}_{i}\}}_{i=1}^{N}$. Under Assumption BA for ${\{\tilde{D}_{i}\}}_{i=1}^{N}$, UIJIVE1’s approximate bias is proportional to

$$tr(C)-\mathcal{L}-1=\sum_{i=1}^{N}\frac{\frac{L_{1}+1}{N}}{1-\tilde{D}_{i}+\frac{L_{1}+1}{N}}-L_{1}-1\to 0$$

Therefore, we have that UIJIVE1’s approximate bias is asymptotically vanishing. See proof in Appendix C.

I define the $\omega_{1}$-class estimator that contains all the following four estimators: JIVE1, IJIVE1, UIJIVE1, and OLS. Since all of them can be expressed as: $(X^{\prime}C^{\prime}X)^{-1}(X^{\prime}C^{\prime}y)$, I define matrix $C$ for the $\omega_{1}$-class estimators:

$$(I-D+\omega_{1}I)^{-1}(P_{Z}-D+\omega_{1}I)$$

when $\omega_{1}=0$, it corresponds to JIVE1; when $\omega_{1}=\infty$, it corresponds to OLS. On the other hand, IJIVE1 is a special case of JIVE1 where $(y,X,Z)$ is replaced with $(\tilde{y},\tilde{X},\tilde{Z})$, and hence is belonged to $\omega_{1}$-class estimator. UIJIVE1 uses $(\tilde{y},\tilde{X},\tilde{Z})$ and sets $\omega_{1}=\frac{L_{1}+1}{N}$. As stated in Section 3.2, this choice of $\omega_{1}=\frac{L_{1}+1}{N}$ outputs UIJIVE1 whose approximate bias is asymptotically vanishing. The information of these four estimators is summarized in Table 2a.

The development from TSLS to JIVE1 to IJIVE1 to UIJIVE1 is depicted in the upper half of Figure 1. TSLS to JIVE1: JIVE1 is a jackknife version of TSLS where the first stage OLS in TSLS is replaced with a jackknife procedure for out of sample prediction. Call the resulting matrix from the first stage $\hat{X}$, the second stage IV estimation is the same for TSLS and JIVE1: $(\hat{X}^{\prime}X)^{-1}(\hat{X}^{\prime}y)$. JIVE1 to IJIVE1: IJIVE1 is a special case of JIVE1. JIVE1 takes $(y,X,Z)$ as the input, IJIVE1 takes $(\tilde{y},\tilde{X},\tilde{Z})$ (the partialled-out version of $(y,X,Z)$) as input. IJIVE1 to UIJIVE1: UIJIVE1 adds a constant $\omega_{1}=\frac{L_{1}+1}{N}$ to the diagonal of $C$ matrix of IJIVE1. We can interpret this addition of constant $\omega_{1}I$ as bridging IJIVE1 and OLS since OLS has its $C$ matrix as $I$.

I define $\omega_{2}$-class of estimator that contains JIVE2 and OLS. The $C$ matrix of $\omega_{2}$-class estimator is

$$P_{Z}-D+\omega_{2}I.$$

The only difference between $C$ matrices for $\omega_{1}$- and $\omega_{2}$- class estimators is the exclusion/inclusion of $(I-D+\omega_{1}I)^{-1}$, which is a rowwise division operation (i.e. dividing $i$th row of $P_{Z}-D+\omega_{1}I$ by $1-D_{i}+\omega_{1}$).

$\omega_{2}=0$ corresponds to JIVE2 and $\omega_{2}=\infty$ corresponds to OLS. By Corollary 1, the approximate bias of $\omega_{2}$-class estimators that takes $(\tilde{y},\tilde{X},\tilde{Z})$ is proportional to $\omega_{2}N-L_{1}-1$. If $\omega_{2}=0$, I obtain a new estimator IJIVE2, corresponding to IJIVE1 from $\omega_{1}$-class estimators. Selecting $\omega_{2}=\frac{L_{1}+1}{N}$, I obtain approximately unbiased $\omega_{2}$-class estimator and name it UIJIVE2. Its closed-form expression is:

$$(\tilde{X}^{\prime}(P_{\tilde{Z}}-\tilde{D}+\frac{L_{1}+1}{N}I)^{\prime}\tilde{X})^{-1}(\tilde{X}^{\prime}(P_{\tilde{Z}}-\tilde{D}+\frac{L_{1}+1}{N}I)^{\prime}\tilde{y})$$

where the transpose on $P_{\tilde{Z}}-\tilde{D}+\frac{L_{1}+1}{N}I$ is not necessary as the matrix is symmetric, but I keep it for coherent notation.

The information of OLS, JIVE2, IJIVE2, and UIJIVE2 are summarized in Table 2b. I also depict the parallel relationship between $\omega_{1}$-class and $\omega_{2}$-class estimators in the lower half of Figure 1. The development from OLS to JIVE2 to IJIVE2 to UIJIVE2 is analogous to the development of the $\omega_{1}$-class estimators. Readers can also interpret Figure 1 from top to bottom, all the estimators in the bottom remove the rowwise division operations in the estimators above them. The removal of rowwise division is important to the comparison between $\omega_{1}$- and $\omega_{2}$-class estimators. The division operation is highly unstable when $D_{i}$ (or $\tilde{D}_{i}$) is close to 1, leading undesirable asymptotic property to $\omega_{1}$-class estimator, but not to $\omega_{2}$-class estimator. This point will be explained formally in Section 6.

Under fixed $K$ and $L$, I show that both UOJIVE1 and UOJIVE2 are consistent as $N\to\infty$ with some assumptions imposed on the moment existence for observable and unobservable variables. It is worth mentioning that Assumption BA is important to the consistency of UOJIVE1, but not to that of UOJIVE2, suggesting that UOJIVE2 is robust to the presence of high leverage points while UOJIVE1 is not. Throughout this subsection, I make the following assumptions

Denote $\frac{L+1}{N}$ as $\omega$. Recall that UOJIVE1’s matrix expression is

	$\displaystyle\hat{\beta}_{UOJIVE1}=$	$\displaystyle(X^{\prime}(P_{Z}-D+\omega I)(I-D+\omega I)^{-1}X)^{-1}(X^{\prime}(P_{Z}-D+\omega I)(I-D+\omega I)^{-1}y)$
	$\displaystyle=$	$\displaystyle\beta+(X^{\prime}(P_{Z}-D+\omega I)(I-D+\omega I)^{-1}X)^{-1}(X^{\prime}(P_{Z}-D+\omega I)(I-D+\omega I)^{-1}\epsilon)$

The proofs for Lemma 6.1 and 6.2 are collected in Appendix E. The importance of Assumption BA stems from the presence of $\{D_{i}\}_{i=1}^{N}$ in the denominators. UOJIVE2 does not have the same problem since $(I-D+\omega I)^{-1}$ does not appear in its analytical form.

Now, we turn to analyzing the consistency of UOJIVE2. The steps for proving its consistency are similar to what we have done for UOJIVE1.

The proofs for Lemma 6.3 and 6.4 are collected in Appendix E. With these two lemmas, we establish the consistency of UOJIVE2.

Proof for Lemma 6.5 can be found in Appendix E. $\frac{1}{\sqrt{N}}X^{\prime}(P_{Z}-D+\omega I)(I-D+\omega I)^{-1}\epsilon\overset{d}{\to}N(0,\sigma_{\epsilon}^{2}H)$. Combining with Lemma 6.1, the asymptotic variance of UOJIVE is $\sigma_{\epsilon}^{2}H^{-1}$.

Proof for the lemma can be found in Appendix E. Lemma 6.3 and 6.6 establish the asymptotic normality of UOJIVE2 without Assumption BA.

The many-instrument asymptotics framework is that both $K_{1}$ and $N$ go to infinity and the ratio $\frac{K_{1}}{N}$ converges to a constant $\alpha$ where $0<\alpha<1$. The motivation behind the many-instrument framework is to provide a better approximation of a situation where the number of instruments is large relative to sample size and first-stage overfitting is concerning. Many papers have looked into many-instrument setup (Bekker, 1994; Kunitomo, 2012; Chao et al., 2012; Hausman et al., 2012). Theorem 1 from Hausman et al. (2012) directly applies to UOJIVE2.

The simulation setup for many-instrument asymptotics is summarized in Table 5. The parameters for all instruments $Z^{*}$ (and for all controls $W$) are set as a constant, so I only report one value for each parameter instead of a vector. $R_{0}=\frac{N\pi E[Z^{\prime}Z]\pi}{\sigma_{\eta}^{2}}$ is the concentration parameter, I set it to be around 150 following Hansen and Kozbur (2014) to maintain a reasonable instrumental variable strength. $Z$ follows standard multivariate normal distributions.⁴⁴4I follow Bekker and Crudu (2015) which also assumes nonrandom $Z$ for analyzing the theoretical property of the IV estimator, but let $Z$ follow a random distribution in their simulation study. The error terms $\epsilon$ and $\eta$ are bivariate normal with mean $(0,0)^{\prime}$ and covariance matrix $\begin{pmatrix}0.8&-0.6\\ -0.6&1\end{pmatrix}$.

I report the simulation results in Table 6, it is clear that OLS, TSLS, and JIVEs suffer from a larger bias than approximately unbiased estimators. In addition, as I increase the degree of overidentification and the number of control variables, the bias of TSLS and JIVEs deteriorates. The sharp worsening of JIVEs’ bias is likely exacerbated by its instability as pointed out by Davidson and MacKinnon (2007). The simulation results align with Corollary 1 which implies that TSLS’ approximate bias is proportional to the degree of overidentification while JIVEs’ approximate biases are proportional to the number of control variables. All approximately unbiased estimators perform well under both setups. For example, UOJIVE1 and UOJIVE2 virtually do not show any bias and have a relatively small variance compared to JIVEs.

Following the setup in Ackerberg and Devereux (2009), I test the performance of estimators with heteroskedastic errors while setting $Z$ to be a group-fixed effect matrix without any control variables $W$. Since there is no $W$, UIJIVEs and UOJIVEs are equivalent and I only report UOJIVEs’ results.

The sample size is set to be 500, $N=500$. The first 115 observations belong to Group 1, the next 115 observations belong to Group 2. Groups 1 and 2 are two big groups. The rest of the 270 observations consist of 18 small groups. Each small group has 15 observations. The first group is excluded from $Z$ and $\pi^{*}=0.3$, meaning that Group 1 on average has its $X$ value 0.3 below other groups conditional on everything else being equal. There are two covariance matrices: $\begin{pmatrix}0.25&-0.1\\ -0.1&0.25\end{pmatrix}$ and $\begin{pmatrix}0.25&0.2\\ 0.2&0.25\end{pmatrix}$, denoted as $-$ and $+$ covariance matrices, respectively. In Setup 1, I let the big groups have the $+$ covariance matrix and the small groups have the $-$ covariance matrix; in Setup 2, I reverse the covariance matrices for the two types of groups.

I summarize the simulation results in Table 7. The result aligns with Ackerberg and Devereux (2009). The Nagar estimator is neither approximately unbiased nor consistent under heteroskedasticity with many instruments. The same can be said about approximately unbiased k-class estimator AUK which is proposed in Appendix D. In contrast, the performance of UOJIVE2 remains stellar, aligning with Theorem 6.5 that consistency of UOJIVE2 is established without assuming homoskedasticity.

In my proofs for the consistency and asymptotic normality results of UOJIVE1, Assumption BA is repeatedly invoked to bound below the denominator. With high leverage points, the asymptotic proofs for UOJIVE1 are no longer valid. In contrast, UOJIVE2’s asymptotic results do not require Assumption BA. In the following simulation setups, I deliberately introduce high leverage points in the DGP, when these high leverage points coincide with large variances of $\epsilon$, the performance of UOJIVE1 is much worse than that of UOJIVE2.

There are 5 instruments $Z^{*}$, one endogenous variable $X^{*}$, and no controls $W$. Again, UIJIVEs and UOJIVEs are equivalent due to the absence of $W$, so I only report UOJIVEs. I set the sample size $N$ to be $\{101,401,901,1601\}$. All these numbers are 1 plus a square number so it is easy to set up the following simulations. The first observation is the high leverage point with its first entry being $(N-1)^{1/3}$. For the rest of the $N-1$ observations, every $\sqrt{N-1}$ observations have their first five rows being the identity matrix, the rest of the rows are all zeroes. This setup is equivalent to the group fixed effect setup for the heteroskedasticity simulation where there are five small groups (indexed 2-6) and one large group (Group 1), except the first row which is supposed to belong to Group 2 is contaminated, and has its value multiplied by $N^{1/3}$. I include Table 11 in the Appendix F to illustrate this simulation setup. The error terms $\epsilon$ and $\eta$ are bivariate normal with mean $(0,0)^{\prime}$ and covariance matrix $\begin{pmatrix}0.8&-0.6\\ -0.6&1\end{pmatrix}$, except that $\epsilon_{1}$, the error terms for the high leverage point, is multiplied by $N^{1/3}$. The coincidence between high leverage and large variance of $\epsilon$ generates an outlier. Intuitively, the first observation has a high influence on its fitted value and has a large probability of deviating a lot from the regression line. $\pi^{*}$ is set to be one.

The results for the simulations with outliers are summarized in Table 8. Throughout the four simulations, UOJIVE2 outperforms UOJIVE1 and TSJI2 outperforms TSJI1 substantially. The simulation result points to the importance of Assumption BA to the consistency of UOJIVE1.

Moreover, the consistency results of UOJIVE2 seem to be preserved under this simulation setup even though $\Sigma_{ZZ}^{-1}$ does not exist. As shown by Figure 3, the empirical distribution of the 1000 UOJIVE2 estimates concentrates around $\beta^{*}=0.3$ more and more as the sample size increases, whereas that of UOJIVE1 does not change much across the four simulations. It can be an interesting future research direction to show that UOJIVE2 is consistent under such a weak-instrument setup whereas UOJIVE1 is not.

The quarter of birth example has been repeatedly cited by many-instrument literature. Here I apply TSJIs and UOJIVEs the famous example in Angrist and Krueger (1991).

Many states in the US have a compulsory school attendance policy. Students are mandated to stay in school until their 16th, 17th, or 18th birthday depending on which state they are from. As such, students’ quarter of birth may induce different quitting-school behaviors. This natural experiment makes a quarter of birth a valid IV to estimate the marginal earnings brought by an additional school year for those who are affected by the compulsory attendance policy.

Angrist and Krueger (1991) interacts quarter of birth with other dummy variables to generate a large number of IVs,

1. 1.

   Quarter of birth $\times$ Year of birth

2. 2.

   Quarter of birth $\times$ Year of birth $\times$ State of birth

where case 1 contains 30 instruments, and case 2 contains 180 instruments. The results are reported in Table 9.

Bedard and Deschênes (2006) use year of birth and its interaction with gender as instruments to estimate how much enlisting for WWII and the Korean War increases the veterans’ probability of smoking during the later part of their lives. The result can be interpreted as LATE.

1. 1.

   Birth year $\times$ gender

2. 2.

   Birth year

where case 1 uses all data and case 2 uses only data for male veterans. The results are summarized in Table 10.

The results of all estimators are close except for Case 2 with CPS90. In this setup of Case 2 CPS90, JIVE’s result, which is negative and counterintuitively large in magnitude (larger than 1), is driven by its instability. TSLS’ estimate is about 10% below the estimates of other approximately unbiased estimators. Compared to other setups (Case 1 CPS60, Case 1 CPS90, Case 2 CPS90), the TSLS estimate for this case is the smallest. In contrast, for other approximately unbiased estimators, their estimates throughout all four cases are closer to each other. Bedard and Deschênes (2006) claims that the four TSLS estimates “are almost identical”. Table 10 gives stronger evidence for this claim with even closer estimates from other approximately unbiased estimators.