Estimation and Inference of Average Treatment Effect in Percentage Points under Heterogeneity

Interpreting estimates of $\tilde{\tau}$, the weighted average of treatment effect in log points, as approximations of $\bar{\rho}$ is common in empirical research. This practice is particularly prevalent when treatment effects in log-transformed outcomes are heterogeneous, yet estimation assumes a constant effect. For example, consider a semi-log regression model $\ln(y_{i})=\alpha+\tau s_{i}+x_{i}^{\prime}\beta+\epsilon_{i},$ where $s_{i}$ is the treatment dummy and $x_{i}$ is a column vector of covariates. Under the assumption $E\left[\ln(y_{0,i})|x_{i},s_{i}\right]=E(\ln(y_{0,i})|x_{i})$, where $y_{0,i}$ is the potential outcome if not treated, the OLS estimator of $\tau$ is a convex average of covariate-specific treatment effect in $\ln(y_{i})$ (Angrist, 1998). Another example is the two-way fixed effects (TWFE) estimator for the staggered difference-in-differences designs, where different cohorts start to get treated at different times. Treatment effects can be heterogeneous across cohorts and across event time (time since treatment). Consider the following TWFE model:

$$\ln(y_{it})=\alpha_{i}+\gamma_{t}+\tau s_{it}+x_{it}^{\prime}\beta+\epsilon_{it},$$

(2)

where $y_{it}$ is the outcome of individual $i$ at time $t$, $s_{it}$ is the indicator that $i$ has already been treated in period $t$, $\alpha_{i}$, $\gamma_{t}$ and $x_{it}$ are individual fixed effects, time fixed effects and control variables respectively. Under parallel trend and no anticipation assumptions, the OLS estimator for $\tau$ is a weighted average of treatment effect in $\ln(y_{i})$ for different cohort-event time combinations. Notably, some of the weights can potentially be negative (de Chaisemartin and D’Haultfœuille, 2020; Goodman-Bacon, 2021). When outcomes are log-transformed, two-stage least squares (2SLS) estimators are also weighted average of log point effects (see, e.g., Imbens and Angrist 1994; Mogstad et al. 2021).

In these examples, interpreting the coefficient of the treatment dummy or its exponential transformation minus one as the ATE in percentage points implicitly approximates $\bar{\rho}$ with weighted averages of log points effects $\tilde{\tau}$ or $\operatorname{exp}(\tilde{\tau})-1$. Such approximations can lead to misleading interpretations due to several key issues.

First, if $\tilde{\tau}$ is a non-convex average of $\tau_{g}$, as can occur in some staggered difference-in-differences settings, then $\tilde{\tau}$ and $\operatorname{exp}(\tilde{\tau})-1$ may fall out of the convex hull of $\rho_{g}$. For example, if $(\tilde{w}_{1},\tilde{w}_{2})=(-0.1,1.1)$ and $(\tau_{1},\tau_{2})=(0.1,0.05)$, we have $(\rho_{1},\rho_{2})=(0.105,0.051)$, hence $\tilde{\tau}=0.045$ and $\operatorname{exp}(\tilde{\tau})-1=0.046$ are both outside the convex hull of $\rho_{g}$.

Secondly, if $\tilde{\tau}$ is a convex average of $\tau_{g}$, $\tilde{\tau}$ may still be out of the convex hull of $\rho_{g}$, although $\operatorname{exp}(\tilde{\tau})-1$ is in the convex hull of $\rho_{g}$. One example of the former case is when $(\tau_{1},\tau_{2})=(0.09,0.1)$ and $\tilde{w}=(0.9,0.1)$, then $\tilde{\tau}=0.091$ is not in the convex hull of $(\rho_{1},\rho_{2})=(0.094,0.105)$. To see the latter, note that with $0\leqslant\tilde{w}_{g}\leqslant 1$, $\min_{g}(\tau_{g})\leqslant\tilde{\tau}\leqslant\max_{g}(\tau_{g})$, hence $\min_{g}(\rho_{g})=\min(\operatorname{exp}(\tau_{g})-1)\leqslant\operatorname{exp}(\tilde{\tau})-1\leqslant\max_{g}(\operatorname{exp}(\tau_{g})-1)=\max(\rho_{g})$.

Third, even if $\tilde{\tau}$ and $\operatorname{exp}(\tilde{\tau})-1$ are in the convex hull of $\rho_{g}$, they are different from $\bar{\rho}$. Take $\tilde{\tau}$ for example,

$\displaystyle\tilde{\tau}-\bar{\rho}$

$\displaystyle=\sum_{g=1}^{G}\tilde{w}_{g}\tau_{g}-\sum_{g=1}^{G}w_{g}\rho_{g}=\sum_{g=1}^{G}w_{g}(\tau_{g}-\rho_{g})+\sum_{g=1}^{G}(\tilde{w}_{g}-w_{g})\tau_{g},$

which depends on the disparities between $w_{g}$ and $\tilde{w}_{g}$, as well as between $\tau_{g}$ and $\rho_{g}=\operatorname{exp}(\tau_{g})-1$. The difference approaches 0 if $\tau_{g}$ is close to 0 for all $g$, but can be large even if $\tilde{\tau}$ is close to 0. One example is when $(\tilde{w}_{1},\tilde{w}_{2})=(0.2,0.8)$, $(w_{1},w_{2})=(0.8,0.2)$ and $(\tau_{1},\tau_{2})=(0.08,-0.02)$, then $\tilde{\tau}=\operatorname{exp}(\tilde{\tau})-1=0$ but $\bar{\rho}=0.063$, and the difference is $6.3$ percentage points.

The ATE in log points, denoted as $\bar{\tau}$, can be estimated by explicitly accounting for treatment effects heterogeneity. This involves estimating $\tau_{g}$ and $w_{g}$ for each subgroup, then computing their weighted average. For example, in staggered difference-in-differences designs, Callaway and Sant’Anna (2021) develop estimates for aggregate treatment effects that essentially represent a set of ATEs in log points when outcomes are log-transformed. Their method employs doubly robust estimation to estimate each $\tau_{g}$ (i.e., treatment effect in log points for a specific cohort-event time combination), and then aggregates them up using respective weights. Another example is the interaction-weighted estimator developed by Gibbons et al. (2019) and applied to staggered difference-in-differences settings by Sun and Abraham (2021). With log-transformed outcomes, these methods estimate $\tau_{g}$ using OLS and then compute the weighted average using sample size based weights.

Given that $w_{g}>0$, $\bar{\tau}$ is a convex average of $\tau_{g}$. As discussed in the second property of $\tilde{\tau}$, $\operatorname{exp}(\bar{\tau})-1$ lies within the convex hull of $\rho_{g}$, but $\bar{\tau}$ itself may not. Moreover, both $\bar{\tau}$ and $\operatorname{exp}(\bar{\tau})-1$ may be poor approximations of $\bar{\rho}$. The approximation bias of $\bar{\tau}$ is $\bar{\tau}-\bar{\rho}=\sum_{g=1}^{G}w_{g}(\tau_{g}-\rho_{g})=\sum_{g=1}^{G}w_{g}(\tau_{g}-\operatorname{exp}(\tau_{g})+1)$. This bias approaches $0$ if $\tau_{g}$ is close to 0 for all $g$. However, it can be substantial if individual $\tau_{g}$ values are large, even when $\bar{\tau}=0$. For instance, with $w_{1}=w_{2}=0.5$ and $(\tau_{1},\tau_{2})=(-0.2,0.2)$, we obtain $\bar{\tau}=0$ but $\bar{\rho}=0.02$. The approximation $\operatorname{exp}(\bar{\tau})-1$ yields a smaller error, as $\bar{\rho}\geqslant\operatorname{exp}(\bar{\tau})-1\geqslant\bar{\tau}$. The first inequality follows from Jensen’s inequality: $\sum_{g=1}^{G}w_{g}\operatorname{exp}(\tau_{g})\geqslant\operatorname{exp}(\sum w_{g}\tau_{g})$, with equality holds only when treatment effects are homogeneous ($\tau_{g}=\tau$ for all $g$). The larger the heterogeneity in $\tau_{g}$, the greater the bias of $\operatorname{exp}(\bar{\tau})-1$. The second inequality stems from $\operatorname{exp}(x)-x-1\geqslant 0$ for any $x$, with equality holds only when $\bar{\tau}=0$, see footnote 2. Thus, $\operatorname{exp}(\bar{\tau})-1$ provides a better approximation for $\bar{\rho}$ as it always lies within the convex hull of $\rho_{g}$ and yields a smaller bias. However, $\operatorname{exp}(\bar{\tau})-1$ lacks a clear economic interpretation. It can be viewed as an alternative to the geometric mean of the percentage effect, as $\operatorname{exp}(\bar{\tau})=\operatorname{exp}\left(\sum_{g=1}^{G}w_{g}\ln(\rho_{g}+1)\right)$ represents the weighted geometric mean of $\rho_{g}+1$, which is the ratio of the potential outcomes $y_{i,1}/y_{i,0}$ for subgroup $g$.

While $\bar{\tau}$ or $\operatorname{exp}(\bar{\tau})-1$ improve upon $\tilde{\tau}$ or $\operatorname{exp}(\tilde{\tau})-1$, they still fail to fully capture the ATE in percentage points. These limitations underscore the importance of developing consistent and, under certain circumstances, unbiased estimation and inference methods for $\bar{\rho}$, which will be addressed in the subsequent section.

This example demonstrates how $\hat{w}$ and $\hat{\tau}$ satisfying Assumption 3 can be obtained as OLS estimators, applicable in research designs such as (conditional) randomized controlled trials.

The objective is to estimate the average treatment effect on the treated (ATT) in percentage points. The population of interest is the entire treatment group, comprising $G$ sub-treatment groups, with homogeneous treatment effects within each sub-treatment group but potential heterogeneity across groups. In this setting, $d_{i}^{(g)}=1$ if $i$ is in sub-treatment group $g$ and $0$ otherwise. The indicator of treatment is $s_{i}=\sum_{g=1}^{G}d_{i}^{(g)}=\mathbf{1}^{\prime}d_{i}$, where $\mathbf{1}_{G}$ is a $G\times 1$ vector of ones.

I obtain $\hat{\tau}$ as the OLS estimator from a semi-log regression model. Note that $s_{i}d_{i}=d_{i}$ as $d_{i}^{(g)}$ are mutually exclusive dummies that sum up to $s_{i}$. Also recall that $\tau_{i}\equiv\ln(y_{1,i})-\ln(y_{0,i})=d_{i}^{\prime}\tau$. Thus the observed log-transformed outcome is

$\displaystyle\ln(y_{i})$

$\displaystyle=\ln(y_{0,i})(1-s_{i})+\ln(y_{1,i})s_{i}=\ln(y_{0,i})+s_{i}\tau_{i}=\ln(y_{0,i})+d_{i}^{\prime}\tau.$

Let $x$ denote a $k_{x}\times 1$ vector of observed covariates including a constant term. Assuming that $E\left(\ln(y_{0,i})\left|x_{i},d_{i}\right.\right)=x_{i}^{\prime}\beta$, a semi-log linear regression model that allows for treatment effect heterogeneity is specified as

$$\ln(y_{i})=x_{i}^{\prime}\beta+d_{i}^{\prime}\tau+\epsilon_{i},$$

(3)

where $\epsilon_{i}=\ln(y_{0,i})-E\left[\ln\left(y_{0,i}\right)|x_{i},d_{i}\right]$ and by construction $E(\epsilon_{i}|x_{i},d_{i})=0$. ⁵⁵5The specification can also be justified by other assumptions and alternative estimation approaches are available, see Goldsmith-Pinkham et al. (2024) for a comprehensive discussion of the estimation of heterogeneous treatment effects in linear regressions.

Under more restrictive conditions, Assumption 4 holds.

This example considers a staggered difference-in-differences design where multiple groups start receiving treatment at different times. I focus on the case with a never-treated group and no treatment exit and use panel data for illustration.

Let $c(i)$ denote the period when individual $i$ first receives treatment, with $c(i)=\infty$ for never-treated individuals. Cohort $c$ is defined as the group of individuals with $c(i)=c$, and the event time $r_{it}=t-c(i)$ is time relative to treatment. Each $c,r$ combination is treated as a sub-group, with $\tau(c,r)$ denoting the treatment effect on the log-transformed outcome for cohort $c$ at event time $r$. Recent literature has demonstrated that if treatment effects are heterogeneous across $c$ and $r$, i.e., $\tau(c,r)\neq\tau(c^{\prime},r^{\prime})$ if $c\neq c^{\prime}$ or $r\neq r^{\prime}$, traditional two-way fixed effects (TWFE) estimators can produce biased results (Goodman-Bacon, 2021; Sun and Abraham, 2021; de Chaisemartin and D’Haultfœuille, 2020; Borusyak et al., 2024). In response, researchers have proposed heterogeneity robust estimators (de Chaisemartin and D’Haultfœuille, 2020; Sun and Abraham, 2021; Callaway and Sant’Anna, 2021; Borusyak et al., 2024, etc.). With log-transformed outcomes, these estimators typically involve estimating $\tau(c,r)$ and the corresponding weights $w(c,r)$, then computing the ATE in log points $\bar{\tau}=\sum_{c}\sum_{r}w(c,r)\tau(c,r)$. As discussed in Section 2.2, $\bar{\tau}$ and $\operatorname{exp}(\bar{\tau})-1$ differ from the ATE in percentage points, $\bar{\rho}$. However, the estimators of $w$ and $\tau$ in these studies usually satisfy Assumption 3,⁶⁶6See, e.g., Theorems 2 and 3 in Callaway and Sant’Anna (2021), Propositions 5 and 6 in Sun and Abraham (2021). and therefore form the basis of the estimation and inference methods of $\bar{\rho}$ in this paper.

Below I discuss estimators of $\tau(c,r)$ and $w(c,r)$ in Sun and Abraham (2021), which are adaptions of the OLS estimators in Example 1 to staggered difference-in-differences settings, if we view each $(c,r)$ combination as a sub-treatment group. For the estimation of $\tau(c,r)$, consider the model

$$\ln(y_{it})=\alpha_{i}+\beta_{t}+x_{it}^{\prime}\gamma+\sum_{c\neq\infty}\sum_{r\neq-1}d_{it}(c,r)\tau(c,r)+\epsilon_{it},$$

(6)

where $y_{it}$, $\alpha_{i}$, $\beta_{t}$ and $x_{it}$ are defined as in equation (2), and $d_{it}(c,r)=1$ if individual $i$ belongs to cohort $c$ and $r_{it}=r$. The never-treated group ($c=\infty$) serves as the control group, and the period immediately preceding treatment ($r=-1$) serves as the base period. Under assumptions of conditional parallel trends and no anticipation, OLS estimation of equation (6) yields an estimator of $\tau(c,r)$ that satisfies Assumption 3, see Propositions 5 and 6 in Sun and Abraham (2021).

The estimation of $w(c,r)$ depends on the parameter of interest. Define $\mathcal{A}\subset\{(c,r):c\neq\infty,r\neq-1\}$ as a subset of the sub-treatment groups ($c,r$ combinations) for which to calculate the ATT on percentage change. Following Callaway and Sant’Anna (2021), I define $\mathcal{A}=\{(c,r):c\neq\infty,r=r^{\ast}\}$ for ATT of event time $r^{\ast}$, $\mathcal{A}=\{(c,r):c=c^{\ast},r\geqslant 0\}$ for ATT of cohort $c^{\ast}$, $\mathcal{A}=\{(c,r):c+r=t,r\geqslant 0\}$ for ATT of calendar time $t$, and $\mathcal{A}=\{(c,r):c\neq\infty,r\geqslant 0\}$ for ATT of all treated units. Let $w^{\mathcal{A}}(c,r)$ be the share of sub-treatment group $(c,r)$ in $\mathcal{A}$, with $w^{\mathcal{A}}(c,r)=0$ if $(c,r)\notin\mathcal{A}$. Then $w^{\mathcal{A}}(c^{\ast},r^{\ast})=E\left[(c,r)=(c^{\ast},r^{\ast})|(c,r)\in\mathcal{A}\right]$.

Treating each $(c,r)$ combination in $\mathcal{A}$ as a sub-treatment group and estimate $w^{\mathcal{A}}(c,r)$ as in Example 1:

$$\hat{w}^{\mathcal{A}}(c,r)=\sum_{i}\sum_{t}d_{it}(c,r)/\left(\sum_{(c^{\ast},r^{\ast})\in\mathcal{A}}\sum_{i}\sum_{t}d_{it}(c,r)\right),$$

(7)

where the numerator is the sample size of cohort $c$ at event time $r$, and the denominator is the sample size of all units in set $\mathcal{A}$. Equivalently, $\hat{w}^{\mathcal{A}}(c,r)$ is the OLS estimator in the regression $d_{it}(c,r)=w^{\mathcal{A}}(c,r)s_{it}^{\mathcal{A}}+v_{it},$ where $s_{it}^{\mathcal{A}}=\sum_{(c,r)\in\mathcal{\mathcal{A}}}d_{it}(c,r)$ is the indicator that individual $i$ in period $t$ belongs to set $\mathcal{A}$. Let $\hat{w}^{\mathcal{A}}$ be the vector of $\hat{w}^{\mathcal{A}}(c,r)$ of all $(c,r)$ combinations in $\mathcal{A}$. The asymptotic covariance matrix of $\sqrt{N}\left(\hat{w}^{\mathcal{A}}-w^{\mathcal{A}}\right)$ is estimated as $\hat{\bar{\Sigma}}_{w}^{\mathcal{A}}=N/N_{\mathcal{A}}\left(\operatorname{diag}(\hat{w}^{\mathcal{A}})-\hat{w}^{\mathcal{A}}\hat{w}^{\mathcal{A}\prime}\right)$, where $N_{\mathcal{A}}$ is the total sample size of all units in $\mathcal{A}$. The estimators $\hat{w}^{\mathcal{A}}$ and $\hat{\bar{\Sigma}}_{w}^{\mathcal{A}}$ satisfy Assumption 3 by Lemma 1 and are identical to those in Sun and Abraham (2021).