Strategy to select most efficient RCT samples based on observational data

There is growing interest in combining observational and experimental data to draw causal conclusions (Hartman et al., 2015; Athey et al., 2020; Yang and Ding, 2020; Chen et al., 2021; Oberst et al., 2022; Rosenman et al., 2020). Experimental data from randomized controlled trials (RCTs) are considered the gold standard for causal inference and can provide unbiased estimates of average treatment effects. However, the scale of the experimental data is usually limited and the trial participants might not represent those in the target cohort. For example, the recruitment criteria for an RCT may prescribe that participants must be less than 65 years old and satisfy certain health criteria, whereas the target population considered for treatment may cover all age groups. This problem is known as the lack of transportability (Pearl and Bareinboim, 2011; Rudolph and van der Laan, 2017), generalizability (Cole and Stuart, 2010; Hernán and VanderWeele, 2011; Dahabreh and Hernán, 2019), representativeness (Campbell, 1957) and external validity (Rothwell, 2005; Westreich et al., 2019). By contrast, observational data usually has both the scale and the scope desired, but one can never prove that there is no hidden confounding. Any unmeasured confounding in the observational data may lead to a biased estimate of the causal effect. When it comes to estimating the causal effect in the target population, combining obervational and experimental data provides an avenue to exploit the benefits of both.

Existing literature has proposed several methods of integrating RCT and observational data to address the issue of the RCT population not being representative of the target cohort. Kallus et al. (2018), considered the case where the supports do not fully overlap, and proposed a linear correction term to approximate the difference between the causal estimates from observational data and experimental data caused by hidden confounding.

Sometimes even though the domain of observational data overlaps with the experimental data, sub-populations with certain traits may be over- or under-represented in the RCT compared to the target cohort. This difference can lead to a biased estimate of the average treatment effect, and as a result, the causal conclusion may not be generalizable to the target population. In this case, reweighting the RCT population to make it applicable to the target cohort is a common choice of remedy (Hartman et al., 2015; Andrews and Oster, 2017). In particular, Inverse Probability of Sampling Weighting (IPSW) has been a popular estimator for reweighting (Cole and Hernán, 2008; Cole and Stuart, 2010; Stuart et al., 2011). In this paper, we base our theoretical results on the IPSW estimator.

Most of the existing literature, including those discussed above, focuses on the analysis stage after RCTs are completed, and propose methods to analyse the data as given. This means, the analysis methods, including reweighting through IPSW, are to passively deal with the RCT data as they are. However, the quality of the causal inference is largely predetermined by the data collected. ‘Design trumps analysis’ (Rubin, 2008); a carefully designed experiment benefits the causal inference by far more than the analysis that follows. Instead of marginally improving through analyses, we focus on developing a strategy for the design phase, specifically the selection of RCT participants with different characteristics , to fundamentally improve the causal inference.

When designing an RCT sample to draw causal conclusions on the target cohort, a heuristic strategy that practitioners tend to opt for is to construct the RCT sample that looks exactly like a miniature version of target cohort. For example, suppose that we want to examine the efficacy of a drug on a target population consisting $30\%$ women and $70\%$ men. If the budget allows us to recruit 100 RCT participants in total, then the intuition is to recruit exactly $30$ females and $70$ males. This intuition definitely works, yet, is it efficient? We refer to the efficiency of the reweighted causal estimator for the average treatment effect in the target population, and specifically, its variance ¹¹1We note that the efficiency of an unbiased estimator $T$ is formally defined as $e(T)=\frac{\mathcal{I}^{-1}(\theta)}{var(T)}$, that is, the ratio of its lowest possible variance over its actual variance. For our purpose, we do not discuss the behaviour of the Fisher information of the data but rather focus on reducing the variance of the estimators. With slight abuse of terminology, in this paper, when we say that one estimator is more efficient than another, we mean that the variance of the former is lower. Similarly, we say that an RCT sample is more efficient if it eventually leads to an estimator of lower variance..

In fact, we find that RCTs following the exact covariate distribution of the target cohort do not necessarily lead to the most efficient estimates after reweighting. Instead, our result suggests that the optimal covariate allocation of experiment samples is the target cohort distribution adjusted by the conditional variability of potential outcomes. Intuitively, this means that an optimal strategy is to sample more from the segments where the causal effect is more volatile or uncertain, even if they do not make up a large proportion of the target cohort.

In this work, we focus on the common practice of generalizing the causal conclusions from an RCT to a target cohort. We aim at fundamentally improving the estimation efficiency by improving the selection of individuals into the trial, that is, the allocation of a certain number of places in the RCT to individuals of certain characteristics. We derive the optimal covariate allocation that minimizes the variance of the causal estimate of the target cohort. Practitioners can use this optimal allocation as a guide when they decide ‘who’ to recruit for the trial. We also formulate a deviation metric that quantifies how far a given RCT allocation is from optimal, and practitioners can use this metric to decide when they are presented with several candidate RCT allocations to choose from.

We develop variations of the main results to cater for various practical scenarios such as where the total number of participants in the trial is fixed, or the total recruitment cost is fixed while unit costs can differ, or with different precision requirements: best overall precision, equal segment precision or somewhere in between. In this paper, we provide practitioners with a clear strategy and versatile tools to select the most efficient RCT samples.

The remainder of this paper is organized at follows: In Section 2, we introduce the problem setting, notations, provide the main assumptions and provide more details on the IPSW estimator that we consider. In Section 3, we derive the optimal covariate allocation for RCT samples to improve estimation efficiency, propose a deviation metric to assess candidate experimental designs and illustrate how this metric influences estimation efficiency. Section 4 provides an estimate of the optimal covariate allocation and the corresponding assumptions to ensure consistency. Section 5 extends the main results and propose design strategies under other practical scenarios like heterogeneous unit cost and same precision requirement. In Section 6, we use two numerical studies, a synthetic simulation and a semi-synthetic simulation with real-word data, to corroborate our theoretical results.

In this paper, we based our notations and assumptions on the potential outcome framework (Rubin, 1974). We assume to have two datasets: a RCT and an observational data. We also make the assumption that the target cohort of interest is contained in the observational data.

Define $S\in\{0,1\}$ as the sample indicator where $s=1$ indicates membership of the experimental data and $s=0$ the target cohort, where $T\in\{0,1\}$ as the treatment indicator and $t=1$ indicates treatment and $t=0$ indicates control. Let $Y_{is}^{(t)}$ denotes potential outcome for a unit $i$ assigned to data set $s$ and treatment $t$. We define $X$ as a set of observable pre-treatment variables, which can consist discrete and/or continuous variables. Let $n_{0}$, $n_{1}$, $n=n_{0}+n_{1}$ denote the number of units in the target cohort, RCT, and the combined dataset, respectively. We use $f_{1}(x)$ and $f_{0}(x)$ to denote the distribution of $X$ in the RCT population and target cohort, respectively.

The ignorability condition assumes that the experimental data is unconfounded and the positivity condition is guaranteed to hold in conditionally randomized experiments.The igonrability and positivity assumptions combined is also referred to as strong ignorability. Under Assumption 1, the causal effect conditioned on $X=x$ in the experimental sample can be estimated without bias using:

where $e(x)=\mathbb{P}(T=1\mid X=x,S=1)$ is the probability of treatment assignment in the experimental sample. This estimator is also known as the Horvitz-Thompson estimator (Horvitz and Thompson, 1952), which we will provide more details later in this section.

Assumption 2 can be interpreted from several perspectives, as elaborated in Hernán and Robins (2010). First, it assumes that all the effect modifiers are captured by the set of observable covariates $X$. Second, it also ensures that the treatment $T$ for different data stays the same. If the assigned treatment differs between the study population and the target population, then the magnitude of the causal effect of treatment will differ too. Lastly, the transportability assumption prescribes that there is no interference across the two populations. That is, treating one individual in one population does not interfere with the outcome of individuals in the other population.

Furthermore, we require the trial population fully overlaps with the the target cohort, so that we can reweight the CATE in the experimental sample to estimate the ATE in the target cohort. That is, for each individual in the target cohort, we want to make sure that we can find a comparable counterpart in the experimental sample with the same characteristics.

In Assumption 3, $\text{supp}(X\,|\,S=0)$ denotes the support of the target cohort, in other words, the set of values that $X$ can take for individuals in the target cohort. Mathematically, $x\in\text{supp}(X\,|\,S=0)$ is equivalent to $\mathbb{P}\left(\|X-x\|\leq\delta\mid S=0\right)>0$, $\forall\delta>0$. Assumption 3 requires that the support of the experimental sample includes the target cohort of interest.

where the probability of treatment $e(X_{i})$ is assumed to be known as we focus on the design phase of experiments. In practice, we can extend the Horvitz-Thompson estimator by replacing $e(x)$ with an estimate $\hat{e}(x)$, for example the Hajek estimator (Hájek, 1971) and the difference-in-means estimator.

The estimator $\hat{\tau}_{\text{AIPW}}$ is doubly robust: $\hat{\tau}_{\text{AIPW}}$ is consistent if either (1) $\hat{e}\left(X_{i}\right)$ is consistent or (2) $\hat{m}^{(t)}(x)$ is consistent.

We can see that the IPSW estimator extends the Horvitz-Thompson estimator by adding a weight $w(x)$, which is the ratio between the probably of observing an individual with characteristics $X=x$ in the trial population that in the target population (Stuart et al., 2011)²²2The definition of the weight $w(x)$ differs slightly from that in Stuart et al. (2011), where $w(x)$ is defined as $\frac{P\left(S=1\mid X=x\right)}{P\left(S=0\mid X=x\right)}$. That is, the ratio of the distribution of being selected into the trail over being selected into the target cohort. This definition is based on the problem setting where there is a super population which the target cohort and trial cohort are sampled from. Our definition here agrees with that in Colnet et al. (2022).. We use an asterisk in the notation to denote that it is an oracle definition where we assume both $f_{1}(X)$ and $f_{0}(X)$ are known, which is probably unrealistic. The IPSW estimator of average treatment effect on target cohort is proven to be unbiased under Assumptions 1– 3.

A concurrent study of high relevance to our work by Colnet et al. (2022) investigated performance of IPSW estimators. In particular, they defined different versions of IPSW estimators, where $f_{1}(X)$ and $f_{0}(X)$ are either treated as known or estimated, and derived the expressions of asymptotic variance for each version. They concluded that the semi-oracle estimator, where $f_{1}(X)$ is estimated and $f_{0}(X)$ is treated as known, outperforms the other two versions giving the lowest asymptotic variance.

The re-weighted ATE estimator we use in this paper, $\hat{\tau}$, coincides with their semi-oracle IPSW estimator defined above, where $f_{1}(X)$ is estimated from the RCT data.

We first consider the more straight-forward case, where the number of possible covariate values is finite. Recall that $e(x)$ denotes the propensity score, We assume that the exact value of $e(x)$ is known for the RCT.

When units in the experimental dataset cover all the possible covariate values, for $m=1,\ldots,M$, recall the Horvitz-Thompson inverse-propensity weighted estimators (Horvitz and Thompson, 1952) of CATE:

Discrete covariates can be furthered divided into two types: ordinal, for example, test grade, and categorical such as blood type. For ordinal covariates, we can construct a smoother estimator by applying kernel-based local averaging:

where $K(\cdot)$ is kernel function and $h$ is the smoothing parameter. Conceptually, the kernel function measures how individuals with covariates in proximity to $x_{m}$ influence the estimation of $\hat{\tau}_{\text{K}}(x_{m})$. This kernel-based estimator works even if the observational data does not fully overlap with the experimental data. The estimator $\hat{\tau}_{\text{K}}$ is inspired by Abrevaya et al. (2015), who used it to estimate the CATE. Specifically, if the covariate is ordinal and the sample size of a sub-population with a certain covariate value is small or even zero, we can consider $\hat{\tau}_{\text{K}}(x)$, as it applies local averaging so that each CATE is informed by more data.

To study the variance of CATE estimates $\hat{\tau}(x_{m})$ and $\hat{\tau}_{\text{K}}(x_{m})$, we define the following terms:

The random vector $\psi(X,Y,T)$ is the influence function of the AIPW estimator (Bang and Robins, 2005). Term $\sigma_{\psi}^{2}(x)$ measures the conditional variability of the difference in potential outcomes given covariate $X=x$, and $m^{(t)}(x)$ denotes the average outcome with treatment $t$ given covariate $X=x$.

Assumption 4 suggests that when we consider the asymptotic behavior of our estimators, sample sizes for both experimental data and observational data go to infinity, though usually there is more observational samples than experimental samples.

Theorem 1 shows the asymptotic distribution of the two CATE estimators for every possible covariate value. Complete randomization in experiments ensures that $\hat{\tau}(x)$ is unbiased. Based on the idea of IPSW estimator, we then construct the following two reweighted estimators for ATE:

It is easy to see that the $\hat{\tau}$ above is the same as the semi-oracle IPSW estimator defined in Definition 2.6 once we substitute in the expression of $\hat{\tau}(x_{m})$.

Theorem 2 indicates that even if the covariate distribution of experimental data is exactly the same as that of the target cohort, it does not necessarily produce the most efficient estimator. The optimal RCT covariate distribution also depends on the conditional variability of potential outcomes. In fact, $f_{1}^{*}$ is essentially the target covariate distribution adjusted by the variability of conditional causal effects. This result suggests that we should sample relatively more individuals from sub-populations where the causal effect is more volatile, even if they do not take up a big proportion of the target cohort. Moreover, the two estimators, $\hat{\tau}$ and $\hat{\tau}_{\text{K}}$, share the same optimal covariate weight no matter whether local averaging is applied.

In practice, if the total number of samples is fixed, experiment designers can select RCT samples with covariate allocation identical to $f_{1}^{*}$ to improve the efficiency of IPSW estimate.

Accoring to Corollary 1, the variance of $\hat{\tau}$ depends on two parts: the first part $\sum_{m=1}^{M}f_{0}(x_{m})\sigma_{\psi}(x_{m})$ depends on the true distribution of the target population, while the second part $\mathcal{D}(f_{1})$ is a measure of the deviation of the RCT sample allocation $f_{1}$ compared to the optimal variability-adjusted allocation $f_{1}^{*}$, and can thus reflect the representativeness of our RCT samples. As Corollary 1 shows, the variance of IPSW estimator for the population, $\hat{\tau}$, is proportional to $\mathcal{D}(f_{1})$.

The deviation metric equips us with a method to compare candidate experiment designs. To be specific, if experiment designers have several potential plans for RCT samples, they can choose one with the smallest deviation metric to maximize the estimation efficiency.

For continuous covariates, for instance, body mass index (BMI), we apply stratification based on propensity score. By considering an appropriate partition of the support $\{A_{1},\ldots,A_{L}\}$ with finite $L\in\mathbb{N}$, we can turn it into the discrete case above.

Assumption 5 assumes that units within each stratum share the same propensity score and CATE. This is a strong but reasonable condition if we make each stratum $A_{l}$ sufficiently small. Under Assumption 5, let $\hat{\tau}(A_{l})$, $\hat{\tau}_{\text{K}}(A_{l})$, $\sigma_{\psi}^{2}(A_{l})$, $e(A_{l})$ denote the causal effect estimate, causal effect estimate with kernel-based local averaging, variance of influence function, propensity score, that are conditioned on $X\in A_{l}$. Let $f_{0}(A_{l})=\mathbb{P}(X\in A_{l}\mid S=0)$ and $f_{1}(A_{l})=\mathbb{P}(X\in A_{l}\mid S=1)$. We can then construct two IPSW estimators:

As shown in Corollary 2, we have similar results to Theorem 2, but instead of the optimal covariate distribution, we derive the optimal probability on each covariate set $A_{l}$.

In the sections that follow, for simplicity, we illustrate our method in the scenario where the covariates are all discrete with finite possible values. The results can easily be extended to include continuous covariates following the same logic as descibed in this section.

We also consider the experimental design with a cost constraint and heterogeneous cost for different sub-populations. The goal is to find the optimal sample allocation for RCT that minimizes the variance of the proposed estimator subject to a cost constraint. For $m=1,\ldots,M$, let $C_{m}$ denote the cost to collect a sample in the sub-population with $X=x_{m}$.

Theorem 4 suggests that the optimal RCT covariate allocation under the given cost constraint is the covariate allocation of target cohort adjusted by both the heterogeneous costs for sub-populations and the conditional variability of potential outcomes. Intuitively, compared to the case without heterogeneous costs, we should include more RCT samples with lower unit cost.

Theorem 2 shows the optimal sample allocation to maximize the efficiency of average treatment effect estimator for the target cohort. If we require the same precision for estimators in each domain, we need the sample allocation as follows:

where we use the superscript s to denote the requirement of same precision for the CATE estimate in each segment.

Intuitively, to take both objectives into consideration, we propose a compromised allocation that falls between the two optimum allocations $\forall k\in[0,1]$:

In this section, we conduct a simulation study to illustrate how representativeness of experiment samples influences the estimation efficiency of average treatment effect for a target cohort, and demonstrate how the representativeness metric $\mathcal{D}(f_{1})$ can facilitate the selection from candidate RCT sample designs. We set the size of observational dataset $n_{0}=10000$ and the size of experimental dataset $n_{1}=200$. For the units in the observational data, we draw covariates $x$ from $\{1,2,3\}$ with probability 0.3, 0.2 and 0.5 respectively. We then set

where $\epsilon\sim\mathcal{N}(0,1)$. We can then compute the conditional variability of potential outcomes $\sigma^{2}_{\psi}(x)$ and thus the optimal covariate distribution $f^{*}_{1}$ from the true population. Our model engenders distinctive conditional variability $\sigma^{2}_{\psi}(x)$ given different $x$, making the optimal covariate distribution $f_{1}^{*}$ very different from the target covariate distribution $f_{0}$.

For experimental data, we simulate 100 different candidate experimental sample designs. In each design, we randomly draw experiment samples from the target cohort with probability

where $p_{1},p_{2},p_{3}$ are i.i.d. samples drawn from standard normal distribution. We can then compute the real covariate distribution $f_{1}(x)$ and the repressiveness metric $\mathcal{D}(f_{1})$. To estimate the efficiency of average treatment effect estimator, we conduct 1000 experiments for each design. In each experiment, the treatment for each unit follows a Bernoulli distribution with probability 0.5. The simulation result is shown in Fig 1. The relationship between the variance and representativeness can be fit into a line, which is consistent with our result that $\text{var}(\hat{\tau})\propto\mathcal{D}(f_{1})$. The red line shows the value $R(f_{1})$ for the naive strategy mimicking exactly the target cohort distribution, which is not zero, and we can see that it is not the optimal RCT sample and does not produce the most efficient causal estimator.

For the case with heterogeneous unit cost, we set the costs for sub-populations with covariate $X$ being 1, 2, 3 to be 20, 30, 40, respectively. The total capital available is 30000. Instead of randomly drawing experiment samples from the target cohort with a fixed number of total subjects, we randomly assign capitals for different sub-populations with a fixed amount of total cost. Given the budget assigned to each sub-population, we then draw subjects randomly from the sub-population, where the amount of subjects can be determined by the assigned budget. The simulation result is illustrated by Figure 2. We can see that under the cost constraint, experiment samples that follow a distribution closer to $f_{1}^{c,*}$ lead to causal estimator with better efficiency.

We use the well-cited Tennessee Student/Teacher Achievement Ratio (STAR) experiment to assess how covariate distribution of experiment samples influences the estimation efficiency of average treatment effect for target cohort. STAR is a randomized experiment started in 1985 to measure the effect of class size on student outcomes, measured by standardized test scores. Similar to the exercise in Kallus et al. (2018), we focus a binary treatment: $T=1$ for small classes (13-17 pupils), and $T=0$ for regular classes (22-25 pupils). Since many students only started the study at first grade, we took as treatment their class-type at first grade. The outcome $Y$ is the sum of the listening, reading, and math standardized test at the end of first grade. We use the following covariates $X$ for each student: student race ($X_{1}\in\{1,2\}$) and school urbanicity ($X_{2}\in\{1,2,3,4\}$). We exclude units with missing covariates. The records of 4584 students remain, with 1733 assigned to treatment (small class, $T=1$), and 2413 to control (regular size class, $T=0$). Before analysis we fill the missing outcome by linear regression based on treatments and two covariates so that both two potential outcomes $Y_{0}$ and $Y_{1}$ for each student are known.

We simulate 500 candidate experiment sample allocations. For each allocations, we select $n_{1}=500$ experiment units from the dataset with probability

where $p_{11},p_{12},p_{13},p_{14},p_{21},p_{22},p_{23},p_{24}$ are i.i.d. samples drawn from standard normal distribution. We can then compute the real covariate distribution $f_{1}(x)$ and the repressiveness of the experiment samples $\mathcal{D}(f_{1})=\text{var}_{1}(f_{1}^{*}(X)/f_{1}(X))$. To estimate the efficiency of average treatment effect estimator, we conduct 200 experiments for each design. In each experiment, the treatment follows a Bernoulli distribution with probability $1733/4584=0.378$. The simulation result is shown in Fig 3. The relationship between the variance and the deviation metric $\mathcal{D}(f)$ can be fit into a line, which is consistent with our result that $\text{var}(\hat{\tau})\propto\mathcal{D}(f_{1})$.