Optimizing Multi-Scale Representations to Detect Effect Heterogeneity Using Earth Observation and Computer Vision: Applications to Two Anti-Poverty RCTs

Earth Observation (EO) data play an increasingly important role in policy analysis by providing researchers with contextual information to estimate treatment effects at a more granular level, revealing characteristics of environmental conditions, land use patterns, economic development, and climate variables (Anderson et al., 2017; Daoud et al., 2023; Burke et al., 2021b; Pettersson et al., 2023; Kino et al., 2021). A growing body of work therefore focuses on estimating household or neighborhood-specific Conditional Average Treatment Effects (CATE) (Sakamoto et al., 2024; Jerzak et al., 2023; Serdavaa, 2023; Giannarakis et al., 2023; Go et al., 2022; Daoud and Johansson, 2024; Shiba et al., 2022).

An important application of EO-based policy analysis lies in anti-poverty Randomized Controlled Trials (RCTs), where the outcome $\mathbf{Y}$ is the household income for the unit that the satellite imagery is centered on, and the treatment $\mathbf{W}$ is cash transfer to the household. The researcher’s goal is to estimate effects conditioned on available satellite imagery $\mathbf{M}$. We denote this estimate for a given image, $\hat{\tau}(\mathbf{m})$. In order to do so, we encode $\mathbf{M}$ into a sufficiently low-dimensional representation, $\phi$, that preserves maximal information about $\mathbf{Z}$, the set of unobserved background features that are relevant to treatment (e.g., the village geography/baseline wealth). See the white nodes of Figure 2 for illustration.

A key characteristic of EO-based images in the context of effect heterogeneity estimation is the inherent multi-scale nature of the relevant background features $\mathbf{Z}$ for observational units, a phenomenon termed “multi-scale dynamics” (Xiong et al., 2022; Reed et al., 2023). Figure 1 provides some intuition for such multi-scale dynamics in EO-based inference, where information relevant to heterogeneity is encoded in both the local area around a household and the broader neighborhood around which the household is situated.

Along with Figure 1, we provide a visualization in Figure 2 that provides a more formal illustration of our setting. Parts highlighted in blue refer to multi-scaled components of the DAG that were previously not considered in effect heterogeneity estimation methodologies.

Our contribution is to propose Multi-Scale Representation Concatenation, a procedure that transforms single-scale EO-based CATE estimation algorithms into multi-scale ones by concatenating the image representation tensors. We benchmark using a single-scale CATE estimation algorithm previously used in anti-poverty RCTs (Jerzak et al., 2023). A remote-sensing fine-tuned image model, CLIP-RSICD, is used to generate lower-dimensional representations of single-scaled satellite imagery (Lu et al., 2017; Arutiunian et al., 2021; Radford et al., 2021). A causal forest (CF) model then produces the final CATE estimates.¹¹1We also report results for SWIN and Clay model architectures, but use CLIP results as baseline due to the possibility of text-based interpretability analyses and the wide use of this model class.

Beyond building on top of previous work in Causal Inference, there is a rich body of work investigating multi-scale phenomena in Computer Vision which we have drawn inspiration from to develop Multi-Scale Representation Concatenation. We survey them in §8.4.

To quantify performance in real randomized controlled trials (RCTs) lacking ground truth CATE data, we use a measure of the degree of effect heterogeneity detected by a model, Rank Average Treatment Effect Ratio (RATE Ratio), as our metric (Yadlowsky et al., 2021). We find evidence that applying Multi-Scale Representation Concatenation improves the RATE Ratio, and, thus, potentially improves accuracy of CATE estimates and policy targeting.

As extensions, simulation studies are used to validate results from the RCT analysis due to the difficulty in evaluating model performance on RCT data where ground-truth CATE is unknown (§4); a scaling analysis demonstrates that Multi-Scale Representation Concatenation can significantly boost the performance of CATE estimation models in EO-based causal inference beyond just two scales (§5.3); and an analysis where we assume the CATE analyst has limited prior information on the geographic distribution of household units finds that multi-scale inference improves the RATE metric under this weaker assumption (§7). These suggest promising further areas of exploration.

An enduring question in EO-based causal inference is how to best estimate $\tau(\mathbf{m})\coloneqq\mathbb{E}[Y_{i}(1)-Y_{i}(0)\mid\mathbf{M}_{i}=\mathbf{m}]$, i.e., the CATE for pre-treatment image array, $\mathbf{M}_{i}$ (Athey et al., 2018; Jerzak et al., 2023). In theory, one would provide all available covariates to an oracle function generating estimated $\tau(\mathbf{m})$’s to obtain the most accurate causal estimates. For EO-based causal inference, this would ideally involve using the largest, highest-resolution satellite image available. However, in practice, this function is difficult to estimate due to the high dimensionality of $\mathbf{M}_{i}$.

In this context, because there are well-understood methods to estimate CATEs from $\mathbf{W}$, $\mathbf{Y}$, and pre-treatment covariate vectors Chernozhukov et al. (2024), a natural approach in image-based CATE estimation would be to introduce an encoder $\phi$ that maps high-dimensional imagery into a lower-dimensional representation. However, in an EO-based causal inference context, a pre-trained image encoder may struggle to capture both household-specific and neighborhood-level contextual information. In some cases, adding additional contextual information can even degrade performance if the model fails to distinguish relevant from non-relevant signals (D’Amario et al., 2022). Model fine-tuning could address this issue, yet it remains difficult due to the small size of most RCTs.

Our main contribution therefore is to develop Multi-Scale Representation Concatenation, a family of procedures that transform any previous single-scale procedure into a multi-scale one through tensor concatenation. It is especially suitable in data-constrained settings (e.g. RCTs), as it requires minimal additional data requirements. The strong interpretability of the procedure also makes it simple to explain to relevant stakeholders. Further, it requires limited model experimentation: The optimal concatenation procedure can be found by systematically varying the image size and combining image representations from different scales. Multi-Scale Representation Concatenation thus transforms the problem of designing architectures specific to the multi-scale dynamics of the problem at hand to a less complex inference-time computational search (Snell et al., 2024). We examine the improvements in model performance when incorporating Multi-Scale Representation Concatenation in the context of randomized controlled trials (RCTs) conducted in Peru and Uganda.

Let $i$ index the experimental units in the study. Each $i$ has a geolocation denoted by $\mathbf{x}_{i}\in\mathbb{R}^{2}$, representing spatial coordinates (e.g., latitude/longitude), and a binary treatment indicator $W_{i}$ encoding whether the unit received treatment. The observed outcome from the RCT for unit $i$ is $Y_{i}\in\mathbb{R}$. Let $Y_{i}(1)$ and $Y_{i}(0)$ denote potential outcomes under treatment and control. Identification will be performed assuming unconfoundedness and SUTVA (Imbens and Rubin, 2016; Chernozhukov et al., 2024). Now define an image fetcher that generates an image from a coordinate ${\mathbf{x}_{i}}\in\mathbb{R}^{2}$ with a specified size $s\in\mathbb{N}_{+}$: $f_{I}:\mathbb{R}^{2}\times\mathbb{N}_{+}\to\mathcal{M}$, where $\mathcal{M}$ is the space of possible imagery:

where $\mathbf{M}_{i,s}$ is the image of size $s>0$ centered at $\mathbf{x}_{i}$. Here, size refers to the width of the raw satellite imagery in terms of pixel number, with pixel resolution fixed.

Next, because of the high-dimensionality of $\mathbf{M}_{i,s}$, we need to introduce a causal representation extraction function $\phi:\mathcal{M}^{c}\to\mathbb{R}^{d}$, which takes $c\in\mathbb{N}_{+}$ images as input and outputs a $d$-dimensional feature vector. The projection of the image to a lower dimensionality makes estimation tractable in finite samples. We develop a Multi-Scale Representation Concatenation procedure that constructs $\phi$ from base image encoders, $f_{\phi_{s_{k}}^{\prime}}:\mathcal{M}\to\mathbb{R}^{d^{\prime}}$, to easily adapt single-scale representation extraction functions into a multi-scale setting.

Although our algorithm is generalizable to arbitrary $c$, for concreteness, our exposition will be limited to the case where $c=2$, with $f_{\phi}:\mathcal{M}\times\mathcal{M}\to\mathbb{R}^{2d^{\prime}}$:

where $s_{1}$ and $s_{2}$ are two different image sizes. In this paper, $f_{\phi}$ is constructed through representation concatenation of the outputs of base image encoders $f_{\phi_{s_{k}}}^{\prime}$ along with (optionally) a dimensionality reduction function $r$, so that $f_{\phi}\left(\mathbf{M}_{i,s_{1}},\,\mathbf{M}_{i,s_{2}}\right)=r((f_{\phi_{s_{1}}}^{\prime}(\mathbf{M}_{i,s_{1}}),f_{\phi_{s_{2}}}^{\prime}(\mathbf{M}_{i,s_{2}})))$. Here, $r:\mathbb{R}^{2d^{\prime}}\to\mathbb{R}^{l}$, $l<2d^{\prime}$.

To estimate CATEs, we need to estimate both the representation extraction function $\phi$ and the estimation function $h_{\theta}$. Our goal is to learn a function $f_{\phi}$ that extracts the most causally relevant covariates from the image at multiple levels of representation (e.g., household and village). Having found $f_{\phi}$, we then draw upon the CF approach, a well-established method for estimating $h_{\theta}$ under unconfoundedness (Athey and Wager, 2019).

With this algorithm for estimating CATEs, we then seek to maximize the heterogeneity signal of different multi-scale representations using a metric, $\mu(\cdot)$, designed to quantify the extent of effect heterogeneity detected given input features. In our case, $\mu(\cdot)$ will denote the RATE Ratio as a principled metric that allows one to evaluate model performance without ground truth individual treatment effects, overcoming the unobservability of true CATE values (Holland, 1986; Yadlowsky et al., 2021). For a set of image conditioning variables, $\mathbf{M}_{i}$, the RATE is:

A more detailed exposition is in §8.3. The RATE Ratio (the measure $\mu$) is defined as $\frac{\text{RATE}}{\text{sd(RATE)}}$. Motivation for using the RATE Ratio lies in the fact that (1) it is interpretable and comparable across contexts as a scale-free quantification of heterogeneity signal and (2) it can be used as an asymptotic $t$-statistic of the existence of treatment heterogeneity in the population given the conditioning data (here, pre-treatment satellite image arrays). The detailed algorithm for RATE Ratio estimation is presented in §8.3.

The RATE Ratio is chosen as the primary metric for model evaluation due to its comparability across RCTs and outcomes. However, RATE Ratio is not as interpretable as the more policy-oriented metrics of RATE and Qini Curves (Yadlowsky et al., 2021; Sverdrup et al., 2024). RATE alone is a summary measure of heterogeneity in the data captured by the model and downstream policy relevance of the current heterogeneity model, and Qini Curves plot the gain over randomization of assigning treatment according to CATE estimates from a model. We validate the RATE Ratio metric with the values produced by these two alternatives. We find a high correlation between the three metrics, indicating that in our context RATE Ratio also quantifies the policy benefits of a specific CATE estimation model. Due to the high correlation, we leave RATE and Qini figures for the Appendix (Figure 12, Figure 13, Figure 14). For a discussion of Qini Curves, see §8.5.

While the RATE Ratio is a useful theoretical metric, contextual-scale evaluation in an applied context is difficult due to the lack of ground-truth CATE data. In developing our methods, we thus use a simulation to supplement findings from the later RCT analysis. Our simulation employs images of size 32$\times$32 and 256$\times$256 pixels drawn from the Peru RCT. We design three image perturbations corresponding to three scales of causal features (see Figure 3). We perturb images’ household-level features by masking the center of the image, neighborhood-level features by adding an image fading to the edge of the larger scale image, and global context features by applying image contrast. For each of our experiments, we choose a subset of these perturbations, each applied independently to half of the images. The different experiments help us discern how our model will perform when differently scaled covariates are present. Synthetic outcomes are then constructed by adding a deterministic signal with Gaussian noise (see Figure 3).

In our simulation, we design a setup that mimics an RCT. Each unit has an associated image, and we apply various perturbations—such as masking or edge fading—to these images. The specific type of perturbation applied to a unit’s image determines its outcome (and associated CATE). We systematically test the ability of different modeling strategies to detect and leverage these image-based signals in estimation. We measure how well we identify features driving the outcome of interest with the 5-fold cross-validated out-of-sample $R^{2}$.

With the outcome and image data defined, we then train a Multi-Layer Perceptron (MLP) on top of representations generated by the CLIP model to predict the outcome of the units. We defer further details to the Appendix (§8.6). For a single-scale approach, the input to the MLP is the representation generated by the CLIP image encoder of an image of a fixed size. We then apply Multi-Scale Representation Concatenation with no dimensionality reduction to generate the multi-scale analog.

Three sets of experiments are performed on the perturbed dataset to explore performance of a multi-scale modeling approach when (a) only household or neighborhood level information is present in the dataset, (b) when both levels of information are present in the dataset, and (c) when the global feature is present in the dataset. We found that our simple concatenation-based multi-scale modeling can recover most of the information present on one level of resolution and captures signal simultaneously from both scales if present. Perhaps more surprisingly, multi-scale modeling also better captures global features (see Figure 4).

While the investigations present here isolate the influence of specific local, intermediate, and global perturbations, we investigate all possible combinations in the Appendix (Table 2), where an additional global perturbation of 90-degree image rotation is included. The results accord with those presented in Figure 4.

While the simulation results are suggestive of potential benefits of multi-scale analysis, we now turn to quantify its benefits in the context of real RCTs. Our analysis here draws on unique experimental datasets from diverse country contexts: Peru and Uganda. For both, there is evidence that SUTVA is satisfied as spillover effects are determined to be unlikely and treatment implementation is standardized. These datasets therefore provide a solid foundation for exploring the impacts of scale across different societal settings.

Larger Images Are Not Always Better. A counterintuitive finding from our analyses is that using the largest possible satellite image context (349 pixels) never maximizes the heterogeneity signal in any of our experimental conditions. This challenges the intuitive assumption that incorporating more contextual information necessarily improves causal inference. We find this “bigger is not better” principle manifests in three ways.

First, in single-scale analyses with unperturbed geolocations, smaller images often generate elevated heterogeneity signals, likely because they better capture household-specific characteristics without dilution from broader contexts. Second, even in multi-scale analyses, the optimal combinations typically pair a small or medium-sized image (around 64 pixels) with a large but not maximum-sized image, rather than using the largest available scale. Third, multi-scale analysis can provide for robust inferences even when only approximate location information is available (as evidenced by the displaced locations analysis).

These findings have implications for EO-based causal inference. Rather than defaulting to the largest possible image context, researchers should carefully consider the trade-off between capturing relevant local heterogeneity and broader contextual information, either employing Multi-Scale Representation Concatenation or incorporating multi-scale dynamics into the inferential procedure in other ways to be robust to the intricacies of EO data.

Towards Adaptive Multi-Scale Representation Concatenation. Despite grid search’s promising performance, its computational requirement is exponential in $C$, the number of scales considered for combined analysis. Further, the current Multi-Scale Representation Concatenation technique limits itself to a fixed image-scale combination for each observational unit, whilst in practice the optimal scale for each observational unit may differ.

Our finding that Multi-Scale Representation Concatenation outperforms single-scale analysis suggests that there may be a way to optimize Multi-Scale Representation Concatenation further by estimating an additional function $l:\mathcal{M}\to\mathbb{R}^{2}$ that takes in a large image over an entire region around an experimental unit and outputs locations $\mathbf{x_{i}}$ where an image at each scale is centered. This would involve evaluating each geographic location for suitability of analysis, where dense prediction techniques could prove helpful (Zuo et al., 2022). We present the estimation of $l$ as an open problem.

Multi-scale dynamics beyond EO. Multi-scale dynamics are present in diverse geospatial, biological, and public health contexts. Beyond CATE estimation, these dynamics are relevant for environmental cost-benefit analysis (Druckenmiller et al., 2024), EO-based poverty imputation (Burke et al., 2021a), and confounder control—where the latent geospatial confounding structure may operate with multi-scale dynamics (Jerzak et al., 2022). We invite further work to incorporate multi-scale dynamics in these diverse settings.

This paper addressed the methodological challenge of capturing multi-scale dynamics in EO-based causal inference. By combining representations across scales, our approach captures both fine-grained individual-level details and broader contextual information, enhancing the estimation of CATEs. Simulation studies and analysis of two RCTs demonstrate the promise of multi-scale inference in outperforming single-scale-only methods when effect heterogeneity information exists at multiple levels. This offers a promising solution to finding a trade-off between individual heterogeneity and neighborhood-level context, and contributes to the growing literature on deep learning for causal inference (Daoud and Dubhashi, 2023).

Limitations remain. First, the approach here assumes SUTVA and unconfoundedness for identification, and thus, an extension to observational inference requires additional assumptions. Second, images used in our study have low resolution, and resolution likely interacts with the gain from doing multi-scale analysis. Third, using high-resolution images in a multi-scale approach raises privacy concerns. Fourth, the approach taken to assess the validity of Multi-Scale Representation Concatenation is empirical; theoretical results would strengthen our understanding of the method. Fifth, the approach is currently only validated on anti-poverty RCT datasets. Further research could be done to address these limitations of the general Multi-Scale Representation Concatenation approach. $\square$

We thank members of the AI & Global Development Lab for helpful feedback, resources, and inspiration. We thank Dean Karlan and Andre Nickow for assistance in accessing data for the Peru experiment. We thank Hannah Druckenmiller, Antonio Linero, and SayedMorteza Malaekeh for helpful discussions. All remaining limitations are our own.