Counterfactual Generation Under
Confounding

A confounder is a variable that causally influences two or more variables that are not necessarily directly causally dependent (Pearl, 2001). Often, the presence of confounders in a data-generating process is the reason for spurious correlations among variables in the observational data. The bias caused by such confounders is inevitable in observational data, making it challenging to identify invariant features representative of a target variable (Rothenhäusler et al., 2021; Meinshausen & Bühlmann, 2015; Wang et al., 2022). For example, the demographic area an individual resides in often confounds the race and perhaps the level of education that individual receives. Using such observational data, if the goal is to predict an individual’s salary, a machine learning model may exploit the spurious correlation between education and race even though those two variables should ideally be treated as independent variables. Removing the effects of confounding in trained machine learning models has shown to be helpful in various applications such as zero or few-shot learning, disentanglement, domain generalization, counterfactual generation, algorithmic fairness, healthcare, etc. (Suter et al., 2019; Kilbertus et al., 2020; Atzmon et al., 2020; Zhao et al., 2020; Yue et al., 2021; Sauer & Geiger, 2021; Goel et al., 2021; Dash et al., 2022; Reddy et al., 2022; Dinga et al., 2020).

In observational data, confounding may be observed or unobserved and can pose various challenges in learning models depending on the task. For example, disentangling spuriously correlated features using generative modeling when there are confounders is challenging (Sauer & Geiger, 2021; Reddy et al., 2022; Funke et al., 2022). As stated earlier, a classifier may rely on non-causal features to make predictions in the presence of confounders (Schölkopf et al., 2021). Recent years have seen a few efforts to handle the spurious correlations caused by confounding effects in observational data (Träuble et al., 2021; Sauer & Geiger, 2021; Goel et al., 2021; Reddy et al., 2022). However, these methods either make strong assumptions on the underlying causal generative process or require strong supervision. In this paper, we study the adversarial effect of confounding in observational data on a classifier’s performance and propose a mechanism to marginalize such effects when performing data augmentation using counterfactual data. Counterfactual data generation provides a mechanism to address such issues arising from confounding and building robust learning models without the additional task of building complex generative models.

The causal generative processes considered throughout this paper are shown in Figure 1(a). We assume that a set of generative factors (attributes) $Z_{1},Z_{2},\dots,Z_{n}$ (e.g., background, shape, texture) and a label $Y$ (e.g., cow) cause a real-world observation $X$ (e.g., an image of a cow in a particular background) through an unknown causal mechanism $g$ (Peters et al., 2017b). To study the effects of confounding, we consider $Y,Z_{1},Z_{2},\dots,Z_{n}$ to be confounded by a set of confounding variables $C_{1},\dots,C_{m}$ (e.g., certain breeds of cows appear only in certain shapes or colors and appear only in certain countries). Such causal generative processes have been considered earlier for other kinds of tasks such as disentanglement Suter et al. (2019); Von Kügelgen et al. (2021); Reddy et al. (2022). The presence of confounding variables results in spurious correlations among generative factors in the observed data, whose effect we aim to remove using counterfactual data augmentation.

A related recent effort by (Sauer & Geiger, 2021) proposes Counterfactual Generative Networks (CGN) to address this problem using a data augmentation approach. This work assumes each image to be composed of three Independent Causal Mechanisms (ICMs) (Peters et al., 2017a) responsible for three fixed factors of variations: shape, texture, and background (as represented by $Z_{1},Z_{2}$, and $Z_{3}$ in Figure 1(b). This work then trains a generative model that learns three ICMs for shape, texture, and background separately, and combines them in a deterministic fashion to generate observations. Once the ICMs are learned, sampling images by making interventions to these mechanisms give counterfactual data that can be used along with training data to improve classification results. However, fixing the architecture to specific number and types of mechanisms (shape, texture, background) is not generalizable, and may not directly be applicable to settings where the number of underlying generative factors is unknown. It is also computationally expensive to train different generative models for each aspect of an image such as texture, shape or background.

In this work, we begin with quantifying confounding in observational data that is generated by an underlying causal graph (more general than considered by CGN) of the form shown in Figure 1(a). We then provide a counterfactual data augmentation methodology called CONIC (COunterfactual geNeratIon under Confounding). We hypothesize that the counterfactual images generated using the proposed CONIC method provide a mechanism to marginalize the causal mechanisms responsible for spurious correlations (i.e., causal arrows from $C_{i}$ to $Z_{j}$ for some $i,j$). We take a generative modeling approach and propose a neural network architecture based on conditional CycleGAN (Zhu et al., 2017) to generate counterfactual images. The proposed architecture improves CycleGAN’s ability to generate quality counterfactual images under confounded data by adding additional contrastive losses to distinguish between fixed and modified features, while learning the cross domain translations. To demonstrate the usefulness of such counterfactual images, we consider classification as a downstream task and study the performance of various models on unconfounded test set. Our key contributions include:

• •

  We formally quantify confounding in causal generative processes of the form in Fig  1(a), and study the relationship between correlation and confounding between any pair of generative factors.

• •

  We present a counterfactual data augmentation methodology to generate counterfactual instances of observed data, that can work even under highly confounded data ($\sim 95\%$ confounding) and provides a mechanism to marginalize the causal mechanisms responsible for confounding.

• •

  We modify conditional CycleGAN to improve the quality of generated counterfactuals. Our method is computationally efficient and easy to implement.

• •

  Following previous work, we perform extensive experiments on well-known benchmarks – three MNIST variants and CelebA datasets – to showcase the usefulness of our proposed methodology in improving the accuracy of a downstream classifier.

Counterfactual Inference: (Pearl, 2009), in his seminal text on causality, provided a three-step procedure for generation of a counterfactual data instance, given an observed instance: (i) Abduction: abduct/recover the values of exogenous noise variables; (ii) Action: perform the required intervention; and (iii) Prediction: generate the counterfactual instance. One however needs access to the underlying structural causal model (SCM) to perform the above steps for counterfactual generation. Since real-world data do not come with an underlying SCM, many recent efforts have focused on modeling the underlying causal mechanisms generating data under various assumptions. These methods then perform the required intervention on specific variables in the learned model to generate counterfactual instances that can be used for various downstream tasks such as classification, fairness, explanations etc. (Kusner et al., 2017; Joo & Kärkkäinen, 2020; Denton et al., 2019; Zmigrod et al., 2019; Pitis et al., 2020; Yoon et al., 2018; Bica et al., 2020; Pawlowski et al., 2020).

Generating Counterfactuals by Learning ICMs: In a more recent effort, assuming any real-world image is generated with three independent causal mechanisms for shape, texture, background, and a composition mechanism of the first three, (Sauer & Geiger, 2021) developed Counterfactual Generative Networks (CGN) that generate counterfactual images of a given image. CGN trains three Generative Adversarial Networks (GANs) (Goodfellow et al., 2014b) to learn shape, texture, background mechanisms and combine these three mechanisms using a composition mechanism $g$ as $g(shape,texture,background)=shape\odot texture+(1-shape)\odot background$ where $\odot$ is the Hadamard product. Each of these independent mechanisms is given an input of noise vector $u$ and a label $y$ specific to that independent mechanism while training. Once the independent mechanisms are trained, counterfactual images are generated by sampling a label and a noise vector corresponding to each mechanism and then feeding the input to CGN. Finally, a classifier is trained with both original and counterfactual images to achieve better test time accuracy, showing the usefulness of CGN. However, such deterministic nature of the architecture is not generalizable to the case where the number of underlying generative factors are unknown and it is computationally infeasible to train generative models for specific aspect of an image such as texture/background.

Disentanglement and Data Augmentation: The spurious correlations among generative factors have been considered in disentanglement (Funke et al., 2022; von Kügelgen et al., 2021). The general idea in these efforts is to separate the causal predictive features from non-causal/spurious predictive features to predict an outcome. Our goal is different from disentanglement, and we focus on the performance of a downstream classifier instead of separating the sources of generative factors. Traditional data augmentation methods such as rotation, scaling, corruption, etc. (Hendrycks et al., 2020; Devries & Taylor, 2017; Zhang et al., 2018; Yun et al., 2019) do not consider the causal generative process and hence they can not remove the confounding in the images via data augmentation (e.g., color and shape of an object can not be separated using simple augmentations). We hence focus on counterfactual data augmentations that is focused on marginalizing the confounding effect caused by confounders.

A similar effort to our paper is by (Goel et al., 2021) who use CycleGAN to generate counterfactual data points. However, they focus on the performance of a subgroup (a subset of data with specific properties) which is different from our goal of controlling confounding in the entire dataset. Another recent work by (Wang et al., 2022) considers spurious correlations among generative factors and uses CycleGAN to generate counterfactual images. Compared to these efforts, rather than using CycleGAN directly, we propose a CycleGAN-based architecture that is optimized for controlled generation using contrastive losses.

Applications of Counterfactuals: Augmenting the training data with appropriate counterfactual data has shown to be helpful in many applications ranging from vision to natural language tasks (Joo & Kärkkäinen, 2020; Lample et al., 2017; Kusner et al., 2017; Kaushik et al., 2019; Dash et al., 2022).  (Joo & Kärkkäinen, 2020) identified existing biases in computer vision APIs deployed in the real world by Amazon, Google, IBM, and Clarifai by looking at the differences made by those APIs on counterfactual images that differ by protected/sensitive attributes (e.g., race and gender). Using locally independent causal mechanisms, (Pitis et al., 2020) augmented training data with counterfactual data points in a model-free reinforcement learning setting. Here, the idea is to use any two factual trajectories of an episode and combine the two trajectories at a particular point in time to generate the counterfactual data point, which will then be added to the replay buffer. Independently factored samples are essential to get plausible and realistic counterfactual instances.

Background and Problem Formulation: Let $\{Z_{1},Z_{2},\dots,Z_{n}\}$ be a set of random variables denoting the generative factors of an observed data point $X$, and $Y$ be the label of the observation $X$. Each generative factor $Z_{i}$ (e.g., color) can take on a value form a discrete set of values $\{z_{i}^{1},\dots,z_{i}^{d}\}$ (e.g., red, green etc.). Let the set $S=\{Y,Z_{1},\dots,Z_{n}\}$ generates $N$ real-world observations $\{X_{i}\}_{i=1}^{N}$ through an unknown causal mechanism $g$. Each $X_{i}$ can be thought of as an observation generated using the causal mechanism $g$ with certain intervention on the variables in the set $S$. Variables in $S$ may potentially be confounded by a set of confounders $C=\{C_{1},\dots,C_{m}\}$ that denote real-world confounding such as selection bias. Let $\mathcal{D}$ be the dataset of real-world observations along with corresponding values taken by $\{Y,Z_{1},\dots,Z_{n}\}$. Causal graph in Figure 1(a) shows the general form of this setting. From a causal effect perspective, each variable in $S$ has a direct causal influence on the observation $X$ (e.g., the causal edge $Z_{i}\rightarrow X$) and also has non-causal influence on $X$ via the confounding variables $C_{1},\dots,C_{m}$ (e.g., $Z_{i}\leftarrow C_{j}\rightarrow Z_{k}\rightarrow X$ for some $C_{j}$ and $Z_{k}$). These paths via the confounding variables, in which there is an incoming arrow to the variables in $S$, are also referred to as backdoor paths (Pearl, 2001). Due to the presence of backdoor paths, we may observe spurious correlations among the variables in $S$ in the observational data $\mathcal{D}$.

If we observe spurious correlation between two generative factors $Z_{i},Z_{j}$ when they take on the values $z_{i}$ and $z_{j}$ respectively, generating counterfactual instances w.r.t. $Z_{j}$ with the intervention $do(Z_{j}=z_{j}^{\prime})$ and adding the counterfactual instances to original data breaks the correlation between $Z_{i},Z_{j}$. With this idea, we now present our algorithm to generate counterfactual images in a systematic manner remove confounding from observational data.

Our goal is to remove the effect of confounding in the observational data on a downstream task such as classification. To this end, we propose a way to systematically generate counterfactual data that can marginalize the effect of any confounding edge $C_{i}\rightarrow Z_{j}$ in Fig. 1 (a) as explained below.

Removing The Confounding Effect of $C_{i}\rightarrow Z_{j}$: In the causal graphs of the form 1(a), for paths of the form $Z_{j}\leftarrow C_{i}\rightarrow Z_{l}$, we call the edges $C_{i}\rightarrow Z_{j}$ and $C_{i}\rightarrow Z_{l}$ as confounding edges since together, their existence is the reason for confounding in the data. Also, let $(z_{j}^{p},z_{l}^{q})$ is one pair of attribute values taken by the variable pair $(Z_{j},Z_{l})$ under extreme confounding (e.g., in the training set of colored MNIST dataset, correlation coefficient of $0.99$ between color and digit is observed such that whenever color is red, digit is $7$ etc.). To remove the effect of the confounding edge $C_{i}\rightarrow Z_{j}$ w.r.t. the another confounding edge $C_{i}\rightarrow Z_{l}$ (recall that confounding between $Z_{j},Z_{l}$ is present if and only if there exists a pair of causal arrows $C_{i}\rightarrow Z_{j}$ and $C_{i}\rightarrow Z_{l}$ for some $i$; due to this reason we consider the confounding effect of the confounding edge $C_{i}\rightarrow Z_{j}$ w.r.t. another confounding edge $C_{i}\rightarrow Z_{l}$), we consider two subsets $T_{1},T_{2}$ of the observational data $\mathcal{D}$ which are constructed as follows. $T_{1}$ consists of the set of instances for which $Z_{j}\neq z_{j}^{p}$ and $Z_{l}=z_{l}^{q}$, $T_{2}$ consists of the set of instances for which $Z_{j}=z_{j}^{p}$ and $Z_{l}=z_{l}^{q}$. The size of $T_{1}$ is usually much smaller than the size of $T_{2}$ because of high correlation between $Z_{j}$ and $Z_{l}$ (e.g., there are more red $7$’s than non-red $7$’s).

Now, we learn a mapping $\mathcal{M}$ from the set $T_{1}$ to the set $T_{2}$ that changes the attribute $Z_{j}$ while fixing the value of $Z_{l}$ at $z_{l}^{q}$. That is, for any given instance $X\in T_{1}$, for which $Z_{j}\neq z_{j}^{p}$, $\mathcal{M}$ maps $X$ to a different instance $X^{\prime}$ in which the value of the generative factor $Z_{j}$ is changed to $z_{j}^{p}$ (e.g., $\mathcal{M}$ takes red $9$ as input and returns red $7$ as output). This mapping $\mathcal{M}$ can be thought of as a function performing the 3-step counterfactual inference: learning the underlying generative factors, performing the intervention $do(Z_{j}=z_{j}^{p})$ and then generating the counterfactual instance $X^{\prime}$. Now, given an instance $X$ for which $Z_{j}\neq z_{j}^{p}$ and $Z_{l}\neq z_{l}^{q}$, using $\mathcal{M}$, we can generate counterfactual instance $X^{\prime}$ in which $Z_{j}=z_{j}^{p}$ and $Z_{l}\neq z_{l}^{q}$. These counterfactual instances, when augmented with the original observed dataset $\mathcal{D}$, removes the effect of the confounding edge $C_{i}\rightarrow Z_{j}$ w.r.t. the edge $C_{i}\rightarrow Z_{l}$. That is, the counterfactual instances, when augmented with original data, breaks the correlation between $Z_{j}$ and $Z_{l}$. This process can now be repeated systematically for each confounding edge to generate counterfactual instances that remove the spurious correlations. Such augmented data points which differ from original data points w.r.t. only one feature (e.g., if original image is a male with blond hair color, augmented image is same male with black hair color) are referred as coupled sets by (Goel et al., 2021), images generated by causal essential transformations by (Wang et al., 2022). The overall procedure to generate counterfactual instances is summarized in Algorithm 1.

Earlier works use CycleGAN to generate counterfactual images that differ from original image by a single attribute/feature (Wang et al., 2022; Goel et al., 2021). Given two domains/sets of images that differ w.r.t. only one generative factor $Z_{j}$, a CycleGAN can learn to translate between the two domains by changing the attribute value of $Z_{j}$. In this case, one can think of CycleGAN as a function performing the required intervention $Z_{j}$ and generating counterfactual instance without modeling the underlying causal process. Concretely, CycleGAN is an architecture used to perform unsupervised domain translation using unpaired images. In a CycleGAN, a generator $G_{1}$ first transforms a given image $X$ from a domain/set $T_{1}$ into $X^{\prime}$ so that $X^{\prime}$ appears to come from another domain/set $T_{2}$ such that certain features from input $X$ are preserved in the output $X^{\prime}$. A discriminator $D_{T_{2}}$ then classifies whether the translated image $X^{\prime}$ is original (i.e., sampled from $T_{2}$) or fake (i.e., generated by $G_{1}$). A second generator $G_{2}$ transforms the image $X^{\prime}$ back to original image $X$ to ensure that $G_{1}$ is using the contents of $X$ to generate $X^{\prime}$. The same procedure is repeated to translate images from domain $T_{2}$ into domain $T_{1}$. The loss function of CycleGAN can be written as follows.

Where $\mathcal{L}_{GAN}$ is simple Generative Adversarial Network (GAN) (Goodfellow et al., 2014a) loss and $\mathcal{L}_{cycle}$ is cycle consistency loss measuring how well the output of $G_{2}$ is matching with the original input $X$. For example, $\mathcal{L}_{cycle}(G_{1},G_{2})=\mathbb{E}_{X\sim\mathcal{D}}[||G_{2}(G_{1}(X))-X||_{1}]$ can ensure that $G_{2}(G_{1}(X))=X$. In this work, to learn the mapping function $\mathcal{M}$, we use conditional variant of CycleGAN to leverage the supervision in terms of attribute values. For each generator, along with input, we also feed a desired target attribute as shown in the Figure 2.

To improve the quality of counterfactual images generated by conditional CycleGAN under extreme confounding, we propose a modification to conditional CycleGAN as detailed below. As discussed earlier, $X^{\prime}$, the output of $G_{1}$, can be thought of as a counterfactual image of $X$. When changing the feature $Z_{j}$ of $X$, we keep the feature $Z_{l}$ fixed. That is, the representation for $Z_{j}$ in both $X$ and $X^{\prime}$ should be different and the representation for $Z_{l}$ in both $X$ and $X^{\prime}$ should be same. To ensure this, as shown in Figure 2, along with two generators $G_{1},G_{2}$ and a discriminator $D_{T_{2}}$ that are part of conditional CycleGAN, we add two pre-trained discriminators $L_{1},L_{2}$ (shown in green color in Fig. 2). $L_{1}$ takes two images $X,X^{\prime}$ as input and returns high penalty if the representation of $Z_{j}$ is similar in $X,X^{\prime}$ and small penalty otherwise. $L_{2}$ takes two images $X,X^{\prime}$ as input and returns high penalty if the representation of $Z_{l}$ is different and small penalty otherwise. Thus, our overall objective to generate good quality counterfactual images is to train the modified conditional CycleGAN by minimizing the following objective.

Where $\alpha$ is a hyperparameter and $\mathcal{L}_{contrastive}$ is the contrastive loss (Hadsell et al., 2006). For a pair of images $(X,X^{\prime})$, $\mathcal{L}_{contrastive}$ defined as follows.

Where $A=1$ if $X,X^{\prime}$ belong to same class (or have same attribute values), $A=0$ if $X,X^{\prime}$ belong to different classes (or have different attribute values). $D$ is the distance between the representations of $X,X^{\prime}$ (e.g., Euclidean distance). $\epsilon$ is the margin of error allowed between two representations of the images of different classes. $L_{1}$ and $L_{2}$ are pre-trained models and the parameters of $L_{1}$ and $L_{2}$ are fixed. That is, the loss values returned by $\mathcal{L}_{contrastive}$ are only used to update the trainable parameters of conditional CycleGAN.

A Downstream Task - Image Classification: To measure the goodness of counterfactual generation under confounding using Algorithm 1, we consider the classification task on the unconfounded test set as a downstream task. Let $\mathcal{D}^{aug}=\{(X_{i},Y_{i})\}_{i=1}^{M}$ be the dataset consisting of original data points from $\mathcal{D}$ and corresponding counterfactual data points. Usual empirical risk minimizer minimizes the following loss over $\mathcal{D}$.

Where $l$ is cross entropy loss. Using $\mathcal{D}^{aug}$, we minimize the following loss $\mathcal{L}_{aug}$:

To further improve the performance of a classifier using $\mathcal{D}^{aug}$, for each pair of images $X_{i},X_{j}$ we minimize the contrastive loss $\mathcal{L}_{contrastive}(X_{i},X_{j})$ on the logits in the final layer. Now, the final objective to optimize for classification task is to minimize the following loss.

Where $\lambda>0$ is a regularization hyperparameter.

In this section, we present the experimental results on both synthetic (MNIST variants) and real world (CelebA) datasets. Having access to the ground truth generative factors (i.e., $Z_{1},\dots,Z_{n}$) of images,we artificially create confounding in the training data and we leave test data to be unconfounded (i.e., no correlation among generative factors). We compare CONIC with various baselines including traditional Empirical Risk Minimizer (ERM), Conditional GAN (CGAN)  (Goodfellow et al., 2014a), Conditional VAE (CVAE)  (Kingma & Welling, 2013), Conditional-$\beta$-VAE (C-$\beta$-VAE)  (Higgins et al., 2017), AugMix (Hendrycks et al., 2020), CutMix (Yun et al., 2019), Invariant Risk Minimization (IRM) (Arjovsky et al., 2019), and Counterfactual Generative Networks (CGN) (Sauer & Geiger, 2021). To check the goodness of each of these methods, we check how well the performance of the downstream classifier on the test set is improved using the augmented images.

MNIST Variants: We construct the following three synthetic datasets based on MNIST dataset (Lecun et al., 1998) and its colored, texture, and morpho (where the digit thickness is controlled; Fig. 3) variants (Arjovsky et al., 2019; Castro et al., 2019; Sauer & Geiger, 2021): (i) colored morpho MNIST (CM-MNIST), (ii) double colored morpho MNIST (DCM-MNIST), and (iii) wildlife morpho MNIST (WLM-MNIST). We consider extreme confounding among generative factors as explained below.

For the experimental results shown in Table 1, in the training set of CM-MNIST dataset, the correlation coefficient between digit label and digit color $r(label,color)$ is $0.95$ and the digits from $0$ to $4$ are thin and digits from $5$ to $9$ are thick (see Figure 3). That is, $r(label,thin)=1$ if the digit is in [0,1,2,3,4] else $r(label,thick)=1$. In the training set of DCM-MNIST dataset, digit label, digit color, and background color jointly take a fixed set of values 95% of the time. That is, $r(label,color)=r(color,background)=r(label,background)=0.95$ and the digits from $0$ to $4$ are thin and digits from $5$ to $9$ are thick. In the training set of WLM-MNIST dataset digit shape, digit texture, and background texture jointly take a fixed set of attribute values 95% of the time and the digits from $0$ to $4$ are thin and digits from $5$ to $9$ are thick.

In all of these MNIST variants, test set images are unconfounded (e.g., in the test set of DCM-MNIST, any digit can be thin or think, can be in any background color, can be in any foreground color). In these experiments, under extreme confounding, our goal is to generate counterfactual images that break the confounding among generative factors. We evaluate models on this grounds by training a classifier using the augmented data and testing it on the unconfounded test data. Table 1 shows the results in which CONIC outperforms all the baselines. See Appendix for comparison of augmented images by various baselines. Coninc uses only $10000$, $15000$, $15000$ counterfactual images in CM-MNIST, DCM-MNIST, and WLM-MNIST experiments as augmented images respectively to get improved performance. The regularization hyperparameter $\lambda$ in Equation 8 set to $0.5$ for all MNIST experiments.

CelebA: Unlike MNIST variants, CelebA (Liu et al., 2015) dataset implicitly contains confounding (e.g., the percentage of males with blond hair is different from the percentage of females with blond hair, in addition to the difference in the total number of males and females in the dataset). In this experiment, we consider the performance of a classifier trained on the augmented data that predicts hair color given an image. Our test set is the set of males with blond hair.

We train models on the train set and test the performance on the set of males with blond hair. Since the number of males with blond hair is very low in the dataset (approximately 4% of males have blond hair), we show that the augmenting the train set with only 10000 images of males with blond hair improves the performance over baselines (see Table 1) whereas other baselines require more than 50000 augmented images to get minor improvement over ERM. Given a male image with non-blond hair, CONIC generates the counterfactual image with blond hair without changing the male attribute (see Figure 4 for sample counterfactual images). We also note that the deterministic models such as CGN fail when they are applied to a different task where the number and type of generative factors are not fixed and are difficult to separate (e.g., CelebA). CGN results in table 1 are obtained with only 1000 counterfactual images as augmented data points. When we increase the number of counterfactual instances, performance of CGN reduces further.

Time Complexity Analysis: Apart from its simple methodology, CONIC brings

additional advantages in terms of computing time required to train the model that generates counterfactual images. As shown in Table 2, the time required to run our method to generate counterfactual images w.r.t. a generative factor $Z_{j}$ is significantly less than CGN that learns deterministic causal mechanisms as discussed in Section 2. Even though we used CycleGAN in this work, for the cases where the number of generative factors are more, StarGAN (Choi et al., 2018) can be used to minimize the time required to learn the mappings from one domain to another domain (Wang et al., 2022; Goel et al., 2021).

We studied the adverse effects of confounding in observational data on the performance of a classifier. We showed the relationship between confounding and correlation in the causal processes considered, and we proposed a methodology to remove the correlation between the target variable and generative factors that works even when the dataset is highly confounded. Specifically, we proposed a counterfactual data augmentation method that systematically removes the confounding effect rather than addressing the confounding problem through random augmentations. Using the generated counterfactuals leads to substantial increase in a downstream classifier’s accuracy. That said, we observed that the counterfactual quality can still be improved, which will be interesting future work.