Reinforcement Learning with Continuous Actions Under Unmeasured Confounding

In practical applications of reinforcement learning (RL) (Sutton and Barto, 2018), the evaluation and optimization of policies using only pre-collected datasets has become essential. This need arises from the potential costs and safety risks associated with frequent interactions with the environment, such as real-world testing in autonomous driving (Zhu et al., 2020) and treatment selection in precision medicine (Luckett et al., 2019; Zhou et al., 2024a). As a result, there has been a growing interest in offline RL (Precup, 2000; Levine et al., 2020), which focuses on policy evaluation and optimization without requiring additional environment interactions.

Recent years have seen significant advancements in both off-policy evaluation (OPE) and off-policy learning (OPL). In OPE, popular methodologies include value-based approaches (Le et al., 2019; Liao et al., 2021), importance sampling techniques (Liu et al., 2018; Xie et al., 2019), and doubly robust methods that integrate value-based and importance sampling estimators (Uehara et al., 2020; Kallus et al., 2022). Additionally, confidence intervals for a target policy have been developed using bootstrapping (Hao et al., 2021), asymptotic properties (Shi et al., 2022b), and finite-sample error bounds (Zhou et al., 2023). In OPL, various algorithms have been proposed, demonstrating notable success across both discrete (Mnih et al., 2015; Luckett et al., 2019; Zhou et al., 2024b) and continuous action spaces (Kumar et al., 2020; Fujimoto and Gu, 2021; Li et al., 2023; Zhou, 2024).

A common assumption in the works mentioned above is the absence of unmeasured confounders. Specifically, these studies assume that all state variables are fully observed, with no unmeasured variables that might confound the observed actions. However, this assumption is unverifiable with offline data and is often violated in practical settings. Real-world instances include the influence of genetic factors in personalized medicine (Fröhlich et al., 2018) and the complexities of path planning in robotics (Zhang et al., 2020). To address these challenges, a recent line of research has focused on OPE within the framework of a confounded partially observable Markov Decision Process (POMDP), where the behavior policy generating the batch data may depend on unobserved state variables (Zhang and Bareinboim, 2016).

Various methods have been developed to identify the true policy value with the presence of unmeasured confounders. Within the contextual bandit setting (i.e., one decision point), causal inference frameworks have been established to identify the treatment effect by sensitivity analysis (Bonvini and Kennedy, 2022), and through the use of instrumental variables (Cui and Tchetgen Tchetgen, 2021) and negative controls (Miao et al., 2018; Cui et al., 2023). Existing methods for extensions to multiple decision points or the infinite horizon setting fall into two main categories. The first type considers i.i.d. confounders in the dynamic system and thus preserves the Markovian property (Zhang and Bareinboim, 2016). Under this framework, OPE methods have been formulated under various identification conditions, such as partial identification via sensitivity analysis (Kallus and Zhou, 2020; Bruns-Smith, 2021) and approaches that utilize instrumental variables or mediators (Li et al., 2021; Shi et al., 2022c). The second category explores the estimation of policy values in more general confounded POMDP models, where the Markovian assumption does not hold. These methods span a wide array of strategies, including the use of proxy variables (Shi et al., 2022a; Miao et al., 2022; Uehara et al., 2024) or instrumental variables (Fu et al., 2022; Xu et al., 2023), spectral methods for undercomplete POMDPs (Jin et al., 2020), and techniques focusing on predictive state representation (Cai et al., 2022; Guo et al., 2022).

However, there are two less investigated issues in confounded POMDP settings. Firstly, existing methods mainly focus on discrete action spaces (Miao et al., 2022; Shi et al., 2022a; Bennett and Kallus, 2023) despite many real-world scenarios requiring decision making on a continuous action space (Lillicrap et al., 2015). A straightforward workaround in adapting existing methods to continuous domains is to discretize the continuous action space. However, this approach either introduces significant bias when using coarse discretization (Lee et al., 2018; Cai et al., 2021) or encounters the the curse of dimensionality when applied to fine grids (Chou et al., 2017). Secondly, the challenge of learning an optimal policy from a batch dataset, particularly in the presence of unmeasured confounders, is paramount in various fields, including personalized medicine (Lu et al., 2022) and robotics (Brunke et al., 2022). A majority of existing methods, however, focus on policy evaluation rather than optimization. There are some recent efforts to address this gap. For instance, Qi et al. (2023) explore policy learning under the contextual bandit setting, and Hong et al. (2023) investigate policy gradient methods under finite-horizon confounded POMDPs. In terms of the infinite-horizon, Kallus and Zhou (2021); Fu et al. (2022) consider a restrictive memoryless confounders setting where the Markovian property is preserved, while Guo et al. (2022); Lu et al. (2022) focus on theoretical properties of the induced estimator under more general confounded POMDP settings, yet practical computational algorithms remain elusive. Thus, the development of computationally viable policy learning algorithms under infinite-horizon confounded MDPs, where the Markovian property is violated, remains a significant challenge.

Motivated by these, in this paper, we study the policy learning with continuous actions for confounded POMDPs over an infinite horizon. Our main contribution to the literature is threefold. First, relying on some time-dependent proxy variables, we extend the proximal causal inference framework (Miao et al., 2018; Tchetgen Tchetgen et al., 2020) to infinite horizon setting, and establish a nonparametric identification result for OPE using $V$-bridge functions with continuous actions for confounded POMDPs. Leveraging the identification result, we propose an unbiased minimax estimator for the $V$-bridge function and introduce a computationally efficient Fitted-Q Evaluation (FQE)-type algorithm for estimating the policy value. Second, we develop a novel policy gradient algorithm that searches the optimal policy within a specified policy class by maximizing the estimated policy value. The proposed algorithm is tailored for continuous action spaces and provides enhanced interpretability of the optimal policy. Third, we thoroughly investigate the theoretical properties of the proposed methods on both policy evaluation and policy learning, including the estimator consistency, finite-sample bound for performance error, and sub-optimality of the induced optimal policy. We validate our proposed method through extensive numerical experiments and apply it to the German Family Panel (Pairfam) dataset (Brüderl et al., 2023), where we aim at identifying optimal strategies to enhance long-term relationship satisfaction.

We consider an infinite-horizon confounded Partially Observable Markov Decision Process (POMDP) defined as $\mathcal{M}=\{\mathcal{S},\mathcal{O},\mathcal{A},\mathbf{P},R,\gamma\}$, where $\mathcal{S}$ and $\mathcal{O}$ denote the unobserved and observed continuous state space respectively, $\mathcal{A}$ is the action space, $\mathbf{P}:\mathcal{S}\times\mathcal{O}\times\mathcal{A}\to\Delta(\mathcal{S}\times\mathcal{O})$ is the unknown transitional kernel, $R:\mathcal{S}\times\mathcal{O}\times\mathcal{A}\to\mathbb{R}$ is a bounded reward function, and $\gamma\in[0,1)$ is the discounted factor that balances the immediate and future rewards. $\mathcal{O}$ can also be treated as the observation space in the classical POMDP, then the process $\mathcal{M}$ can be summarized as $\{S_{t},O_{t},A_{t},R_{t}\}^{T}_{t=1}$, with $S_{t}$ and $O_{t}$ as unobserved and observed state variables, $A_{t}$ as the action, and $R_{t}$ as the reward.

The objective of policy learning is to search an optimal policy, $\pi^{*}$, which maximizes the expected discounted sum of rewards, using batch data obtained from a behavior policy $\pi^{b}$. We assume the batch data consists of $n$ i.i.d. trajectories, i.e.,

where the length of trajectory $T$ is assumed to be fixed for simplicity, and the information on unobserved state $S^{i}_{t}$ is not available. In this paper, we focus on scenarios where the behavior policy, $\pi^{b}$, maps both unobserved and observed state spaces to the action space, that is, $\pi^{b}:\mathcal{S}\times\mathcal{O}\to\mathcal{A}$. Meanwhile, the target optimal policy only depends on the observed state: $\pi^{*}:\mathcal{O}\to\mathcal{A}$. For a given target policy $\pi:\mathcal{O}\to\mathcal{A}$ , its state-value function is denoted as

where $\mathbb{E}_{\pi}$ denotes the expectation with respect to the distribution whose actions at each decision point $t$ follow policy $\pi$. Our goal is to utilize the batch data $\mathcal{D}_{n}$ to find the optimal policy $\pi^{*}$ which maximizes the target policy value defined as

with $\mathbb{E}$ representing the expectation in accordance with the behavior policy. Due to the unobserved state $S_{t}$, standard policy learning approach based on the Bellman optimality equation yields biased estimates. Thus, we first introduce an identification result to estimate the policy value for any target policy $\pi$ with the help of some proxy variables. Subsequently, we employ policy gradient techniques to find the optimal policy $\pi^{*}$.

Inspired by the proximal causal inference framework introduced by Tchetgen Tchetgen et al. (2020), we initially present the identification result for policy evaluation in Section 3.1. Following this, its associated minimax estimator for any given target policy $\pi$ is discussed in Section 3.2. Building on these policy evaluation findings, we detail a policy-gradient based approach in Section 3.3 to identify the optimal policy within a policy class by maximizing the estimated policy value.

In this section, we first derive the global rate of convergence for $\hat{Q}^{\pi}$ in (3.5), and the finite-sample error bound for the estimated policy value $\hat{J}(\pi)$ in (3.6). We then extend the results to the estimated optimal policy, where we derive the regret bound for $\hat{\pi}^{*}$ defined in (3.10). We note that while we use kernel representation to obtain the closed-form solution for the inner-maximization problem as demnstrated in (3.7) and (3.8), our theoretical results are based on the general form of (3.5). To simplify notation, we denote the $L_{2}$ norm with respect to the average state-action distribution in the trajectory $\mathcal{D}$ as $\|f\|^{2}=\mathbb{E}\Big{[}\frac{1}{T}\sum^{T}_{t=1}f^{2}(\cdot)\Big{]}$, and the Bellman operator $\mathcal{T}_{\pi}$ with respect to the target policy $\pi$ as $\mathcal{T}_{\pi}(s,o,a;q)=\mathbb{E}\big{[}R_{t+1}+\gamma\int_{a\in\mathcal{A}}\pi(a^{\prime}|O_{t+1})q(O_{t+1},W_{t+1},a^{\prime})da\mid S_{t}=s,O_{t}=o,A_{t}=a\big{]}$. Before presenting our main results, we first state several standard assumptions on policy class $\Pi$, reward function $R_{t}$, and function class $\mathcal{Q}$ and $\mathcal{F}$.

Assumption 5 is used to control the complexity of policy class, which are commonly assumed in policy learning problems (Liao et al., 2022; Wang and Zou, 2022). Assumption 6 is a regular assumptions to impose boundedness condition on reward and function class $\mathcal{Q}$ (Antos et al., 2008; Bennett et al., 2021). Assumption 7 similarly ensures the boundedness of function class $\mathcal{F}$. Additionally, the value of $\kappa$ measures how well the function class $\mathcal{F}$ approximates the Bellman error for all $q\in\mathcal{Q},\pi\in\Pi$. This concept has been explored as the well-posedness condition within the field of off-policy evaluation (Chen and Qi, 2022; Miao et al., 2022). A strictly positive value of $\kappa$ indicates a substantial overlap between the behavior and target policies in terms of the state-action distribution, ensuring the identification of the true policy value $J(\pi)$.

Assumption 8 characterizes the complexity of the function class $\mathcal{Q}$ and $\mathcal{F}$. The condition that the regularizers be pseudo-norms is satisfied for common function classes such as RKHS and Sobolev spaces (Geer, 2000; Thomas and Brunskill, 2016). The upper bound on $h_{2}$ is realistic when the transition kernel is sufficiently smooth (Farahm et al., 2016). We use a common $\alpha\in(0,1)$ for both $\mathcal{Q}$ and $\mathcal{F}$ to simplify the proof.

In supplementary material, we have proved that $\|\hat{Q}^{\pi}-Q^{\pi}\|^{2}\lesssim\|\mathcal{T}_{\pi}(\cdot,\cdot,\cdot;\hat{Q}^{\pi})-\hat{Q}^{\pi}\|^{2}$. Consequently, Theorem 4.1 demonstrates that $\hat{Q}^{\pi}$ serves as a consistent estimator for $Q^{\pi}$, provided that $\lambda_{n}=o_{P}(1)$. When the tuning parameter is selected such that $\lambda_{n}\asymp\mu_{n}$ and $\lambda_{n}\asymp n^{-1/(1+\alpha)}$, we achieve the optimal convergence rate of the Bellman error at $\hat{Q}^{\pi}$, which is $O_{P}(n^{-1/(1+\alpha)})$. Before delving into the finite-sample error bound for $\hat{J}(\pi)$, we first introduce some additional notations.

We define the discounted state-action visitation of target policy $\pi$ as $d^{\pi}(o,s,a)=(1-\gamma)\sum^{\infty}_{t=0}\gamma^{t}d^{\pi}_{t}(o,s,a)$, where $d^{\pi}_{t}$ is the density of the state-action pair at $t^{\text{th}}$ time point under the target policy $\pi$, and denote the average density over $T$ decision times under the behavior policy as $\bar{d}_{T}(o,s,a)$. We further define the direction function $e^{\pi}(o^{-},a^{-},o,a)$ by

The direction function $e^{\pi}$ is used to control the bias $|\hat{J}(\pi)-J(\pi)|$ caused by the penalization of $Q$-bridge in (3.5). As demonstrated in Uehara et al. (2020); Shi et al. (2022a), the direction function $e^{\pi}$ enjoys the following property in our setting

Next, we define $\mathbb{E}[\xi^{\pi}(O_{t},W_{t},A_{t})|S_{t},O_{t},A_{t}]=\mathbb{E}[\sum^{\infty}_{t=0}\gamma^{t}\frac{d^{\pi}(O_{t},S_{t},A_{t})}{\bar{d}_{T}(O_{t},S_{t},A_{t})}]$. Note that $\xi^{\pi}$ has a similar structure to the $Q$-bridge function, where the “reward” at time $t$ is defined as $\frac{d^{\pi}(O_{t},S_{t},A_{t})}{\bar{d}_{T}(O_{t},S_{t},A_{t})}$. Similar to the $Q$-bridge, $\xi^{\pi}$ satisfies the following Bellman-like equation as demonstrated in Theorem 3.1: