On Instrumental Variable Regression for
Deep Offline Policy Evaluation

Reinforcement learning considers a Markov decision process $\left\langle\mathcal{S},\mathcal{A},P,R,\mu_{0},\gamma\right\rangle$, where $\mathcal{S}$ is the state space, $\mathcal{A}$ is the action space, and $P(s^{\prime}|a,s)$ is the probability or probability density of making a transition to state $s^{\prime}$ when taking action $a$ in state $s$. $R(r|s,a)$ denotes the probability density of observing reward $r$ after having taken action $a$ from $s$, and $\mu_{0}(s)$ is the initial state distribution. Let $\pi$ be a policy of an agent, and denote $\pi(a|s)$ as the probability or probability density of selecting action $a$ in state $s\in\mathcal{S}$. With a discount factor $\gamma\in(0,1]$, the state-action value function - i.e. $Q$ function - is defined by

$$Q(s,a)=\mathbb{E}\left[\sum_{t=0}^{\infty}\gamma^{t}r_{t}|s_{0}=s,a_{0}=a\right],$$

(1)

with $a_{t}\sim\pi(\cdot\mid s_{t}),s_{t+1}\sim P(\cdot|s_{t},a_{t}),r_{t}\sim R(\cdot|s_{t},a_{t})$ for $t\geq 0$.

Common tasks in reinforcement learning including estimating the value of a given target policy $\pi$ or optimizing the policy value with respect to $\pi$, where the policy value is defined by the expected sum of discounted rewards from the initial state distribution

$$\rho(\pi)=\mathbb{E}_{s_{0}\sim\mu_{0}}\left[\sum_{t=0}^{\infty}\gamma^{t}r_{t}\right]=\mathbb{E}_{s\sim\mu_{0},a|s\sim\pi}[Q(s,a)],$$

(2)

When an agent is prohibited to interact with the environment directly, one has to rely on an existing dataset of trajectories or transition tuples $(s,a,r,s^{\prime})$, to estimate the policy value or learn the optimal policy. The dataset could have been collected by one or a mixture of potentially unknown policies of potentially unknown analytical form, denoted by $\pi_{b}(\cdot|s)$, and the corresponding state action distribution is denoted by $\mu_{b}$.

The goal of offline policy evaluation (OPE) is to evaluate the value of a target policy, $\pi$, based on the offline behavior dataset. This problem has been extensively studied in the literature. The readers are referred to Levine et al. (2020) for a review and Voloshin et al. (2019b); Fu et al. (2021) for benchmarks of recent OPE algorithms.

One family of OPE approaches is to estimate the value function based on the Bellman equation,

$$Q(s,a)=\mathbb{E}_{r\sim R(\cdot|s,a)}\left[r|s,a\right]+\gamma\mathbb{E}_{s^{\prime}\sim P(\cdot|s,a),a^{\prime}\sim\pi(\cdot|s^{\prime})}\left[Q(s^{\prime},a^{\prime})|s,a\right],\forall s\in\mathcal{S},a\in\mathcal{A}.$$

(3)

We can solve the Bellman equation as a least square regression problem for the reward $r$ on the state-action pair $(s,a)$

$\displaystyle\mathbb{E}\left[r|s,a\right]=Q(s,a)-\gamma\mathbb{E}\left[Q(s^{\prime},a^{\prime})|s,a\right],$

(4)

and find the function $Q$ to minimize the mean squared Bellman error (MSBE) (Sutton and Barto, 2018, p. 268) with respect to the behavior distribution

$\displaystyle Q=\underset{Q}{\arg\min}\leavevmode\nobreak\ \mathbb{E}_{(s,a)\sim\mu_{b},r\sim R}\left[\left(r-Q(s,a)+\gamma\mathbb{E}_{s^{\prime}\sim P(\cdot|s,a),a^{\prime}\sim\pi(\cdot|s^{\prime})}\left[Q(s^{\prime},a^{\prime})\right]\right)^{2}\right].$

(5)

We then use Equation 2 for the evaluation policy $\pi$.

Instrumental variable (IV) regression methods (Stock and Trebbi, 2003) are standard techniques developed in the causal inference and econometrics literature, which are used to predict the effect of actions $X$ (called treatment) on the world when the treatment affects the distribution of the variable of interest $Y$, which is called the outcome. IV regression provides a framework to assess this effect (called the structural function) by using an instrumental variable $Z$, which only affects the treatment directly, but has no direct effect on the outcome.

Instrumental variables can be found in many contexts, and IV regression is extensively used by economists and epidemiologists. For example, (Wright, 1928; Blundell et al., 2012) used supply cost shifters as instrumental variables, and estimate the effect of price on the demand to correct for confounders such as the time of the year. IV regression has also been used to measure the effect of a drug in the scenario of imperfect compliance (Angrist et al., 1996), or the influence of military service on lifetime earnings (Angrist, 1990).

Formally, we aim to learn a relationship between $X$ and $Y$, which is generated from

$\displaystyle Y=f_{\mathrm{struct}}(X)+\varepsilon,\quad\mathbb{E}\left[\varepsilon\right]=0,\quad\mathbb{E}\left[\varepsilon|X\right]\neq 0,$

(9)

where $f_{\mathrm{struct}}$ is the structural function, which we assume to be continuous, and $\varepsilon$ is an additive noise term. The challenge is that $\mathbb{E}\left[\varepsilon|X\right]\neq 0$, which reflects the existence of a latent confounder. Hence, we cannot use ordinary supervised learning techniques since $f_{\mathrm{struct}}(x)\neq\mathbb{E}\left[Y|X=x\right]$.

To deal with the confounder $\varepsilon$, we assume to have access to an instrumental variable $Z\in\mathcal{Z}$ which satisfies the following assumption.

Intuitively, Assumption 1 means that the instrument $Z$ induces variation in the treatment $X$ but is uncorrelated with the hidden confounder $\varepsilon$. The causal graph describing these relationships is shown in Figure 1.¹¹1We show the simplest causal graph in Figure 1 It entails $Z\mathop{\perp\!\!\!\perp}\varepsilon$, but we only require $Z$ and $\varepsilon$ to be uncorrelated in Assumption 1. Of course, this graph also says that $Z$ is not independent of $\varepsilon$ when conditioned on observations $X$. Note that the instrument $Z$ cannot have an incoming edge from the latent confounder that is also a parent of the outcome. It follows directly from 1 that

$\displaystyle\mathbb{E}\left[Y|Z\right]=\mathbb{E}\left[f(X)|Z\right]+\mathbb{E}\left[\varepsilon|Z\right]=\int_{X}f(X)P(X|Z)\text{d}X.$

(10)

Classically, IV regression is solved by the two-stage least squares (2SLS) algorithm; we learn a mapping from the instrument to the treatment in the first stage, and learn the structural function in the second stage as the mapping from the conditional expectation of the treatment given the instrument (obtained from stage 1) to the outcome. Originally, 2SLS assumes linear relationships in both stages, i.e.,

$$X=Z\omega+\delta,\quad f_{\mathrm{struct}}(X)=X\theta\,,$$

where $\omega$ and $\theta$ are unknown regression coefficients and $\delta$ is a random variable satisfying $\mathbb{E}\left[\delta|Z\right]=0$. In the case when the dimension of $X$ equals that of $Z$, the 2SLS estimator has the following simple form

$$\hat{\theta}=\left(Z^{\top}X\right)^{-1}\left(Z^{\top}Y\right)\,,$$

(11)

where we somewhat abuse notation and denote by $X$, $Z$ the matrices of observed treatment and instrumental variables, by $Y$ the vector of outcomes, and each row corresponds to one observation.

Instrumental variable methods have been extended to non-linear settings first in economics (Newey and Powell, 2003; Carrasco et al., 2007; Darolles et al., 2011; Blundell et al., 2012; Chen and Christensen, 2018) and more recently in machine learning (Hartford et al., 2017a; Lewis and Syrgkanis, 2018; Singh et al., 2019a; Muandet et al., 2019; Bennett et al., 2019a; Dikkala et al., 2020; Luofeng et al., 2020; Xu et al., 2021). One approach has been to use non-linear feature maps. Sieve IV (Newey and Powell, 2003; Chen and Christensen, 2018) uses a finite number of explicitly specified basis functions. Kernel IV (KIV) (Singh et al., 2019a) and Dual IV regression (Muandet et al., 2019) extend sieve IV to allow for an infinite number of basis functions using reproducing kernel Hibert spaces (RKHS). Although these methods enjoy desirable theoretical properties, the flexibility of the model is limited due to the prespecified features. To mitigate this limitation, Xu et al. (2021) proposes Deep Feature IV (DFIV) method, in which one learns features adaptively using neural networks while preserving the two-stage nature of 2SLS.

Another non-linear approach to the stage 1 regression is to estimate the conditional distribution $P(X|Z)$ (Carrasco et al., 2007; Darolles et al., 2011; Hartford et al., 2017a). This allows flexible models, including deep neural nets, as proposed in the DeepIV algorithm of (Hartford et al., 2017a). However, the conditional density estimation can be costly, and can suffer from high variance when the treatment is high-dimensional.

As a further alternative, several recent works (Lewis and Syrgkanis, 2018; Bennett et al., 2019a; Dikkala et al., 2020; Luofeng et al., 2020) have been inspired by another instrumental variable technique, the Generalized Method of Moments (GMM) (Hansen, 1982), and find non-linear structural functions to ensure that the regression residual and the instrument are uncorrelated. These works do not require two stage regression, and the resulting methods are often formulated as solving a minimax optimization problem.

When the value function $Q$ is a non-linear function of $(s,a)$, it was shown recently by Xu et al. (2021) that we can estimate it by solving a non-linear IV regression problem following Equation 13 with the corresponding causal graphical model in Figure 3,

$\displaystyle\underbrace{r}_{Y}$

$\displaystyle=\underbrace{Q(s,a)-\gamma Q(s^{\prime},a^{\prime})}_{f(X)}+\underbrace{\left(\gamma Q(s^{\prime},a^{\prime})-\gamma\mathbb{E}\left[Q(s^{\prime},a^{\prime})|s,a\right]\right)+\left(r-\mathbb{E}\left[r|s,a\right]\right)}_{\varepsilon},$

(19)

where $(s,a)\sim\mu_{b},\leavevmode\nobreak\ \leavevmode\nobreak\ s^{\prime}\sim P(s^{\prime}|s,a),\leavevmode\nobreak\ \leavevmode\nobreak\ a^{\prime}\sim\pi(a^{\prime}|s^{\prime})$ and the structural function is $f(s,a,s^{\prime},a^{\prime})=Q(s,a)-\gamma Q(s^{\prime},a^{\prime})$ with $X=(s,a,s^{\prime},a^{\prime})$. Choosing the instrument $Z=(s,a)$, it is easy to show that $Z$ satisfies 1 as $P(X|Z)$ is not constant in $Z$, where $Z$ is a subset of $X$ and $\mathbb{E}(\varepsilon|Z)=0$.

In practice, we need to parameterize the structural function $f(s,a,s^{\prime},a^{\prime})$. It is sensible to use a parameterization of the form

$$f_{\theta}(s,a,s^{\prime},a^{\prime})=Q_{\theta}(s,a)-\gamma Q_{\theta}(s^{\prime},a^{\prime})$$

(20)

instead of parameterizing $f$ directly. We should also keep in mind that when we use approximate $Q$ functions it will not be true that $\mathbb{E}(\varepsilon|Z)=0$ exactly. The induced bias will be illustrated later in the experiments with an under-fitted function approximator.

The dependency of the output noise $\varepsilon$ on the next state $s^{\prime}$ bears resemblance to the “colored” noise in the GPTD formulation (Engel et al., 2005). However, we emphasize here that the GPTD formulation fails to reflect the confounding issue inherent to the Bellman residual minimization problem because the noise $N(s,s^{\prime})$ in GPTD is still assumed to be uncorrelated with $s^{\prime}$ even though dependent. Therefore, their solution may still lead to a biased estimator.

Depending on the function family of $Q_{\theta}$ and how to apply the instrumental variables, we will present a few representative non-linear IV methods in the setting of offline policy evaluation in the following sections.

The Deep IV method in Hartford et al. (2017b) is based on the identity Equation 10

$\displaystyle\mathbb{E}\left[Y|Z\right]=\int_{X}f(X)P(X|Z)\text{d}X.$

(21)

It is a two-stage regression approach that estimates the nonlinear relationships between $Z$ and $X$, and $X$ and $Y$, using a neural network function approximator in each stage. In the first stage, Deep IV trains a treatment network to estimate the conditional distribution of treatment $P_{\phi}(X|Z)$ by maximum likelihood estimation. The network outputs a categorical distribution if the treatment variable $X$ is discrete and a mixture of Gaussian distributions if it is continuous. In the second stage, it estimates the structural function using an outcome network $f_{\theta}(X)$ by regressing $Y$ on the conditional expectation $\mathbb{E}_{P_{\phi}(X|Z)}\left[f_{\theta}(X)|Z\right]$ where the estimate of the expectation is obtained by Monte Carlo samples from $P_{\phi}(X|Z)$.

In our RL context, recall that $X=(s,a,s^{\prime},a^{\prime})$ while $Z=(s,a)$, so the non-degenerate part of the conditional distribution of the treatment is given by

$$P(s^{\prime},a^{\prime}|s,a)=P(s^{\prime}|s,a)\pi(a^{\prime}|s^{\prime}).$$

(22)

Thus, applying Deep IV in the RL context consists first of estimating the transition distribution $P(s^{\prime}|s,a)$ using a generative model (as we know the target policy $\pi(a^{\prime}|s^{\prime})$). We then compute a Monte Carlo estimate of $\mathbb{E}(f_{\theta}(X)|Z)$ with multiple samples, and estimate $Q$ by minimizing the approximate MSBE in Equation 5.

Hartford et al. (2017b) also allow for observable confounders, but this extension is unnecessary in our scenario. Deep IV is closely related to the model-based reinforcement learning algorithm Dyna-Q (Sutton, 1990), which also learns the transition distribution and then $Q$ by minimizing the TD error. The difference is that Dyna-Q uses Q-learning to minimize the TD error in the Bellman optimal equation in order to improve the policy instead of estimation, and it requires only a single sample of $s^{\prime}$ to provide an unbiased estimate of the gradient, similar to the iterative FQE algorithm.

KIV (Singh et al., 2019a) and DFIV (Xu et al., 2021) introduce nonlinear feature maps in the IV formulation. Similar to Deep IV, this approach is also based on Equation 10, and solves for $f$ by minimizing

$\displaystyle\mathcal{L}(f)=\mathbb{E}_{YZ}\left[(Y-\mathbb{E}_{X|Z}\left[f(X)\right])^{2}\right]+R(f),$

where $R(f)$ is the regularization for $f$. KIV and DFIV regress to the expected features of $X$ on $Z$, however, in contrast to Deep IV, which estimates the conditional distribution of $P(Z|X)$; density estimation is not necessary for estimating $\mathbb{E}_{X|Z}\left[f(X)\right],$ and can be more difficult in practice. KIV and DFIV employ the following models:

$\displaystyle f(X)=w^{\top}\phi(X),\quad\mathbb{E}\left[\phi(X)|Z\right]=V\psi(Z),$

where $w,V$ are the learnable linear weights and $\phi(X),\psi(Z)$ are the nonlinear feature maps for the treatment and the instrument, respectively.

KIV considers static feature maps from a Reproducing Kernel Hilbert Space (RKHS) for $\phi(X),\psi(Z)$ and learns weights $w,V$ by a two-stage regression: stage 1 performs the regression from the instrument $Z$ to the treatment features $\phi(X)$ to learn weight $V$; then in stage 2, weights $w$ are learned by minimizing the loss $\mathcal{L}(f)$ using the predicted treatment features $V\psi(Z)$. DFIV additionally learns feature maps $\phi(X),\psi(Z)$ using neural networks in the same two-stage regression.

Note that in the RL context where $X=(s,a,s^{\prime},a^{\prime})$ and $Z=(s,a)$, the loss $\mathcal{L}$ is identical to MSBE defined in eq. 5 apart from the regularization term. This two-stage regression proceeds as follows. As in eq. 12, KIV and DFIV model $Q(s,a)=\phi(s,a)^{\top}\theta$ where $\phi$ is a feature map and $\theta$ are the parameters. Furthermore, they model the conditional expectation as $\mathbb{E}_{s^{\prime},a^{\prime}|s,a}\left[\phi(s^{\prime},a^{\prime})\right]=V\psi(s,a)$, where $\psi(s,a)$ is another feature map and $V$ is the parameter matrix to be learned.

In stage 1, $V$ is learned by minimizing the following loss,

$\displaystyle\hat{V}=\operatorname*{arg\,min}_{V}\mathcal{L}_{1}(V),\quad\mathcal{L}_{1}(V)=\mathbb{E}_{s,a,s^{\prime},a^{\prime}}\left[\|\phi(s^{\prime},a^{\prime})-V\psi(s,a)\|^{2}\right]+\lambda_{1}\|V\|^{2},$

(23)

where $\lambda_{1}>0$ is a regularization parameter. This is a linear ridge regression problem with multiple targets, which can be solved analytically. In stage 2, given $\hat{V}$, $\theta$ is obtained by minimizing the loss

$\displaystyle\hat{\theta}=\operatorname*{arg\,min}_{\theta}\mathcal{L}_{2}(\theta),\quad\mathcal{L}_{2}(\theta)=\mathbb{E}_{r,s,a}\left[\|r-\theta^{\top}(\phi(s,a)-\hat{V}\psi(s,a))\|^{2}\right]+\lambda_{2}\|\theta\|^{2},$

(24)

where $\lambda_{2}>0$ is another regularization parameter. Stage 2 corresponds to a ridge linear regression from $\phi(s,a)-\hat{V}\psi(s,a)$ to $r$, and also has a closed-form solution.

In KIV, from the characteristics of RKHS functions one can learn a non-linear $Q$ function while retaining the closed-form solution of the two-stage regressions. In DFIV, because of the use of adaptive features for $\phi(s,a)$ and $\psi(s,a)$ parameterized with neural networks they learn those features by alternating the two regression stages, which enables one to learn a more flexible $Q$ function compared to KIV. In this paper, we consider a variant of DFIV that regresses $\phi(s,a)-\gamma\phi(s^{\prime},a^{\prime})$ instead of $\phi(s^{\prime},a^{\prime})$ on instrumental variables in stage 1, with details explained in Appendix A. We found this approach is more stable than the original version in the experiments. While we can derive a similar variant for KIV, we did not notice an improvement in performance.

A family of non-linear IV methods is based on the moment restrictions derived from 1 of instrumental variables,

$$\mathbb{E}(\varepsilon|Z)=\mathbb{E}(Y-f(X)|Z)=0,\forall Z.$$

(25)

In the linear setting, we require the following unconditional moment to be zero, whose solution is Equation 11,

$$\mathbb{E}(\varepsilon Z)=\mathbb{E}[(Y-f(X))Z]=0.$$

(26)

In the non-linear setting, by defining a set of potentially infinitely many test functions of $Z$, $g\in\mathcal{G}$, we require all the unconditional moments to be zero,

$$\Psi(f,g)=\mathbb{E}_{X,Y,Z}\left[(Y-f(X))g(Z)\right]=0,\forall g.$$

(27)

The unified solution in the family of Generalized Method of Moments (GMM) (Hansen, 1982) is a saddle-point of the following minimax objective function,

$$f^{*}=\underset{f\in\mathcal{F}}{\arg\inf}\sup_{g\in\mathcal{G}}\Psi_{n}(f,g)+R_{f}(f)-R_{g}(g)\,,$$

(28)

where $g$ is optimized to find the largest violation of the moment condition for the current estimate of $f$ and the expectation in $\Psi$ is the empirical estimate from a dataset of size $n$. $R_{f}(f)$ and $R_{g}(g)$ are regularization terms for $f$ and $g$, respectively, for identifiability.

In the context of RL, this objective is thus given by

	$\displaystyle Q^{*}$	$\displaystyle=\underset{Q\in\mathcal{Q}}{\arg\inf}\sup_{g\in\mathcal{G}}\Psi_{n}(Q,g)+R_{f}(Q)-R_{g}(g)\,,$
	$\displaystyle\text{with }\Psi_{n}(Q,g)$	$\displaystyle=\mathbb{E}_{s,a\sim\mu_{b},s^{\prime}\sim P,r\sim R,a^{\prime}\sim\pi}\left[(r-Q(s,a)+\gamma Q(s^{\prime},a^{\prime}))g(s,a)\right].$		(29)

Here $g$ acts to find the largest moment of the TD error. Variants of GMM method differ in the choice of the function space $\mathcal{Q}$, $\mathcal{G}$ and the regularization functions.

Let us consider a simple MDP with 100 discrete states allocated uniformly along the interval $[-2,2]$: $s_{i}=-2+\frac{4}{100}i,i\in\{0,1,\dots,99\}$. The agent always starts at the first state $s_{0}$ in every episode and terminates at the last state $s_{99}$. There is only a single action, $a=\text{right}$, in every state to move to right, and therefore the policy is always fixed. The state is transitioned to the right neighboring state with a probability of $p>0$ or stays in the same location. It is easy to show that the resulting state distribution $\mu(s)$ pooled from different steps across the trajectory, i.e. the offline data distribution, is uniform among all the non-terminating states $i\in\{0,1,\dots,98\}$, whatever the value of $p$ and $\mu(s_{99})=p\mu(s_{98})$. The reward function is defined with a Gaussian kernel as $R=\exp(-\frac{s^{2}}{0.2^{2}})$, and is illustrated in Figure 4 together with $Q(s,a=\text{right})$.

As there is only a single policy in this environment, the target policy is the same as the behavior policy. We sample a dataset of $N=10^{5}$ transitions with $p=0.5$, and estimate the state value using a fixed set of $D=90$ Gaussian kernel features $\phi_{j}(s)=\exp\left(\frac{(s-(-2+(4/D)j))^{2}}{0.1^{2}}\right),j=\{0,1,\dots,89\}$. For this linear instrumental variable regression problem, we compare the LSTD-Q method in Section 3 with DBRM, which reduces to a naive least square minimization algorithm in this case, and FQE, which alternates solving a least squared minimization in Equation 18 with the linear weights $\theta$ in $Y$ being fixed and replacing those weights with the solution.

The estimates of the $Q$ function for all methods are shown in Figure 4. The estimate of LSTD-Q matches the ground truth value very well, while DBRM - which ignores the confounding problem - leads to a heavily biased estimate as expected. The estimate of FQE also matches the ground truth, but requires multiple iterations to converge to the solution as shown in Figure 6 (right). This is because the TD formulation of FQE can only propagate the state value information along the reverse order of state dynamics $s^{\prime}\rightarrow s$ by one step at every iteration.

Next, we conduct an ablation study to investigate how the advantage of IV method depends on the following four variables: dataset size $N$, feature dimensions $D$, transition randomness ($1-p$), and the extent of off-policyness. While the target policy always matches the behavior policy in this problem, we create an offline dataset with a shifted distribution by sampling the states with the following distribution $\mu(s)\propto\exp(\alpha s)$, where $\alpha=0$ corresponds to the original uniform distribution, and a larger value of $\alpha$ leads to a shifted distribution towards the right end of the state space.

We display the absolute error of the estimated state value at the initial state $Q(s_{0},a=right)$ by LSTD-Q and DBRM in Figure 5. The error of FQE is not shown because it converges to the same solution as LSTD-Q in the linear case. We find the error of LSTD-Q increases and eventually becomes on par with DBRM when we reduce the size of the dataset, or increase the data distribution shift, which also decreases the effective dataset size. The error of LSTD-Q also increases when we reduce the number of features, which will lead to model misspecification, and could violate the IV requirement that the residual needs to have zero mean. Lastly, we see that DBRM is unbiased when the dynamics are deterministic, $p=1$, but the bias increases quickly when $p$ decreases. In contrast, the error of LSTD-Q increases but much more slowly, which we suspect is due to an increasingly diverse distribution of transitions $(s,s^{\prime})$ that requires more data to learn the estimate accurately.

When we use a non-linear value function, the solutions given from IV techniques do not coincides with FQE, and both methods require iterative optimization. Nonetheless, the policy evaluation algorithms based on non-linear IV (e.g. DFIV) preserve the relatively faster convergence rate than FQE in this example, as shown in Figure 6 (right). Here we estimate the value function from the raw state $s$ and estimate an multi-layer perceptron (MLP) with two hidden layers, each with 50 units and ReLU activation function. Both methods use a learning rate of $10^{-4}$.

From this ablation study, we demonstrate that the advantages of using an IV method over a simple method like DBRM and FQE can be significant, but that the magnitude of this benefit depends on multiple variables. On a more complex and comprehensive OPE benchmark, we study in the following section whether recent non-linear IV methods are competitive with state-of-the-art OPE methods such as FQE and Distributional FQE (Fu et al., 2021).

We compare a list of representative non-linear IV methods, including Kernel IV (KIV), Deep IV, Deep Feature IV (DFIV) and three adversarial IV methods: Deep GMM, Adversarial GMM Networks (AGMM), Adversarial approach to structural equation models (ASEM). We also include as baselines the deterministic Bellman residual minimization (DBRM) and two variants of the fitted Q evaluation methods with a deterministic (FQE) and distributional (DFQE) Q representation respectively. (D)FQE was shown to be the best performing OPE algorithm in a recent benchmark paper (Fu et al., 2021) under a similar task setting as in this paper.

All algorithms except KIV use the same network architecture to estimate the Q function as in the trained agent for a fair comparison. For BSuite tasks, the $Q$ network is an MLP with layer size 50-50-1 and ReLU activation. The input is a concatenation of the flattened observation and one-hot encoding of the discrete action variable. For DM Control tasks, it is an MLP with layer size 512-512-256-1, ELU activation and a layer normalization after the first hidden layer. The input is the concatenation of the flattened observation and action variables. The architecture of additional networks in each algorithm, such as the generative model in Deep IV and the adversarial function network in AGMM, ASEM and DeepGMM are selected as part of the hyper-parameter search procedure that will be discussed later. We use OAdam for adversarial methods as suggested by Bennett et al. (2019b); Dikkala et al. (2020) and Adam for other methods.

We compare all the algorithms with respect to the accuracy of estimating the target policy value $\rho(\pi)$ (Eq. 2), i.e., the expected cumulative discounted reward from the initial state distribution. The estimate is computed as

$$\hat{\rho}(\pi)=\mathbb{E}_{s\sim\mu_{0},a|s\sim\pi}[\hat{Q}^{\pi}(s,a)],$$

(40)

where $\hat{Q}^{\pi}$ is given by each algorithm under comparison. We normalize the policy value into a range of $[0,1]$ for ease of comparison across environments:

$$\rho_{\mathrm{Norm}}=\frac{\rho-\rho_{\mathrm{min}}}{\rho-\rho_{\mathrm{max}}}$$

(41)

where $\rho_{\min/\max}:=-1/1$ for BSuite Catch and $\rho_{\min/\max}:=R_{\min/\max}\sum_{t=0}^{1000}\gamma^{t}$ for other environments. We measure the accuracy in terms of the absolute error $|\hat{\rho}_{\mathrm{Norm}}-\rho_{\mathrm{Norm}}|$ in this paper. Other metrics such as the policy ranking correlation and regret have been considered in the literature (Paine et al., 2020; Fu et al., 2021) when multiple target policies are available in the same environment. Our experiment setup does not meet that condition and those metrics are hence not included.