Convergence of a Human-in-the-Loop Policy-Gradient Algorithm With Eligibility Trace Under Reward, Policy, and Advantage Feedback

In this section we show that the gradient of the objective function $\Sigma_{t=0}^{\infty}\gamma^{t}\mathbb{E}[\frac{\nabla_{\theta}\pi_{\theta}(s_{t},a_{t})}{\pi_{\theta}(s_{t},a_{t})}Q^{\pi}(s_{t},a_{t})]$ is what REINFORCE performs gradient ascent on. Note that this quantity is what E-COACH is estimating (via $\theta_{L}$ parameter) and performing gradient ascent on.

Consider the REINFORCE algorithm (Algorithm 2). Here, $G_{t}$ is a Monte Carlo estimate of $Q^{\pi}(s_{t},a_{t})$. Hence, at any terminal time $L$, it is clear that the expected value of $\theta_{L}$ obtained by REINFORCE is equal to that of E-COACH.

Consider the unnormalised state visitation distribution, such that $\forall s\in S$, $d^{\pi}(s)=\mathbb{P}_{0}^{\pi}(s)+\gamma\mathbb{P}_{1}^{\pi}(s)+\ldots+\gamma^{i}\mathbb{P}_{i}^{\pi}(s)+\ldots$ where $\mathbb{P}_{t}^{\pi}(s)$ denotes the probability of arriving in state $s$ at time $t$ following policy $\pi$. The objective to maximize, as described in the Policy Gradient Theorem (Sutton et al., 2000), is

and its gradient is

The gradient can be rewritten as $\nabla_{\theta}\rho^{\pi}=\mathbb{E}_{s\sim d^{\pi},a\sim\pi(s,\cdot)}[\frac{\nabla_{\theta}\pi_{\theta}(s_{t},a_{t})}{\pi_{\theta}(s_{t},a_{t})}Q^{\pi}(s,a)]$.

Expanding with respect to $d^{\pi}(s)$ yields

which is exactly what E-COACH and REINFORCE are estimating. Since E-COACH is performing gradient ascent on the policy gradient objective, we can use results from (Agarwal et al., 2020) to say that E-COACH converges to local optima or saddle points. Although, recent work has shown that policy gradient methods can escape saddle points under mild assumptions on the rewards and minor modifications to existing algorithms (Zhang et al., 2020) (Jin et al., 2017).
$\Box$

Something to note is that behavioral evidence indicates that one-step reward is not a typical choice of human trainers (Ho et al., 2019).

Let us start by considering two similar MDPs $M_{1}=\langle S,A,R,T,\gamma\rangle$ and $M_{2}=\langle S,A,f,T,\gamma\rangle$. Note the differing reward functions $R$ and $f$ in the two MDPs.

We will denote the value functions for $M_{1}$ and $M_{2}$ as $V_{1}$ and $V_{2}$, respectively. We will say that the starting state for both of our MDPs is $s_{0}$. Define $V_{1}^{\min}=\min_{\pi\in\Pi}V_{1}^{\pi}(s_{0})$, $V_{1}^{*}=\max_{\pi\in\Pi}V_{1}^{\pi}(s_{0})$.

The following theorem will have the following assumption:

1. 1.

   E-COACH (see algorithm 1) will give us a policy $\pi_{2}(s,a)$ such that $\mathbb{E}_{s\sim d^{\pi_{2}^{*}}}\big{[}\sum_{a}\big{|}\pi_{2}^{*}(s,a)-\pi_{2}(s,a)\big{|}\big{]}\leq\delta$ for some optimal policy $\pi_{2}^{*}$ on the domain $M_{2}$. The proof in section 3 strengthens this assumption by showing that E-COACH optimizes the policy gradient objective. Note $\pi_{2}^{*}$ may not be the only optimal policy; instead, it is a single optimal policy.

2. 2.

   We also assume that $\gamma\neq 1$ for the case where the MDP has an infinite horizon, which will we will justify later on.

Theorem 4.1 requires the condition that all optimal policies for $M_{2}$ are also optimal for $M_{1}$. We will later show that this condition holds true for the case of policy feedback in theorem 4.2, allowing us to leverage these results.

Theorem 4.1: If all optimal policies for $M_{2}$ are also optimal for $M_{1}$ (optimal policies of $M_{2}$ are a subset of those for $M_{1}$), then running E-COACH on $M_{2}$ will result in a policy that is close to an optimal policy on $M_{2}$, which will also be close to an optimal policy for $M_{1}$. Let’s define $W=\max(|V_{1}^{*}|,|V_{1}^{\min}|)$. Then we find that,

Proof: We have to show that running E-COACH in $M_{2}$ will yield a policy that is not too far off from an optimal policy for $M_{1}$. We would like to run E-COACH on $M_{2}$, using the alternate form of feedback as the reward function, and for any good policy (as per assumption 1) we get from E-COACH on $M_{2}$, we would like for that policy to also be good on $M_{1}$, the original MDP we are trying to solve.

The lower-bound in the theorem statement is immediate.

For the upper-bound, let’s let $\pi^{(n)}$ denote a policy that follows/simulates $\pi_{2}^{*}$ for the first $n-1$ time-steps and $\pi_{2}$ for the rest. Hence, on the $n^{\textnormal{th}}$ time-step, $\pi^{(n)}$ will follow/simulate $\pi_{2}$ and not $\pi_{2}^{*}$. Let $V^{(n)}$ denote the value of policy $\pi^{(n)}$. Therefore, we can say that $V_{1}^{\pi_{2}}=V^{(0)}$ and $V_{1}^{*}=V^{(\infty)}$. Remember that $\pi_{2}^{*}$ is optimal on $M_{1}$ and $M_{2}$ by the condition above, and thus has value $V^{*}$.

We’ll start by considering $V^{(t)}-V^{(t-1)}$. Both $\pi^{(t)}$ and $\pi^{(t-1)}$ accumulate the same expected reward for the first $t-2$ steps and so these rewards cancel out. Note that the $\mathbb{P}$ we use below is the same as that defined in section 3.1. We find the following:

Now we’ll use the above fact when considering $V_{1}^{*}-V_{1}^{\pi_{2}}$.

As $\delta\rightarrow 0$, we have that $V_{1}^{*}-V_{1}^{\pi_{2}}\rightarrow 0$.
$\Box$

Note that our theorem says something different than the Simulation Lemma (Kearns & Singh, 1998) as we make no assumptions about how close the reward functions of $M_{1}$ and $M_{2}$ are. Instead our theorem requires optimal policies in $M_{2}$ be optimal in $M_{1}$ and bounds the return of a policy learnt by E-COACH in $M_{2}$.

Let $M_{1}=\langle S,A,R,T,\gamma\rangle$ be an MDP without any specific reward function. Under policy feedback, a trainer has a target stationary deterministic policy $\pi^{*}_{1}$ in mind and delivers feedback based on whether the trainer’s decision agrees with $\pi^{*}_{1}$. When an agent takes an action $a$ in state $s$, the trainer will give feedback

with $I(s,a)$ defined as,

Theorem 4.2: E-COACH converges under feedback $f(s,a)=I(s,a),\forall\>s\times a\in S\times A$.

Proof: Consider the case of replacing the reward function $R(s,a)$ with $I(s,a)$ in MDP $M_{1}$, constructing a new MDP $M_{2}=\langle S,A,f,T,\gamma\rangle$. We would like to show that, in this setting, the E-COACH algorithm converges to the optimal solution. $M_{1}$ and $M_{2}$ satisfy the prerequisites for theorem 4.1.

Consider the optimal policy for $M_{2}$. The best policy will select the best action in every state. We have that $V^{*}_{2}(s_{0})=\sum_{i=0}^{\infty}1\cdot\gamma^{i}$. The optimal policy for $M_{2}$ will achieve this value function because, if not, then we have a policy such that $V_{2}^{\prime}(s_{0})=\sum_{i=0}^{\infty}t(i)\cdot 1\cdot\gamma^{i}$, where $t(i)\in\{0,1\}\forall\>i$ and $t(j)=0$ for some $j$. Take the smallest value $k\in\mathbb{N}$ such that $t(k)=0$, then $V^{*}_{2}(s_{0})-V_{2}^{\prime}(s_{0})\geq\gamma^{k}$ so then the policy achieving $V_{2}^{\prime}$ is sub-optimal. We can conclude that $V^{*}_{2}(s_{0})$ is the value function for the optimal policy. Therefore, the policy that always chooses the action that gives a value of one is optimal. Also, note that always choosing the action that results in a feedback of one corresponds exactly to the decision of $\pi_{1}^{*}$ by construction of $f(s,a)$. So, we obtain that $\pi_{1}^{*}(s,a)=\pi_{2}^{*}(s,a),\forall\>(s,a)\in S\times A$. In other words, an optimal policy in the new domain is equivalent to the target policy from the original one.

We can leverage Theorem 4.1 to show that the algorithm converges under policy feedback.
$\Box$

The algorithm will converge, but the policy it converges to will be suboptimal for $\gamma\neq 1$ because COACH does not estimate the gradient of the policy gradient objective, $\nabla_{\theta}\rho$. Because COACH does not incorporate the discount factor, it behaves as if the domain has $\gamma=1$, even if it isn’t necessarily the case. If the domain has a $\gamma\in[0,1)$, then the policy will have a long-term view because it will ignore this discount factor. Ignoring discounts can lead to suboptimal behavior.

Consider the five-state domain in Figure 2. It shows how the optimal decision in a state can change with the discount factor.

COACH will converge, but to a poor performing policy for the same reason given in Section 6.1.

COACH should converge under this feedback type as per the argument given by MacGlashan et al..

TAMER expects the human trainer to take each action’s long-term implications into account when providing feedback. TAMER learns the trainer’s feedback function, then returns the policy that maximizes one-step feedback in each state.

The pseudocode as described in algorithm 3 is the TAMER algorithm. See (Knox & Stone, 2008) for more details. The $t$ represents the time, the weights $\overrightarrow{w}$ is used for the reward model, and the feature vectors $\overrightarrow{f_{t-2}}$ and $\overrightarrow{f_{t-1}}$ are state feature vectors. It takes input $\alpha$, a learning rate.

Note that there are several different versions of TAMER. The one we are analyzing is the original by Knox & Stone (2008).

Q-learning (Watkins, 1989) is an algorithm that expects feedback in the form of immediate reward and calculates long-term value from these signals. Specifically, $Q_{k}(s,a)$ is its estimate of long-term value and, when it is informed of a transition from $s_{t}$ to $s_{t+1}$ via action $a_{t}$ and feedback $f_{t}$, it makes the update: