The Reinforce Policy Gradient Algorithm Revisited

Policy gradient methods [25, 26] are a popular class of approaches in reinforcement learning. The policy in these approaches is considered parameterized and one updates the policy parameter along a gradient search direction where the gradient is of a performance objective, normally the value function. The policy gradient theorem [26, 20, 10] which is a fundamental result in these approaches relies on an interchange of the gradient and expectation operators and in such cases turns out to be the expectation of the gradient of noisy performance functions much like the previously studied perturbation analysis based sensitivity approaches for optimization via simulation [13, 11].

The Reinforce algorithm [27, 25] is a noisy gradient scheme for which the expectation of the gradient is the policy gradient, i.e., the gradient of the expected objective w.r.t. the policy parameters. The updates of the policy parameter are however obtained once after the full return on an episode has been found. Actor-critic algorithms [25, 16, 17, 7, 6] have been presented in the literature as alternatives to the Reinforce algorithm as they perform incremental parameter updates at every instant but do so using two-timescale stochastic approximation algorithms.

In this paper, we revisit the Reinforce algorithm and present a new algorithm for the case of episodic tasks or the stochastic shortest path setting. Our algorithm performs parameter updates upon termination of episodes, that is when goal or terminal states are reached. In this setting, as mentioned above, updates are performed only at instants of visit to a prescribed recurrent state [10, 20]. This algorithm is based on a single function measurement or simulation at a perturbed parameter value where the perturbations are obtained using independent Gaussian random variates.

Gradient estimation in this algorithm is performed using the smoothed functional (SF) technique for gradient estimation [22, 4, 3, 5]. The basic problem in this setting is the following: Given an objective function $J:\mathcal{R}^{d}\rightarrow\mathcal{R}$ such that $J(\theta)=E_{\xi}[h(\theta,\xi)]$, where $\theta\in\mathcal{R}^{d}$ is the parameter to be tuned and $\xi$ is the noise element, the goal is to find $\theta^{*}\in\mathcal{R}^{d}$ such that

Since the objective function $J(\cdot)$ can be highly nonlinear, one often settles for a lesser goal – that of finding a local instead of a global minimum. In this setting, the Kiefer-Wolfowitz [15] finite difference estimates for the gradient of $J$ would correspond to the following: For $i=1,\ldots,d$,

where $e_{i}$ is the unit vector with 1 in the $i$th place and $0$ elsewhere. Further, $\xi^{+}_{i}$ and $\xi^{-}_{i}$, $i=1,\ldots,d$ are independent noise random variables having a common distribution. The expectation above is taken w.r.t this common distribution on the noise random variables. This approach does not perform well in practice when $d$ is large, since one needs $2d$ function measurements or simulations for a parameter update.

Random search methods such as simultaenous perturbation stochastic approximation (SPSA) [23, 24, 2], smoothed functional (SF) [14, 4, 3] or random directions stochastic approximation (RDSA) [18, 21] typically require much less number of simulations. For instance, the gradient based algorithms in these approaches require only one or two simulations regardless of the parameter dimension $d$, while their Newton-based counterparts usually involve one to four system simulations for any parameter update (again regardless of $d$). A textbook treatment of random search approaches for stochastic optimization is available in [5].

We consider here a one-simulation SF algorithm where the gradient of $J(\theta)$ is estimated using a noisy function measurement, at the parameter $\theta+\delta\Delta$, where $\delta>0$ and $\Delta\stackrel{{\scriptstyle\triangle}}{{=}}(\Delta_{1},\ldots,\Delta_{d})^{T}$ with each $\Delta_{i}\sim N(0,1)$ with $\Delta_{i}$ being independent of $\Delta_{j}$, $\forall j\not=i$. Further, $\Delta$ is independent of the measurement noise as well. The gradient estimate in this setting is the following: For $i=1,\ldots,d$,

In the above, $\xi$ denotes the measurement noise random variable. The expectation above is with respect to the joint distribution of $\xi$ and $\Delta$. Before we proceed further, we present the basic Markov decision process (MDP) setting as well as recall the Reinforce algorithm that we consider for the episodic setting. We remark here that there are not many analyses of Reinforce type algorithms in the literature in the episodic or stochastic shortest path setting.

By a Markov decision process, we mean a controlled stochastic process $\{X_{n}\}$ whose evolution is governed by an associated control-valued sequence $\{Z_{n}\}$. It is assumed that $X_{n},n\geq 0$ takes values in a set $S$ called the state-space. Let $A(s)$ be the set of feasible actions in state $s\in S$ and ${\displaystyle A\stackrel{{\scriptstyle\triangle}}{{=}}\cup_{s\in S}A(s)}$ denote the set of all actions. When the state is say $s$ and a feasible action $a$ is chosen, the next state seen is $s^{\prime}$ with a probability $p(s^{\prime}|s,a)\stackrel{{\scriptstyle\triangle}}{{=}}P(X_{n+1}=s^{\prime}\mid X_{n}=s,Z_{n}=a)$, $\forall n$. We assume these probabilities do not depend on $n$. Such a process satisfies the controlled Markov property, i.e.,

By an admissible policy or simply a policy, we mean a sequence of functions $\pi=\{\mu_{0},\mu_{1},\mu_{2},\ldots\}$, with $\mu_{i}:S\rightarrow A$, $i\geq 0$, such that $\mu_{i}(s)\in A(s)$, $\forall s\in S$. The policy $\pi$ is a decision rule which specifies that if at instant $k$, the state is $i$, then the action chosen under $\pi$ by the decision maker would be $\mu_{k}(i)$. A stationary policy $\pi$ is one for which $\mu_{k}=\mu_{l}\stackrel{{\scriptstyle\triangle}}{{=}}\mu$, $\forall k,l=0,1,\ldots$. In other words, under a stationary policy, the function that decides the action-choice in a given state does not depend on the instant $n$ at which the action is chosen.

Associated with any transition to a state $s^{\prime}$ from a state $s$ under action $a$, is a ‘single-stage’ cost $g(s,a,s^{\prime})$ where $g:S\times A\times S\rightarrow\mathcal{R}$ is called the cost function. The goal of the decision maker is to select actions $a_{k},k\geq 0$ in response to the system states $s_{k},k\geq 0$, observed one at a time, so as to minimize a long-term cost objective. We assume here that the number of states and actions is finite.

We consider here the case of episodic problems and the model-free setting whereby we do not assume any knowledge of the system model, i.e., the transition probabilities $p(s^{\prime}\mid s,a)$, and in their place, we assume that we have access to data (either real or simulated). The data that is available is over trajectories of states, actions, single-stage costs and next states until termination. We assume that multiple trajectories of data can be made available and the data on the $m$th trajectory can be represented in the form of the tuples $(s^{m}_{k},a^{m}_{k},g^{m}_{k},s^{m}_{k+1})$, $k=0,1,\ldots,T_{m}$ with $T_{m}$ being the termination instant on the $m$th trajectory, $m\geq 1$. Also, $s^{m}_{j}$ is the state at instant $j$, $j=k,k+1$ in the $m$th trajectory. Further, $a^{m}_{k}$ and $g^{m}_{k}$ are the action chosen and the cost incurred, respectively, at instant $k$ in the $m$th trajectory. As mentioned before, we consider a class of stationary randomized policies that are parameterized by $\theta$ and satisfy Assumption 1.

Let $\Gamma:{\cal R}^{d}\rightarrow C$ denote a projection operator that projects any $x=(x_{1},\ldots,x_{d})^{T}\in{\cal R}^{d}$ to its nearest point in $C$. Thus, if $x\in C$, then $\Gamma(x)\in C$ as well. For ease of exposition, we assume that $C$ is a $d$-dimensional rectangle having the form ${\displaystyle C=\prod_{i=1}^{d}[a_{i,\min},a_{i,\max}]}$, where $-\infty<a_{i,\min}<a_{i,\max}<\infty$, $\forall i=1,\ldots,d$. Then $\Gamma(x)=(\Gamma_{1}(x_{1}),\ldots,\Gamma_{d}(x_{d}))^{T}$ with $\Gamma_{i}:{\cal R}\rightarrow[a_{i,\min},a_{i,\max}]$ such that ${\displaystyle\Gamma_{i}(x_{i})=\min(a_{i,\max},\max(a_{i,\min},x))}$, $i=1,\ldots,d$. Also, let ${\cal C}(C)$ denote the space of all continuous functions from $C$ to ${\cal R}^{d}$.

In what follows, we present a procedure that incrementally updates the parameter $\theta$. Let $\theta(n)$ denote the parameter value obtained after the $n$th update of this procedure which depends on the $n$th episode and which is run using the policy parameter $\Gamma(\theta(n)+\delta_{n}\Delta(n))$, for $n\geq 0$, where $\theta(n)=(\theta_{1}(n),\ldots,\theta_{d}(n))^{T}\in\mathcal{R}^{d}$, $\delta_{n}>0$ $\forall n$ with $\delta_{n}\rightarrow 0$ as $n\rightarrow\infty$ and $\Delta(n)=(\Delta_{1}(n),\ldots,\Delta_{d}(n))^{T},n\geq 0$, where $\Delta_{i}(n),i=1,\ldots,d,n\geq 0$ are independent random variables distributed according to the $N(0,1)$ distribution.

Algorithm (5) below is used to update the parameter $\theta\in C\subset\mathcal{R}^{d}$. Let $\chi^{n}$ denote the $n$th state-action trajectory $\chi^{n}=\{s^{n}_{0},a^{n}_{0},s^{n}_{1},a^{n}_{1},\ldots,s^{n}_{T-1},a^{n}_{T-1},s^{n}_{T}\}$, $n\geq 0$ where the actions $a^{n}_{0},\ldots,a^{n}_{T-1}$ in $\chi^{n}$ are obtained using the policy parameter $\theta(n)+\delta_{n}\Delta(n)$. The instant $T$ denotes the termination instant in the trajectory $\chi^{n}$ and corresponds to the instant when the terminal or goal state $t$ is reached.. Note that the various actions in the trajectory $\chi^{n}$ are chosen according to the policy $\phi_{(\theta(n)+\delta_{n}\Delta(n))}$. The initial state is assumed to be sampled from a given initial distribution $\nu=(\nu(i),i\in S)$ over states.

Let ${\displaystyle G^{n}=\sum_{k=0}^{T-1}g^{n}_{k}}$ denote the sum of costs until termination on the trajectory $\chi^{n}$, with $g^{n}_{k}\equiv g(X^{n}_{k},Z^{n}_{k},X^{n}_{k+1})$ and where the superscript $n$ on the various quantities indicates that these correspond to the $n$th episode. The update rule that we consider here is the following: For $n\geq 0,i=1,\ldots,d$,

We assume that the step-sizes $a(n),n\geq 0$ satisfy the following requirement:

Once the $n$th update of the parameter (i.e., $\theta(n)$) is obtained, the perturbed parameter $\theta(n)+\delta_{n}\Delta(n)$ is obtained after sampling $\Delta(n)$ from the multivariate Gaussian distribution as explained previously and thereafter a new trajectory governed by this perturbed parameter is generated with the initial state in each episode sampled according to a given distribution $\nu$. This is because each episode ends in the terminal state $t$ and any fresh episode starts from a non-terminal initial state sampled from the distribution $\nu$.

We begin by rewriting the recursion (5) as follows:

where $M^{i}_{n+1}=\Delta_{i}(n)\frac{G^{n}}{\delta_{n}}-E\left[\Delta_{i}(n)\frac{G^{n}}{\delta_{n}}|\mathcal{F}_{n}\right],n\geq 0.$ Here, we let $\mathcal{F}_{n}\stackrel{{\scriptstyle\triangle}}{{=}}\sigma(\theta(m),m\leq n,\Delta(m),\chi^{m},m<n),n\geq 1$ as a sequence of increasing sigma fields with $\mathcal{F}_{0}=\sigma(\theta(0))$. Let $M_{n}\stackrel{{\scriptstyle\triangle}}{{=}}(M^{1}_{n},\ldots,M^{d}_{n})^{T}$, $n\geq 0$.

In the light of Proposition 1, we can rewrite (5) as follows:

where ${\displaystyle\beta_{i}(n)=E\left[\Delta_{i}(n)\frac{G_{n}}{\delta}\mid\mathcal{F}_{n}\right]}$ $-\sum_{s}\nu(s)\nabla_{i}V_{\theta(n)}(s)$ and $\beta(n)=(\beta_{1}(n),\ldots,\beta_{d}(n))^{T}$. From Proposition 1, it follows that $\beta(n)=o(\delta_{n})$.

Define now a sequence $Z_{n},n\geq 0$ according to

$n\geq 1$ with $Z_{0}=0$.

Consider now the following ODE:

where $\bar{\Gamma}:{\cal C}(C)\rightarrow{\cal C}({\cal R}^{d})$ is defined according to

Let $H\stackrel{{\scriptstyle\triangle}}{{=}}\{\theta\mid\bar{\Gamma}(-\sum_{s}\nu(s)\nabla V_{\theta}(s))=0\}$ denote the set of all equilibria of (9). By Lemma 11.1 of [9], the only possible $\omega$-limit sets that can occur as invariant sets for the ODE (9) are subsets of $H$. Let $\bar{H}\subset H$ be the set of all internally chain recurrent points of the ODE (9). Our main result below is based on Theorem 5.3.1 of [18] for projected stochastic approximation algorithms. Before we proceed further, we recall that result below.

Let $C\subset{\cal R}^{d}$ be a compact and convex set as before and $\Gamma:{\cal R}^{d}\rightarrow C$ denote the projection operator that projects any $x=(x_{1},\ldots,x_{d})^{T}\in{\cal R}^{d}$ to its nearest point in $C$.

Consider now the following the $d$-dimensional stochastic recursion

under the assumptions listed below. Also, consider the following ODE associated with (11):

Let ${\cal C}(C)$ denote the space of all continuous functions from $C$ to ${\cal R}^{d}$. The operator $\bar{\Gamma}:{\cal C}(C)\rightarrow{\cal C}({\cal R}^{d})$ is defined according to

for any continuous $v:C\rightarrow{\cal R}^{d}$. The limit in (13) exists and is unique since $C$ is a convex set. In case this limit is not unique, one may consider the set of all limit points of (13). Note also that from its definition, $\bar{\Gamma}(v(x))=v(x)$ if $x\in C^{o}$ (the interior of $C$). This is because for such an $x$, one can find $\eta>0$ sufficiently small so that $x+\eta v(x)\in C^{o}$ as well and hence $\Gamma(x+\eta v(x))=x+\eta v(x)$. On the other hand, if $x\in\partial C$ (the boundary of $C$) is such that $x+\eta v(x)\not\in C$, for any small $\eta>0$, then $\bar{\Gamma}(v(x))$ is the projection of $v(x)$ to the tangent space of $\partial C$ at $x$.

Consider now the assumptions listed below.

• (B1)

  The function $h:{\cal R}^{d}\rightarrow{\cal R}^{d}$ is continuous.

• (B2)

  The step-sizes $a(n),n\geq 0$ satisfy

  $$a(n)>0\forall n,\mbox{ }\sum_{n}a(n)=\infty,\mbox{ }a(n)\rightarrow 0\mbox{ as }n\rightarrow\infty.$$

• (B3)

  The sequence $\beta_{n},n\geq 0$ is a bounded random sequence with $\beta_{n}\rightarrow 0$ almost surely as $n\rightarrow\infty$.

• (B4)

  There exists $T>0$ such that $\forall\epsilon>0$,

  $$\lim_{n\rightarrow\infty}P\left(\sup_{j\geq n}\max_{t\leq T}\left|\sum_{i=m(jT)}^{m(jT+t)-1}a(i)\xi_{i}\right|\geq\epsilon\right)=0.$$

• (B5)

  The ODE (12) has a compact subset $K$ of ${\cal R}^{N}$ as its set of asymptotically stable equilibrium points.

Let $t(n),n\geq 0$ be a sequence of positive real numbers defined according to $t(0)=0$ and for $n\geq 1$, ${\displaystyle t(n)=\sum_{j=0}^{n-1}a(j)}$. By Assumption (B2), $t(n)\rightarrow\infty$ as $n\rightarrow\infty$. Let ${\displaystyle m(t)=\max\{n\mid t(n)\leq t\}}$. Thus, $m(t)\rightarrow\infty$ as $t\rightarrow\infty$. Assumptions (B1)-(B3) correspond to A5.1.3-A5.1.5 of [18] while (B4)-(B5) correspond to A5.3.1-A5.3.2 there.