On the Stochastic (Variance-Reduced) Proximal Gradient Method for Regularized Expected Reward Optimization

Reinforcement learning (RL) Sutton & Barto (2018) has recently become a highly active research area of machine learning that learns to make sequential decisions via interacting with the environment. In recent years, RL has achieved tremendous success so far in many applications such as control, job scheduling, online advertising, and game-playing Zhang & Dietterich (1995); Pednault et al. (2002); Mnih et al. (2013), to mention a few. One of the central tasks of RL is to solve a certain (expected) reward optimization problem for decision-making. Following the research theme, we consider the following problem of maximizing the regularized expected reward:

where $\mathcal{G}:\mathbb{R}^{n}\to\mathbb{R}\cup\{+\infty\}$ is a closed proper convex (possibly nonsmooth) function, $x\in\mathbb{R}^{d}$, $\mathcal{R}_{\theta}:\mathbb{R}^{d}\to\mathbb{R}$ is the reward function depending on the parameter $\theta$, and $\pi_{\theta}$ denotes the probability distribution over a given subset $\mathcal{S}\subseteq\mathbb{R}^{d}$ parameterized by $\theta\in\mathbb{R}^{n}$. By adapting the convention in RL, we call $\pi_{\theta}$ a policy parameterized by $\theta$. Moreover, for the rest of this paper, we denote $\mathcal{J}(\theta):=\mathbb{E}_{x\sim\pi_{\theta}}\left[\mathcal{R}_{\theta}(x)\right]$ as the expected reward function in the non-oblivious setting. The learning objective is to learn a decision rule via finding the policy parameter $\theta$ that maximizes the regularized expected reward. To the best of our knowledge, the study on the general model (1) has been limited in the literature. Hence, developing and analyzing algorithmic frameworks for solving the problem is of great interest.

There are large body of works in supervised learning focusing on the oblivious setting Zhang (2004); Hastie et al. (2009); Shapiro et al. (2021), i.e., $\mathcal{J}(\theta):=\mathbb{E}_{x\sim\pi}\left[\mathcal{R}_{\theta}(x)\right]$, where $x$ is sampled from an invariant distribution $\pi$. Clearly, problem (1) can be viewed as a generalization of those machine learning problems with oblivious objective functions. In the literature, an RL problem is often formulated as a discrete-time and discounted Markov decision processes (MDPs) Sutton & Barto (2018) which aims to learn an optimal policy via optimizing the (discounted) cumulative sum of rewards. We can also see that the learning objective of an MDP can be covered by the problem (1) with the property that the function $\mathcal{R}(x)$ does not depend on $\theta$ (see Example 3.3). Recently, the application of RL for solving combinatorial optimization (CO) problems which are typically NP-hard has attracted much attention. These CO problems may include the traveling salesman problem and related problems Bello et al. (2016); Mazyavkina et al. (2021), the reward optimization problem arising from the finite expression method Liang & Yang (2022); Song et al. (2023), and the general binary optimization problem Chen et al. (2023), to name just a few. The common key component of the aforementioned applications is the reward optimization, which could also be formulated as problem (1). There also exist problems with general reward functions that are outside the scope of cumulative sum of rewards of trajectories that are used in MDPs. An interesting example is the MDP with general utilities; see, e.g., Zhang et al. (2020a); Kumar et al. (2022); Barakat et al. (2023) and references therein.

Adding a regularizer to the objective function is a commonly used technique to impose desirable structures to the solution and/or to greatly enhance the expression power and applicability of RL Lan (2023); Zhan et al. (2023). When one considers the direct/simplex parameterization Agarwal et al. (2021) of $\pi_{\theta}$, a regularization function using the indicator function for the standard probability simplex is needed. Moreover, by using other indicator functions for general convex sets, one is able to impose some additional constraints on the parameter $\theta$. For the softmax parameterization, one may also enforce a bounded constraint to $\theta$ to prevent it taking values that are too large. This can avoid potential numerical issues, including the overflow error on a floating point system. On the other hand, there are incomplete parametric policy classes, such as the log-linear and neural policy classes, that are often formulated as $\{\pi_{\theta}|\theta\in\Theta\}$, where $\Theta$ is a closed convex set Agarwal et al. (2021). In this case, the indicator function is still necessary and useful. Some recent works (see, e.g., Ahmed et al. (2019); Agarwal et al. (2020); Mei et al. (2020); Cen et al. (2022)) have investigated the impact of the entropy regularization for MDPs. Systematic studies on general convex regularization for MDPs have been limited until the recent works Pham et al. (2020); Lan (2023); Zhan et al. (2023). Finally, problem (1) takes the same form as the stochastic optimization problem with decision-dependent distributions (see e.g., Drusvyatskiy & Xiao (2023) and references therein), leading to numerous real-world applications such as performative prediction Mendler-Dünner et al. (2020); Perdomo et al. (2020), concept drift Gama et al. (2014), strategic classification Tsirtsis et al. (2024); Milli et al. (2019), and casual inference Yao et al. (2021). Consequently, we can see that problem (1) is in fact quite general and has promising modeling power, as it covers many existing problems in the literature.

The purpose of this paper is to leverage existing tools and results in MDPs and nonconvex optimization for solving the general regularized expected reward optimization problem (1) with general policy parameterization, which, to the best of our knowledge, has not been formally considered in the RL literature. It is well known that the policy gradient method Williams (1992); Sutton et al. (1999); Baxter & Bartlett (2001), which lies in the heart of RL, is one of the most competitive and efficient algorithms due to its simplicity and versatility. Moreover, the policy gradient method is readily implemented and can be paired with other effective techniques. In this paper, we observe that the stochastic proximal gradient method, which shares the same spirit of the policy gradient method, can be applied directly for solving the targeted problem (1) with convergence guarantees to a stationary point. Since the classical stochastic gradient estimator typically introduces a large variance, there is also a need to consider designing advanced stochastic gradient estimators with smaller variances. To this end, we shall also look into a certain stochastic variance-reduced proximal gradient method and analyze its convergence properties. In particular, the contributions of this paper are summarized as follows.

• •

  We consider a novel and general regularized reward optimization model (1) that covers many existing important models in the machine learning and optimization literature. Thus, problem (1) admits a promising modeling power which encourages potential applications.

• •

  In order to solve our targeted problem, we consider applying the classical stochastic proximal gradient method and analyze its convergence properties. We first demonstrate that the gradient of $\mathcal{J}(\cdot)$ is Lipschitz continuous under standard conditions with respect to the reward function $\mathcal{R}_{\theta}(\cdot)$ and the parameterized policy $\pi_{\theta}(\cdot)$. Using the L-smoothness of $\mathcal{J}(\cdot)$, we then show that the classical stochastic proximal gradient method with a constant step-size (depending only on the Lipschitz constant for $\nabla_{\theta}\mathcal{J}(\cdot)$) for solving problem (1) outputs an $\epsilon$-stationary point (see Definition 3.4) within $T:=O(\epsilon^{-2})$ iterations, and the sample size for each iteration is $O(\epsilon^{-2})$, where $\epsilon>0$ is a given tolerance. Thus, the total sample complexity becomes $O(\epsilon^{-4})$, which matches the current state-of-the-art sample complexity of the classical stochastic policy gradient for MDPs; see e.g., Williams (1992); Baxter & Bartlett (2001); Zhang et al. (2020b); Xiong et al. (2021); Yuan et al. (2022).

• •

  Moreover, in order to further reduce the variance of the stochastic gradient estimator, we utilize an importance sampling based probabilistic gradient estimator which leads to an efficient single-looped variance reduced method. The application of this probabilistic gradient estimator is motivated by the recent progress in developing efficient stochastic variance-reduced gradient methods for solving stochastic optimization Li et al. (2021b) and (unregularized) MDPs Gargiani et al. (2022). We show that, under additional technical conditions, the total sample complexity is improved from $O(\epsilon^{-4})$ to $O(\epsilon^{-3})$. This result again matches the results of some existing competitive variance-reduced methods for MDPs Papini et al. (2018); Xu et al. (2019); Pham et al. (2020); Huang et al. (2021); Yang et al. (2022); Gargiani et al. (2022). Moreover, to the best of our knowledge, the application of the above probabilistic gradient estimator is new for solving the regularized expected reward optimization (1).

The rest of this paper is organized as follows. We first summarize some relative works in Section 2. Next, in Section 3, we present some background information that are needed for the exposition of this paper. Then, in Section 4, we describe the classical stochastic proximal gradient method for solving (1) and present the convergence properties of this method under standard technical conditions. Section 5 is dedicated to describing and analyzing the stochastic variance-reduced proximal gradient method with an importance sampling based probabilistic gradient estimator. Finally, we make some concluding remarks, and list certain limitations and future research directions of this paper in Section 6.

The policy gradient method. One of the most influential algorithms for solving RL problems is the policy gradient method, built upon the foundations established in Williams (1992); Sutton et al. (1999); Baxter & Bartlett (2001). Motivated by the empirical success of the policy gradient method and its variants, analyzing the convergence properties for these methods has long been one of the most active research topics in RL. Since the objective function $\mathcal{J}(\theta)$ is generally nonconcave, early works Sutton et al. (1999); Pirotta et al. (2015) focused on the asymptotic convergence properties to a stationary point. By utilizing the special structure in (entropy regularized) MDPs, recent works Liu et al. (2019); Mei et al. (2020); Agarwal et al. (2021); Li et al. (2021a); Xiao (2022); Cen et al. (2022); Lan (2023); Fatkhullin et al. (2023) provided some exciting results on the global convergence. Meanwhile, since the exact gradient of the objective function can hardly be computed, sampling-based approximated/stochastic gradients have gained much attention. Therefore, many works investigated the convergence properties, including the iteration and sample complexities, for these algorithms with inexact gradients; see e.g., Zhang et al. (2020b); Liu et al. (2020); Zhang et al. (2021b); Xiong et al. (2021); Yuan et al. (2022); Lan (2023) and references therein.

Variance reduction. While the classical stochastic gradient estimator is straightforward and simple to implement, one of its most critical issues is that the variance of the inexact gradient estimator can be large, which generally slows down the convergence of the algorithm. To alleviate this issue, an attractive approach is to pair the sample-based policy gradient methods with certain variance-reduced techniques. Variance-reduced methods were originally developed for solving (oblivious) stochastic optimization problems Johnson & Zhang (2013); Nguyen et al. (2017); Fang et al. (2018); Li et al. (2021b) typically arising from supervised learning tasks. Motivated by the superior theoretical properties and practical performance of the stochastic variance-reduced gradient methods, similar algorithmic frameworks have recently been applied for solving MDPs Papini et al. (2018); Xu et al. (2019); Yuan et al. (2020); Pham et al. (2020); Huang et al. (2021); Yang et al. (2022); Gargiani et al. (2022).

Stochastic optimization with decision-dependent distributions. Stochastic optimization is the core of modern machine learning applications, whose main objective is to learn a decision rule from a limited data sample that is assumed to generalize well to the entire population Drusvyatskiy & Xiao (2023). In the classical supervised learning framework Zhang (2004); Hastie et al. (2009); Shapiro et al. (2021), the underlying data distribution is assumed to be static, which turns out to be a crucial assumption when analyzing the convergence properties of the common stochastic optimization algorithms. On the other hand, there are problems where the distribution changes over the course of iterations of a specific algorithm, and these are closely related to the concept of performative prediction Perdomo et al. (2020). In this case, understanding the convergence properties of the algorithm becomes more challenging. Toward this, some recent progress has been made on (strongly) convex stochastic optimization with decision-dependent distributions Mendler-Dünner et al. (2020); Perdomo et al. (2020); Drusvyatskiy & Xiao (2023). Moreover, other works have also considered nonconvex problems and obtained some promising results; see Dong et al. (2023); Jagadeesan et al. (2022) and references therein. Developing theoretical foundation for these problems has become a very active field nowadays.

RL with general utilities. It is known that the goal of an agent associated with an MDP is to seek an optimal policy via maximizing the cumulative discounted reward Sutton & Barto (2018). However, there are decision problems of interest having more general forms. Beyond the scope of the expected cumulative reward in MDPs, some recent works also looked into RL problems with general utilities; see e.g., Zhang et al. (2020a); Kumar et al. (2022); Barakat et al. (2023) as mentioned previously. Global convergence results can also be derived via investigating the hidden convex structure Zhang et al. (2020a) inherited from the MDP.

The first condition on the boundedness of the function $\mathcal{R}_{\theta}(\cdot)$, which is commonly assumed in the literature Sutton & Barto (2018), ensures that $\mathcal{J}(\theta)$ is well-defined. And the second condition will be used to guarantee the well-definiteness and L-smoothness of the gradient $\nabla_{\theta}\mathcal{J}(\theta)$. We remark here that when the reward function $\mathcal{R}_{\theta}(x)$ does not depend on $\theta$ (see e.g., Example 3.3), then the second assumption holds automatically.

This assumption is a standard one and commonly employed in the literature when studying the convergence properties of the policy gradient method for MDPs; see e.g., Pirotta et al. (2015); Papini et al. (2018); Xu et al. (2020); Pham et al. (2020); Zhang et al. (2021a); Yang et al. (2022) and references therein.

Under Assumption 3.1 and Assumption 3.2, it is easy to verify that the gradient for the expected reward function $\mathcal{J}(\theta)$ can be written as:

Being a composite optimization problem, problem (1) admits the following first-order stationary condition

Here, $\partial\mathcal{G}(\cdot)$ denotes the subdifferential of the proper closed and convex function $\mathcal{G}(\cdot)$ which is defined as

It is well-known that $\partial\mathcal{G}(\theta)$ is a nonempty closed convex subset of $\mathbb{R}^{n}$ for any $\theta\in\mathbb{R}^{n}$ such that $\mathcal{G}(\theta)<\infty$ (see e.g., Rockafellar (1997)). Note that any optimal solution of problem (1) satisfies the condition (3), while the reverse statement is generally not valid for nonconcave problems, including the problem (1). The condition (3) leads to the following concept of stationary points for problem (1).

The main objective of this paper is to study the convergence properties, including iteration and sample complexities, of the stochastic (variance-reduced) proximal gradient method to a $\epsilon$-stationary point with a pre-specified $\epsilon>0$. Note that all proofs of our results are presented in the appendix. Moreover, we acknowledge that our analysis is drawn upon classical results in the literature.

In this section, we present and analyze the stochastic proximal gradient method for solving the problem (1). The fundamental idea of the algorithm is to replace the true gradient $\nabla_{\theta}\mathcal{J}(\theta)$, which are not available for most of the time, with a stochastic gradient estimator in the classical proximal gradient method Beck (2017). The method can be viewed as extensions to the projected policy gradient method with direct parameterization Agarwal et al. (2021) and the stochastic policy gradient method for unregularized MDPs Williams (1992). The detailed description of the algorithm is presented in Algorithm 1.

For notational simplicity, we denote

From Algorithm 1, we see that at each iteration, $N$ data points, namely $\{x^{t,1},\dots,x^{t,N}\}$, are sample according to the current probability distribution $\pi_{\theta^{t}}$. Using these data points, we can construct a REINFORCE-type stochastic gradient estimator $g^{t}$. Then, the algorithm just performs a proximal gradient ascent updating. Let $T>0$ be the maximal number of iterations, then a sequence $\{\theta^{t}\}_{t=1}^{T}$ can be generated, and the output solution is selected randomly from this sequence. Next, we shall proceed to answer the questions that how to choose the learning rate $\eta>0$, how large the sample size $N$ should be, and how many iterations for the algorithm to output an $\epsilon$-stationary point for a given $\epsilon>0$, theoretically. The next lemma establishes the L-smoothness of $\mathcal{J}(\cdot)$ whose proof is given at Appendix A.1.

As a consequence of the L-smoothness of the function $\mathcal{J}(\cdot)$, we next show that the learning rate can be chosen as a positive constant upper bounded by a constant depends only on the Lipschitz constant of $\nabla_{\theta}\mathcal{J}(\cdot)$. For notational complicity, we denote $\Delta:=\mathcal{F}^{*}-\mathcal{F}(\theta^{0})>0$ for the rest of this paper.

The proof of the above theorem is provided in Appendix A.2. From this theorem, if one sets $g^{t}=\nabla_{\theta}\mathcal{J}(\theta^{t})$, i.e., $\left\lVert g^{t}-\nabla_{\theta}\mathcal{J}(\theta^{t})\right\rVert^{2}=0$, then there is no randomness along the iterations and the convergence property is reduced to

which is implied by classical results on proximal gradient method (see e.g., Beck (2017)). However, since the exact full gradient $\nabla_{\theta}\mathcal{J}(\theta)$ is rarely computable, it is common to require the variance (i.e., the trace of the covariance matrix) of the stochastic estimator to be bounded. The latter condition plays an essential role in analyzing stochastic first-order methods for solving nonconvex optimization problems, including RL applications; see, e.g., Beck (2017); Papini et al. (2018); Shen et al. (2019); Lan (2020); Yang et al. (2022).

As already mentioned in the introduction, the total sample complexity of Algorithm 1 to an $\epsilon$-stationary point is shown to be $O(\epsilon^{-4})$, which matches the most competitive sample complexity of the classical stochastic policy gradient for MDPs Williams (1992); Baxter & Bartlett (2001); Zhang et al. (2020b); Xiong et al. (2021); Yuan et al. (2022).

Recall from Theorem 4.3 that, there is a trade-off between the sample complexity and the iteration complexity of Algorithm 1. In particular, while there is little room for us to improve the term $\frac{\Delta}{T}\left(\frac{2}{\eta}+\frac{4}{\eta(1-2\eta L)}\right)$ which corresponds to the iteration complexity, it is possible to construct $g^{t}$ in an advanced manner to improve the sample complexity. Therefore, our main goal in this section is to reduce the expected sample complexity while keeping the term $\displaystyle\frac{1}{T}\sum_{t=0}^{T-1}\mathbb{E}_{T}\left[\left\lVert\nabla_{\theta}\mathcal{J}(\theta^{t})-g^{t}\right\rVert^{2}\right]$ small. We achieve this goal by considering the stochastic variance-reduced gradient methods that have recently attracted much attention. Among these variance-reduced methods, as argued in Gargiani et al. (2022), the ProbAbilistic Gradient Estimator (PAGE) proposed in Li et al. (2021b) has a simple structure, and can lead to optimal convergence properties. These appealing features make it attractive in machine learning applications. Therefore, in this section, we also consider the stochastic variance-reduced proximal gradient method with PAGE for solving the problem (1).

PAGE is originally designed for the stochastic nonconvex minimization in the oblivious setting:

where $\pi$ is a fixed probability distribution and $F:\mathbb{R}^{d}\times\mathbb{R}^{n}\to\mathbb{R}$ is a certain differentiable (and possibly nonconvex) loss function. For stochastic gradient-type methods, a certain stochastic gradient estimator for $f$ is required for performing the optimization. At the $t$-th iteration, given a probability $p_{t}\in[0,1]$ and the current gradient estimator $g^{t}$, PAGE proposed to replace the vanilla mini-batch gradient estimator with the following unbiased stochastic estimator:

where $\{x^{j}\}$ are sampled from $\pi$, $N_{1},N_{2}$ denote the sample sizes. Some key advantages of applying PAGE are summarized as follows. First, the algorithm is single-looped, which admit simpler implementation compared with existing double-looped variance reduced methods. Second, the probability $p_{t}$ can be adjusted dynamically, leading to more flexibilities. Third, one can choose $N_{2}$ to be much smaller than $N_{1}$ to guarantee the same iteration complexity as the vanilla SGD. Thus, the overall sample complexity can be significantly reduced. However, the application of PAGE to our setting needs significant modifications and extensions, which we shall demonstrate below. To the best of our knowledge, the application of PAGE for solving the general regularized reward optimization problem in the non-oblivious setting considered in this paper is new.

For notational simplicity, for the rest of this section, we denote

for $\theta,\theta^{\prime}\in\mathbb{R}^{n},\;x\in\mathbb{R}^{d}$, where $\frac{\pi_{\theta}(x)}{\pi_{\theta^{\prime}}(x)}$ denotes the importance weight between $\pi_{\theta}$ and $\pi_{\theta^{\prime}}$. Note also that

The description of the proposed PAGE variance-reduced stochastic proximal gradient method is given in Algorithm 2.

It is clear that the only difference between Algorithm 1 and Algorithm 2 is the choice of the gradient estimator. At each iteration of the latter algorithm, we have two choices for the gradient estimator, where, with probability $p$, one chooses the same estimator as in Algorithm 1 with a sample size $N_{1}$, and with probability $1-p$, one constructs the estimator in a clever way which combines the information of the current iterate and the previous one. Since the data set $\{x^{t+1,1},\dots,x^{t+1,N_{2}}\}$ is sampled according to the current probability distribution $\pi_{\theta^{t+1}}$, we need to rely on the importance weight between $\theta^{t}$ and $\theta^{t+1}$ and construct the gradient estimator $\displaystyle\frac{1}{N_{2}}\sum_{j=1}^{N_{2}}g_{w}(x^{t+1},\theta^{t},\theta^{t+1})$, which is an unbiased estimator for $\nabla_{\theta}\mathcal{J}(\theta^{t})$, so that $g^{t+1}$ becomes an unbiased estimator of $\nabla_{\theta}\mathcal{J}(\theta^{t+1})$. Indeed, one can easily verify that for any $\theta,\theta^{\prime}\in\mathbb{R}^{n}$, it holds that

i.e., $g(x,\theta,\theta^{\prime})$ is an unbiased estimator for $\nabla_{\theta}\mathcal{J}(\theta)$ provided that $x\sim\pi_{\theta^{\prime}}$.

Clearly, the significance of the constant $C_{w}$ (if exists) may depend sensitively on $\theta$ and $\theta^{\prime}$. To see this, let us assume that for any $\theta\in\mathbb{R}^{n}$, $\pi_{\theta}=\theta$ is a discrete distribution over a set of finite points $\{x_{k}\}_{k=1}^{n}$ for which $\pi_{\theta}(x_{k})=\theta_{k}>0$ for all $k=1,\dots,n$. Now, suppose that $\theta=\theta^{\prime}+\Delta\theta$ with $|\Delta\theta_{k}|\leq 1$. Then, a simple calculation shows that

However, it is possible that there exists a certain $\theta_{k}^{\prime}=0$ or tiny. In this case, $C_{w}$ can be huge or even infinity. Fortunately, the regularization term $\mathcal{G}(\theta)$ can help to avoid such undesired situations via imposing the lower-bounded constraints $\theta_{k}\geq\delta>0$ for all $k$. In this case, we see that $\sum_{k=1}^{n}\frac{\theta_{k}}{\theta^{\prime}_{k}}\leq\sum_{k=1}^{n}\frac{\theta_{k}}{\delta}=\frac{1}{\delta}$.

The bounded variance of the importance weight implies that the (expected) distance between $g(x,\theta^{\prime})$ and $g_{w}(x,\theta,\theta^{\prime})$ is controlled by the distance between $\theta$ and $\theta^{\prime}$, for any given $\theta,\theta^{\prime}\in\mathbb{R}^{d}$. In particular, we have the following lemma, whose proof is provided in Appendix A.5.

Under the considered assumptions, we are able to provide an estimate for the term $\displaystyle\sum_{t=0}^{T-1}\mathbb{E}_{T}\left[\left\lVert g^{t}-\nabla_{\theta}\mathcal{J}(\theta^{t})\right\rVert^{2}\right]$, which plays an essential role in deriving an improved sample complexity of Algorithm 2. The results are summarized in the following Lemma 5.4; see Appendix A.6 for a proof which shares the same spirit as (Li et al., 2021b, Lemma 3 & 4).

We are now ready to present the main result on the convergence property of the Algorithm 2 by showing how to select the sample sizes $N_{1}$ and $N_{2}$, probability $p$, and the learning rate $\eta$. Intuitively, $N_{1}$ is typically a large number and one does not want to perform samplings with $N_{1}$ samples frequently, thus the probability $p$ and the sample size $N_{2}$ should both be small. Given $N_{1}$, $N_{2}$ and $p$, we can then determine the value of $\eta$ such that $\eta<\frac{1}{2L}$. Consequently, the key estimate in Theorem 4.3 can be applied directly. Our results are summarized in the following theorem. Reader is referred to Appendix A.7 for the proof of this result.

By using the stochastic variance-reduce gradient estimator with PAGE and the importance sampling technique, we have improved the total sample complexity from $O(\epsilon^{-4})$ to $O(\epsilon^{-3})$, under the considered conditions. This result matches with the current competitive results established in Xu et al. (2019); Yuan et al. (2020); Pham et al. (2020); Gargiani et al. (2022) for solving MDPs and is applicable to the general model (1). Finally, as mentioned in Remark 4.7, by assuming or verifying stronger conditions, such as the gradient domination and its extensions, it is also possible to derive some global convergence results. Again, such a possibility is left to a future research direction.

We have studied the stochastic (variance-reduced) proximal gradient method addressing a general regularized expected reward optimization problem which covers many existing important problem in reinforcement learning. We have established the $O(\epsilon^{-4})$ sample complexity of the classical stochastic proximal gradient method and the $O(\epsilon^{-3})$ sample complexity of the stochastic variance-reduced proximal gradient method with an importance sampling based probabilistic gradient estimator. Our results match the sample complexity of their most competitive counterparts under similar settings for Markov decision processes.

Meanwhile, we have also suspected some limitations in the current paper. First, due to the nonconcavity of the objective function, we found it challenging to derive global convergence properties of the stochastic proximal gradient method and its variants without imposing additional conditions. On the other hand, analyzing the sample complexity for achieving convergence to second-order stationary points—thereby avoiding saddle points—may be more realistic and feasible Arjevani et al. (2020). Second, the bounded variance condition for the importance weight turns out to be quite strong and can not be verified in general. How to relax this condition for our general model deserves further investigation. Last but not least, since we focus more on the theoretical analysis in this paper and due to the space constraint, we did not conduct any numerical simulation to examine the practical efficiency of the proposed methods. We shall try to delve into these challenges and get better understandings of the proposed problem and algorithms in a future research.

Finally, this paper has demonstrated the possibility of pairing the stochastic proximal gradient method with efficient variance reduction techniques Li et al. (2021b) for solving the reward optimization problem (1). Beyond variance-reduced methods, there are other possibilities that allow one deriving more sophisticated algorithms. For instance, one can also pair the stochastic proximal gradient method with the ideas of the actor-critic method Konda & Tsitsiklis (1999), the natural policy gradient method Kakade (2001), policy mirror descent methods Tomar et al. (2020); Lan (2023), trust-region methods Schulman et al. (2015); Shani et al. (2020), and the variational policy gradient methods Zhang et al. (2020a). We think that these possible generalizations can lead to more exciting results and make further contributions to the literature.