Concave Utility Reinforcement Learning with Zero-Constraint Violations

In many applications where a learning agent uses reinforcement learning to find optimal policies, the agent optimizes a concave function of the expected rewards or the agent must satisfy certain constraints while maximizing an objective (Altman & Schwartz, 1991; Roijers et al., 2013). For example, in network scheduling, a controller can maximize fairness of the users using a concave function of the average reward of each of the users (Chen et al., 2021). Consider a scheduler which allocates a resource to users. Each user obtains some reward based on their current state. The goal of the scheduler is to maximize fairness among the users. However, there are certain preferred users for which some service level agreements (SLA) must be made. For this setup, the scheduler aims to find a policy which maximizes the fairness while ensuring the SLA constraints of the preferred users are met. Note that, here, the objective is a non-linear concave utility in the presence of constraints on service level agreement. Setups with constraints also exist in autonomous vehicles where the goal is to reach the destination quickly while ensuring the safety of the surroundings (Le et al., 2019; Tessler et al., 2018). Further, an agent may aim to efficiently explore the environment by maximizing the entropy, which is a concave function of the distribution induced over the state and action space Hazan et al. (2019).

Owing to the variety of the use cases, recently, there has been significant effort to make RL algorithms for setups with constraints, or concave utilties, or both. For episodic setup, works range from model based algorithms (Brantley et al., 2020; Yu et al., 2021) to primal-dual based model-free algorithms (Ding et al., 2021). Recently, there has been a thrust towards developing algorithms which can also achieve zero-constraint violations in the learning phase as well Wei et al. (2022a); Liu et al. (2021); Bai et al. (2022b). However, for the episodic setup, the majority of the current works consider the weaker regret definition specified by Efroni et al. (2020) and only achieve zero expected constraint violations. Further, these algorithms require the knowledge of a safe policy following which the agent does not violate constraints, or the knowledge of the Slater’s gap $\delta$ which determines how far a safe policy is from the constraint boundary.

The definition which considers the average over time makes sense for an infinite horizon setup as the long-term average is naturally defined (Puterman, 2014). For a tabular infinite-horizon setup, Singh et al. (2020) proposed an optimistic epoch-based algorithm. Much recently, Chen et al. (2022) proposed an Optimistic Online Mirror Descent based algorithm. In this work, we consider the problem of maximizing concave utility of the expected rewards while also ensuring that a set of convex constraints of the expected rewards are also satisfied. Moreover, we aim to develop algorithms that can also ensure that the constraints are not violated during the training phase as well. We work with tabular MDP with infinite horizon. For such setup, our algorithm updates policies as it learns the system model. Further, our approach also bounds the accumulated observed constraint violations as compared to the expected constraint violations.

For infinite horizon setups for non-constrained setup, the regret analysis has been widely studied Fruit et al. (2018); Jaksch et al. (2010). However, we note that the dealing with constraints and non-linear setup requires additional attention because of the stochastic policies. Further, unlike episodic setup, the distribution at the epoch is not constant and hence the policy switching cost has to be accounted explicitly. Prior works in infinite horizon also faced this issue and provide some tools to overcome this limitation. Singh et al. (2020) builds confidence intervals for transition probability for every next state given the current state-action pair and obtains a regret bound of $O(T^{2/3})$. Chen et al. (2022) obtains a regret bound of $O(T_{M}\sqrt{T})$ with $O(T_{M}^{2}S^{3}A)$ constraint violations for ergodic MDPs with $T_{M}$ mixing time following an analysis which works with confidence intervals on both transition probability vectors and value functions.

To overcome the limitations mentioned in previous analysis and to obtain a tighter result, we propose an optimism based UC-CURL algorithm which proceeds in epochs $e$. At each epoch, we solve for an policy which considers constraints tighter by $\epsilon_{e}$ than the true bounds for the optimistic MDP in the confidence intervals for the transition probabilities. Further, as the knowledge of the model improves with increased interactions with the environment, we reduce this tightness. This $\epsilon_{e}$-sequence is critical to our algorithm as, if the sequence decays too fast, the constraints violations cannot be bounded by zero. If this sequence decays too slow, the objective regret may not decay fast enough. Further, using the $\epsilon_{e}$-sequence, we do not require the knowledge of the total time $T$ for which the algorithm runs.

We bound our regret by bounding the gap between the optimal policy in the feasible region and the optimal policy for the optimization problem with $\epsilon_{e}$ tight constraints. We bound this gap with a multiplicative factor of $O(1/\delta)$, where $\delta$ is Slater’s parameter. Based on our analysis using the Slater’s parameter $\delta$, we consider a case where a lower bound $T_{l}$ on the time horizon $T$ is known. This knowledge of $T_{l}$ allows us to relax our assumption on $\delta$.

Further, for the regret analysis of the proposed UC-CURL algorithm, we use Bellman error for infinite horizon setup to bound the difference between the performance of optimistic policy on the optimistic MDP and the true MDP. Compared to analysis of Jaksch et al. (2010), this allows us to work with stochastic policies. We bound our regret as $\tilde{O}(\frac{1}{\delta}LdT_{M}S\sqrt{A/T}+CT_{M}S^{2}A/T(1-\rho))$ and constraint violations as $0$, where $S$ and $A$ are the number of states and actions respectively, $L$ is the Lipschitz constant of the objective and constraint functions, $d$ is the number of costs the agent is trying to optimize, and $T_{M}$ is the mixing time of the MDP. The Bellman error based analysis along with Slater’s slackness assumption also allows to develop posterior sampling based methods for constrained RL (see Appendix G) by showing feasibility of the optimization problem for the sampled MDPs.

To summarize our contributions, we improve prior results on infinite horizon concave utility reinforcement learning setup on multiple fronts. First, we consider convex function for objectives and constraints. Second, even with a non-linear function setup, we reduce the regret order to $O(T_{M}S\sqrt{A/T})$ and bound the constraint violations with $0$. Third, our algorithm does not require the knowledge of the time horizon $T$, safe policy, or Slater’s gap $\delta$. Finally, we provide analysis for posterior sampling algorithm which improves both empirical performance and computational complexity.

Constrained RL: Altman (1999) builds the formulation for constrained MDPs to study constrained reinforcement learning and provides algorithms for obtaining policies with known transition models. Zheng & Ratliff (2020) considered an episodic CMDP (Constrained Markov Decision Processes) and use an optimism based algorithm to bound the constraint violation as $\tilde{O}(1/T^{0.25})$ with high probability. Kalagarla et al. (2021) also considered the episodic setup to obtain PAC-style bound for an optimism based algorithm. Ding et al. (2021) considered the setup of $H$-episode length episodic CMDPs with $d$-dimensional linear function approximation to bound the constraint violations as $\tilde{O}(d\sqrt{H^{5}/T})$ by mixing the optimal policy with an exploration policy. Efroni et al. (2020) proposes a linear-programming and primal-dual policy optimization algorithm to bound the regret as $O(S\sqrt{H^{3}/T})$. Wei et al. (2022a); Liu et al. (2021) considered the problem of ensuring zero constraint violations using a model-free algorithm for tabular MDPs with linear rewards and constraints. However, for infinite horizon setups, the analysis from finite horizon algorithms does not directly hold. This is because finite horizon setups can update the policy after every episode. But this policy switch modifies the induced Markov chains which takes time to converge to a stationary distribution.

Xu et al. (2021) considered an infinite horizon discounted setup with constraints and obtain global convergence using policy gradient algorithms. Bai et al. (2022b) proposed a conservative stochastic model-free primal-dual algorithm for infinite horizon discounted setup. Ding et al. (2020); Bai et al. (2023) also considered an infinite horizon discounted setup with parametrization. They used a natural policy gradient to update the primal variable and sub-gradient descent to update the dual variable. In addition to the above results on discounted MDPs, the long-term rewards have also been considered. Singh et al. (2020) considered the setup of infinite-horizon ergodic CMDPs with long-term average constraints with an optimism based algorithm. Gattami et al. (2021) analyzed the asymptotic performance for Lagrangian based algorithms for infinite-horizon long-term average constraints, however they only show convergence guarantees without explicit convergence rates. Chen et al. (2022) provided an optimistic online mirror descent algorithm for ergodic MDPs which obtain a regret bound of $O(T_{M}S\sqrt{SAT})$, and Wei et al. (2022b) provided a model free SARSA algorithm which obtains a regret bound of $O(\sqrt{SA}T^{5/6})$ for constrained MDPs. Agarwal et al. (2022b) proposed a posterior sampling based algorithm for infinite horizon setup with a regret of $\tilde{O}(T_{M}S\sqrt{AT})$ and constraint violation of $\tilde{O}(T_{M}S\sqrt{AT})$.

Concave Utility RL: Another major research area related to this work is concave utility RL (Hazan et al., 2019). Along this direction, Cheung (2019) considered a concave function of expected per-step vector reward and developed an algorithm using Frank-Wolfe gradient of the concave function for tabular infinite horizon MDPs. Agarwal & Aggarwal (2022); Agarwal et al. (2022a) also considered the same setup using a posterior sampling based algorithm. Recently, Brantley et al. (2020) combined concave utility reinforcement learning and constrained reinforcement learning for an episodic setup. Yu et al. (2021) also considered the case of episodic setup with concave utility RL. However, both (Brantley et al., 2020) and (Yu et al., 2021) consider the weaker regret definition by Efroni et al. (2020), and Cheung (2019); Yu et al. (2021) do not target the convergence of the policy. Further, these works do not target zero-constraint violations. Recently, policy gradient based algorithms have also been studied for discounted infinite horizon setup Bai et al. (2022a).

Another parallel line of work in RL which deals with concave utilities is variational policy gradient (Zhang et al., 2021; 2020). However, they consider discounted MDPs whereas we consider undiscounted setup for our work.

Compared to prior works, we consider the constrained reinforcement learning with convex constraints and concave objective function. Using infinite-horizon setup, we consider the tightest possible regret definition. Further, we achieve zero constraint violations with objective regret tight in $T$ using an optimization problem with decaying tightness. A comparative survey of key prior works and our work is also presented in Table 1.

We consider an ergodic tabular infinite-horizon constrained Markov Decision Process $\mathcal{M}=(\mathcal{S},\mathcal{A},r,f,c_{1},\cdots,c_{d},g,P,\phi)$. $\mathcal{S}$ is finite set of $S$ states, and $\mathcal{A}$ is a finite set of $A$ actions. $P:\mathcal{S}\times\mathcal{A}\to\Delta(\mathcal{S})$ denotes the transition probability distribution such that on taking action $a\in\mathcal{A}$ in state $s\in\mathcal{S}$, the system moves to state $s^{\prime}\in\mathcal{S}$ with probability $P(s^{\prime}|s,a)$. $r:\mathcal{S}\times\mathcal{A}\to[0,1]$ and $c_{i}:\mathcal{S}\times\mathcal{A}\to[0,1],i\in{1,\cdots,d}$ denotes the average reward obtained and average costs incurred in state action pair $(s,a)\in\mathcal{S}\times\mathcal{A}$, and $\phi$ is the distribution over the initial state.

The agent interacts with $\mathcal{M}$ in time-steps $t\in{1,2,\cdots}$ for a total of $T$ time-steps. We note that $T$ is possibly unknown and $s_{1}\sim\phi$. At each time $t$, the agent observes state $s_{t}$ and plays action $a_{t}$. The agent selects an action on observing the state $s$ using a policy $\pi:\mathcal{S}\to\Delta(\mathcal{A})$, where $\Delta(\mathcal{A})$ is the probability simplex on the action space. On following a policy $\pi$, the long-term average reward of the agent is denoted as:

where $\mathbb{E}_{\pi,P}[\cdot]$ denotes the expectation over the state and action trajectory generated from following $\pi$ on transitions $P$. The long-term average reward can also be represented as:

where $V_{\gamma}^{\pi,P}(s)$ is the discounted cumulative reward on following policy $\pi$, and $\rho_{\pi}^{P}\in\Delta(\mathcal{S}\times\mathcal{A})$ is the steady-state occupancy measure generated from following policy $\pi$ on MDP with transitions $P$ Puterman (2014). Similarly, we also define the long-term average costs as follows:

The agent interacts with the CMDP $\mathcal{M}$ for $T$ time-steps in an online manner and aims to maximize a function $f:[0,1]\to\mathbb{R}$ of the average per-step reward. Further, the agent attempts to ensure that a function of average per-step costs $g:[0,1]^{d}\to\mathbb{R}$ is at most $0$. In the hindsight, the agents wants to play a policy $\pi^{*}$ which which satisfies:

Let $P^{t}_{\pi,s}{=\prod_{t^{\prime}=1}^{t}{P_{\pi}}}$ denote the $t$-step transition probability on following policy $\pi$ in MDP $\mathcal{M}$ starting from some state $s$ where $P_{\pi}(\cdot|s)=\sum_{a}\pi(a|s)P(\cdot|s,a)$. Let $T_{s\to s^{\prime}}^{\pi}$ denote the time taken by the Markov chain induced by the policy $\pi$ to hit state $s^{\prime}$ starting from state $s$. Further, let $T_{M}:=\max_{\pi}\mathbb{E}[T^{\pi}_{s\to s^{\prime}}]$ be the mixing time of the MDP $\mathcal{M}$. We now introduce our assumptions on the MDP $\mathcal{M}$.

This assumption is again a standard assumption in the constrained RL literature (Efroni et al., 2020; Ding et al., 2021; 2020; Wei et al., 2022a). $\delta$ is referred as Slater’s constant. Ding et al. (2021) assumes that the Slater’s constant $\delta$ is known. Wei et al. (2022a) assumes that the number of iterations of the algorithm is at least $\tilde{\Omega}(SAH/\delta)^{5}$ for episode length $H$. On the contrary, we simply assume the existence of $\delta$ and a lower bound on the value of $\delta$ which gets relaxed as the agent acquires more time to interact with the environment.

Any online algorithm starting with no prior knowledge will need to obtain estimates of transition probabilities $P$ and obtain reward $r$ and costs $c_{k},\forall\ k\in\{1,\cdots,d\}$, for each state action pair. Initially, when algorithm does not have good estimate of the model, it accumulates a regret as well as violates constraints as it does not know the optimal policy. We define reward regret $R(T)$ as the difference between the average cumulative reward obtained vs the expected rewards from running the optimal policy $\pi^{*}$ for $T$ steps, or

Additionally, we define constraint regret $C(T)$ as the gap between the constraint function and incurred and constraint bounds, or

In the following section, we present a model-based algorithm to obtain this policy $\pi^{*}$, and reward regret and the constraint regret accumulated by the algorithm.

We now present our algorithm UC-CURL and the key ideas used in designing the algorithm. Note that if the agent is aware of the true transition $P$, it can solve the following optimization problem for the optimal feasible policy.

with the following set of constraints,

for all $s^{\prime}\in\mathcal{S},~{}\forall~{}s\in\mathcal{S},$ and $\forall~{}a\in\mathcal{A}$. Equation (11) denotes the constraint on the transition structure for the underlying Markov Process. Equation (10) ensures that the solution is a valid probability distribution. Finally, Equation (12) are the constraints for the constrained MDP setup which the policy must satisfy. Using the solution for $\rho$, we can obtain the optimal policy as:

However, the agent does not have the knowledge of $P$ to solve this optimization problem, and thus starts learning the transitions with an arbitrary policy. We first note that if the agent does not have complete knowledge of the transition $P$ of the true MDP $\mathcal{M}$, it should be conservative in its policy to allow room to violate constraints. Based on this idea, we formulate the $\epsilon$-tight optimization problem by modifying the constraint in Equation (12) as.

Let $\rho_{\epsilon}$ be the solution of the $\epsilon$-tight optimization problem, then the optimal conservative policy becomes:

We are now ready to design our algorithm UC-CURL which is based on the optimism principle (Jaksch et al., 2010). The UC-CURL algorithm is presented in Algorithm 1. The algorithm proceeds in epochs $e$. The algorithm maintains three key variables $\nu_{e}(s,a)$, $N_{e}(s,a)$, and $\hat{P}(s,a,s^{\prime})$ for all $s,a$. $\nu_{e}(s,a)$ stores the number of times state-action pair $(s,a)$ are visited in epoch $e$. $N_{e}(s,a)$ stores the number of times $(s,a)$ are visited till the start of epoch $e$. $\hat{P}(s,a,s^{\prime})$ stores the number of times the system transitions to state $s^{\prime}$ after taking action $a$ in state $s$. Another key parameter of the algorithm is $\epsilon_{e}=K\sqrt{(\log t_{e})/t_{e}}$ where $t_{e}$ is the start time of the epoch $e$ and $K$ is a configurable constant. Using these variables, the agent solves for the optimal $\epsilon_{e}$-conservative policy for the optimistic MDP by replacing the constraints in Equation (11) by:

for all $s^{\prime}\in\mathcal{S},\forall s\in\mathcal{S},$ and $\forall a\in\mathcal{A}$ and $x\vee y=\max(x,y)$. Equation (18) ensures that the agent searches for optimistic policy in the confidence intervals of the transition probability estimates.

Combining the right hand side of (16) with (10) gives

Thus, joint with (16), we see that equality in (16) will be satisfied at the boundary as $\sum_{a}\rho(s^{\prime},a)$ for some $s^{\prime}$ can never exceed the boundary to compensate for another $s^{\prime}$ and hence, for all $s^{\prime}$, $\sum_{a}\rho(s^{\prime},a)$ will lie on the boundary. In other words, the above constraints give $\sum\nolimits_{a\in\mathcal{A}}\rho(s^{\prime},a)=\sum\nolimits_{s,a}\tilde{P}_{e}(s^{\prime}|s,a)\rho(s,a)$. Further, we note that the region for the constraints is convex. This is because the set $\{x,y,z:xy\geq z\}$ is convex when $x,y,z\geq 0$. We note that even though the optimization problem may look non-convex due to constraints having product of two variables, we see Equations (9), (14), and (16)-(18) form a convex optimization problem. We expand more on this in Appendix B. We note that (Rosenberg & Mansour, 2019) provide another approach to obtain a convex optimization problem for optimistic MDP.

Let $\rho_{e}$ be the solution for $\epsilon_{e}$-tight optimization equation for the optimistic MDP. Then, we obtain the optimal conservative policy for epoch $e$ as:

The agent plays the optimistic conservative policy $\pi_{e}$ for epoch $e$. Note that the conservative parameter $\epsilon_{e}$ decays with time. As the agent interacts with the environment, the system model improves and the agent does not need to be as conservative as before. This allows us to bound both constraint violations and the objective regret. Further, if during the initial iterations of the algorithms a conservative solution is not feasible, we can ignore the constraints completely. We will show that the conservation behavior is required when $t=\Theta(T)$ to compensate for the violations in the initial period of the algorithm E.2.

For the UC-CURL algorithm described in Algorithm 1, we choose $\{\epsilon_{e}\}=\{K\sqrt{(\log t_{e})/t_{e}}\}$. However, if the agent has access to a lower bound $T_{l}$ (Assumption 3.7) on the time horizon $T$, the algorithm can change the $\epsilon_{e}=K\sqrt{(\ln(t_{e}\vee T_{l}))/(t_{e}\vee T_{l})}\leq\delta$ in each epoch $e$ as follows. Note that if $T_{l}=0$, $\epsilon_{e}$ becomes as specified in Algorithm 1 and if $T_{l}=T$, $\epsilon_{e}$ becomes constant for all epochs $e$.

After describing UC-CURL algorithm, we now perform the regret and constraint violation analysis. We note that the standard analysis for infinite horizon tabular MDPs of UCRL2 Jaksch et al. (2010) cannot be directly applied as the policy $\pi_{e}$ is possibly stochastic for every epoch. Another peculiar aspect of the analysis of the infinite horizon MDPs is that the regret grows linearly with the number of epochs (or policy switches). This is because a new policy induces a new Markov chain and this chain take time to converge to the stationary distribution. The analysis still bounds the regret by $\tilde{O}(T_{M}S\sqrt{A/T})$ as the number of epochs are bounded by $O(SA\log T)$.

Before diving into the details, we first define few important variables which are key to our analysis. The first variable is the standard $Q$-value function. We define $Q_{\gamma}^{\pi,P}$ as the long term expected reward on taking action $a$ in state $s$ and then following policy $\pi$ for the MDP with transition $P$. Mathematically, we have

We also define Bellman error $B^{\pi,\tilde{P}}(s,a)$ for the infinite horizon MDPs as the difference between the cumulative expected rewards obtained for deviating from the system model with transition $\tilde{P}$ for one step by taking action $a$ in state $s$ and then following policy $\pi$. We have:

After defining the key variables, we can now jump into bounding the objective regret $R(T)$. Intuitively, the algorithm incurs regret on three accounts. First source is following the conservative policy which we require to limit the constraint violations. Second source of regret is solving for the policy which is optimal for the optimistic MDP. Third source of regret is the stochastic behavior of the system. We also note that the constraints are violated because of the imperfect MDP knowledge and the stochastic behavior. However, the conservative behavior actually allows us to violate the constraints within some limits which we will discuss in the later part of this section.

We start by stating our first lemma which bounds the regret due to solving for a conservative policy. We define $\epsilon_{e}$-tight optimization problem as optimization problem for the true MDP with transitions $P$ with $\epsilon=\epsilon_{e}$. We bound the gap between the value of function $f$ at the long-term expected reward of the policy for $\epsilon_{e}$-tight optimization problem and the true optimization problem (Equation (9)-(12)) in the following lemma.

Lemma 5.1 and our construction of $\epsilon_{e}$ sequence allows us to limit the growth of regret because of conservative policy by $\tilde{O}(LdT_{M}S\sqrt{A/T})$.

The regret analysis framework also prepares us to bound the constraint violations as well. We again start by quantifying the reasons for constraint violations. The agent violates the constraint because 1. it is playing with the imperfect knowledge of the MDP and 2. the stochasticity of the MDP which results in the deviation from the average costs. We note that the conservative policy $\pi_{e}$ for every epoch does not violate the constraints, but instead allows the agent to manage the constraint violations because of the imperfect model knowledge and the system dynamics.

After giving the details on bounds on the possible sources of regret and constraint violations, we can formally state the result in the form of following theorem.

To validate the performance of the UC-CURL algorithm and the PS-CURL algorithm, we run the simulation on the flow and service control in a single-serve queue, which was introduced in (Altman & Schwartz, 1991). Along with validating the performance of the proposed algorithms, we also compare the algorithms against the algorithms proposed in Singh et al. (2020) and in Chen et al. (2022) for model-based constrained reinforcement learning for infinite horizon MDPs. Compared to these algorithms, we note that our algorithm is also designed to handle concave objectives of expected rewards with convex constraints on costs with $0$ constraint violations.

In the queue environment, a discrete-time single-server queue with a buffer of finite size $L$ is considered. The number of customers waiting in the queue is considered as the state in this problem and thus $|S|=L+1$. Two kinds of the actions, service and flow, are considered in the problem and control the number of customers together. The action space for service is a finite subset $A$ in $[a_{min},a_{max}]$, where $0<a_{min}\leq a_{max}<1$. Given a specific service action $a$, the service a customer is successfully finished with the probability $b$. If the service is successful, the length of the queue will reduce by 1. Similarly, the space for flow is also a finite subsection $B$ in $[b_{min},b_{max}]$. In contrast to the service action, flow action will increase the queue by $1$ with probability $b$ if the specific flow action $b$ is given. Also, we assume that there is no customer arriving when the queue is full. The overall action space is the Cartesian product of the $A$ and $B$. According to the service and flow probability, the transition probability can be computed and is given in the Table 2.

Define the reward function as $r(s,a,b)$ and the constraints for service and flow as $c^{1}(s,a,b)$ and $c^{2}(s,a,b)$, respectively. Define the stationary policy for service and flow as $\pi_{a}$ and $\pi_{b}$, respectively. Then, the problem can be defined as

According to the discussion in (Altman & Schwartz, 1991), we define the reward function as $r(s,a,b)=5-s$, which is an decreasing function only dependent on the state. It is reasonable to give higher reward when the number of customers waiting in the queue is small. For the constraint function, we define $c^{1}(s,a,b)=-10a+6$ and $c^{2}=-8(1-b)^{2}+2$, which are dependent only on service and flow action, respectively. Higher constraint value is given if the probability for the service and flow are low and high, respectively.

In the simulation, the length of the buffer is set as $L=5$. The service action space is set as $[0.2,0.4,0.6,0.8]$ and the flow action space is set as $[0.4,0.5,0.6,0.7]$. We use the length of horizon $T=5\times 10^{5}$ and run $50$ independent simulations of all algorithms. The experiments were run on a $36$ core Intel-i9 CPU @$3.00$ GHz with $64$ GB of RAM. The result is shown in the Figure 1(d). The average values of the cumulative reward and the constraint functions are shown in the solid lines. Further, we plot the standard deviation around the mean value in the shadow to show the random error. In order to compare this result to the optimal, we assume that the full information of the transition dynamics is known and then use Linear Programming to solve the problem. The optimal cumulative reward for the constrained optimization is calculated to be 4.48 with both flow constraint and service constraint values to be $0$. The optimal cumulative reward for the unconstrained optimization is 4.8 with service constraint being $-2$ and flow constraint being $-0.88$.

We now discuss the performance of all the algorithms starting with our algorithms UC-CURL and PS-CURL. In Figure 1(d), we observe that the proposed UC-CURL algorithm in Algorithm 1 does not perform well initially. We observe that this is because the confidence interval radius $\sqrt{S\log(At)/N(s,a)}$ for any $(s,a)$ are not tight enough in the initial period. After the algorithms collects sufficient samples to construct tight confidence intervals around the transition probabilities, the algorithm starts converging towards the optimal policy. We also note that the linear epoch modification of the algorithm works better than the doubling epoch algorithm presented in Algorithm 1. This is because the linear epoch variant updates the policy quickly whereas the doubling epoch algorithm works with the same policy for too long and thus looses the advantages of collected samples. For our implementation, we choose the value of parameter $K$ in Algorithm 1 as $K=1$, using which we observe that the constraint values start converging towards zero.

We now analyse the performance of the PS-CURL algorithm. For our implementation of the PS-CURL algorithm, we sample the transition probabilities using Dirichlet distribution. Note that the true transition probabilities were not sampled from a Dirichlet distribution and hence this experiment also shows the robustness against misspecified priors. We observe that the algorithm quickly brings the reward close to the optimal rewards. The performance of the PS-CURL algorithm is significantly better than the UC-CURL algorithm. We suspect this is because the UC-CURL algorithm wastes a large-number of steps to find optimistic policy with a large confidence interval. This observation aligns with the TDSE algorithm Ouyang et al. (2017), where they show that the Thompson sampling algorithm with $O(\sqrt{SAT})$ epochs performs empirically better than the optimism based UCRL2 algorithm Jaksch et al. (2010) with $O(\sqrt{SA\log T)}$ epochs. Osband et al. (2013) also made a similar observation where their PSRL algorithm worked better than the UCRL2 algorithm. Again, we set the value of parameter $K$ as $1$ and with $K=1$, the algorithm does not violate constraints. We also observe that the standard deviation of the rewards and constraints are higher for the PS-CURL algorithm as compared to the UC-CURL algorithm as the PS-CURL algorithm has an additional stochastic component which arises from sampling the transition probabilities.

After analysing the algorithms presented in this paper, we now analyse the performance of the algorithm by Chen et al. (2022). They provide an optimistic online mirror descent algorithm which also works with conservative parameter to tightly bound constraint violations. Their algorithm also obtains a $O(\sqrt{T})$ regret bound. However, their algorithm is designed for a linear reward/constraint setup with a single constraint, and empirically the algorithm is difficult to tune as it requires additional knowledge of $T_{M}$, $\rho$, $\delta$, and $T$ to fine tune parameters used in their algorithm. We set the value of the learning rate $\theta$ for online mirror descent as $5\times 10^{-2}$ with an episode length of $5\times 10^{3}$. Further, we scale the rewards and costs to ensure that they lie between $0$ and $1$. We analyze the behavior of the optimistic online mirror descent algorithm in Figure 1(b). We observe that the algorithm has three phases. The first phase is the first episodes where the algorithm uses a uniform policy which is the initial flat area till first $5000$ steps. In the second phase, the algorithm updates the policy for the first time and starts converging to the optimal policy with a convergence rate which matches to that of the PS-CURL algorithm. However, after few policy updates, we observe that the algorithm has oscillatory behavior which is because the dual variable updates require online constraint violations.

Finally, we analyze the the algorithm by Singh et al. (2020). They also provide an algorithm which proceeds in epochs and solves an optimization problem at every epoch. The algorithm considers a fixed epoch length $T^{1/3}$. Further, the algorithm considers a confidence interval on each estimate of $P(s^{\prime}|s,a)$ for all $s,a,s^{\prime}$ triplet. The algorithm does not perform well even though it updates the policy most frequently because of creating confidence intervals on individual transition probabilities $P(s^{\prime}|s,a)$ instead of the probability vector $P(s^{\prime}|s,a)$.

From the experimental observations, we note that the proposed UC-CURL algorithm is suitable in cases where the parameter tuning is not possible and the system requires tighter bounds on deviation of the performance of the algorithm. The PS-CURL algorithm can be used in cases where the variance in algorithm’s performance can be tolerated or computational complexity is a constraint. Further, for both the algorithms, it is beneficial to use the linear increasing epoch lengths. Additionally, the algorithm by Chen et al. (2022) is suitable for cases where solving an optimization equation is not feasible, for example an embedded system, as the algorithm updates policy using exponential function which can be easily computed. However, this algorithm is only applicable in applications with linear reward/constraint and single constraint.

We considered the problem of Markov Decision Process with concave objective and convex constraints. For this problem, we proposed UC-CURL algorithm which works on the principle of optimism. To bound the constraint violations, we solve for a conservative policy using an optimistic model for an $\epsilon$-tight optimization problem. Using an analysis based on Bellman error for infinite-horizon MDPs, we show the UC-CURL algorithm achieves $0$ constraint violations with a regret bound of $\tilde{O}(LdT_{M}S\sqrt{A/T}+(CSA\log T)/(T(1-\rho))$. Further, to reduce the computation complexity of finding optimistic MDP, we also propose a posterior sampling algorithm which finds the optimal policy for a sampled MDP. We provide a Bayesian regret bound of $\tilde{O}(LdT_{M}S\sqrt{A/T}+(CT_{M}S^{2}A\log T)/(T(1-\rho))$ for the posterior sampling algorithm by considering a stronger Slater’s condition to solve for constrained optimization for sampled MDPs as well. As part of potential future works, we consider dynamically configuring $K$ to be an interesting and important direction to reduce the requirement of problem parameters.