A Distributed Primal-Dual Method for Constrained Multi-agent Reinforcement Learning with General Parameterization

Reinforcement Learning (RL) has gained significant attention in recent years due to its success in solving complex decision-making problems across diverse domains [1], [2]. Traditional RL focuses on minimizing cumulative costs for a single agent or a set of agents acting independently. However, many real-world scenarios—such as traffic control [3], finance [4], and smart grids [5]—involve multiple agents that must cooperate or compete in a shared environment, leading to the study of Multi-agent Reinforcement Learning (MARL). In this framework, agents interact with each other and the environment, seeking to optimize their individual or joint policies while accounting for the influence of other agent’s actions (see, e.g., [6], [7], [8]). Moreover, a critical requirement for multi-agent systems in real-world scenarios is ensuring the independence of agents during online learning and operation. In many practical applications, centralized controllers face challenges related to scalability, real-time processing, and communication efficiency. To address these issues, distributed methods have been developed [9], [10], [11], [12].

While MARL holds great promise, many practical applications impose constraints that must be respected to ensure safety, fairness, or efficiency. For example, in networked microgrid management, maintaining power balance and preventing system overloads are critical for system stability [13]; similarly, in electric vehicle rebalancing systems, agents must consider factors such as battery life and access to charging stations [14]. Incorporating these types of constraints into reinforcement learning leads to the development of CMARL, which extends MARL frameworks to handle complex environments where satisfying constraints is as crucial as maximizing objective costs.

CMARL is framed within the Constrained Markov Game (CMG) framework [15], where each agent has its own local constraints and objective costs. In this work, we address a cooperative CMARL problem in which all agents aim to minimize a global objective function subject to global constraints, which are composed of local costs. We propose and analyze a distributed online method for solving this cooperative CMARL problem.

Although recent advancements in constrained single-agent reinforcement learning have demonstrated a zero duality gap [16], [17], the duality gap for the cooperative CMARL problem can be non-zero [18]. This inherent complexity poses significant challenges for algorithms attempting to satisfy global constraints [18]. In our study, we provide an analysis on the feasibility and sub-optimality of the equilibrium point of the proposed online algorithm.

Related Works: A widely used approach for solving single-agent constrained reinforcement learning problems is the primal-dual method, which demonstrates a zero duality gap by converting the original problem into an unconstrained Markov decision process with a Lagrangian cost [16], [19], [17]. This relaxed MDP is solved through alternating updates of the primal and dual variables. This approach has been extended to cooperative CMARL by relaxing the constrained problem into an unconstrained cooperative MARL [20], [18], [21]. Such relaxation enables the use of existing MARL algorithms, such as distributed actor-critic methods for networked agents [11].

Although there are many works on CMARL, only a few address the general problem where agent’s local costs are coupled through global constraint functions. For instance, in the CMARL formulation of [22], each agent’s actions indirectly influence others through state transition dynamics, but the method requires some coordination, preventing full decentralization. In contrast, [21] achieves decentralization through parameter sharing among agents, though it solves a distributed constrained MDP with networked agents rather than a true CMARL problem. This approach assumes homogeneous agents, with policies converging to a consensus. A recent extension by [23] builds on this by reducing gradient estimation variance for improved scalability.

For general CMARL formulation, [24] adopts the centralized training, decentralized execution framework, improving computational efficiency and scalability in large-scale multi-agent environments. It separates an agent’s policy into two components: a base policy for reward maximization and a perturbation policy for constraint satisfaction. However, it requires communication between agents during execution, distinguishing it from fully distributed methods.

Another notable work is [25], which operates in distributed settings. They propose a scalable method for general utility and constraint functions, modeled as nonlinear functions of the state-action occupancy measure. Their approach decomposes the state space for each agent, directly estimating local state-action occupancy measures while leveraging spatial correlation decay and truncated policy gradient estimators for scalability and convergence. However, despite its general policy parameterization, directly estimating local occupancy measures remains challenging in large state-action spaces.

Contributions: In this work, we propose a distributed algorithm for the CMARL problem with networked agents, where each agent maintains its own local primal and dual variables. Specifically, we estimate the Lagrangian multipliers of the global CMARL problem using only local information, enabling a fully distributed policy update method. To the best of our knowledge, this problem has not been addressed previously. Our approach builds on the method of [11] for unconstrained MARL. As such, our method can be viewed as a constrained extension of [11]. The main contributions of the article are as follows.

1. 1.

   We propose a fully distributed formulation for Lagrange multipliers in the cooperative CMARL problem over networked agents.

2. 2.

   We develop a distributed online primal-dual algorithm based on actor-critic methods with general function approximation.

3. 3.

   We prove the consensus of the Lagrange multipliers and the convergence of the proposed algorithm to an equilibrium point.

4. 4.

   We analyze feasibility of constraints, and sub-optimality of the algorithm’s equilibrium point compared to the exact solution of the unparameterized primal CMARL problem.

Notation: Let $\mathbb{R}^{d}$ denote the $d$-dimensional Euclidean space, and let $\|\cdot\|$ represent the Euclidean norm for vectors and the spectral norm for matrices. The $L_{1}$ norm, denoted by $\|\cdot\|_{1}$, is the sum of the absolute values of the vector components. We define the gradient operator by $\nabla$. The notation $\mathbf{1}$ represents a vector of ones, and $\mathds{1}_{A}$ denotes the indicator function for the set $A$. Additionally, $\mathbf{I}_{d}$ denotes the identity matrix of size $d\times d$. We define the matrix $J$ as the projection matrix onto the consensus subspace, given by $J:=\frac{1}{N}\mathbf{1}\mathbf{1}^{\top}\otimes\mathbf{I}_{d},$ where $\otimes$ denotes the Kronecker product. The orthogonal projection onto the subspace orthogonal to the consensus subspace is given by: $J_{\perp}:=\mathbf{I}_{Nd}-J.$ For any vector $x\in\mathbb{R}^{Nd}$, the consensus component is defined by $\langle x\rangle:=\frac{1}{N}(\mathbf{1}^{\top}\otimes\mathbf{I}_{d})x=\frac{1}{N}(x_{1}+\dots+x_{N})$, where $x_{i}\in\mathbb{R}^{d}$ for $\{i=1,\dots,N\}$. The disagreement component of $x$ relative to the consensus subspace is given by $x_{\perp}:=J_{\perp}x$. Thus, any vector $x$ can be decomposed as $x=\mathbf{1}\otimes\langle x\rangle+x_{\perp}.$ Furthermore, we denote the projected dynamical system corresponding to the projection operator $\Gamma_{C}:\mathbb{R}^{m}\rightarrow C$ using $\dot{x}=\hat{\Gamma}_{C}[f(x)],$ where $\hat{\Gamma}_{C}[f(x)]=\lim_{0<\eta\rightarrow 0}\left[\frac{\Gamma_{C}[x+\eta f(x)]-x}{\eta}\right]$.

In this section, we formally define the cooperative CMARL problem based on the CMG framework. We then propose a novel distributed problem formulation for CMARL, in which networked agents update their policies using only local information. To begin, we consider a CMG, which is represented as a tuple $\text{CMG}=\{\mathcal{S},\mathcal{A},P,\mathcal{C},\mathcal{G},b\}$. This framework involves $N$ agents, subject to $K$ constraints, where agents and constraints are indexed by $n\in\mathcal{N}=\{1,\cdots,N\}$ and $k\in\mathcal{K}=\{1,\cdots,K\}$, respectively. The components of the CMG tuple include the following: $\mathcal{S}$ is the finite state space. At each time step $t$, a random state $s^{t}\in\mathcal{S}$ is drawn. $\mathcal{A}=\prod_{n=1}^{N}\mathcal{A}_{n}$ represents the finite joint action space, where $\mathcal{A}_{n}$ is the action space for agent $n$. The joint action of all agents is denoted by $a=(a_{1},a_{2},\ldots,a_{N})^{\top}$, where $a_{n}$ is the action of agent $n$, and $a_{-n}=(a_{1},\ldots,a_{n-1},a_{n+1},\ldots,a_{N})^{\top}$ denotes the joint action vector of all agents except agent $n$, i.e., $a_{-n}\in\mathcal{A}_{-n}=\prod_{j\neq n}\mathcal{A}_{j}$. Thus, the joint action can be succinctly expressed as $a=(a_{n},a_{-n})$. $P(s^{\prime}|s,a)$ is the probability transition kernel, representing the probability of transitioning from state $s\in\mathcal{S}$ to state $s^{\prime}\in\mathcal{S}$ under the joint action $a\in\mathcal{A}$. $\mathcal{C}=\{c_{n}(s,a)\mid n\in\mathcal{N}\}$ represents the set of local costs for each agent. The local cost $c_{n}(s,a):\mathcal{S}\times\mathcal{A}\rightarrow\mathbb{R}$ describes the expected cost for agent $n$ at state $s$ under the joint action $a$. Specifically, $c_{n}(s,a)=\mathbb{E}[c_{n}^{t+1}\mid s^{t}=s,a^{t}=a]$, where $c_{n}^{t+1}$ is the immediate cost incurred by agent $n$ at time step $t+1$. $\mathcal{G}=\{g_{n,k}(s,a)\mid n\in\mathcal{N},k\in\mathcal{K}\}$ represents the set of local constraint cost for each agent. The local constraint cost $g_{n,k}(s,a):\mathcal{S}\times\mathcal{A}\rightarrow\mathbb{R}$ describes the expected $k$-th constraint cost for agent $n$ under state $s$ and joint action $a$, i.e., $g_{n,k}(s,a)=\mathbb{E}[g_{n,k}^{t+1}\mid s^{t}=s,a^{t}=a]$, where $g_{n,k}^{t+1}$ is the immediate $k$-th constraint cost incurred by agent $n$ at time step $t+1$. $\mathbf{b}=(b_{1},b_{2},\ldots,b_{K})^{\top}$ is the upper bound vector for the set of constraints, where each $b_{k}$ corresponds to the upper bound for the $k$-th cost function.

CMARL is formulated within the CMG framework, where each agent $n$ operates under an individual policy $\pi_{n}\in\mathcal{P}_{n}:=\Delta(\mathcal{A}_{n})^{\mathcal{S}}$, with $\Delta(\mathcal{A}_{n})$ representing the probability simplex over the action space $\mathcal{A}_{n}$. Since the agents make decisions independently, the joint policy of the all agents is expressed as the product of individual policies, $\pi=\prod_{n\in\mathcal{N}}\pi_{n}\in\mathcal{P}=\prod_{n\in\mathcal{N}}\mathcal{P}_{n}$.

In this study, we focus on a general class of parameterized policies. Each agent $n$ operates with a set of policy parameters $\theta_{n}\in\Theta_{n}$, where $\Theta_{n}\subseteq\mathbb{R}^{m}$ is a compact and convex set. This assumption is particularly beneficial in environments with large state spaces $\mathcal{S}$, as it facilitates more efficient policy representation. The policy parameters for all agents are concatenated as $\theta=\left[\left(\theta_{1}\right)^{\top},\cdots,\left(\theta_{N}\right)^{\top}\right]^{\top}\in\Theta$, where $\Theta=\prod_{n\in\mathcal{N}}\Theta_{n}$, forming the product parameter space. The corresponding product parameterized policy is denoted by $\pi_{\theta}(a|s)=\prod_{n\in\mathcal{N}}\pi_{\theta_{n}}\left(a_{n}|s\right)$. Furthermore, we define $\theta_{-n}=\left[\left(\theta_{1}\right)^{\top},\cdots,\left(\theta_{n-1}\right)^{\top},\left(\theta_{n+1}\right)^{\top},\cdots,\left(\theta_{N}\right)^{\top}\right]^{\top}\in\Theta_{-n}$, where $\Theta_{-n}=\prod_{j\neq n}\Theta_{j}$ represents the concatenated parameters of all agents except agent $n$. Therefore, the overall parameter vector can be expressed as $\theta=(\theta_{n},\theta_{-n})$.

In CMARL, agents collaborate to find an optimal, feasible policy that minimizes the global objective cost function, which is defined by the total average of the local objective costs across all agents. Although each agent operates independently and utilizes only its local information, their collective goal is to minimize this global objective function. The formal definition of the global objective cost $J(\pi_{\theta})$ is expressed as:

Here, the local cost function is defined as $c(s,a)=\frac{1}{N}\sum_{n\in\mathcal{N}}c_{n}(s,a)$, representing the average cost across all agents. In a similar manner to the global objective cost, we define the global constraint functions, which agents are expected not to violate. The global constraint functions are given by:

where $g_{k}(s,a)=\frac{1}{N}\sum_{n\in\mathcal{N}}g_{n,k}(s,a)$ represents the average constraint cost across all agents for the $k$-th constraint. Additionally, we denote $G(\pi_{\theta})=(G_{1}(\pi_{\theta}),\cdots,G_{K}(\pi_{\theta}))^{\top}$. It is important to note that the global constraint functions must be satisfied collectively by all agents and are not specific to any single agent. In fact, the constraint functions in (2) impose implicit restrictions on the product parameter space $\Theta$. Therefore, we define the feasible product parameter space as $\Theta_{C}$, where $\Theta_{C}:=\{\theta\in\Theta\mid G_{k}(\pi_{\theta})\leq b_{k},\,\forall k\in\mathcal{K}\}$. The global Lagrangian function for this stochastic multi-agent optimization problem is then expressed as:

where, $L(s,a;\lambda)=c(s,a)+\sum_{k=1}^{K}\lambda_{k}\left(g_{k}(s,a)-b_{k}\right)$ represents the expected global Lagrangian cost, where $\lambda=(\lambda_{1},\ldots,\lambda_{K})^{\top}$ is the global Lagrange multiplier vector associated with the global constraint functions, and $\lambda_{k}\geq 0$ for all $k\in\mathcal{K}$. Using this formulation, the cooperative CMARL problem can be formulated as a minimax optimization problem $\min_{\theta\in\Theta}\max_{\lambda\in\mathbb{R}_{+}^{K}}\mathcal{L}(\pi_{\theta},\lambda)$. Thus, the corresponding dual problem is expressed as $\max_{\lambda\in\mathbb{R}_{+}^{K}}\min_{\theta\in\Theta}\mathcal{L}(\pi_{\theta},\lambda)$. furthermore it is worth noting, the primal and dual problem of CMARL may have different solutions since the cooperative CMARL problem has a nonzero duality gap[18]. Nevertheless, we are interested in solving the dual problem, as it is more suitable for distributed constrained optimization [26], [27]. For a given Lagrange multiplier vector, the inner minimization in the dual problem reduces to an unconstrained MARL problem. In this framework, each agent is assigned a local Lagrangian cost and works collaboratively to minimize the global Lagrangian function, relying solely on local information. By utilizing this approach, we can alternately update the primal variables and the Lagrange multipliers. In this work, we propose a distributed method for updating the Lagrange multipliers to efficiently solve the dual problem.

To address the CMARL problem in a distributed manner, we introduce the concept of locally estimated Lagrangian multipliers, denoted as $\{\hat{\lambda}_{n,k}\mid n\in\mathcal{N},k\in\mathcal{K}\}$. where, $\hat{\lambda}_{n,k}$ represents the local estimation by the $n$-th agent of the global Lagrange multiplier $\lambda_{k}$. The vector of local estimates for the $n$-th agent is denoted as $\hat{\lambda}_{n}=(\hat{\lambda}_{n,1},\cdots,\hat{\lambda}_{n,K})^{\top}$. Additionally, the augmented vector of all local estimates across all agents is represented as $\hat{\lambda}=(\hat{\lambda}_{1}^{\top},\cdots,\hat{\lambda}_{N}^{\top})^{\top}$.

By defining $\hat{\lambda}_{n}$ and considering that each agent operates with local parameters $(\theta_{n},\hat{\lambda}_{n})$, we introduce the decomposed Lagrangian objective function as follows:

where $L_{n}(s,a;\hat{\lambda}_{n})=c_{n}(s,a)+\sum_{k=1}^{K}\hat{\lambda}_{n,k}\left(g_{n,k}(s,a)-b_{k}\right)$ represents the expected local Lagrangian cost for agent $n$. Accordingly, we define the immediate local Lagrangian cost for agent $n$ at time step $t+1$ as $l_{n}^{t+1}(\hat{\lambda}_{n})$, and the global Lagrangian cost at time step $t+1$, aggregated over all agents, as $l^{t+1}(\hat{\lambda})=\frac{1}{N}\sum_{n\in\mathcal{N}}l_{n}^{t+1}(\hat{\lambda}_{n})$.

For a given set of locally estimated Lagrange multipliers, the decomposed Lagrangian objective function can be treated as an unconstrained objective cost function, which is solvable using distributed actor-critic methods, as described in [11]. Furthermore, when the locally estimated Lagrange multipliers satisfy the consensus condition $\hat{\lambda}_{n,k}=\lambda_{k}$ for all $n\in\mathcal{N}$ and $k\in\mathcal{K}$, the decomposed Lagrangian objective function becomes equivalent to the global Lagrangian function in (3), i.e., $\hat{\mathcal{L}}(\pi_{\theta},\hat{\lambda}_{1},\dots,\hat{\lambda}_{N})=\mathcal{L}(\pi_{\theta},\lambda)$.

Accordingly, for a continuously differentiable policy $\pi_{\theta}$ and given $\hat{\lambda}$, the gradient of the decomposed Lagrangian objective function $\hat{\mathcal{L}}(\theta,\pi_{\theta})$ with respect to the parameters $\theta_{n}$ is given by the policy gradient theorem for MARL [11]:

where $A_{n}^{\pi_{\theta}}(s,a)$ is the advantage function for agent $n$, defined as:

The term $Q^{\pi_{\theta}}(s,a)$ refers to the global differential expected action-value function of the decomposed Lagrangian objective, and is defined as follows [28]:

We parameterize the function $Q^{\pi_{\theta}}(s,a)$ linearly as follows $Q^{\pi_{\theta}}(s,a;w)=\phi(s,a)^{\top}w,$ where $\phi(s,a)$ represents the feature vector corresponding to each state-action pair, and each agent maintains its own local set of weights $w_{n}\in\mathbb{R}$.

According to the CMARL problem formulation, the proposed algorithm, outlined in Algorithm 1, operates in two phases during each iteration. In the first phase, for a given $\hat{\lambda}_{n}^{t}$, agents update their critic and actor weights using local information to solve an unconstrained multi-agent problem for a decomposed Lagrangian objective function. In this phase, each agent computes and utilizes $l_{n}^{t+1}(\hat{\lambda}_{n})$ as a surrogate immediate objective cost. This phase is based on the Algorithm 1 presented in [11]. Each agent maintains its critic parameters, denoted by $w_{n}$, which must be shared locally according to the communication graph, with the weight matrix denoted by $\mathrm{C}^{t}=[c_{n,n^{\prime}}^{t}]$ for all $n,n^{\prime}\in\mathcal{N}$. The critic update rule for each agent $n$ in this phase is given by:

Here, $\alpha^{t}$ represents the critic learning rate, and the temporal difference (TD) error is updated using the following rule:

Additionally, in the first phase, we update the actors locally using the policy gradient theorem for MARL, as described in (5) for sample based gradient estimation of Lagrangian objective cost. The actor update is defined as follows:

where, $\beta^{t}$ represents the actor learning rate. The parameter $\theta_{n}$ is updated in the descent direction within the set $\Theta$, where $P_{\Theta_{n}}$ denotes the projection onto the set $\Theta_{n}$. Next, we propose the second phase, which focuses on updating the locally estimated Lagrange multipliers. This phase also involves estimating the constraint cost function. Since agents do not have access to the global constraint cost, these functions are estimated locally using a recursive update as follows:

where, $\hat{G}_{n}=(\hat{G}_{n,1},\cdots,\hat{G}_{n,K})^{\top}$ and $g^{t}_{n}=(g_{n,1}^{t},\cdots,g_{n,K}^{t})^{\top}$ are defined for notational simplicity. Finally, we present the update rule for the Lagrange multipliers, which involves taking a step in the ascent direction with respect to the decomposed Lagrangian objective function using the estimated values:

where, $\gamma^{t}$ represents the learning rate for the Lagrange multipliers, and $P_{\Lambda}$ denotes the projection onto the set $\Lambda\subseteq\mathbb{R}_{+}^{K}$.

This section consists of two parts. In the first part, we prove the convergence of Algorithm 1 to a stationary point of the update rules in (3)-(10). In the second part, we analyze the sub-optimality of the convergent point by deriving an upper bound on the gap between the obtained solution and the optimal solution.