Learning with Linear Function Approximations in Mean-Field Control^††thanks: E. Bayraktar is partially supported by the National Science Foundation under grant DMS-2106556 and by the Susan M. Smith chair.

Learning for multi agent control problems is a practically relevant and a challenging problem where there has been as a growing interest in recent years. A general solution methodology for multi-agent control problems is difficult to obtain and the solution, in general, is intractable except for special information structures between the agents. We refer the reader to the survey paper by [43] for a substantive summary of learning methods in the context of multi-agent decision making problems.

In this paper, we study a particular case of multi-agent problems in which both the agents and their interactions are symmetric and homogeneous. For these mean-field type decision making problems, the agents are coupled only through the so-called mean-field term. These problems can be broadly divided into two categories; mean-field game problems where the agents are competitive and interested in optimizing their self objective functions, and mean-field control problems, where the agents are interested in a common objective function optimization. We cite some papers by [21, 12, 11, 42, 26, 2, 18, 19, 24, 35, 40, 38, 39] and references therein, for papers in mean-field game setting. We do not discuss these in detail as our focus will be on mean-field control problems which are significantly different in both analysis and the nature of the problems of interest.

For mean-field control problems, where the agents are cooperative and work together to minimize (or maximize) a common cost (or reward) function, see [8, 17, 30, 14, 36, 13, 20, 7, 10] and references therein for the study of dynamic programming principle and learning methods in continuous time. In particular, we point out the papers [28, 17] which provide the justification for studying the centralized limit problem by rigorously connecting the large population decentralized setting and the infinite population limit problem.

For papers studying mean-field control in discrete time, we refer the reader to [33, 5, 22, 23, 32, 15]. [33, 32] study existence of solutions to the control problem in both infinite and finite population settings, and they rigorously establish the connection between the finite and infinite population problems. [5] studies the finite population mean-field control problems and their infinite population limit, and provide solutions of the ergodic control problems for some special cases.

In the context of learning, [22, 23] study dynamic programming principle and Q learning methods directly for the infinite population control problem. The value functions and the Q functions are defined for the lifted problem, where the state process is the measure-valued mean-field flow. They consider dynamics without common noise, and thus the learning problem from the perspective of a coordinator becomes a deterministic one.

[15] also considers the limit (infinite population) problem and studies different classes of policies that achieve optimal performance for the infinite population (limit problem) and focuses on Q learning methods for the problem after establishing the optimality of randomized feedback policies for the agents. The learning problem considers the state as the measure valued mean-field term and defines a learning problem over the set of probability measures where various approximations are considered to deal with the high dimension issues.

[4, 3] have studied learning methods for the mean-field game and control problems from a joint lens. However, for the control setup, they consider a different control objective compared to the previously cited papers. In particular, they aim to optimize the asymptotic phase of the control problem where the agents are assumed to reach to their stationary distributions under joint symmetric policies. Furthermore, the agents only use their local state variables, and thus the objective is to find a stationary measure for the agents where the cost is minimized under this stationary regime. Since the agents only use their local state variables (and not the mean-field term) for their control, the authors can define a Q function over the finite state and action spaces of the agents.

[34] consider a closely related problem to our setting, where they propose model-based learning methods for the mean-field control. Similar to [22, 23], they directly work with the infinite population dynamics without analyzing the approximation consistency between the finite-population dynamics and their infinite-population counterpart. Furthermore, they restrict the dynamics to the models with additive noise, and the optimality search is within deterministic and Lipschitz continuous controls.

We also note that there are various studies that focus on the application of the mean-field modeling using numerical methods based on machine learning techniques, see e.g. the works by [37, 1, 29].

In this paper, we will consider the learning problem using an alternative formulation where the state is represented as the measure valued mean-field term. To approximate this uncountable space, and the cost and transition functions, different from the previous works in the mean-field control setting, we will consider linear function approximation methods. These methods have been studied well for single agent discrete time stochastic control problems. We cite papers by [31, 16, 41, 27] in which reinforcement learning techniques are used to study Markov decision problems with continuous state spaces using linear function approximations.

Contributions.

• •

  In Section 2, we present the learning methods using linear function approximation. We focus on various scenarios.

  • –

    We first consider the ideal case where we assume that the team has infinitely many agents. For this case, we study; (i) learning by a coordinator who has access to information about every agent in the team, and estimates a model from a data set by fitting a linear model that minimizes the $L_{2}$ distance between the training data and the estimate linear model, (ii) each agent estimates their own linear model using their local information via an iterative algorithm from a single sequence of data.

  • –

    In Section 2.3, we consider the practical case, where the team has finitely many agents, and they aim to estimate a linear model from a single sequence of data, using their local information variables.

• •

  The methods we study in Section 2 minimize the $L_{2}$ distance between the learned linear model and the actual model under a probability measure that depends on the training data. However, to find upper bounds for the performance loss of the policies designed for the learned linear estimates in any scenario, we need uniform estimation errors rather than $L_{2}$ estimation errors. In Section 3, we generalize $L_{2}$ error bounds to uniform error bounds.

• •

  The proposed learning methods do not match the true model perfectly in general, due to linear approximation mismatch. Therefore, finally, in Section 4, we provide upper bounds on the performance of the policies that are designed for the learned models when they are applied on the true control problem. We note that the flow of the mean-field term is deterministic for infinitely many agents, and thus can be estimated using the dynamics without observing the mean-field term. Therefore, for the execution of the policies we focus on two methods, (i) open loop control, where the agents only observe their local states and estimate the mean-field term with the learned dynamics, (ii) closed loop control where the agents observe both their local information and the mean-field term. For each of these execution procedures, we provide upper bounds for the performance loss. As in Section 2, we first consider the ideal case where it is assumed that the system has infinitely many agents. In this case, the error bound depends on the uniform model mismatch between the learned model and the true model. We then consider the case with finitely many agents. We assume that each agent follows the policy that they calculate considering the limit (infinite population) model. In this case, the error upper bounds depend on both the uniform model mismatch, and an empirical concentration bound since we estimate the finitely many agent model with the infinite population limit problem.

The dynamics for the model are presented as follows: suppose $N$ agents (decision-makers or controllers) act in a cooperative way to minimize a cost function, and the agents share a common state and an action space denoted by $\mathds{X}$ and $\mathds{U}$. We assume that $\mathds{X}$ and $\mathds{U}$ are finite. We refer the reader to the paper by [6], for finite approximations of mean-field control problems where the state and actions spaces of the agents are continuous. For any time step $t$, and agent $i\in\{1,\dots,N\}$ we have

$\displaystyle x^{i}_{t+1}=f(x_{t}^{i},u_{t}^{i},\mu_{\bf x_{t}},w_{t}^{i})$

(1)

for a measurable function $f$, where $\{w_{t}^{i}\}$ denotes the i.i.d. idiosyncratic noise process.

Furthermore, $\mu_{\bf x}\in\mathcal{P}_{N}(\mathds{X})$ denotes the empirical distribution of the agents on the state space $\mathds{X}$ such that for a given joint state of the team of agents ${\bf x}:=(x^{1},\dots,x^{N})\in\mathds{X}^{N}$

$\displaystyle\mu_{\bf x}(\cdot):=\frac{1}{N}\sum_{i=1}^{N}\delta_{x^{i}}(\cdot)$

where $\delta_{x^{i}}$ represents the Dirac measure centered at $x^{i}$. Throughout this paper, we use the notation

$\displaystyle\mathds{X}^{N}:=\underbrace{\mathds{X}\times\dots\times\mathds{X}}_{\text{ $N$ times}}$

to denote the space of all joint state variables of the team equipped with the product topology on $\mathds{X}^{N}$. We further define $\mathcal{P}_{N}(\mathds{X})$, the set of all empirical measures on $\mathds{X}$ constructed using sequences of $N$ states in $\mathds{X}$, such that

$\displaystyle\mathcal{P}_{N}(\mathds{X}):=\{\mu_{\bf x}:{\bf x}=(x^{1},\dots,x^{N})\in\mathds{X}^{N}\}.$

Note that $\mathcal{P}_{N}(\mathds{X})\subset{\mathcal{P}}(\mathds{X})$ where ${\mathcal{P}}(\mathds{X})$ denotes the set of all probability measures on $\mathds{X}$ equipped with the weak convergence topology.

Equivalently, the next state of the agent $i$ is determined by some stochastic kernel, that is, a regular conditional probability distribution:

$\displaystyle\mathcal{T}(\cdot|x_{t}^{i},u_{t}^{i},\mu_{\bf x_{t}}).$

(2)

At each time stage $t$, each agent receives a cost determined by a measurable stage-wise cost function $c:\mathds{X}\times\mathds{U}\times\mathcal{P}_{N}(\mathds{X})\to\mathds{R}$. If the state, action, and empirical distribution of the agents are given by $x_{t}^{i},u_{t}^{i},\mu_{\bf x_{t}}$, then the agent receives the cost.

$\displaystyle c(x_{t}^{i},u_{t}^{i},\mu_{\bf x_{t}}).$

For the remainder of the paper, by an abuse of notation, we will sometimes denote the dynamics in terms of the vector state and action variables, ${\bf x}=(x^{1},\dots,x^{N})$, and ${\bf u}=(u^{1},\dots,u^{N})$, and vector noise variables ${\bf w}=(w^{1},\dots,w^{N})$ such that

$\displaystyle{\bf x_{t+1}}=f({\bf x_{t},u_{t},w_{t}}).$

For the initial formulation, every agent is assumed to know the state and action variables of every other agent. We define an admissible policy for an agent $i$, as a sequence of functions $\gamma^{i}:=\{\gamma^{i}_{t}\}_{t}$, where $\gamma^{i}_{t}$ is a $\mathds{U}$-valued (possibly randomized) function which is measurable with respect to the $\sigma$-algebra generated by

$\displaystyle I_{t}=\{{\bf x}_{0},\dots,{\bf x}_{t},{\bf u}_{0},\dots,{\bf u}_{t-1}\}.$

(3)

Accordingly, an admissible team policy, is defined as $\gamma:=\{\gamma^{1},\dots,\gamma^{N}\}$, where $\gamma^{i}$ is an admissible policy for the agent $i$. In other words, agents share the complete information.

The objective of the agents is to minimize the following cost function

$\displaystyle J^{N}_{\beta}({\bf x}_{0},\gamma)=\sum_{t=0}^{\infty}\beta^{t}E_{\gamma}\left[{\bf c}({\bf x}_{t},{\bf u}_{t})\right]$

where $E_{\gamma}$ denotes the expectation with respect to the probability measure induces by the team policy $\gamma$, and where

$\displaystyle{\bf c}({\bf x}_{t},{\bf u}_{t}):=\frac{1}{N}\sum_{i=1}^{N}c(x^{i}_{t},u^{i}_{t},\mu_{{\bf x}_{t}}).$

The optimal cost is defined by

$\displaystyle J_{\beta}^{N,*}({\bf x}_{0}):=\inf_{\gamma\in\Gamma}J_{\beta}^{N}({\bf x}_{0},\gamma)$

(4)

where $\Gamma$ denotes the set of all admissible team policies.

We note that this information structure (3) will be our benchmark for evaluating the performance of the approximate solutions using simpler information structures presented in the paper. In other words, the value function that is achieved when the agents share full information and full history will be taken to be our reference point for simpler information structures.

For example, one immediate observation is that the problem under full information sharing can be reformulated as a centralized control problem where the state and action spaces are $\mathds{X}^{N}$ and $\mathds{U}^{N}$. Therefore, one can consider Markov policies such that $I_{t}=\{{\bf x}_{t}\}$ without loss of optimality.

However, if the problem is modeled as an MDP with state space $\mathds{X}^{N}$ and action space $\mathds{U}^{N}$, we face some computational challenges:

• (i)

  the curse of dimensionality when $N$ is large, since $\mathds{X}^{N}$ and $\mathds{U}^{N}$ might be too large even when $\mathds{X,U}$ are of manageable size,

• (ii)

  the curse of coordination: even if the optimal team policy is found, its execution at the agent level requires coordination among the agents. In particular, the agents may need to follow asymmetric policies to achieve optimality, even though we assume full symmetry for the dynamics and the cost models. The following simple example from [9] shows that the agents may need to follow asymmetric policies to achieve optimality which requires coordination among the agents.

A standard approach to deal with mean-field control problems when $N$ is large is to consider the infinite population problem, i.e. taking the limit $N\to\infty$. A propagation of chaos argument can be used to show that in the limit, the agents become asymptotically independent. Hence, the problem can be formulated from the perspective of a representative single agent. This approach is suitable to deal with coordination challenges, as the correlation between the agents vanish in the limit, and thus the symmetric policies can achieve optimal performance for the infinite population control problem. In particular, for Example 1.1 in the infinite population setting, the optimal policy is to follow a randomized policy such that $Pr(u=1)=Pr(u=0)=\frac{1}{2}$. We will introduce the limit problem in Section 1.5 and make the connections between the limit problem and the finite population problem rigorous.

Recall that we assume that the state and action spaces of agents $\mathds{X,U}$ are finite (see [6] for finite approximations of continuous space mean-field control problems).

Note. Even though we assume that $\mathds{X}$ and $\mathds{U}$ are finite, we will continue using integral signs instead of summation signs for expectation computations due to notation consistency, by simply considering Dirac delta measures.

We metrize $\mathds{X}$ and $\mathds{U}$ so that $d(x,x^{\prime})=1$ if $x\neq x^{\prime}$ and $d(x,x^{\prime})=0$ otherwise. Note that with this metric, for any $\mu,\nu\in{\mathcal{P}}(\mathds{X})$ and for any coupling $Q$ of $\mu,\nu$, we have that

$\displaystyle E_{Q}\left[|X-Y|\right]=P_{Q}(X\neq Y)$

which in particular implies via the optimal coupling that

$\displaystyle W_{1}(\mu,\nu)=\|\mu-\nu\|_{TV}$

where $W_{1}$ denotes the first order Wasserstein distance, or the Kantorovich–Rubinstein metric, and $\|\cdot\|_{TV}$ denotes the total variation norm for signed measures.

Note further that for measures defined on finite spaces, we have that

$\displaystyle\|\mu-\nu\|_{TV}=\frac{1}{2}\|\mu-\nu\|_{1}=\frac{1}{2}\sum_{x}\left|\mu(x)-\nu(x)\right|.$

(5)

Hence, in what follows we will simply write $\|\mu-\nu\|$ to refer to the distance between $\mu$ and $\nu$, which may correspond to the total variation distance, the first order Wasserstein metric, or the normalized $L_{1}$ distance.

We also define the following Dobrushin coefficient for the kernel $\mathcal{T}$:

$\displaystyle\sup_{\mu,\gamma,x,\hat{x}}\|\int_{\mathds{U}}\mathcal{T}(\cdot|x,u,\mu)\gamma(du|x)-\int_{\mathds{U}}\mathcal{T}(\cdot|\hat{x},u,\mu)\gamma(du|\hat{x})\|=:\delta_{T}$

(6)

Realize that we always have $\delta_{T}\leq 1$. In certain cases, we can also have strict inequality, e.g. if there exists some $x^{*}\in\mathds{X}$ such that

$\displaystyle\mathcal{T}(x^{*}|x,u,\mu)<1-\alpha,\quad\forall x,u,\mu$

then one can show that $\delta_{T}\leq 1-\alpha<1.$

For the remaining part of the paper, we will often consider an alternative formulation of the control problem for the finitely many agent case where the controlled process is the state distribution of the agents, rather than the state vector of the agents. We refer the reader to [6] for the full construction; in this section, we will give an overview.

We define an MDP for the distribution of the agents, where the control actions are the joint distribution of the state and action vectors of the agents.

We let the state space be $\mathcal{Z}={\mathcal{P}}_{N}(\mathds{X})$ which is the set of all empirical measures on $\mathds{X}$ that can be constructed using the state vectors of $N$-agents. In other words, for a given state vector ${\bf x}=\{x^{1},\dots,x^{N}\}$, we consider $\mu_{\bf x}\in{\mathcal{P}}_{N}(\mathds{X})$ to be the new state variable of the team control problem.

The admissible set of actions for some state $\mu\in\mathcal{Z}$, is denoted by $U(\mu)$, where

$\displaystyle U(\mu)=\{\Theta\in{\mathcal{P}}_{N}(\mathds{U}\times\mathds{X})|\Theta(\mathds{U},\cdot)=\mu(\cdot)\},$

(7)

that is, the set of actions for a state $\mu$, is the set of all joint empirical measures on $\mathds{X}\times\mathds{U}$ whose marginal on $\mathds{X}$ coincides with $\mu$.

We equip the state space $\mathcal{Z}$, and the action sets $U(\mu)$, with the norm $\|\cdot\|_{TV}$ (see (5)) .

One can show that [6, 9] the empirical distributions of the states of agents $\mu_{t}$, and of the joint state and actions $\Theta_{t}$ define a controlled Markov chain such that

		$\displaystyle Pr(\mu_{t+1}\in B|\mu_{t},\dots\mu_{0},\Theta_{t},\dots,\Theta_{0})=Pr(\mu_{t+1}\in B|\mu_{t},\Theta_{t})$
		$\displaystyle:=\eta(B|\mu_{t},\Theta_{t})$		(8)

where $\eta(\cdot|\mu,\Theta)\in{\mathcal{P}}({\mathcal{P}}_{N}(\mathds{X}))$ is the transition kernel of the centralized measure valued MDP, which is induced by the dynamics of the team problem.

We define the stage-wise cost function $k(\mu,\Theta)$ by

$\displaystyle k(\mu,\Theta):=\int c(x,u,\mu)\Theta(du,dx)=\frac{1}{N}\sum_{i=1}^{N}c(x^{i},u^{i},\mu).$

(9)

Thus, we have an MDP with state space $\mathcal{Z}$, action space $\cup_{\mu\in\mathcal{Z}}U(\mu)$, transition kernel $\eta$ and the stage-wise cost function $k$.

We define the set of admissible policies for this measured valued MDP as a sequence of functions $g=\{g_{0},g_{1},g_{2},\dots\}$ such that at every time $t$, $g_{t}$ is measurable with respect to the $\sigma$-algebra generated by the information variables

$\displaystyle\hat{I}_{t}=\{\mu_{0},\dots,\mu_{t},\Theta_{0},\dots,\Theta_{t-1}\}.$

We denote the set of all admissible control policies by $G$ for the measure valued MDP.

In particular, we define the infinite horizon discounted expected cost function under a policy $g$ by

$\displaystyle K^{N}_{\beta}(\mu_{0},g)=E_{\mu_{0}}^{\eta}\left[\sum_{t=0}^{\infty}\beta^{t}k(\mu_{t},\Theta_{t})\right].$

We also define the optimal cost by

$\displaystyle K_{\beta}^{N,*}(\mu_{0})=\inf_{g\in G}K^{N}_{\beta}(\mu_{0},g).$

(10)

The following result shows that this formulation is without loss of optimality:

We now introduce the control problem for infinite population teams, i.e. for $N\to\infty$. For some agent $i\in\mathds{N}$, we define the dynamics as

$\displaystyle x^{i}_{t+1}=f(x_{t}^{i},u_{t}^{i},\mu_{t}^{i},w_{t}^{i})$

where $x_{0}\sim\mu_{0}$ and $\mu_{t}^{i}=\mathcal{L}(X_{t}^{i})$ is the law of the state at time $t$. The agent tries to minimize the following cost function:

$\displaystyle J_{\beta}^{\infty}(\mu_{0},\gamma)=\sum_{t=0}^{\infty}\beta^{t}E\left[c(X_{t}^{i},U_{t}^{i},\mu_{t}^{i})\right]$

where $\gamma=\{\gamma_{t}\}_{t}$ is an admissible policy such that $\gamma_{t}$ is measurable with respect to the information variables

$\displaystyle I_{t}^{i}=\left\{x_{0}^{i},\dots,x_{t}^{i},u_{0}^{i},\dots,u_{t-1}^{i},\mu_{0}^{i},\dots,\mu_{t}^{i}\right\}.$

Note that the agents are no longer correlated and they are indistinguishable. Hence, in what follows we will drop the dependence on $i$ when we refer to the infinite population problem.

The problem is now a single agent control problem; however, the state variable is not Markovian. However, we can reformulate the problem as an MDP by viewing the state variable as the measure valued $\mu_{t}$.

We let the state space to be ${\mathcal{P}}(\mathds{X})$. Different from the measure valued construction we have introduced in Section 1.4, we let the action space to be $\Gamma={\mathcal{P}}(\mathds{U})^{|\mathds{X}|}$. In particular, an action $\gamma(\cdot|x)\in\Gamma$ for the team is a randomized policy at the agent level. We equip $\Gamma$ with the product topology, where we use the weak convergence for each coordinate. We note that each action $\gamma(du|x)$ and state $\mu(dx)$ induce a distribution on $\mathds{X}\times\mathds{U}$, which we denote by $\Theta(du,dx)=\gamma(du|x)\mu(dx)$.

Recall the notation in (2); at time $t$, we can use the following stochastic kernel for the dynamics:

$\displaystyle x_{t+1}\sim\mathcal{T}(\cdot|x_{t},u_{t},\mu_{t})$

which is induced by the idiosyncratic noise $w^{i}_{t}$. Hence, we can define

$\displaystyle\mu_{t+1}=F(\mu_{t},\gamma_{t}):=\int\mathcal{T}(\cdot|x,u,\mu_{t})\gamma_{t}(du|x)\mu_{t}(dx).$

(11)

Note that the dynamics are deterministic for the infinite population measure valued problem. Furthermore, we can define the stage-wise cost function as

$\displaystyle k(\mu,\gamma):=\int c(x,u,\mu)\gamma(du|x)\mu(dx).$

(12)

Hence, the problem is a deterministic MDP for the measure valued state process $\mu_{t}$. A policy, say $g:{\mathcal{P}}(\mathds{X})\to\Gamma$ for the measure-valued MDP, can be written as

$\displaystyle g(\mu)=\gamma(du|x)$

for some $\mu\in{\mathcal{P}}(\mathds{X})$. That is, an agent observes $\mu$ and chooses their actions as an agent-level randomized policy $\gamma(du|x)$.

We reintroduce the infinite horizon discounted cost of the agents for the measure valued formulation:

$\displaystyle K_{\beta}(\mu_{0},g)=\sum_{t=0}^{\infty}\beta^{t}k(\mu_{t},\gamma_{t})$

for some initial mean-field term $\mu_{0}$ and under some policy $g$. Furthermore, the optimal policy is denoted by

$\displaystyle K_{\beta}^{*}(\mu_{0})=\inf_{g}K_{\beta}(\mu_{0},g).$

At a given time $t$, the pair $(x_{t},\mu_{t})$ can be used as sufficient information for decision making by the agent $i$. Furthermore, if the model is fully known by the agents, then the mean-field flow $\mu_{t}$ can be perfectly estimated if every agent agrees and follows the same policy $g(\mu)$, since the dynamics of $\mu_{t}$ is deterministic.

We note that for the infinite population control problem, the coordination requirement between the agents may be relaxed, though cannot be fully abandoned in general (see Section 1.6). In particular, if the agents agree on a common policy $g(\mu)=\gamma(du|x,\mu)$, then for the execution of this policy, no coordination or communication is needed since every agent can estimate the mean-field term $\mu_{t}$ independently and perfectly. Furthermore, every agent can use the same agent-level policy $\gamma(du|x,\mu)$ symmetrically, without any coordination with the other agents.

The following result makes the connection between the finite population and the infinite population control problem rigorous [32, 5, 9].

We have argued in the previous section that the team control problem can be solved near optimally by using the infinite population control solution. Furthermore, if the agents agree on the application of a common optimal policy, the resulting team policy can be executed independently in a decentralized way and achieves near-optimal performance.

The following example shows that if the agents do not coordinate on which policy to follow, i.e. if they are fully decentralized, then the resulting team policy will not achieve the desired outcome.

We note that the issue with the previous example results from the fact that the optimal policy is not unique. If the optimal policy can be guaranteed to be unique, then the agents can act fully independently.

In this section, we will consider an idealized scenario, where there are infinitely many agents, and a coordinator learns the model by linear function approximation.

We define the following sets for which the model and the cost functions can be learned perfectly within the training data: let $x\in\mathds{X}$, we define

$\displaystyle P_{x}:=\{\mu\in T:\mu(x)>0\}.$

(14)

$P_{x}\subset{\mathcal{P}}(\mathds{X})$ denotes the set of probability measures which assign positive measure to a particular state $x\in\mathds{X}$ that are also in the training data for the mean-field terms. In particular, for a given $(x,u)$ pair, the kernel $\mathcal{T}(\cdot|x,u,\mu)$ and the cost $c(x,u,\mu)$ can be learned perfectly for every $\mu\in P_{x}$ with Assumption 3.

For a given $x\in\mathds{X}$, we denote by $M_{x}$ the number of mean-field terms within the set $P_{x}$ (see (14)). For every $(x,u)\in\mathds{X}\times\mathds{U}$ pair, the coordinator aims to find ${\bf\theta}_{(x,u)}$ and ${\bf Q}_{(x,u)}$ such that

	$\displaystyle\frac{1}{M_{x}}\sum_{j=1}^{M_{x}}\left|c(x,u,\mu_{j})-{\bf\Phi^{\intercal}_{(x,u)}}(\mu_{j}){\bf\theta_{(x,u)}}\right|^{2}$
	$\displaystyle\frac{1}{M_{x}}\sum_{j=1}^{M_{x}}\left\|\mathcal{T}(\cdot|x,u,\mu_{j})-{\bf\Phi^{\intercal}_{(x,u)}}(\mu_{j}){\bf Q_{(x,u)}(\cdot)}\right\|$

is minimized.

The least squares linear models can be estimated in closed form for ${\bf\theta}_{(x,u)}$ and ${\bf Q}_{(x,u)}(\cdot)$ using the training data. We define the following vector and matrices to present the closed form solution in a more compact form: for each $(x,u)\in\mathds{X}\times\mathds{U}$ we introduce ${\bf b}_{(x,u)}\in\mathds{R}^{M_{x}}$, and ${\bf d}_{(x,u)}\in\mathds{R}^{M_{x}\times|\mathds{X}|}$,

$\displaystyle{\bf b}_{(x,u)}=\begin{bmatrix}c(x,u,\mu_{1})\\ c(x,u,\mu_{2})\\ \vdots\\ c(x,u,\mu_{M_{x}})\end{bmatrix},\quad{\bf d}_{(x,u)}=\begin{bmatrix}\mathcal{T}(x^{1}|x,u,\mu_{1})&\mathcal{T}(x^{2}|x,u,\mu_{1})&\dots&\mathcal{T}(x^{|\mathds{X}|}|x,u,\mu_{1})\\ \mathcal{T}(x^{1}|x,u,\mu_{2})&\mathcal{T}(x^{2}|x,u,\mu_{2})&\dots&\mathcal{T}(x^{|\mathds{X}|}|x,u,\mu_{2})\\ \vdots\\ \mathcal{T}(x^{1}|x,u,\mu_{M_{x}})&\mathcal{T}(x^{2}|x,u,\mu_{M_{x}})&\dots&\mathcal{T}(x^{|\mathds{X}|}|x,u,\mu_{M_{x}})\end{bmatrix}.$

(15)

Furthermore, we also define ${\bf A}_{(x,u)}\in\mathds{R}^{d\times M_{x}}$

$\displaystyle{\bf A}_{(x,u)}=\left[{\bf\Phi}_{(x,u)}(\mu_{1}),\dots,{\bf\Phi}_{(x,u)}(\mu_{M_{x}})\right].$

(16)

Assuming that ${\bf A}_{(x,u)}$ has linearly independent columns, i.e. ${\bf\Phi}_{(x,u)}(\mu_{i})$ and ${\bf\Phi}_{(x,u)}(\mu_{j})$ are linearly independent for $\mu_{i}\neq\mu_{j}$, the estimates for $\theta_{(x,u)}$ and ${\bf Q}_{(x,u)}$ can be written as follows

	$\displaystyle\theta_{(x,u)}$	$\displaystyle=\left({\bf A}^{\intercal}_{(x,u)}{\bf A}_{(x,u)}\right)^{-1}{\bf A}^{\intercal}_{(x,u)}{\bf b}_{(x,u)}$
	$\displaystyle{\bf Q}_{(x,u)}$	$\displaystyle=\left({\bf A}^{\intercal}_{(x,u)}{\bf A}_{(x,u)}\right)^{-1}{\bf A}^{\intercal}_{(x,u)}{\bf d}_{(x,u)}.$		(17)

Note that above, each row of ${\bf Q}_{(x,u)}$ represents a signed measure on $\mathds{X}$.

In this section, we will introduce a learning method where the agents perform independent learning to some extent. Here, rather than using a training set, we will focus on an online learning algorithm where at every time step, agent $i$ observes $x^{i},u^{i},X_{1}^{i},c(x^{i},u^{i},\mu),\mu$. That is, each agent has access to their local state, action, cost realizations, one-step ahead state, and the mean-field term. However, they do not have access to local information about the other agents.

We first argue that full decentralization is usually not possible in the context of learning either. Recall that the mean-field flow is deterministic if every agent follows the same independently randomized agent-level policy. Furthermore, the flow of the mean-field terms remains deterministic even when the agents choose different exploration policies if the randomization is independent. To see this, assume that each agent picks some policy $\gamma_{w}(du|x)$ randomly by choosing $w\in\mathds{W}$ from some arbitrary distribution, where the mapping $w\to\gamma_{w}(du|x)$ is predetermined. If the agents pick $w\in\mathds{W}$ independently, the mean-field dynamics is given by

$\displaystyle\mu_{t+1}(\cdot)=\int\mathcal{T}(\cdot|x,u,\mu_{t})\gamma_{w}(du|x)P_{w}(dw)\mu_{t}(dx)$

where $P_{w}(dw)$ is the distribution by which the agents perform their independent randomization for the policy selection. Hence, the mean-field term dynamics follow a deterministic flow. Note that for the above example, for simplicity, we assume that the agents pick according to the same distribution. In general, even if the agents follow different distributions for $w\in\mathds{W}$, the dynamics of the mean-field flow would remain deterministic according to a mixture distribution.

Deterministic behavior might cause poor exploration performance. There might be cases where the mean-field flow gets stuck at a fixed distribution without learning or exploring the ‘important’ parts of the space ${\mathcal{P}}(\mathds{X})$ sufficiently. To overcome this issue, and to make sure that the system is stirred sufficiently well during the exploration, one option is to introduce a common randomness for the selection of the exploration policies. In particular, each agent follows a randomized policy $\gamma_{w}(du|x)$ where the common randomness $w\in\mathds{W}$ is mutual information. Then the dynamics of the mean-field flow can be written as

$\displaystyle\mu_{t+1}(\cdot)=F(\mu_{t},w):=\int\mathcal{T}(\cdot|x,u,\mu_{t})\gamma_{w}(du|x)\mu_{t}(dx).$

(18)

The common noise variable ensures a level of coordination among the agents. However, this is still a significant relaxation compared to full coordination where the agents share their full state or control data. In this section, we show that agents can construct independent learning iterates that converge by coordinating through an arbitrary common source of randomness.

We assume the following for the mean-field flow during the exploration:

We now define the trained measures, $P_{x}\in{\mathcal{P}}({\mathcal{P}}(\mathds{X}))$ for each $(x,u)$ pair based on the invariant measure for the mean-field flow. Under Assumption 3, that is assuming $Pr(u|x)=\int\gamma_{w}(u|x)P_{w}(dw)>0$ for all $(x,u)$ pairs, we can write

	$\displaystyle P(\mu\in A|x,u)$	$\displaystyle=\frac{Pr(x,u,\mu\in A)}{Pr(x,u)}$
		$\displaystyle=\frac{\int_{\mu\in A}\int_{w}\gamma_{w}(u|x)P_{w}(dw)\mu(x)P(d\mu)}{\int_{\mu\in{\mathcal{P}}(\mathds{X})}\int_{w}\gamma_{w}(u|x)P_{w}(dw)\mu(x)P(d\mu)}$
		$\displaystyle=\frac{\int_{\mu\in A}\mu(x)P(d\mu)}{\int_{\mu\in{\mathcal{P}}(\mathds{X})}\mu(x)P(d\mu)}=:P_{x}(\mu\in A)$		(21)

Note that the trained sets of mean-field terms are independent of the control action $u$, as the exploration policies are independent of the mean-field terms given the state $x$. These sets have similar implications as the sets defined in (14). In particular, they indicate for which mean-field terms, one can estimate the kernel $\mathcal{T}(\cdot|x,u,\mu)$ and the cost function $c(x,u,\mu)$ via the training process.