Diverse Policies Converge in Reward-free Markov Decision Processes

Although there has been a lot of work on exploring diversity, we find that these algorithms lack a unified framework. So we propose a unified framework for diversity algorithms in Algorithm 1 to pave the way for further research.

We use $Div$ to measure the diversity distance between two policies and we abbreviate policy $\pi_{\theta}(\cdot|s,z_{i})$ as $\pi^{i}$. Vector $z_{i}$ can be thought of as a skill unique to each policy $\pi^{i}$. Moreover, we define $U\in\mathbb{R}^{N\times N}$ as diversity matrix where $U_{ij}=Div(\pi^{i},\pi^{j})$ and $N$ denotes the number of policies.
For each episode, we first sample $z_{i}$ to decide which policy to update. Then we interact the chosen policy with the environment to get trajectory $\tau$, which is used to calculate intrinsic reward $r^{in}$ and update diversity matrix $U$. We then store tuple ($s,a,s^{\prime},r^{in},z_{i}$) in replay buffer $\mathcal{D}$ and update $\pi^{i}$ through any reinforcement learning algorithm.
Here we abstract the procedure of selecting $z_{i}$ and calculating $r^{in}$ as $SelectZ$ and $CalR$ functions respectively, which are usually the most essential differences between diversity algorithms. We summarize the comparison of some diversity algorithms in Table 1. Now we describe these two functions in more detail.
Policy Selection. Note that we denote by $p(z)$ the distribution of $z$. We can divide means to select $z_{i}$ into three categories in general, namely iteration fashion, uniform sample and bandit selection:

(1) Iteration fashion. Diversity algorithms such as RSPO [26] and SIPO [9] obtain diverse policies in an iterative manner. In the $k$-th iteration, policy $\pi^{k}$ will be chosen to update, and the target of optimization is to make $\pi^{k}$ sufficiently different from previously discovered policies $\pi^{1},...,\pi^{k-1}$. This method doesn’t ensure optimal performance and is greatly affected by policy initialization.

(2) Uniform sample. Another kind of popular diversity algorithm such as DIAYN [7] and DGPO [4], samples $z_{i}$ uniformly to maximize the entropy of $p(z)$. Due to the method’s disregard for the differences between policies, it often leads to slower convergence.

(3) Bandit selection. We frame obtaining diverse policies as a contextual bandit problem. Sampling $z_{i}$ corresponds to minimizing regret in this context. This approach guarantees strong performance and rapid convergence.
Reward Calculation. Diversity algorithms differ in intrinsic reward calculation. Some, like [4, 7, 19], use mutual information theory and a discriminator $\phi$ to distinguish policies. DIAYN[7] emphasizes deriving skill $z$ from the state $s$, while [19] suggests using state-action pairs. On the other hand, algorithms like [15, 26] aim to make policies’ action or reward distributions distinguishable, known as behavior-driven and reward-driven exploration. DGPO[4] maximizes the minimal diversity distance between policies.

In this section, we will show the convergence of diversity algorithms under a reasonable diversity target. We define $\mathcal{P}=\{\pi^{1},\pi^{2},...,\pi^{N}\}$ as the set of independent policies, or policy population.

Definition 1. $g:\{\pi^{1},\pi^{2},...,\pi^{N}\}\to\mathbb{R}^{N\times N}$ is a function that maps population $\mathcal{P}$ to diversity matrix $U$ which is defined in section 4.1. Given a population $\mathcal{P}$, we can calculate pairwise diversity distance under a certain diversity metric, which indicates that $g$ is an injective function.

Definition 2. Note that in the iterative process of the diversity algorithm, we update $\mathcal{P}$ directly instead of $U$. So if we find a valid $U$ that satisfies the diversity target, then the corresponding population $\mathcal{P}$ is exactly our target diverse population. We refer to this process of finding $\mathcal{P}$ backward as $g^{-1}$.

Definition 3. $f:\mathbb{R}^{N\times N}\to R$ is a function that maps $U$ to a real number. While $U$ measures the pairwise diversity distance between policies, $f$ measures the diversity of the entire population $\mathcal{P}$. As the diversity of the population increases, the diversity metric calculated by $f$ will increase as well.

Definition 4. We further define $\delta$-target population set $\mathcal{T}_{\delta}=\{g^{-1}(U)|f(U)>\delta,U\in\mathbb{R}^{N\times N}\}$. $\delta$ is a threshold used to separate target and non-target regions. The meaning of this definition is that, during the training iteration process, when the diversity metric closely related to $U$ exceeds a certain threshold, or we say $f(U)>\delta$, the corresponding population $\mathcal{P}$ is our target population.
Note two important points: (1) The population meeting the diversity requirement should be a set, not a fixed point. (2) Choose a reasonable threshold $\delta$ that ensures both sufficient diversity and ease of obtaining the population.

Remark. Careful selection of threshold $\delta$ is crucial for diversity algorithms. Reasonable diversity goals should be set to avoid difficulty or getting stuck in the training process. This hyperparameter can be obtained through empirical experiments or methods like hyperparameter search. In certain diversity algorithms, both $\delta$ and $\mathcal{P}$ may change during training. For instance, in iteration fashion algorithms (Section 4.1), during the $k$-th iteration, $\mathcal{P}=\{\pi^{1},\pi^{2},...,\pi^{k}\}$ with a target threshold of $\delta_{k}$. If policy $\pi^{k}$ becomes distinct from $\pi^{1},...,\pi^{k-1}$, meeting the diversity target, policy $\pi_{k+1}$ is added to $\mathcal{P}$ and the threshold changes to $\delta_{k+1}$.

As mentioned in Section 4.1, we can sample $z_{i}$ via bandit selection. In this section, we formally define $K$-armed contextual bandit problem [14], and show how it models diversity optimization procedure.

We show the procedure of the contextual bandit problem in Algorithm 2. In each iteration, we can observe feature vectors $x_{t,a}$ for each $a\in\mathcal{A}$, which are also denoted as context. Note that context may change during training. Then, $Algo$ will choose an arm $a_{t}\in\mathcal{A}$ based on contextual information and will receive reward $r_{t,a_{t}}$. Finally, tuple ($x_{t,a_{t}},a_{t},r_{t,a_{t}}$) will be used to update $Algo$.
We further define T-Reward [14] of $Algo$ as $\sum_{t=1}^{T}r_{t}$. Similarly, we define the optimal expected T-Reward as ${\bf E}[\sum_{t=1}^{T}r_{t,a_{t}^{*}}]$, where $a_{t}^{*}$ denotes the arm with maximum expected reward in iteration $t$. To measure $Algo$’s performance, we define T-regret $R_{T}$ of $Algo$ by

Our goal is to minimize $R_{T}$.
In the diversity optimization problem, policies are akin to arms, and context is represented by visited states or $\rho^{\pi}(s)$. Note that context may change as policies evolve. When updating a policy, the reward is the difference in diversity metric before and after the update, linked to the diversity matrix $U$ (Section 4.1). Our objective is to maximize policy diversity, equivalent to maximizing expected reward or minimizing $R_{T}$ in contextual bandit formulation.
Here’s an example to demonstrate the effectiveness of bandit selection. In some cases, a policy $\pi^{i}$ may already be distinct enough from others, meaning that selecting $\pi^{i}$ for an update wouldn’t significantly affect policy diversity. To address this, we should decrease the probability of sampling $\pi^{i}$. Fixed uniform sampling fails to address this issue, but bandit algorithms like UCB[2] or LinUCB[14] consider both historical rewards and the number of times policies have been chosen. This caters to our needs in such cases.

In this section, we provide the regret bound for bandit selection in the diversity algorithms.

Problem Setting. We define $T$ as the number of iterations. In each iteration $t$, we can observe $N$ feature vectors $x_{t,a}\in\mathbb{R}^{d}$ and receive reward $r_{t,a_{t}}$ with $\|x_{t,a}\|\leq 1$ for $a\in\mathcal{A}$ and $r_{t,a_{t}}\in[0,1]$, where $\|\cdot\|$ means $l_{2}$-norm, $d$ denotes the dimension of feature vector and $a_{t}$ is the chosen action in iteration $t$.

Linear Realizability Assumption. Similar to lots of theoretical analyses of contextual bandit problems [1, 5], we propose linear realizability assumption to simplify the problem. We assume that there exists an unknown weight vector $\theta^{*}\in\mathbb{R}^{d}$ with $\|\theta^{*}\|\leq 1$ s.t.

for all t and a.
We now analyze the rationality of this assumption in practical diversity algorithms. Reward $r_{t,a}$ measures the changed value of overall diversity metric $\bigtriangleup f(U)$ of policy population $\mathcal{P}$ after an update. Suppose $\pi^{i}_{t}$ is the policy corresponding to the feature vector $x_{t,a}$ in the iteration $t$. While $x_{t,a}$ encodes state features of $\pi^{i}_{t}$, it can encode the diversity information of $\pi^{i}_{t}$ as well. Therefore, we can conclude that $r_{t,a}$ is closely related to $x_{t,a}$. So given that $x_{t,a}$ contains enough diversity information, we can assume that the hypothesis holds.

To visualize the policy evolution process, we use DIAYN [7] as our diversity algorithm and construct a simple 3-state MDP [8] to conduct the experiment. The set of feasible state marginal distributions is described by a triangle $[(1,0,0),(0,1,0),(0,0,1)]$ in $\mathbb{R}^{3}$. And we use state occupancy measure $\rho^{\pi_{i}}(s)$ to represent policy $\pi^{i}$. Moreover, we project the state occupancy measure onto a two-dimensional simplex for visualization.

Let $\rho(s)$ be the average state marginal distribution of all policies. Figure 1(a) shows policy evolution during training. Initially, the state occupancy measures of different policies are similar. However, as training progresses, the policies spread out, indicating increased diversity. Figure 1(a) highlights that diversity [8] ensures distinct state occupancy measures among policies.

We use $I(\cdot;\cdot)$ to denote mutual information. The diversity metric in unsupervised skill discovery algorithms is based on the mutual information of states and latent variable $z$. Furthermore, the mutual information can be viewed as the average divergence between each policy’s state distribution $\rho(s|z)$ and the average state distribution $\rho(s)$ [8]:

Figure 1(b) shows the policy evolution process and the diversity metric $I(s;z)$. We find that the diversity metric increased gradually during the training process, which is in line with our expectation.

We continue to use 3-state MDP [8] as the experimental environment. Whereas, in order to get closer to the complicated practical environment, we set specific $\delta$-target and increased the number of policies. Moreover, when a policy that hasn’t met the diversity requirement is chosen to update, we will receive a reward $r=1$, otherwise, we will receive a reward $r=0$. We use $I(s;z)$ as the diversity metric and use LinUCB[14] as our contextual bandit algorithm.

Figure 2 shows the training curves under different numbers of policies and different $\delta$-target over six random seeds. The results show that bandit selection not only always reaches the convergence fastest, but also achieves the highest overall diversity metric of the population when it converges. We now empirically analyze the reasons for this result:
Drawbacks of uniform sample. In many experiments, we observe that uniform sample has similar final performance to bandit selection, but significantly slower convergence. This is because after several iterations, some policies become distinct enough to prioritize updating other policies. However, uniform sample treats all policies equally, resulting in slow convergence.
Drawbacks of iteration fashion. In experiments, the iteration fashion converges quickly but has lower final performance than the other two methods. It’s greatly affected by initialization. Each policy update depends on the previous one, so poor initialization can severely impact subsequent updates, damaging the overall training process.
Advantages of bandit selection. Considering historical rewards and balancing exploitation and exploration, bandit selection quickly determines if a policy is different enough to adjust the sample’s probability distribution. Unlike iteration fashion, all policies can be selected for an update in a single iteration, making bandit selection not limited by policy initialization.