ASE: Large-Scale Reusable Adversarial Skill Embeddings for Physically Simulated Characters

Since computing the Jensen-Shannon divergence can be computationally challenging, we will leverage a variational approximation in the form of a GAN-like framework (Goodfellow et al., 2014). First, we introduce a discriminator $D({\mathbf{s}},{\mathbf{s}}^{\prime})$, which is trained to classify if a given state transition $({\mathbf{s}},{\mathbf{s}}^{\prime})$ is from the dataset or was produced by the agent,

The policy can then be trained using rewards specified by $r_{t}=-\mathrm{log}\left(1-D\left({\mathbf{s}}_{t},{\mathbf{s}}_{t+1}\right)\right)$. It can be shown that this adversarial training procedure minimizes the Jensen-Shannon divergence between $d^{\pi}({\mathbf{s}},{\mathbf{s}}^{\prime})$ and $d^{\mathcal{M}}({\mathbf{s}},{\mathbf{s}}^{\prime})$ (Nowozin et al., 2016; Ke et al., 2021). This objective is similar to the one presented by Peng et al. (2021), and can be interpreted as training a policy to produce behaviors that appear to the discriminator as being indistinguishable from motions shown in the dataset. But unlike Peng et al. (2021), which leverages a discriminator to shape the behavioral style of policies trained from scratch for a particular task, our framework utilizes a discriminator to learn reusable skill embeddings, which can later be used to perform new tasks. However, simply matching the behavioral distribution of the data, does not ensure that the low-level policy learns a directable skill representation that can be effectively reused for new tasks. In the next subsection, we will present a skill discovery objective for acquiring more directable skill representations.

The imitation objective encourages the policy to produce behaviors that resemble the dataset, but it does not ensure that $\pi$ learns a skill representation that is easy to control and amenable for reuse on downstream tasks. Furthermore, the adversarial training procedure described in Section 5.1 is prone to mode-collapse, where the policy reproduces only a narrow range of behaviors depicted in the original dataset. To address these shortcomings, we incorporate a skill discovery objective, commonly used in unsupervised reinforcement learning, to encourage the policy to model more diverse and distinct behaviors. The objective aims to maximize the mutual information $I({\mathbf{s}},{\mathbf{s}}^{\prime};{\mathbf{z}}|\pi)$ between the state transitions $({\mathbf{s}},{\mathbf{s}}^{\prime})$ produced by a policy $\pi$, and a latent skill variable ${\mathbf{z}}$, drawn from a distribution of skills ${\mathbf{z}}\sim p({\mathbf{z}})$. The intuition for this choice of objective can be made more apparent if we consider the definition of mutual information,

Maximizing the mutual information can be interpreted as maximizing the marginal state-transition entropy $\mathcal{H}({\mathbf{s}},{\mathbf{s}}^{\prime}|\pi)$ induced by $\pi$, while minimizing the conditional state-transition entropy $\mathcal{H}({\mathbf{s}},{\mathbf{s}}^{\prime}|{\mathbf{z}},\pi)$ produced by a particular skill ${\mathbf{z}}$. In other words, the policy should learn a set of skills that produces diverse behaviors, while each individual skill should produce a distinct behavior.

Unfortunately, computing Equation 4 is intractable in most domains. Therefore, we will approximate $I({\mathbf{s}},{\mathbf{s}}^{\prime};{\mathbf{z}}|\pi)$ using a variational lower bound presented by Gregor et al. (2017) and Eysenbach et al. (2019). But unlike these prior methods, instead of learning a discrete set of skills, our policy will be trained to model a continuous latent space of skills, which is more amenable to interpolation and blending between different behaviors. To derive this objective, we first note that determining the marginal entropy $\mathcal{H}({\mathbf{s}},{\mathbf{s}}^{\prime})$ is also generally intractable. A more practical objective can be obtained by taking advantage of the symmetry of mutual information,

This decomposition removes the need to estimate $\mathcal{H}({\mathbf{s}},{\mathbf{s}}^{\prime}|\pi)$, and instead we now only need to determine the entropy over skills $\mathcal{H}({\mathbf{z}})$. Note that, since we are free to define $p({\mathbf{z}})$ and it is independent of $\pi$, $\mathcal{H}({\mathbf{z}})$ is effectively a constant and does not influence the optimization process. Then, to obtain a tractable approximation of $\mathcal{H}({\mathbf{z}}|{\mathbf{s}},{\mathbf{s}}^{\prime},\pi)$, we will introduce a variational approximation $q({\mathbf{z}}|{\mathbf{s}},{\mathbf{s}}^{\prime})$ of the conditional skill distribution $p({\mathbf{z}}|{\mathbf{s}},{\mathbf{s}}^{\prime},\pi)$. Since the cross-entropy between $p$ and $q$ is an upper bound on the entropy of $p$, $\mathcal{H}(p)\leq\mathcal{H}(p,q)$, we obtain the following variational lower bound on $I({\mathbf{s}},{\mathbf{s}}^{\prime};{\mathbf{z}}|\pi)$,

where the lower bound is tight if $q=p$. We will refer to $q({\mathbf{z}}|{\mathbf{s}},{\mathbf{s}}^{\prime})$ as the encoder. This skill discovery objective encourages a policy to produce distinct behaviors for each skill ${\mathbf{z}}$, such that the encoder can easily recover the ${\mathbf{z}}$ that produced a particular behavior.

Using the above variational approximations, we can construct a surrogate objective that approximates Equation 2,

The reward for the policy at each timestep is then specified by

This objective in effect encourages a model to develop a set of distinct skills, which also produce behaviors that resemble the dataset. A similar objective was previously proposed by Chen et al. (2016) for learning disentangled representations for image synthesis with adversarial generative networks. Similar techniques have also been applied to acquire disentangled skill representations for imitation learning (Li et al., 2017; Hausman et al., 2017). However, these methods have generally only been effective for low-dimensional systems, such as driving and simple RL benchmarks, and have not shown to be effective for more complex domains. Furthermore, the skill embeddings learned by these prior methods have not been demonstrated as being effective for hierarchical control. In this work, we show that our method can in fact learn a rich repertoire of sophisticated motor skills for complex physically simulated characters, and in the following sections, we propose a number of design decisions that improve the quality of the resulting motions, as well as allow the skills to be effectively utilized for hierarchical control.

First, we consider the design of the latent space of skills $\mathcal{Z}$ and the prior over skills $p({\mathbf{z}})$. In GAN frameworks, a common choice is to model the latent distribution using a Gaussian $p({\mathbf{z}})=\mathcal{N}\left(0,I\right)$. However, this design results in an unbounded latent space, where latents that are far from the origin of $\mathcal{Z}$ may produce low-quality samples. Since the latent space will be used as the action space for a high-level policy $\omega({\mathbf{z}}|{\mathbf{s}},{\mathbf{g}})$, an unbounded $\mathcal{Z}$ can lead $\omega$ to select latents that are far from the origin, which may result in unnatural motions. To better ensure high-quality samples, bounded latent spaces have also been used, where $p({\mathbf{z}})$ can be modeled with a uniform distribution $\mathcal{U}[-1,1]$ or a truncated Gaussian (Brock et al., 2019). In this work, we will model the latent space as a hypersphere $\mathcal{Z}=\{{\mathbf{z}}:||{\mathbf{z}}||=1\}$ (Karras et al., 2019), and $p({\mathbf{z}})$ will be a uniform distribution on the surface of the sphere. Samples can be drawn from $p({\mathbf{z}})$ by normalizing samples from a standard Gaussian distribution,

This provides the model with a bounded latent space, which can reduce the likelihood of unnatural behaviors arising from out-of-distribution latents. As we will discuss in Section 7, this choice of latent space can also facilitate exploration for downstream tasks.

Since the the latent space is modeled as a hypersphere, the skill encoder will be modeled using a von Mises-Fisher distribution,

which is the analogue to the Gaussian distribution on the surface of a sphere. $\mu_{q}({\mathbf{s}},{\mathbf{s}}^{\prime})$ is the mean of the distribution, which must be normalized $||\mu_{q}({\mathbf{s}},{\mathbf{s}}^{\prime})||=1$, $Z$ is a normalization constant, and $\kappa$ is a scaling factor. The encoder can then be trained by maximizing the log-likelihood of samples $({\mathbf{z}},{\mathbf{s}},{\mathbf{s}}^{\prime})$ collected from the policy,

where $d^{\pi}({\mathbf{s}},{\mathbf{s}}^{\prime}|{\mathbf{z}})$ represents the likelihood of observing a state transition under $\pi$ given a particular skill ${\mathbf{z}}$.

Adversarial imitation learning is known to be notoriously unstable. To improve training stability and quality of the resulting motions, we incorporate the gradient penalty regularizers used by Peng et al. (2021). The discriminator is trained using the following objective,

where $w_{\mathrm{gp}}$ is a manually specified coefficient.

When reusing pre-trained skills to perform new tasks, the high-level policy $\omega({\mathbf{z}}|{\mathbf{s}},{\mathbf{g}})$ specifies latents ${\mathbf{z}}_{t}$ at each timestep to control the behavior of the low-level policy $\pi({\mathbf{a}}|{\mathbf{s}},{\mathbf{z}})$. A responsive low-level policy should change its behaviors according to changes in ${\mathbf{z}}$. However, the objective described in Equation 9 can lead to unresponsive behaviors, where $\pi$ may perform different behaviors depending on the initial ${\mathbf{z}}_{0}$ selected at the start of an episode, but if a new latent ${\mathbf{z}}^{\prime}$ is selected at a later timestep, then the policy may ignore ${\mathbf{z}}^{\prime}$ and continue performing the same behavior specified by ${\mathbf{z}}_{0}$. This lack of responsiveness can hamper a character’s ability to perform new tasks and agilely respond to unexpected perturbations.

To improve the responsiveness of the low-level policy, we propose two modifications to the objective in Equation 9. First, instead of conditioning $\pi$ on a fixed ${\mathbf{z}}$ over an entire episode, we will instead construct a sequence of latents ${\mathbf{Z}}=\{{\mathbf{z}}_{0},{\mathbf{z}}_{1},...,{\mathbf{z}}_{T-1}\}$, and condition $\pi$ on a different latent ${\mathbf{z}}_{t}$ at each timestep $t$. The sequence of latents is constructed such that a latent ${\mathbf{z}}$ is repeated for multiple timesteps, before a new latent is sampled from $p({\mathbf{z}})$ and repeated for multiple subsequent timesteps. This encourages the model to learn to transition between different skills.

To further encourage the model to produce different behaviors for different latents, we incorporate a diversity objective similar to the one proposed by  Yang et al. (2019) to mitigate mode-collapse of conditional GANs. These modifications lead to the following pre-training objective,

with $w_{\mathrm{div}}$ being a manually specified coefficient. The last term is the diversity objective, which stipulates that if two latents ${\mathbf{z}}_{1}$ and ${\mathbf{z}}_{2}$ are similar under a distance function $D_{\mathbf{z}}$, then the policy should produce similar action distributions, as measured under the KL-divergence. Conversely, if ${\mathbf{z}}_{1}$ and ${\mathbf{z}}_{2}$ are different, then the resulting action distributions should also be different. In our implementation, the distance function is specified by $D_{\mathbf{z}}({\mathbf{z}}_{1},{\mathbf{z}}_{2})=0.5(1-{\mathbf{z}}_{1}^{T}{\mathbf{z}}_{2})$, which reflects the cosine distance between ${\mathbf{z}}_{1}$ and ${\mathbf{z}}_{2}$, since the latents lie on a sphere. This diversity objective is reminiscent of the loss used in multidimensional scaling (Kruskal, 1964), and we found it to be more stable than the objective proposed by Yang et al. (2019). Note, the reward for the policy at each time step depends only on the encoder $q$ and discriminator $D$. The diversity objective is only applied during gradient updates.

A common failure case for physically simulated characters is losing balance and falling when subjected to perturbations. In these situations, it would be favorable to have characters that can automatically recover and resume performing a task. Therefore, in addition to training the low-level policy to imitate behaviors from a dataset, $\pi$ is also trained to recover from a large variety of fallen configurations. During pre-training, the character has a $10\%$ probability of being initialized in a random fallen state at the start of each episode. The fallen states are generated by dropping the character from the air at random heights and orientations. This simple strategy then leads to robust recovery strategies that can consistently recover from significant perturbations. By incorporating these recovery strategies into the low-level controller, $\pi$ can be conveniently reused to allow the character to automatically recover from perturbations when performing downstream tasks, without requiring the character to be explicitly trained to recover from falling for each new task.

Algorithm 1 provides an overview of the ASE pre-training process for the low-level policy. At the start of each episode, a sequence of latents ${\mathbf{Z}}=\{{\mathbf{z}}_{0},{\mathbf{z}}_{1},...,{\mathbf{z}}_{T-1}\}$ is sampled from the prior $p({\mathbf{z}})$. A trajectory $\tau^{i}$ is collected by conditioning the policy $\pi$ on ${\mathbf{z}}_{t}$ at each timestep $t$. The agent receives a reward $r_{t}$ at each timestep, calculated from the discriminator $D$ and encoder $q$ according to Equation 10. Once a batch of trajectories has been collected, minibatches of transitions $({\mathbf{s}}_{j},{\mathbf{s}}^{\prime}_{j},{\mathbf{z}}_{j})$ are sampled from the trajectories and used to update the encoder according to Equation 13. The discriminator is updated according to Equation 14 using minibatches of transitions $({\mathbf{s}}_{j},{\mathbf{s}}^{\prime}_{j})$ sampled from the agent’s trajectories and the motion dataset $\mathcal{M}$. Finally, the recorded trajectories are used to update the policy. The policy is trained using proximal policy optimization (PPO) (Schulman et al., 2017), with advantages computed using GAE($\lambda$) (Schulman et al., 2015), and the value function is updated using TD($\lambda$) (Sutton and Barto, 1998).

When the character is first presented with a new task, it has no knowledge of which skill will be most effective. Therefore, during early stages of training, the high-level policy should sample skills uniformly from $\mathcal{Z}$ in order to explore a diverse variety of behaviors. As training progresses, the policy should hone in on the skills that are more effective for the task, and assign lower likelihoods to skills that are less effective. Our choice of a spherical latent space provides a convenient structure that can directly encode this exploration-exploitation trade-off into the action space for the high-level policy. This can be accomplished by using the unnormalized latents $\bar{{\mathbf{z}}}\in\bar{\mathcal{Z}}$ as the action space for $\omega$. The high-level policy is then defined as a Guassian in the unnormalized space $\omega(\bar{{\mathbf{z}}}|{\mathbf{s}},{\mathbf{g}})=\mathcal{N}\left(\mu_{\omega}({\mathbf{s}},{\mathbf{g}}),\Sigma_{\omega}\right)$. The actions from $\omega$ are projected onto $\mathcal{Z}$ by normalizing the actions ${\mathbf{z}}=\bar{{\mathbf{z}}}/||\bar{{\mathbf{z}}}||$, before being passed to the low-level policy. An illustrative example of this sampling scheme is available in Figure 3. At the start of training, the action distribution of $\omega$ is initialized close to the origin of $\bar{\mathcal{Z}}$, $\omega(\bar{{\mathbf{z}}}|{\mathbf{s}},{\mathbf{g}})\approx\mathcal{N}\left(0,\Sigma_{\omega}\right)$, which allows $\omega$ to sample skill uniformly from the normalized latent space $\mathcal{Z}$. As training progress, the mean $\mu_{\omega}({\mathbf{s}},{\mathbf{g}})$ can be shifted away from the origin as needed by the gradient updates, thereby allowing $\omega$ to specialize, and increase the likelihood of more effective skills. By shifting the action distribution closer or further from the origin of $\bar{\mathcal{Z}}$, the $\omega$ can increase or decrease its entropy over skills in $\mathcal{Z}$ respectively.

While $\pi$ is trained to produce natural behaviors for any latent skill ${\mathbf{z}}$, when reusing the low-level policy on new tasks, it is still possible for $\omega$ to specify sequences of latents that lead to unnatural behaviors. This is especially noticeable when the actions from $\omega$ changes drastically between timesteps, leading the character to constantly change the skill that it is executing, which can result in unnatural jittery movements. Motion quality on downstream tasks can be improved by reusing the discriminator $D({\mathbf{s}},{\mathbf{s}}^{\prime})$ from pre-training as a portable motion prior when training the high-level policy. The reward for the high-level policy is then specified by a combination of a task-reward $r^{G}({\mathbf{s}},{\mathbf{a}},{\mathbf{s}}^{\prime},{\mathbf{g}})$ and a style-reward from the discriminator, in a similar manner as Peng et al. (2021),

with $w_{G}$ and $w_{S}$ being manually specified coefficients. Note that the parameters of the discriminator are fixed after pre-training, and are not updated during task-training. Therefore, no motion data is needed when training on new tasks. Prior works have observed that training a policy against a fixed discriminator often leads to unnatural behaviors that exploit idiosyncrasies of the discriminator (Peng et al., 2021). However, we found that the low-level policy $\pi$ sufficiently constrains the behaviors that can be produced by the character, such that this kind of exploitation is largely eliminated.

The state ${\mathbf{s}}_{t}$ consists of a set of features that describes the configuration of the character’s body. The features include:

• •

  Height of the root from the ground.

• •

  Rotation of the root in the character’s local coordinate frame.

• •

  Linear and angular velocity of the root in the character’s local coordinate frame.

• •

  Local rotation of each joint.

• •

  Local velocity of each joint.

• •

  Positions of the hands, feet, shield, and tip of the sword in the character’s local coordinate frame.

The root is designated to be character’s pelvis. The character’s local coordinate frame is defined with the origin located at the root, the x-axis oriented along the root link’s facing direction, and the y-axis aligned with the global up vector. The 1D rotation of revolute joints are encoded using a scalar value, representing the rotation angle. The 3D rotation of the root and spherical joints are encoded using two 3D vector corresponding to the tangent ${\mathbf{u}}$ and normal ${\mathbf{v}}$ of the link’s local coordinate frame expressed in the link parent’s coordinate frame (Peng et al., 2021). Combined, these features result in a 120D state space. Each action ${\mathbf{a}}_{t}$ specifies target rotations for PD controllers positioned at each of the character’s joints. Like Peng et al. (2021), the target rotation for 3D spherical joints are encoded using a 3D exponential map (Grassia, 1998). These action parameters result in a 31D action space.

A schematic illustration of the network architectures used to model the various components of the system is provided in Figure 5. The low-level policy is modeled by a neural network that maps a state ${\mathbf{s}}$ and latent ${\mathbf{z}}$ to a Gaussian distribution over actions $\pi({\mathbf{a}}|{\mathbf{s}},{\mathbf{z}})=\mathcal{N}\left(\mu_{\pi}({\mathbf{s}},{\mathbf{z}}),\Sigma_{\pi}\right)$, with an input-dependent mean $\mu_{\pi}({\mathbf{s}},{\mathbf{z}})$ and a fixed diagonal covariance matrix $\Sigma_{\pi}$. The mean is specified by a fully-connected network with 3 hidden layers of [1024, 1024, 512] units, followed by linear output units. The value function $V({\mathbf{s}},{\mathbf{z}})$ is modeled by a similar network, but with a single linear output unit. The encoder $q({\mathbf{z}}|{\mathbf{s}},{\mathbf{s}}^{\prime})$ and discriminator $D({\mathbf{s}},{\mathbf{s}}^{\prime})$ are jointly modeled by a single network, with separate output units for the mean of the encoder distribution $\mu_{q}({\mathbf{s}},{\mathbf{s}}^{\prime})$ and the output of the discriminator. The output units of the encoder is normalized such that $||\mu_{q}({\mathbf{s}},{\mathbf{s}}^{\prime})||=1$, and the output of the discriminator consists of a single sigmoid unit. The high-level policy $\omega(\bar{{\mathbf{z}}}|{\mathbf{s}},{\mathbf{g}})=\mathcal{N}\left(\mu_{\omega}({\mathbf{s}},{\mathbf{g}}),\Sigma_{\omega}\right)$ is modeled using 2 hidden layers with [1024, 512] units, followed by linear output units for the mean $\mu_{\omega}({\mathbf{s}},{\mathbf{g}})$. Outputs from the high-level policy specify unnormalized latents $\bar{{\mathbf{z}}}$, which are then normalized ${\mathbf{z}}=\bar{{\mathbf{z}}}/||\bar{{\mathbf{z}}}||$ before being passed to the low-level policy $\pi$. ReLU activations are used for all hidden units (Nair and Hinton, 2010).

All environments are simulated using Isaac Gym (Makoviychuk et al., 2021), a high-performance GPU-based physics simulator. During training, 4096 environments are simulated in parallel on a single NVIDIA V100 GPU, with a simulation frequency of 120Hz. The low-level policy operates at 30Hz, while the high-level policy operates at 6Hz. All neural networks are implemented using PyTorch (Paszke et al., 2019). The low-level policy is trained using a custom motion dataset of 187 motion clips, provided by Reallusion (Reallusion, 2022). The dataset contains approximately 30 minutes of motion data that depict a combination of everyday locomotion behaviors, such as walking and running, as well as motion clips that depict a gladiator wielding a sword and shield. The use of a high-performance simulator allows our models to be trained with large volumes of simulated data. The low-level policy is trained with over 10 billion samples, which is approximately 10 years in simulated time, requiring about 10 days on a single GPU. A batch of 131072 samples is collected per update iteration, and gradients are computed using mini-batches of 16384 samples. Gradient updates are performed using the Adam optimizer with a stepsize of $2\times 10^{-5}$ (Kingma and Ba, 2015). Detailed hyperparameter settings are available in Appendix A.

The ASE pre-training process is able to develop expressive low-level policies that can perform a diverse array of complex skills. Examples of the learned skills are shown in Figure 6. Conditioning the policy on random latents ${\mathbf{z}}$ leads to a large variety of naturalistic and agile behaviors, ranging from common locomotion behaviors, such as walking and running, to highly dynamic behaviors, such as sword swings and shield bashes. All of these skills are modeled by a single low-level policy. During pre-taining, the motion dataset is treated as a homogeneous set of state transitions, without any segmentation or organization of the clips into distinct skills. Despite this lack of prescribed structure, our model learns to organize the different behaviors into a structured skill embedding, where each latent produces a semantically distinct skill, such as sword swings vs. shield bashes. This structure is likely a result of the skill discovery objective, which encourages the low-level policy to produce distinct behaviors for each latent ${\mathbf{z}}$, such that the encoder can more easily recover the original ${\mathbf{z}}$. Changing ${\mathbf{z}}$ partway through a trajectory also leads the character to transition to different skills. The policy is able to synthesize plausible transitions, even when the particular transitions may not be present in the original dataset.

To demonstrate the effectiveness of the pre-trained skill embeddings, we apply the pre-trained low-level policy to a variety of downstream tasks. Separate high-level policies are trained for each task, while the same low-level policy is used for all tasks. Figure 7 illustrates behaviors learned by the character on each task. ASE learns to model a versatile corpus of skills that enables the character to effectively accomplish diverse task objectives. Though each task is represented by a simple reward function that specifies only a minimal criteria for the particular task, the learned skill embedding automatically gives rise to complex and naturalistic behaviors. In the case of the Reach task, the reward simply stipulates that the character should move the tip of its sword to a target location. But the policy then learns to utilize various life-like full-body postures in order to reach target. The skill embedding provides the high-level policy with fine-grain control over the character’s low-level movements, allowing the character to closely track the target, with an average error of $0.088\pm 0.046$m. For the Steering task, the character learns to utilize different forward, backward, and sideways walking behaviors to follow the target directions. When training the character to move at a target speed, the policy is able to transition between various locomotion gaits according to the desired speed. When the target speed is 0m/s, the character learns to stand still. As the target speed increases ($\sim 2$m/s), the character transitions to a crouched walking gait, and then breaks into a fast upright running gait at the fastest speeds ($\sim 7$m/s). This composition of disparate skills is further evident in the Strike task, where the policy learns to utilize running behaviors to quickly move to the target. Once it is close to the target, the policy quickly transitions to a sword swing behavior in order to hit the target. After the target has been successfully knocked over, the character concludes by transitioning to an idle stance.

Learning skill embeddings that can reproduce a wide range of behaviors is crucial for building general and reusable skill models. However, mode-collapse remains a persistent problem for adversarial imitation learning algorithms. To evaluate the diversity of the behaviors produced by our model, we compare motions produced by the low-level policy to motions from the original dataset. First, a trajectory $\tau$ is generated by conditioning $\pi$ on a random latent ${\mathbf{z}}\sim p({\mathbf{z}})$. Then, for every transition $({\mathbf{s}}_{t},{\mathbf{s}}_{t+1})$ in the trajectory, we find a motion clip in the dataset $m^{*}\in\mathcal{M}$ that contains a transition that most closely matches the particular transition from $\pi$,

Note, $\bar{{\mathbf{s}}}$ represents the normalized state features of ${\mathbf{s}}$, normalized using the mean and standard deviation of state features from the motion data. This matching process is repeated for every transition in a trajectory, and the motion clip that contains the most matches will be specified as the clip that best matches the trajectory $\tau$. Figure 10 records the frequencies at which $\pi$ produces trajectories that matched each motion clip in the dataset across 1000 trajectories. We compare the distribution of trajectories generated by our ASE model to an ablated model that was trained without the skill discovery objective (Section 5.2) and the diversity objective (Equation 15). The ASE model produces a diverse variety of behaviors that more evenly covers the motions in the dataset. The model trained without the skill discovery objective and the diversity object produces less diverse behaviors, where a large portion of the trajectories matched a single motion clip, corresponding to an idle standing motion.

In addition to producing diverse skills, the low-level policy should also learn to transition between the various skills, so that the skills can be more easily composed and sequenced to perform more complex tasks. To evaluate a model’s ability to transition between different skills, we generate trajectories by conditioning $\pi$ on two random latents ${\mathbf{z}}_{S}$ and ${\mathbf{z}}_{D}$ per trajectory. A trajectory is then generated by first conditioning $\pi$ on ${\mathbf{z}}_{S}$ for 150 to 200 timesteps, then $\pi$ is conditioned on ${\mathbf{z}}_{D}$ for the remainder for the trajectory. We will refer to the two sub-trajectories produced by the different latents as the source trajectory $\tau_{S}$ and the destination trajectory $\tau_{D}$. The two sub-trajectories are separately matched to motion clips in the dataset, following the same procedure in Equation 22, to identify the source motion $m^{S}$ and destination $m^{D}$. We repeat this process for about 1000 trajectories, and record the probability of transitioning between each pair of motion clips in Figure 9. We compare ASE to models trained without the skill discovery objective (No SD), without the diversity objective (No Div.), and without both objectives (No SD + No Div.). Coverage denotes the portion of transitions out of all possible pairwise transitions observed from trajectories produced by a model $\left(\mathrm{coverage}=\frac{\mathrm{transitions\ from\ model}}{\mathrm{all\ possible\ transitions}}\right)$. Our ASE model generates much denser connections between different skills, and exhibits about $10\%$ of all possible transitions. Removing the skill discovery objective and diversity objective results in less responsive models that exhibit substantially fewer transitions between skills.

To determine the performance impact of these design decisions, we compare the task performance achieved using various low-level policies trained with and without the skill discovery objective and diversity objective. Figure 11 compares the learning curves of the different models, and Table 1 summarizes the average return of each model. The skill discovery objective is crucial for learning an effective skill representation. Without this objective, the skill embedding tends to produce less diverse and distinct behaviors, which in turn leads to a significant deterioration in task performance. While the diversity objective did lead to denser connections between skills (Figure 9), removing this objective (No Div.) did not seem to have a large impact on performance on our suite of tasks. For most of the tasks we considered, the highest returns are achieved by policies that were trained from scratch for each task. These policies often utilize unnatural behaviors in order to better maximize the reward function, such as adopting highly energetic sporadic movements to propel the character more quickly towards the target in the Location and Strike tasks. Examples of these behaviors are shown in Figure 8. Training from scratch also tends to require more samples compared policies trained using ASE, since the model needs to repeatedly relearn common skills for each task, such as maintaining balance and locomotion. ASE is able to achieve high returns on the suite of tasks, while also producing more natural and life-like behaviors.

To evaluate the effects of different choices of action space for the high-level policy, we visualize the behaviors produced by random exploration in different action spaces. Figure 12 illustrates trajectories produced by the different action spaces, and the corresponding motions can be viewed in the supplementary video. First, we consider the behaviors produced by random exploration in the original action space of the system $\mathcal{A}$. This strategy does not produce semantically meaning behaviors, and often just leads to the character falling after a few timesteps. Next, we consider behaviors produced by sampling from a Gaussian distribution in the unnormalized latent space $\tilde{\mathcal{Z}}$, with a standard deviation of 0.1, as described in Section 7.1. Positioning the Gaussian at the origin of $\tilde{\mathcal{Z}}$ allows the policy to uniformly sample latents from the normalized latent space $\mathcal{Z}$, leading to a diverse range of behaviors that travel in different directions. Finally, we have trajectories produced by directly sampling in the normalized latent space $\mathcal{Z}$ with various standard deviations $\sigma=[0.1,0.2,0.5]$. This sampling strategy limits the policy to selecting latents within a local region on the surface of a hypersphere, which can lead to less diverse behaviors. More diverse behaviors can be obtained by drastically increasing the standard deviation of the action distribution. However, a large action standard deviation can hamper learning, and additional mechanisms are needed to decrease the standard deviation over time.

Reusing the pre-trained discriminator as a motion prior when training a high-level policy can improve motion quality. The task-reward function is combined with the style-reward using weights $w_{G}=0.9$ and $w_{S}=0.1$ respectively. A comparison of the motions produced with and without a motion prior is available in the supplementary video. Without the motion prior, the character is prone to producing unnatural jittering behaviors and other extraneous movements. With the motion prior, the character produces much smoother motions, and exhibits more natural transitions between different skills.

During the pre-training stage, the low-level policy is trained to recover from random initial states, which leads the policy to develop robust recovery strategies that enable the character to consistent get back up after falling. These recovery strategies can then be conveniently transferred to new tasks by simply reusing the low-level policy. This allows the character to agilely execute recovery behaviors in order to get up after a fall, and then seamless transition back to performing a given task. These behaviors emerge automatically from the low-level policy, without requiring explicit training to recover from perturbations for each new task. Examples of the learned recovery strategies are available in Figure 13. To evaluate the effectiveness of these recovery strategies, we apply random perturbation forces to the character’s root and record the time required for the character to recover after a fall. Figure 14 shows the amount of time needed for recoveries across 500 trials. A force between [500, 1000]N in a random direction is applied for 0.5s to the character’s root. A fall is detected whenever a body part makes contact with the ground, excluding the feet. A successful recovery is detected once the character’s root recovers back to a height above 0.5m and its head is above 1m. The policy is able to successfully recover from all falls, with a mean recovery time of 0.31s and a maximum recovery time of 4.1s. Even though the dataset does not contain any motion clips that depict get up behaviors, the low-level policy is still able to discover plausible recovery strategies, such as using its hands to break a fall and get back up more quickly. However, the character can still exhibit some unnatural recovery behaviors, such as using overly energetic spins and flips. Including motion data for more life-like recovery strategies will likely help to further improve the realism of these motions.