Dreaming: Model-based Reinforcement Learning
by Latent Imagination without Reconstruction

Let us begin by considering the graphical model illustrated in Fig. 2, whose joint distribution is defined as follows:

$\displaystyle p({\boldsymbol{z}}_{\leq T},\boldsymbol{a}_{<T},\boldsymbol{x}_{\leq T})=\prod_{t}p\left({\boldsymbol{z}}_{t+1}|{\boldsymbol{z}}_{t},\boldsymbol{a}_{t}\right)p\left(\boldsymbol{x}_{t}|{\boldsymbol{z}}_{t}\right),$

(1)

where ${\boldsymbol{z}}$, $\boldsymbol{x}$, and $\boldsymbol{a}$ denote latent state, observation, and action, respectively. As in the case of well-known variational autoencoders (VAEs) [12], generative models $p({\boldsymbol{z}}_{t+1}|{\boldsymbol{z}}_{t},\boldsymbol{a}_{t})$, $p(\boldsymbol{x}_{t}|{\boldsymbol{z}}_{t})$ and inference model $q({\boldsymbol{z}}_{t}|\boldsymbol{x}_{\leq t},\boldsymbol{a}_{<t})$ can be trained by maximizing the evidence lower bound [15]:

	$\displaystyle\log p(\boldsymbol{x}_{\leq T}|\boldsymbol{a}_{\leq T})=\log\int p({\boldsymbol{z}}_{\leq T},\boldsymbol{a}_{<T},\boldsymbol{x}_{\leq T})d{\boldsymbol{z}}_{\leq t}\geq$
	$\displaystyle\sum_{t}\left(\underbrace{\mathbb{E}_{q({\boldsymbol{z}}_{t}|\boldsymbol{x}_{\leq t},\boldsymbol{a}_{<t})}\left[\log p(\boldsymbol{x}_{t}|{\boldsymbol{z}}_{t})\right]}_{\coloneqq\mathcal{J}^{\mathrm{likelihood}}}\right.$		(2)
	$\displaystyle\left.\underbrace{-\mathbb{E}_{q({\boldsymbol{z}}_{t}|\cdot)}\left[{\operatorname{KL}}\left[q({\boldsymbol{z}}_{t+1}|\boldsymbol{x}_{\leq t+1},\boldsymbol{a}_{<t+1})||p({\boldsymbol{z}}_{t+1}|{\boldsymbol{z}}_{t},\boldsymbol{a}_{t})\right]\right]}_{\coloneqq\mathcal{J}^{\mathrm{KL}}}\right).$

If the models are defined to be differentiable and trainable, this objective can be maximized by the stochastic gradient ascent via backpropagation.

Multi-step variational inference is proposed in [15] to improve long-term predictions. This inference, named latent overshooting, involves the multi-step objective $\mathcal{J}^{\mathrm{KL}}_{k}$ defined as:

	$\displaystyle\mathcal{J}^{\mathrm{KL}}\geq\mathcal{J}^{\mathrm{KL}}_{k}\coloneqq\mathbb{E}_{p({\boldsymbol{z}}_{t}|{\boldsymbol{z}}_{t-k},\boldsymbol{a}_{<t})q({\boldsymbol{z}}_{t-k}|\boldsymbol{x}_{\leq t-k},\boldsymbol{a}_{<t-k})}[\hookleftarrow$
	$\displaystyle{\operatorname{KL}}\left[q({\boldsymbol{z}}_{t+1}|\boldsymbol{x}_{\leq t+1},\boldsymbol{a}_{<t+1})||p({\boldsymbol{z}}_{t+1}|{\boldsymbol{z}}_{t},\boldsymbol{a}_{t})\right]],$		(3)

where

$\displaystyle{p}({\boldsymbol{z}}_{t}|{\boldsymbol{z}}_{t-k},\boldsymbol{a}_{<t})\coloneqq\mathbb{E}_{{p}({\boldsymbol{z}}_{t-1}|{\boldsymbol{z}}_{t-k},\boldsymbol{a}_{<t-1})}\left[{p}({\boldsymbol{z}}_{t}|{\boldsymbol{z}}_{t-1},\boldsymbol{a}_{t-1})\right]$

is the multi-step prediction model.

For the purpose of planning or policy optimization, not only for the dynamics model $p({\boldsymbol{z}}_{t+1}|{\boldsymbol{z}}_{t},\boldsymbol{a}_{t})$ but also the reward function $p(r_{t}|{\boldsymbol{z}}_{t})$ is also required. To do this, we can simply regard the rewards as observations and learn the reward function $p(r_{t}|{\boldsymbol{z}}_{t})$ along with the decoder $p(\boldsymbol{x}_{t}|{\boldsymbol{z}}_{t})$. For readability, we omit the specifications of the reward function $p(r_{t}|{\boldsymbol{z}}_{t})$ in the following discussion. Although we remove the decoder later, the reward function and its likelihood objective are kept untouched.

The recurrent state space model (RSSM) is a latent dynamics model equipped with an expressive recurrent neural network, realizing accurate long-term prediction. RSSM is used as an essential component of various MBRL methods from pixels [15, 16, 17, 19, 31] including Dreamer. RSSM assumes the latent ${\boldsymbol{z}}_{t}$ comprises ${\boldsymbol{z}}_{t}=({\boldsymbol{s}}_{t},{\boldsymbol{h}}_{t})$ where ${\boldsymbol{s}}_{t}$, ${\boldsymbol{h}}_{t}$ are the probabilistic and deterministic variables, respectively. RSSM’s generative and inference models are defined as:

	$\displaystyle\mathrm{Generative\ models}$	$\displaystyle:\begin{cases}{\boldsymbol{h}}_{t}=f^{\mathrm{GRU}}({\boldsymbol{h}}_{t-1},{\boldsymbol{s}}_{t-1},\boldsymbol{a}_{t-1})\\ {\boldsymbol{s}}_{t}\sim p({\boldsymbol{s}}_{t}|{\boldsymbol{h}}_{t})\\ \boldsymbol{x}_{t}\sim p(\boldsymbol{x}_{t}|{\boldsymbol{h}}_{t},{\boldsymbol{s}}_{t})\end{cases},$		(4)
	$\displaystyle\mathrm{Inference\ model}$	$\displaystyle:{\boldsymbol{s}}_{t}\sim q({\boldsymbol{s}}_{t}|{\boldsymbol{h}}_{t},\boldsymbol{x}_{t}),$

where deterministic ${\boldsymbol{h}}_{t}$ is considered to be the hidden state of the gated recurrent unit (GRU) $f^{\mathrm{GRU}}(\cdot)$ [32] so that historical information can be embedded into ${\boldsymbol{h}}_{t}$.

Dreamer [16] makes use of RSSM as a differentiable dynamics and efficiently learns the behaviors via backpropagation of Bellman errors estimated from imagined trajectories. Dreamer’s training procedure is simply summarized as follows: (1) Train RSSM with a given dataset by optimizing Eq. (2). (2) Train a policy from the latent imaginations. (3) Execute the trained policy in a real environment and augment the dataset with the observed results. The above steps are iteratively executed until the policy performs as expected.

The original Dreamer paper [16] also introduced a likelihood-free objective by reformulating $\mathcal{J}^{\mathrm{likelihood}}$ of Eq. (2). By adding a constant $\log p(\boldsymbol{x}_{t})$ and applying Bayes’ theorem, we get a decoder-free objective:

	$\displaystyle\mathcal{J}^{\mathrm{likelihood}}\stackrel{{\scriptstyle+}}{{=}}\mathbb{E}_{q({\boldsymbol{z}}_{t}|\cdot)}\left[\log p(\boldsymbol{x}_{t}|{\boldsymbol{z}}_{z})-\log p(\boldsymbol{x}_{t})\right]$
	$\displaystyle=\mathbb{E}_{q({\boldsymbol{z}}_{t}|\cdot)}\left[\log p({\boldsymbol{z}}_{t}|\boldsymbol{x}_{t})-\log p({\boldsymbol{z}}_{t})\right]$
	$\displaystyle\geq\mathbb{E}_{q({\boldsymbol{z}}_{t}|\cdot)}\left[\log p({\boldsymbol{z}}_{t}|\boldsymbol{x}_{t})-\log\sum_{\boldsymbol{x}^{\prime}\in\mathcal{D}}p({\boldsymbol{z}}_{t}|\boldsymbol{x}^{\prime})\right]$
	$\displaystyle\coloneqq\mathcal{J}^{\mathrm{NCE}},$		(5)

where $\mathcal{D}$ denotes the mini-batch and the lower bound in the second line was from the Info-NCE (noise-contrastive estimator) mini-batch bound [33]. Let $B$ be the batch size of $\mathcal{D}$, $\mathcal{J}^{\mathrm{NCE}}$ is considered as a $B$-class categorical cross entropy objective to discriminate the positive pair $({\boldsymbol{z}}_{t},\boldsymbol{x}_{t})$ among the other negative pairs $({\boldsymbol{z}}_{t},\boldsymbol{x}^{\prime}(\neq\boldsymbol{x}_{t}))$. In this interpretation, $p({\boldsymbol{z}}_{t}|\boldsymbol{x}_{t})$ can be considered as a discriminator to discern the positive pairs. Representation learning with this type of objective is known as contrastive learning [22, 23] that encourages the embeddings to be sufficiently seperated from each other in the latent space. However, the experiment in [16] has demonstrated that this objective significantly degrades the performance compared to the original objective $\mathcal{J}^{\mathrm{likelihood}}$.

We propose to further reformulate $\mathcal{J}^{\mathrm{NCE}}$ of Eq. (5) by introducing a multi-step prediction model: $\tilde{p}({\boldsymbol{z}}_{t}|{\boldsymbol{z}}_{t-k},\boldsymbol{a}_{<t})\coloneqq\mathbb{E}_{\tilde{p}({\boldsymbol{z}}_{t-1}|{\boldsymbol{z}}_{t-k},\boldsymbol{a}_{<t-1})}\left[\tilde{p}({\boldsymbol{z}}_{t}|{\boldsymbol{z}}_{t-1},\boldsymbol{a}_{t-1})\right].$ The accent of $\tilde{p}$ implies that an independent dynamics model from $p({\boldsymbol{z}}_{t}|{\boldsymbol{z}}_{t-1},\boldsymbol{a}_{t-1})$ in Eq. (2) can be employed here. By multiplying a constant $\mathbb{E}_{q({\boldsymbol{z}}_{t-k}|\cdot)}[\tilde{p}(\cdot)/\tilde{p}(\cdot)]=1$, we obtain an importance sampling form of $\mathcal{J}^{\mathrm{NCE}}$ as:

	$\displaystyle\mathcal{J}^{\mathrm{NCE}}=$	$\displaystyle\ \mathbb{E}_{\tilde{p}({\boldsymbol{z}}_{t}|{\boldsymbol{z}}_{t-k},\boldsymbol{a}_{<t})q({\boldsymbol{z}}_{t-k}|\cdot)}\left[\frac{q({\boldsymbol{z}}_{t}|\cdot)}{\tilde{p}({\boldsymbol{z}}_{t}|\cdot)}\times\hookleftarrow\right.$
		$\displaystyle\left.\left(\log p({\boldsymbol{z}}_{t}|\boldsymbol{x}_{t})-\log\sum_{\boldsymbol{x}^{\prime}}p({\boldsymbol{z}}_{t}|\boldsymbol{x}^{\prime})\right)\right].$		(6)

For computational simplicity, we approximate the likelihood ratio ${q({\boldsymbol{z}}_{t}|\cdot)}/{\tilde{p}({\boldsymbol{z}}_{t}|\cdot)}$ as a constant and assume that the summation of $\mathcal{J}^{\mathrm{NCE}}$ across batch and time dimension is approximated as: $\sum\mathcal{J}^{\mathrm{NCE}}\stackrel{{\scriptstyle\sim}}{{\propto}}\sum\mathcal{J}^{\mathrm{NCE}}_{k},$ where

	$\displaystyle\mathcal{J}^{\mathrm{NCE}}_{k}\coloneqq$		(7)
	$\displaystyle\mathbb{E}_{\tilde{p}({\boldsymbol{z}}_{t}|{\boldsymbol{z}}_{t-k},\boldsymbol{a}_{<t})q({\boldsymbol{z}}_{t-k}|\cdot)}\left[\log p({\boldsymbol{z}}_{t}|\boldsymbol{x}_{t})-\log\sum_{\boldsymbol{x}^{\prime}}p({\boldsymbol{z}}_{t}|\boldsymbol{x}^{\prime})\right].$

We further import the concept of overshooting and optimize $\mathcal{J}^{\mathrm{NCE}}_{k}$ along with $\mathcal{J}^{\mathrm{KL}}_{k}$ on multi-step prediction of varying $k$s. Finally, the objective we use to train RSSM is:

$\displaystyle\mathcal{J}\coloneqq\textstyle\sum^{K}_{k=0}\left(\mathcal{J}^{\mathrm{NCE}}_{k}+\mathcal{J}^{\mathrm{KL}}_{k}\right).$

(8)

Note that $\mathcal{J}^{\mathrm{NCE}}_{k}$ and $\mathcal{J}^{\mathrm{KL}}_{k}$ have different dynamics model (i.e., $\tilde{p}({\boldsymbol{z}}_{t}|\cdot)$ and $p({\boldsymbol{z}}_{t}|\cdot)$, respectively).

As shown in Appx. A, $\mathcal{J}^{\mathrm{NCE}}$ is a lower bound of the mutual information $I(\boldsymbol{x}_{t};{\boldsymbol{z}}_{t})$, while $\mathcal{J}^{\mathrm{NCE}}_{k}$ is a bound of $I(\boldsymbol{x}_{t};{\boldsymbol{z}}_{t-k})$. Since the latent state sequence is Markovian, we have the data processing inequality as $I(\boldsymbol{x}_{t};{\boldsymbol{z}}_{t})\geq I(\boldsymbol{x}_{t};{\boldsymbol{z}}_{t-k})$. In other words, $\mathcal{J}^{\mathrm{NCE}}$ and approximately derived $\mathcal{J}^{\mathrm{NCE}}_{k}$ share the same InfoMax upper bound metrics. An intuitive motivation to introduce $\mathcal{J}^{\mathrm{NCE}}_{k}$ instead of $\mathcal{J}^{\mathrm{NCE}}$ is so that we can incorporate temporal correlation between $t$ and $t-k$. Another motivation is that we can increase the model capacity of the discriminator $p({\boldsymbol{z}}_{t}|\boldsymbol{x}_{t})$ by incorporating the independent dynamics model $\tilde{p}({\boldsymbol{z}}_{t}|\cdot)$.

This section discusses how we define the discriminator components: $p({\boldsymbol{z}}_{t}|\boldsymbol{x}_{t})$ and $\tilde{p}({\boldsymbol{z}}_{t}|{\boldsymbol{z}}_{t-1},\boldsymbol{a}_{t-1})$. Ref. [34] has empirically shown that the inductive bias from model architectures is a significant factor for contrastive learning. As experimentally recommended in the literature, we define $p({\boldsymbol{z}}_{t}|\boldsymbol{x}_{t})$ as an exponentiated bilinear similarity function parameterized with $W_{{\boldsymbol{z}}|\boldsymbol{x}}$:

$\displaystyle p({\boldsymbol{z}}_{t}|\boldsymbol{x}_{t})\propto\exp({\boldsymbol{z}}_{t}^{\top}W_{{\boldsymbol{z}}|\boldsymbol{x}}\boldsymbol{e}_{t}),$

(9)

where $\boldsymbol{e}_{t}\coloneqq f^{\mathrm{CNN}}(\boldsymbol{x}_{t})$ and $f^{\mathrm{CNN}}(\cdot)$ denotes feature extraction by a CNN unit. With this definition, $\mathcal{J}^{\mathrm{NCE}}_{k}$ is simply a softmax cross-entropy objective with logits ${\boldsymbol{z}}^{\top}W\boldsymbol{e}$. Contrary to the previous contrastive learning literature [22, 34], the definition of newly introduced $\tilde{p}({\boldsymbol{z}}_{t}|{\boldsymbol{z}}_{t-1},\boldsymbol{a}_{t-1})$ is required. Here, we propose to apply linear modeling to define the model deterministically as:

	$\displaystyle\tilde{p}({\boldsymbol{z}}_{t}|{\boldsymbol{z}}_{t-1},\boldsymbol{a}_{t-1})\coloneqq\delta({\boldsymbol{z}}_{t}-{\boldsymbol{z}}^{\prime}_{t}),\$
	$\displaystyle\mathrm{where}\ {\boldsymbol{z}}^{\prime}_{t}\coloneqq W_{{\boldsymbol{z}}}{\boldsymbol{z}}_{t-1}+W_{\boldsymbol{a}}\boldsymbol{a}_{t-1},$		(10)

$\delta$ is the Dirac delta function, and $W_{{\boldsymbol{z}},\boldsymbol{a}}$ are linear parameters.

This linear modeling of $\tilde{p}({\boldsymbol{z}}_{t}|\cdot)$ successfully regularizes $\mathcal{J}^{\mathrm{NCE}}_{k}$ and contributes to construct smooth latent space. We can alternatively define $\tilde{p}({\boldsymbol{z}}_{t}|\cdot)\coloneqq p({\boldsymbol{z}}_{t}|\cdot)$, where $p({\boldsymbol{z}}_{t}|\cdot)$ is generally defined as an expressive model aiming at precise prediction. However, the high model capacity allows to embed temporally consecutive samples too distant from each other to sufficiently optimize $\mathcal{J}^{\mathrm{NCE}}_{k}$, thus yielding unsmooth latent space.

Figure 3 illustrates the architecture to compute $\mathcal{J}^{\mathrm{NCE}}_{k}$. We describe the two paramount components which characterize our proposed contrastive learning scheme as follows:

(i) Independent linear forward dynamics: As previously proposed in Sec. IV-C, we employ a simple linear forward dynamics $\tilde{p}$, which is used only for contrastive learning. During the policy optimization phase, the expressive model with GRU is alternatively utilized to make the most out of its long-term prediction accuracy.

(ii) Data augmentation: We append two independent image preprocessors which process two sets of input images (i.e., $\boldsymbol{x}_{\leq t}$ and $\boldsymbol{x}_{t+1:t+K}$). Considering the empirical success of the previous literature [23, 25, 28, 30, 29], we adopt the random crop of images. In our implementation, the original image shaped $(72,72)$ is cropped to be $(64,64)$. The origin of the crop rectangle is determined at each preprocessor randomly and indenpendently. This makes it difficult for the contrastive learner to discriminate correct positive pairs, encouraging only informative features for control to be extracted.

We propose a decoder-free variant of Dreamer, which we call Dreamer with InfoMax and without generative decoder (Dreaming). Dreaming trains a policy as almost same way with the original Dreamer. The only difference between the methods is that we alternatively use the contrastive learning scheme introduced in the previous section to train RSSM. We implement Dreaming in TensorFlow [35] by modifying the official source code of Dreamer¹¹1https://github.com/google-research/dreamer. We keep all hyperparameters and experimental conditions similar to the original ones. A newly introduced hyperparameter $K$ in Eq. (8) (overshooting distance) is set to be $K=3$ based on the ablation study in Appx. C.

The main objective of this experiment is to demonstrate that Dreaming has advantages over the baseline method Dreamer [16] on difficult 5 manipulation tasks exhibited in Fig. 1, in which Dreamer suffers from object vanishing. We also prepare a likelihood-free variant of Dreamer introduced in Sec. III-C, which utilizes the vanilla contrastive objective $\mathcal{J}^{\mathrm{NCE}}$ instead of $\mathcal{J}^{\mathrm{NCE}}_{k}$ and $\mathcal{J}^{\mathrm{likelihood}}$. The specifications of the two original tasks, UR5-reach and Connector-insert, are described in Appx. B. For the difficult 5 tasks, we also compare the performance with the latest cutting edge MFRL methods, which are CURL [25], DrQ [29] and RAD [28]. In addition, another variety of 10 DMC tasks are evaluated, which are categorized into three classes namely; manipulation, pole-swingup, and locomotion. For the additional 10 tasks, only CURL is selected as an MFRL representative.

Table I summarizes the training results benchmarked at certain environment steps. The results show the mean and standard deviation averaged 4 seeds and 10 consecutive trajectories. This table shows a similar result as in [16] that decoder-free Dreamer with the vanilla contrastive objective $\mathcal{J}^{\mathrm{NCE}}$ degrades the performances on most of tasks than decoder-based Dreamer with $\mathcal{J}^{\mathrm{likelihood}}$. In the following discussions, we use the decoder-based Dreamer as a primary baseline. (A) We put much focus on these difficult tasks and it can be seen that Dreaming consistently achieves better performance than Dreamer. Hence, this indicates that the decoder-free nature of the proposed method successfully surmounts the object vanishing problem. In addition, Dreaming achieves outperforming performance than the leading MFRL methods. (B) On other manipulation tasks, there are no significant difference between Dreaming and Dreamer because the key objects are large enough. (C) Since the pole-swingup tasks also cause vanishing of thin poles, Dreaming takes better performance than Dreamer. (D) Dreaming lags behind the Dreamer on 3 of 4 locomotion tasks, i.e., planar locomotion tasks (Walker-walk, Cheetah-run and Hopper-hop). On these tasks, the cameras always track the center of locomotive robots, and this causes the key control information (i.e., velocity) to be extracted from the background texture. We suppose that this robot-centric nature makes it difficult for the contrastive learner to extract such information because only robots’ attitudes provide enough information to discriminate different samples.

Figure 4 shows video prediction by Dreaming, in which principal features for control (e.g., positions and orientations) are successfully reconstructed from the embeddings learned without likelihood objective. However on Cheeta-run, another kind of object vanishing arises; the checkered floor pattern, which is required to extract the velocity information, is vanished.

This experiment is conducted to analyze how the major components of the proposed representation learning, introduced in Sec. IV-D, contribute to the overall performance. For this purpose, some variants of the proposed method have been prepared: (i) the effect of independent linear dynamics is demonstrated with a variant that has shared dynamics $\tilde{p}({\boldsymbol{z}}_{t}|{\boldsymbol{z}}_{t-1},\boldsymbol{a}_{t-1})\coloneqq{p}({\boldsymbol{z}}_{t}|{\boldsymbol{z}}_{t-1},\boldsymbol{a}_{t-1})$²²2 The prepared variant can be considered as a special case of the contrastive forward model (CFM) [21] as discussed in Sec. II. , (ii) the effect of data augmentation is demonstrated by removing the image preprocessors shown in Fig. 3. We also prepare another data augmentation called color jittering [23, 29, 28], for reference. Only three tasks, which are Cup-catch, Reacher-hard, and Finger-turn-hard, are taken into this experiment. Tables II summarize the results from the performed ablation study, which reveals that both of the proposed components are essential to achieve state-of-the-art results.