Sparse Representation Learning with Modified q-VAE towards Minimal Realization of World Model

Before describing the method of extracting the latent space, which is the main topic of this paper, we briefly introduce the world model for the given state and the use of MPC with it [7]. Here, we first define the state as $s\in\mathcal{S}\subset\mathbb{R}^{|\mathcal{S}|}$, the robot action as $a\in\mathcal{A}\subset\mathbb{R}^{|\mathcal{A}|}$, and the reward (or cost) as $r\in\mathbb{R}$ (with the state and action spaces $\mathcal{S}$ and $\mathcal{A}$, respectively). Note that $|\cdot|$ with space denotes the size of dimensions of the given space. In addition, a discrete-time system is often supposed in the world model and MPC. Therefore, the time step is given as $t=\mathbb{N}$, and it can be noted as a subscript to the above variables to clarify the time of them.

Following the above definitions, we set the world model $\mathcal{W}_{\theta}$ with a set of parameters $\theta$ as follows:

where $p_{\cdot}(\cdot\mid\cdot)$ denotes the conditional probability. That is, with the current state-action pair $(s_{t},a_{t})$, the world model predicts the future state $s_{t}^{\prime}=s_{t+1}$ and evaluates the current situation as $r_{t}$. With this structure (i.e. Markov decision process), a transited state $s^{\prime}$ and an evaluated reward $r$ obtained by an action $a$ given to the “actual” environment in a state $s$ are combined into a tuple $(s,a,s^{\prime},r)$, and a dataset with $N$ tuples, $\mathcal{D}_{\mathcal{W}}=\{(s_{i},a_{i},s^{\prime}_{i},r_{i})\}_{i=1}^{N}$, can be constructed and used to train the world model. Specifically, we can find $\theta\to\theta^{\ast}$ that achieves the following negative log-likelihood minimization problem.

where $\mathbb{E}_{\mathcal{D}_{\mathcal{W}}}[\cdot]$ denotes the expectation operation by randomly sampling tuples from $\mathcal{D}_{\mathcal{W}}$.

It is important to note that the world model only includes the action as one of the conditions, namely, if the robot freely plans and generates the action sequence $a_{t:t+H}=[a_{t},a_{t+1},\ldots,a_{t+H}]$ with $H\in\mathbb{N}$ horizon step, its value can be evaluated via simulating the world model to improve the way to generate $a_{t:t+H}$. This mechanism is utilized in the sampling-based nonlinear MPC used in this paper, so-called cross entropy method (CEM) [7] (see Fig. 2). Based on the evaluation of $a_{t:t+H}$, the optimal $a_{t:t+H}^{\ast}$ is eventually obtained by repeatedly modifying and re-evaluating $a_{t:t+H}$ in the direction of improving the evaluation. Note that although MPC optimizes the whole action sequence, only $a_{t}^{\ast}$ is actually used, since this optimization is conducted at every time step.

Specifically, CEM samples $K$ candidates of $a_{t:t+H}$, $\{a_{t:t+H}^{k}\}_{k=1}^{K}$, from a proposal distribution (or policy), $\pi$, at each iteration (i.e. the evaluation and improvement), and evaluates all of them using the world model. The score of each candidate is given as the sum of rewards $R^{k}=\sum_{h=0}^{H}r_{t+h}$. With this score, $K$ candidates is sorted in descending (or ascending if cost is used instead of reward) order, and then the top $\nu K$ ($\nu\in(0,1)$ denotes the elite ratio) candidates is extracted as the elites. Since these elites should be actively sampled, a new policy $\pi^{\prime}$ is obtained through the following maximum likelihood estimation.

where $R_{\mathrm{threshold}}$ denotes the minimum score in the elites. $\mathbb{I}(\cdot)$ is defined as the indicator function, which returns one if the condition in the bracket is satisfied; otherwise zero. If $\pi$ is modeled as normal distribution with $\mu$ location and $\sigma$ scale, this can be analytically solved by the mean and standard deviation of the elites, respectively.

Note that this improvement $\pi=\pi^{\prime}$ is largely sample-dependent, hence if the samples are biased, $\pi^{\prime}$ would overfit to one of the local optima. To mitigate this issue, the following smooth update is often employed.

where $\theta_{\pi}$ denotes the set of parameters in $\pi$ (in the case of normal distribution, $\theta_{\pi}=[\mu,\sigma]$). Larger $\eta\in(0,1)$ makes the update smoother. The above process (with sampling, evaluation, and improvement) are iterated until the specified number of times or the specified time is exceeded, and the mean of the final $\pi$ updated, or the one with the highest score among the candidates sampled so far, is returned as the optimal action sequence.

Let us briefly introduce several important properties of Tsallis statistics [20, 21], which is utilized in this paper. First of all, well-known natural logarithm $\ln(\cdot)$ is extended to $q$-logarithm $\ln_{q}(\cdot)$ with $q\in\mathbb{R}$ in Tsallis statistics.

where $x>0$. As illustrated in Fig. 3, $q$-logarithm with $q>0$ is concave. While natural logarithm has infinite upper and lower bounds $\pm\infty$, in $q$-logarithm, either is finite according to $q$.

In addition, the following inequality holds for $q_{1}<q_{2}$.

The equality is satisfied only when $x=1$.

In the derivation of q-VAE, several important tricks are described below. First, in $q$-logarithm, pseudo-additivity is established instead of additivity.

For the reciprocal, the following formula holds.

Finally, the $q$-deformed Kullback-Leibler (KL) divergence (or, Tsallis divergence) is given as follows:

Note that some probability distribution models, such as exponential distribution families, have closed-form solutions even for Tsallis divergence [22].

As a comparison to q-VAE, we first derive the original VAE [13]. For VAE, a dataset $\mathcal{D}_{\mathcal{X}}=\{x_{i}\}_{i=1}^{M}$, where $x\in\mathcal{X}$ is the data observed by sensors and $M$ of them are collected. Here, $\mathcal{D}_{\mathcal{X}}$ is distinguished from the dataset for the world model $\mathcal{D}_{\mathcal{W}}$ by definition. However, for practical use, the dataset $\mathcal{D}=\{(x_{i},a_{i},x^{\prime}_{i},r_{i})\}_{i=1}^{N}$ can be reused for both of them by extracting $\mathcal{D}_{\mathcal{X}}\subset\mathcal{D}$ from it to train VAE; and converting it into $\mathcal{D}_{\mathcal{W}}$ by mapping $x_{i},x^{\prime}_{i}\to s_{i},s^{\prime}_{i}$ with VAE.

Anyway, for $\mathcal{D}_{\mathcal{X}}$, in order to obtain a generative distribution $p(x)$, the problem of maximizing its log-likelihood is considered. In variational inference, $x$ is supposed to be generated stochastically depending on the corresponding latent variable $z\in\mathcal{Z}\subset\mathbb{R}^{|\mathcal{Z}|}$ (in general, $|\mathcal{Z}|<|\mathcal{X}|$). In that case, $p(x)$ can be represented to be $p(x)=\int p(x\mid z;\phi)p(z)dz$ with a pre-designed prior distribution $p(z)$ and a decoder $p(x\mid z;\phi)$ with the set of parameters $\phi$. From this relation, the variational lower bound $-\mathcal{L}(\phi;\mathcal{D}_{\mathcal{X}})$ is derived as follows:

where $p(z\mid x_{i};\phi)$ denotes the variational posterior distribution (or encoder). Note that $q(\cdot)$ is generally used instead of $p(\cdot)$ to denote the variational distribution, but since $q$ appears in Tsallis statistics, it is unified with $p(\cdot)$ to avoid confusion. The inequality in the above derivation is given by Jensen’s inequality using the fact that the natural logarithm is a concave function. In order to minimize $\mathcal{L}(\phi;\mathcal{D}_{\mathcal{X}})$, the computational graph for $\phi$ is constructed using the reparameterization trick [13], etc., and one of the stochastic gradient descent methods [23, 24] is used to optimize $\phi$. Furthermore, by considering the minimization problem of $\mathcal{L}(\phi;\mathcal{D}_{\mathcal{X}})$ as a constrained optimization problem with KL divergence, $\beta$-VAE [14], which multiplies KL divergence by a weight $\beta>0$, is derived via Lagrange’s method of undetermined multipliers.

For convenience, the first term is called the reconstruction term to increase the accuracy of reconstructing the observed data from the encoded latent variable, and the second term is called the regularization term that attempts to match the encoder to the prior. The regularization term shapes the latent space according to the prior, and the design of the prior promotes disentangled representation (i.e. independence and sparsification). For implementation, from the many reasons (e.g. the closed-form solution of KL divergence can be obtained, the reparameterization trick is well established, and the computational cost is small), $p(z)$ is frequently given by the standard normal distribution $\mathcal{N}(0,I)$, and $p(z\mid x_{i};\phi)$ is accordingly modeled by a diagonal normal distribution. Note that the model of $p(x\mid z;\phi)$ depends on $x$: for real data such as robot coordinates, a diagonal normal distribution (with fixed variance in some cases) or other real-space distribution like student-t distribution [25] is used; and for image data (normalized to $[0,1]$ for each pixel), Bernoulli distribution (recently, continuous Bernoulli distribution [26]) is adopted.

Finally, we introduce the original q-VAE [19], which uses the same variables and probability distributions, but replaces the starting point for the maximization problem with the $q$-log likelihood. By restricting $q>0$ to make $q$-logarithm concave, the variational lower bound for this problem can be derived as in the usual VAE (note the pseudo-additivity).

where $\rho(x,z)=p(z)/p(z\mid x;\phi)=(1-q)\ln_{q}p(z)/p(z\mid x;\phi)+1>0$. Under the condition on $q<1$, when $\rho$ is small (i.e. $p(z)<p(z\mid x;\phi)$), the influence of the reconstruction term is suppressed and the regularization term dominates, thus promoting $p(z\mid x;\phi)\to p(z)$. Otherwise, the reconstruction term becomes dominant and $p(z\mid x;\phi)$ tries to extract the information needed for the reconstruction. In the original paper, this behavior is regarded as an adaptive $\beta$ in $\beta$-VAE, automatically adjusting the trade-off between the disentangled representation promoted by large $\beta$ and the reconstruction accuracy impaired by it. Indeed, experimental results in that paper showed that the reconstruction accuracy can be retained while increasing the independence among latent variables compared to $\beta$-VAE. Note that, with $q=1$, the above problem reverts to the standard VAE.

In the original q-VAE, numerical stability issues were found, and two ad-hoc cheap tricks were made to address these issues. The first is the removal of the computational graph leading to $\rho$, making $\rho$ a merely adaptive coefficient. In this way, $\rho$ can be regarded as a part of $\beta$ in the original version, but $\phi$ should naturally be updated by the computational graph to $\rho$. This trick may cause large biases in the behavior during training and the obtained latent space.

The other is the limitation of the decoder model. In many cases, the observed data handled by VAE are of very high dimension, and the following another representation of $q$-logarithm for them reveals that it tends to have relatively large values in the exponential function, causing numerical divergence.

That is, if $p(x)$ is given as probability density function (i.e. $x$ is in continuous space), $p(x)$ can be over one, resulting in the positive log-likelihood. Even if the log-likelihood for each dimension is slightly positive, the value accumulated tens of thousands of them would easily diverge the above exponential function numerically. To avoid this issue, the original q-VAE limits the decoder model that cancel $q$-logarithm, such as $q$-Gaussian distribution [21]. Of course, it is not a good idea to do so, because various literatures have reported performance gains by utilizing different decoder models [25, 26], as mentioned above.

For these two open issues, this paper deepens the analysis of the original q-VAE and decomposes it into a new surrogated variational lower bound. In addition, as a part of the flexibility of the decoder model, we also derive a formulation that takes into account the case of mixed observations that should be represented by different models. Note that we take care of making the modified q-VAE a general form by guaranteeing that it reverts to the standard VAE when $q=1$ (and other hyperparameters are appropriately given).

In the original q-VAE, the computational graph of $\rho$ is removed to stabilize the computation, and it becomes a merely adaptive coefficient. Thus, overall, the original q-VAE solves the multi-objective optimization problem of reconstruction and regularization terms by scalarizing them with an (adaptive) linear weighted sum. On the other hand, in the modified q-VAE (with $q<1$), the likelihood of the encoder, which was the denominator of $\rho$, is eliminated, but the regularization by the prior distribution in the numerator is retained, preserving the computational graph. This means that the multi-objective optimization problem with the reconstruction and regularization terms is scalarized and solved in the form of a product. This product is important: both terms must be satisfied simultaneously to obtain a high value.

However, in the maximization problem involving such a product, each term must always be non-negative. That is, if the sign of one term is reversed to negative, the other term has a negative coefficient, converting its maximization problem into a minimization problem. Of course, this switching is also a factor that makes learning unstable. Since the regularization term $p(z_{i})^{1-q}$ is non-negative in definition, we need to reveal the conditions for the non-negative reconstruction term.

Now, we first consider the case with $C=1$ for simplicity. The reconstruction term $\ln_{q_{1}}p(x_{1}\mid z;\phi)$ becomes negative with $p(x_{1}\mid z;\phi)<1$ and non-negative with $p(x_{1}\mid z;\phi)\geq 1$. Since sign reversal may occur depending on the performance of the decoder, we substitute the definition of $q$-logarithm when $q\neq 1$ (see eq. (5)) and reform eq. (23) as follows:

Using this form and the fact that $q$-logarithm with $q<1$ has a finite lower bound as shown in eq. (6), the condition for the values in the brace $\{\cdot\}$ to always be non-negative is revealed.

Similarly, the conditions required when $C>1$ are also considered. The terms inside the brace $\{\cdot\}$ of eq. (24) is derived as follows:

where $q_{0}=q$ and $\zeta_{0}=\beta$ for simplifying the description. The first term is always non-negative, so the required conditions are gained from the second term.

When the above conditions are satisfied, the improvement in the reconstruction accuracy of the observed data (i.e. compression of important information into the latent space) and the regularization of the encoder (i.e. organization of the latent space) should be promoted simultaneously. It is also important that this regularization is given in terms of likelihood rather than log-likelihood to the prior. That is, even if the prior is assumed to be a diagonal distribution such as $p(z)=\mathcal{N}(0,I)$, the likelihood is given by the product of the likelihoods for the respective dimensions. As in the discussion above, if the regularization for each dimension is taken as a multi-objective problem, the regularization for all dimensions should be simultaneously established. This suggests that each dimension of the latent space is sparsified as much as possible to match the prior (basically, centered at the origin), while still extracting sufficient information into the latent space to reconstruct the observed data. In addition, the regularization status of the other dimensions influences each other, thus preventing duplication of information and encouraging independence. As a result, the latent variable $z$ on the latent space constructed by the modified q-VAE is expected to coincide with the state $s$, which is the minimal realization for the whole behavior of the system.

As a statistical validation in the simulation, we use CEM to control highway-env [27]. Specifically, as shown in Fig. 4, this task is to avoid colliding with other blue car(s) and going out of the lane by operating the accelerator and steering wheel (i.e. two-dimensional continuous action space) of a yellow car. Usually, geometric information between cars can be used for observation, but in this verification, the RGB image of 300$\times$150$\times$3 in Fig. 4 is resized to 64$\times$64$\times$3 as an observation. This modification requires to extract the latent state from high-dimensional observation. In addition, to show that multiple types of sensors can be integrated as described above, the yellow car’s velocity $v_{x,y}$ is given as another observation.

The control by CEM is real-time oriented and outputs the currently optimal action in each control period even if the maximum number of iterations has not been finished. In addition, the network architecture of VAE implemented by PyTorch [28] is illustrated in Fig. 5. These details and the way to collect the dataset are described in Appendix A.

Under the common settings described above, three methods shown in Table 1 are compared. Note that these values were adjusted by trial and error to increase the accuracy of observation reconstruction as much as possible. In addition, q-VAE is set to satisfy the sparsity condition indicated by eq. (27) while confirming the stability of the numerical computation.

The learning curves for the statistical reconstruction performance (for image and velocity, respectively) with 25 trials are depicted in Fig. 6. The well-tuned parameters yielded that $\beta$-VAE and q-VAE eventually achieved approximately the same level of reconstruction performance, while the standard VAE (with $\beta=1$) performed poorly in reconstructing images. This is probably due to the strong regularization to the prior distribution $p(z)$. In fact, $\beta$-VAE succeeded in image reconstruction by setting $\beta=0.3<1$.

Another feature of q-VAE is that its learning speed tends to be faster than others. Although it is difficult to make a clear agreement because the parameter settings differ greatly from others, we can consider that $\gamma$ over the entropy term of the encoder can be set separately from $\beta$, and $\gamma<\beta$ mitigates the loss of information from the encoder (and the latent space). In fact, previous studies have pointed out the negative effects of the entropy term found in the decomposition of the regularization term in VAE [15], and this is consistent with those reports.

Next, five samples are selected to illustrate the post-learning reconstruction accuracy in Fig. (7). As can be seen, while all methods succeeded in reconstructing the velocity with good accuracy, in the image reconstruction, the standard VAE in the second row failed to visualize the other blue car(s). In other words, it can be said that the encoder of the standard VAE did not properly incorporate the information of the other blue cars into the latent state. In contrast, $\beta$-VAE and q-VAE properly embedded the important information needed for the reconstruction contained in the observation into the latent space.

Finally, the sparsity of the acquired latent space is evaluated. As sparsity, we use the following definition in the literature [29].

where $z_{i}=\mathbb{E}_{p(z\mid x_{i};\phi)}[z]$ denotes the location of the encoder over $x_{i}$ in the dataset. If each component of $z_{i}$ is with the same magnitude, this definition returns zero; in contrast, if only one component has a non-zero value and others are zero, $\mathrm{sparse}(\mathcal{Z})$ converges to one.

With this definition of the sparsity, we evaluated each method as shown in Fig. 8. As a result, only $\beta$-VAE gained low sparsity. This is due to the fact that $\beta<1$ is used to improve the reconstruction accuracy. The standard VAE with $\beta=1$ achieved the same level of sparsity as the proposed q-VAE, but at the expense of the reconstruction accuracy as mentioned above, extracting the meaningless latent space. This trend is also consistent with the previous study [19]. From the above, it can be concluded that q-VAE mitigates the trade-off between the reconstruction accuracy and sparsity, and increases both of them sufficiently, namely, it enables to acquire the important information contained in the observation with the smallest dimension size of the latent space (i.e. minimal realization).

The control performance by CEM is compared between $\beta$-VAE and q-VAE, both of which obtained the sufficiently meaningful latent space. Here, $|\mathcal{Z}|$ was set to be 50, which cannot reduce the computational cost sufficiently. Therefore, in order to confirm the benefit of sparsification, the unnecessary latent dimensions are excluded by masking, and the state space with minimal realization is extracted. As a criterion for judging unnecessary dimensions, the sample standard deviation of the locations of the encoder is evaluated (see Fig. 9). The dimension with the small sample standard deviation mostly takes zero for most of the data, and can be eliminated as an unnecessary dimension. As can be seen in the figure, it is easy to expect that the top eight dimensions (with a standard deviation over 0.15) are important in the case of q-VAE. In line with this, $\beta$-VAE also extracts the top eight dimensions, but there is concern that the necessary information may be truncated.

The world model constructed on the latent space before and after masking is used to implement control with CEM. Each method was tested with different random seeds for 300 episodes, and if no failure occurred during the episode, it was successfully completed after 200 steps. The statistical number of steps and the number of CEM iterations in each episode are depicted in Fig. 10. It is remarkable that masking reduced the computational cost and increased the number of iterations. In fact, the number of inputs/outputs and parameters of the world model are reduced by masking, as shown in Table 2.

The number of steps reveals the performance difference between the methods. First, $\beta$-VAE clearly reduced the maximum performance by masking. This is because the necessary information was missing due to masking, and the optimization by CEM did not work properly. On the other hand, in q-VAE, masking improved the maximum performance more than in the other cases. One of the reason for this is simply that the necessary information is retained even after masking, facilitating the optimization by increasing the number of iterations.

In addition, it was confirmed that q-VAE without masking requires a larger learning rate than the others (from $10^{-3}$ to $10^{-2}$) for making learning of the world model progress. This may be due to the fact that most of inputs and outputs in the training dataset were zero, resulting in over-learning that generate zero. Therefore, the performance of q-VAE without masking may have been insufficient. In fact, the negative log-likelihood of the world model for the test dataset, as shown in Fig. 11, revealed that q-VAE without masking increased the worst-case loss of the dynamics model. Although the learning rate could be adjusted to make progress, over-learning still occurred in favor of majority zeros, overlooking the important features. Note that the masked $\beta$-VAE was reduced the accuracy of the dynamics model, as expected above.

As a demonstration, we conduct a reaching task, named stretch-reach, with a Stretch RE1 developed by Hello Robot [30]. Specifically, as shown in Fig. 12(a), Stretch RE1 is a kind of mobile manipulator with a camera on its top. For simplicity, the motion of this robot is limited to $yz$-axes arm movement (within $[0.87,1]$ in $y$ direction and $[0,0.5]$ in $z$ direction). The target position is 2 cm above an object randomly placed in the $z$ direction (within $[0.15,0.45]$), and the task is to move the arm to that position (see Fig. 12(b)). Similar to the above simulation, one of the observations is an RGB image (originally, 424$\times$240$\times$3), which is pre-processed to 64$\times$64$\times$3. In addition, since the observation of the arm position and velocity can be easily measured from the encoders for the respective actuators, this 4D information is also added as an observation.

The task is considered successful when the arm stops near the target position. Specifically, the following reward function $r$ is provided, and $r\geq-0.02$ is considered success, and 20 steps without success is considered failure.

where $e$ denotes the distance between the current arm position and the target position, and $v$ denotes the arm velocity. From the above setup, although the task itself is simple, the random target position has to be estimated from the RGB image to compute $e$ and to obtain the world model.

Note that the details of other configurations are described in Appendix A, similar to that for the simulation.

Since the standard VAE is insufficient to extract the important information, only two methods, $\beta$-VAE and q-VAE, are tested as described in Table 3. Note again that q-VAE is set to satisfy the sparsity condition indicated by eq. (27) while confirming the stability of the numerical computation.

As in the simulation, after confirming that the reconstruction accuracy was comparable to each other, the importance of the latent dimensions was evaluated in terms of the sample standard deviation (see Fig. 13). From the figure, q-VAE can select six dimensions (with the same threshold 0.15). In theory, this task may have five dimensions for the minimal realization (i.e. the 2D arm position and velocity and the $z$-axis target position), but in reality, other environmental noise (e.g. misalignment of the target object in the $x$ direction) may occur. Therefore, a total of six dimensions can be reasonably extracted: five dimensions for the theoretical minimal realization and one dimension to handle other noise in practice. On the other hand, $\beta$-VAE is difficult to find the important dimensions.

For practical use, the size of dimensions extracted by q-VAE is unknown when using $\beta$-VAE alone, so masking to $\beta$-VAE is omitted in this experiment. In addition, the simulation results described before indicated that q-VAE without masking is prone to make learning of the world model unstable by unnecessary axes, so we omit it as well. Therefore, the performance comparison in this experiment is limited to $\beta$-VAE without masking and q-VAE with masking.

First, we confirm that the performance of the world models is comparable to each other. The dynamics to $H=5$ horizon ahead handled in CEM is illustrated in Fig. 14. It can be seen that the differences between the predictions and the true observations for $\beta$-VAE and q-VAE are comparable for both the image and the arm state. The predictions and true values of the rewards are also compared in Fig. 15. Similarly, the predictions are mostly consistent with the true values, suggesting that both methods achieved the good accuracy. From the above, it can be expected that the difference in control performance between the two methods (to be confirmed in the next section) would be only due to the masking (from 30 to six dimensions) obtained by the sparsity of q-VAE.

Using the learned world model, 5 trials of reaching were performed for each of the 3 target object positions (i.e. 0.2, 0.3, and 0.4 m, respectively). The number of steps until task termination (20 steps at maximum), average reward, and average number of iterations are listed up in Fig. 16. An example of the trials are shown in Fig. 17 and the attached video.

When the object was placed at 0.2 m, q-VAE succeeded in reaching the target earlier than $\beta$-VAE, although there is little difference in reward due to the overall smaller distance penalty. Even in the case with 0.3 m target object position, q-VAE succeeded in performing the task earlier than $\beta$-VAE, and also increased the reward due to smoother acceleration/deceleration. When the target object was placed as far as 0.4 m, $\beta$-VAE failed in all trials, while q-VAE generally succeeded in most cases.

As noted above, these performance gains are not due to differences in the prediction accuracy of the world model, but rather due to the reduced computational cost by the compact world model, which is with almost the minimal realization. In fact, the number of iterations of q-VAE is more than double that of $\beta$-VAE under all conditions. Thus, we confirmed that q-VAE facilitates the minimal realization of the world model through sparsification and contributes to improving the control performance of computationally expensive optimal control such as CEM in real time.

In this paper, we improved and analyzed q-VAE, a deep learning technique for extracting sparse latent space, aiming at the minimal realization of the world model. In particular, we clarified the hyperparameters condition under which the modified q-VAE always sparsifies the latent space. In both simulations and experiments, the modified q-VAE successfully collapsed many latent dimensions to zero while maintaining the same level of reconstruction accuracy as the baseline method, $\beta$-VAE, by learning according to the condition. The world model with almost the minimal realization was obtained by masking the unnecessary latent dimensions and utilized in CEM, which is a sampling-based optimal control method. Consequently, the optimization of CEM was facilitated by the reduction in computational cost, resulting in better control performance.

Two major open issues can be raised from our investigation. One is how to adjust hyperparameters. The hyperparameters of the modified q-VAE have increased due to the decomposition from the conventional q-VAE, and it is clear that their optimal values are task-specific, although the sparsification condition limits the degree of freedom in design of them. In particular, $q\to 0$ is desirable to strongly promote sparsification, but as the literature [31] reported, a small $q$ would exclude many data from training as outliers. In fact, in highway-env simulations, we observed cases where other blue cars could not be reconstructed as in the standard VAE, depending on the value of $q_{2}$. A framework that is adaptive or robust to this trade-off, such as meta-optimization [32] or ensemble learning with multiple combinations of hyperparameters [33], would be useful.

Another open issue is the simultaneous learning of the latent space and the world model. In this paper, the latent space extraction phase and the world model acquisition phase were conducted independently, giving priority to ease of analysis. However, in order to obtain the state holding the minimal realization, it is desirable to extract the latent space by considering the world model (and controller). Since simultaneous learning of multiple modules tends to be basically difficult, it would be important to introduce curriculum learning [34], etc.

Anyway, we will apply our method to more practical tasks in the near future and enhance its practicality by resolving the above-mentioned open issues.