PEARL: Parallelized Expert-Assisted Reinforcement Learning
for Scene Rearrangement Planning

Sequential decision making procedure is often considered in a Markov Decision Process (MDP). A discounted infinite-horizon MDP is defined as a tuple $(\mathcal{S},\mathcal{A},\mathcal{P},\mathcal{R},\gamma)$ Puterman (1994), where $\mathcal{S}$ is the space of states, $\mathcal{A}$ is the space of actions; $\mathcal{P}$ is the transition function: $\mathcal{P}(s^{\prime}|s,a)$ is the probability of transforming $s$ to $s^{\prime}$ by taking action $a$, $s^{\prime}$ is also written as $\mathcal{P}(s,a)$; $\mathcal{R}$ is the reward function: $\mathcal{R}(s,a)$ represents the reward received from the environment by taking action $a$ at the state $s$; and $\gamma$ is the discount factor. A distribution over the valid actions $a$ given the state $s$ is called a policy, denoted as $\pi(a|s)$. The value function $V^{\pi}(s)$ is the expectation of accumulated discounted reward by following $\pi$ starting in state $s$, i.e.,

$$V^{\pi}(s)=\mathbb{E}_{|s}^{\pi}[\sum_{t=0}\gamma^{t}\mathcal{R}(s_{t},a_{t})\pi(a_{t}|s_{t})],\ s_{0}=s.$$

(1)

The optimal policy $\pi^{*}$ ought to maximize this expectation. Formally, we have $\pi^{*}=\mathop{\arg\max}_{\pi}V^{\pi}(s)$.

Monte Carlo Tree Search (MCTS) Kocsis and Szepesvári (2006); Guo et al. (2014) is a useful strategy to address the challenge of selecting/learning policies for MDP. It is used to estimate the value of states through repeated Monte Carlo sampling and simulations. Each node in the search tree corresponds to a state $s$. The root node represents the current state. The edge from $s_{1}$ to $s_{2}$ represents the execution of action $a$ by which $s_{1}$ can be transformed to $s_{2}$.

In MCTS, four phases are repeated to grow the search tree: a) selection; b) expansion; c) rollout; and d) back-propagation. In the selection phase, a feasible unexpanded edge in the tree is selected according to a tree policy. The commonly used tree policy is the Upper Confidence bounds for Trees (UCT) Auer et al. (2002), i.e.,

$$\text{UCT}(s,a)=\frac{r(s,a)}{n(s,a)}+C\sqrt{\frac{\log n(s)}{n(s,a)}},$$

(2)

where $r(s,a)$ is a prior function that gives suggestion on search. A canonical choice of this function returns a probability of taking action $a$ at state $t$. It could be estimated by simulations or a learnable function Silver et al. (2017), e.g., neural network. $n(s,a)$ is the counter function which returns the visiting times of this edge; $n(s)$ is the counter function for the node corresponding to state $s$; $C$ is the tradeoff parameter to tune the influence of the terms. In the expansion phase, a new node corresponds to state $s^{\prime}\in\{x|\mathcal{P}(x|s,a)\}$, which is expanded from the selected edge. Then, in the rollout phase, a quick simulation under the rollout policy is performed from the state $s^{\prime}$ until a terminal condition is satisfied. The results of simulation are used as the estimate of the node. Finally, in the back propagation phase, the information of nodes, including the visited times $n(s)$, $n(s,a)$ and the estimated value $V(s)$, is updated bottom-up through the tree until it reaches the root node.

When the tree is built, $a^{*}=\mathop{\arg\max}_{a}E(s_{\text{R}},a)$ is the action selected by MCTS, where $s_{\text{R}}$ is the state of the root node and $E(s,a)=\mathcal{R}(s,a)+\gamma V(\mathcal{P}(s,a))$.

The success of employing Deep Reinforcement Learning (DRL) in playing Atari games Mnih et al. (2015) brings a great impact to the field. Various Deep Reinforcement Learning paradigms were proposed to solve MDP and achieved remarkable performance. However, those methods suffer from sparse rewards when facing challenging tasks, commonly resulting in slow convergence and local minima. Those drawbacks can be alleviated by introducing Imitation Learning (IL). In IL, an agent is taught to mimic the actions from experts. Experts can arise from observing humans completing a task and a labeled dataset can be constructed. Nevertheless, acquiring human demonstrations is expensive. The agent trained in static data also has difficulty in generalizing to novel observations.

Expert Iteration provides a method to create an expert which is able to evolve. The expert is composed of a boosting framework and a fast vanilla policy (e.g., neural network). The canonical choice of the boosting framework is a tree search algorithm (e.g., MCTS) which explores the possible states and exploits the gathered information to generate a strong move sequence. The vanilla policy can bias the searching towards more promising states and provide a quick estimation for the explored state. The vanilla policy is also known as the apprentice policy because it can imitate the actions taken by the built expert to accelerate the training. The expert can be improved by embedding the updated apprentice policy.

The Scene Rearrangement Planning (SRP) can be modeled as a MDP. The goal of SRP is to transform the current layout of the scene to the target layout. An agent can achieve this goal by learning a policy to make decisions and maximize the accumulated discounted reward during moving.

In robotics, more atomic action spaces are preferred since they are more tractable. Some robotic works Yuan et al. (2019); Song and Boularias (2019) adopt nonprehensile actions (e.g., push) in the rearrangement problem. Such action definition can be more easily achieved through a real mechanical arm. In this spirit, we define 6 simple nonprehensile actions to transform the positions and orientations of the objects. The first 4 actions are moving an object towards up, down, left or right direction by 1 unit. The last two actions are rotating an object clockwise or anti-clockwise by 15 degrees.

To solve the challenging problem, we propose a novel variation of the expert iteration framework to train the agent efficiently. The boosting framework is a MCTS algorithm. Our agent plays the role of apprentice, which is composed of a policy network $\pi_{\theta}$ to predict the action distribution as well as a value network $v_{\phi}$ to predict the future value. $\theta$ and $\phi$ are parameters of the neural networks.

The expert iteration has two stages in each iteration: 1) Improving stage and 2) Learning stage. In the improving stage, the apprentice is used to build the expert and the expert creates a playing record for the current episode. To construct the expert, the networks are used in the selection and simulation phases of MCTS. Specifically, in the selection phase, we use the policy network and the value network to suggest a more promising action to explore. The formulation of action score can be derived from Eq. 2,

$$\text{UCT}_{\text{NN}}(s,a)=\frac{\pi_{\theta}(a|s)}{n(s,a)}+C\sqrt{\frac{\log n(s)}{n(s,a)}},$$

(3)

In the simulation phase, the value of an expanded tree node is evaluated by the value network. In the learning stage, the policy network and the value network are trained in a reinforced fashion. In contrast to simply imitating expert actions in expert iteration, the learning of the apprentice in our method is reinforced by the rewards from the environment, which means that the apprentice is encouraged or discouraged to take the expert action accordingly. The learning is similar to Advantage Actor-Critic (A2C) Mnih et al. (2016), where the policy network plays the role of actor and the value network is the critic. Since MCTS selects the action by the value of the action that the agent can take at the root node, it provides both an expert action and an expert estimate of the state value. The expert estimate of a state value is $V(s;\theta,\phi)=\max_{a}\mathcal{R}(s,a)+\gamma V(\mathcal{P}(s,a);\theta,\phi)$. Then the loss of the policy network is:

$$\mathcal{L}_{p}=-\sum_{t=0}A(s_{t},a_{t})\log(\pi_{\theta}(a_{t}|s_{t})),$$

(4)

where $A(s_{t},a_{t})$ is the advantage defined as:

	$\displaystyle A(s_{t},a_{t})=$	$\displaystyle\sum_{i=0}^{k}\gamma^{i}\mathcal{R}(s_{t+i},a_{t+i})+\gamma^{k}V(s_{t+k};\theta,\phi)$
		$\displaystyle-v_{\phi}(s_{t}).$

The value network tries to mimic the state value estimated by the expert. The loss of the value network is:

$$\mathcal{L}_{v}=\sum_{t=0}\parallel v_{\phi}(s_{t})-V(s_{t};\theta,\phi)\parallel_{2}.$$

(5)

In this way, our method can potentiallly reduce the bias of learning target on the optimal policy compared to imitation learning for expert iteration.

We train the networks in a parallelized online synchronous manner, where a group of data processes synchronize with the latest network parameters and collect the experiences to support the training of the network in the optimization process. Since the fast convergence is a well known advantage of imitation learning, similar to Wang et al. (2019), we tradeoff the imitation signal and reinforcement signal to speed up the training.

It is worth noting that the improving stage is extremely time consuming compared to the learning stage because of the large simulations in the improving stage. Actually, in the learning stage, it is a waste if the apprentice learns from the selected action and observation records for only once. To improve data utilization, we adopt Prioritized Experience Replay (PER) Schaul et al. (2016) in network training.

We demonstrate our approach on 3D-FRONT Fu et al. (2020), a large-scale indoor scene dataset, where the rooms layout are professionally designed and populated with high-quality 3D models. The interior designs of the rooms are transferred from expert creations. This dataset contains 6,813 distinct houses and 19,062 furnished rooms. It is proposed to support indoor scene tasks like 3D scene understanding, indoor scene synthesis, semantic segmentations, etc.. The diverse variety of furnished rooms and the abundant labels satisfy the demand of the SRP task. In SRP, the complexity of planning highly depends on the number of objects in the scene. The rooms in 3D-FRONT have 6.1 objects in average. More than $99\%$ rooms in the dataset have a number of objects ranging from $1$ to $20$. The statistics of the number of objects in each room is shown in Fig. 2. We split the dataset into the training and test sets. 5% of the data (953 rooms) are randomly sampled as the test set and the rest (18,109 rooms) belong to the training set.

We use the designed layout as the target layout and randomly sample a feasible layout with the same objects as the initial layout. To ensure that the layout pair can be transformed from each other, we sample an initial layout by rounds of random walks starting from the target layout. In each round, a random object is selected to perform a random feasible move. Each initial layout is generated by performing 1,000 rounds of random walks. The configurations of the layouts are extracted from the top-down sihouette of the scene. We first bound the view of the scene to the center of a square. Then the objects are rendered individually and the top-down sihouettes are discretized to extract the shape and position for each object. The same process is also adopted to extract the information of impassable districts. The resolution of the discretization is $64\times 64$.

For a fair comparison, the networks used in our approach, RL, and Expert Iteration share the same architecture. PER strategy is unitilized in the training of those agents.

To analyze the training behavior of the agents, we plot the curve of SR and Length during training. As shown in Figure 4, the x-axis is the number of epsiodes the agent experienced, the y-axes are SR and Length respectively. Compared to RL, the class of expert-assisted algorithms (i.e., Expert Iteration, and PEARL) learn very fast, which illustrates the high efficiency of the actions sampled by expert. Though the performance of our approach grows slower than Expert Iteration at the beginning, it finally achieves higher success rate than Expert Iteration. It shows that the actor trained in a reinforced manner converges to a better policy compared to directly imitating expert actions.