ACuTE: Automatic Curriculum Transfer from
Simple to Complex Environments

An episodic Markov Decision Process (MDP) $M$ is defined as a tuple $(\mathcal{S},\!\mathcal{A},\!p,\!r,\!\gamma)$, where $\!\mathcal{S}\!$ is the set of states, $\!\mathcal{A}\!$ is the set of actions, $\!p(s^{\prime}\!|s,a)\!$ is the transition function,  $\!r(s^{\prime}\!,a,s)\!$ is the reward function and $\gamma\!\in\![0,1]$ is the discount factor. For each timestep $t$, the agent observes a state $s$ and performs an action $a$ given by its policy function $\pi_{\theta}(a|s)$, with parameters $\theta$. The agent’s goal is to learn an optimal policy $\pi^{*}\!$, maximizing its discounted return $G_{0}=\sum^{K}_{k=0}\!\gamma^{k}\!r(s^{\prime}_{k},a_{k},s_{k})$ until the end of the episode at timestep $K$.

We define a task-level curriculum as:
Let $\mathcal{T}$ be a set of tasks, where $M_{i}=(\mathcal{S}_{i},\mathcal{A}_{i},p_{i},r_{i})$ is a task in $\mathcal{T}$. Let $\mathcal{D^{T}}$ be the set of all possible transition samples from tasks in $\mathcal{T}$ : $\mathcal{D^{T}}=\{(s,a,r,s^{\prime})\,|\,\exists\,m_{i}\!\in\!\mathcal{T}\>s.t.\>s\!\in\!\mathcal{S}_{i},\>a\!\in\!\mathcal{A}_{i},\>s^{\prime}\!\sim\!p_{i}(\cdot|s,a),\>r\leftarrow r_{i}(s,a,s^{\prime})\}$. A curriculum $C=[M_{1},M_{2},\ldots,M_{n}]$ is an ordered list of tasks, where $M_{i}$ is the $i^{th}$ task in the curriculum. The ordered list signifies that samples from $M_{i}$ must be used for training a policy before samples from $M_{i+1}$ are used. The sequence of tasks terminates on the target task $M_{n}$.

The aim of CL is to generate a curriculum and train an agent on a sequence of tasks $\{M_{1},M_{2},...,M_{U}\}$, such that the agent’s performance on the final target task $(M_{U})$ improves relative to learning from scratch. The domains $\mathcal{T}^{HF}$ and $\mathcal{T}^{LF}$ of possible tasks are sets of MDPs in the high-fidelity (HF) and the low-fidelity (LF) environments, respectively. An individual task can be realized by varying a set of parametric variables and subjecting the task to a set of constraints. The parametric variables $P$ are a set of attribute-value pair features $[P_{1},...,P_{n}]$ that parameterize the environment to produce a specific task. Each $P_{i}\in P$ has a range of possible values that the feature can take while the $constraints$ of a domain are a set of tasks attained by determining the goal condition $P_{G}$.

Let $\mathcal{C}_{U}^{\mathcal{T}_{LF}}$ be the set of all curricula over tasks $\mathcal{T}^{LF}$ of length $U$ in the LF environment. Similarly, let $\mathcal{C}_{U}^{\mathcal{T}_{HF}}$ be the set of all curricula over tasks $\mathcal{T}^{HF}$ of length $U$ in the HF environment. The goal is to find a curriculum ${c_{U}}^{\mathcal{T}_{LF}}$ in the LF environment that can be transferred through a set of mapping functions $\mathcal{F}:=\{f_{1},f_{2},\ldots,f_{n}\}$ to attain the curriculum in the HF environment ${c_{U}}^{\mathcal{T}_{HF}}$. A mapping function maps the parametric variables in the LF environment ($P^{LF}$) to the parametric variables in the HF environment ($P^{HF}$). We characterize the mapping as an affine transformation given by:

where $A=[a_{1},\ldots,a_{n}]^{T}\in\mathbb{R}^{n}$ and $B=[b_{1},\ldots,b_{n}]^{T}\in\mathbb{R}^{n}$ denote linear mapping and translation vector and $\odot$ is the Hadamard product. Thus, a parameter mapping ($f_{i}:P^{LF}_{i}\rightarrow{}P^{HF}_{i}$) is given by:

The source tasks of the curriculum in the HF environment are learned before final target task as described in Section 3.2.

For the physical environment shown in Fig 2c, we generate Crafter-TurtleBot (Fig 2b), a realistic simulation of the physical environment. The aim is to learn a policy in this high-fidelity (HF) environment, through an automated curriculum transfer from the low-fidelity (LF) environment (Fig 2a), and perform Sim2Real Policy transfer from the HF environment to execute the task in the physical environment.

The agent’s goal is to break 2 trees to collect 2 pieces of wood, break a rock to collect a stone and craft a stone axe at the crafting table. The agent needs to navigate, face the object and perform the break action to collect it in inventory. The parametric variables for this task are the width $(P_{W})$ and height $(P_{H})$ of the navigable area, the number of trees $(P_{T,e})$, rocks $(P_{R,e})$, and crafting table $(P_{CT})$ present, the number of wood $(P_{T,i})$ and stones $(P_{R,i})$ present in the inventory of the agent when the episode starts, and the goal $(P_{G})$ of the task. The goal is drawn from a discrete set, which can be navigating to an item, breaking a subset of the items present in the environment or crafting the stone axe.

As described in Fig 1, the HF target task is mapped to its LF equivalent. The simplified LF dynamics allow efficient curriculum optimization. Once the curriculum is generated, each task of the LF is mapped back to generate an equivalent HF task, which are learned through a curriculum to develop a successful task policy for the target HF task. This policy is then transferred to the Physical Robot through a Sim2Real Transfer.

ACuTE consists of three parts: Generating a LF target task, Curriculum generation in LF, and Task sequencing and learning in HF. Algorithm 1 presents our approach. The first step entails generating the LF target task from the HF target task. To obtain the task parameters for the LF task, we pass the HF target task parameters ($P^{HF_{U}}$) through the inverse of the affine mapping functions $f^{-1}(P^{HF_{U}})\>\forall f\in\mathcal{F}$ followed by Generate_Env (line 1), to obtain the parameters for the target task in the LF environment. The two requirements for obtaining a corresponding mapping are as follows:

• •

  Each task parameter in the HF environment ($P^{HF}$) needs a corresponding task parameter in the LF environment ($P^{LF}$). $\forall\>P^{HF}_{i}\!\in\!P^{HF}\>\exists\>P^{LF}_{i}$ s.t. $f_{i}(P^{LF}_{i})=P^{HF}_{i}$. Varying these task parameters yields different tasks that are sequenced to form a curriculum.

• •

  The final task in the HF environment must be mapped to the final task in the LF environment. $\forall f\in\mathcal{F}\>\exists\>f(P^{LF_{U}})=P^{HF_{U}}$ Thus, we can guarantee that each source task obtained through the curriculum in the LF can be mapped to a corresponding task for the HF environment.

The actions in the HF can be complex, but each action can be simplified to an LF action that need not be correspondingly equivalent. Our approach does not assume an equivalency between the state or the action spaces between the LF and the HF environments, but only on the two requirements listed above. To generate and sequence the source tasks in the LF environment, we compared two approaches. The first approach, called Handcrafted Curriculum Transfer (HC), involves a human expert deciding the parametric variables $P^{HC}$ for the tasks for the curriculum (line 3). The second approach, Automated Curriculum Transfer (AC), automatically generates and optimizes a sequence of source tasks from the parametric variables $P^{AC}$ for the agent to learn the final task with the fewest number of episodes (line 5).

To optimize curricula through Handcrafted Curriculum Transfer (Generate_HC) (HC), a human expert determines the parametric variables $P^{HC}$ and the task sequence for the source tasks in the curriculum. Whereas, for optimizing curriculum using the Automated (AC) procedure (Generate_AC), we use the approach given in Algorithm 2. We start from an empty sequence of source task parameters $P_{W}$. The algorithm calls a parameterizing function (Init_Src) that assigns random values to the parametric variables for the first source task $P^{LF_{1}}$ from the range of values $P^{AC}$ can attain while simultaneously initializing an RL agent (Init_Agent). Based on $P^{LF_{1}}$, the algorithm generates the first task for the agent, $M^{LF}_{1}$, using the function (Generate_Env). The agent attempts learning this source task $({\tt Learn})$ with the initial policy $\pi_{1,w,init}$, until the stopping criterion is met. The stopping criterion determines if the agent’s goal rate ($\delta$) is $\!\geq\!\delta_{G}\!$ in the last $s$ episodes (Algorithm 1 line 12). Failure to meet the stopping criterion implies that the agent has reached maximum permitted episodes (termed budget ($b$)) signifying $\delta\!<\!\delta_{G}$. The value for $\delta_{G}$ is set at $0.85$ for our experiments.

In AC, the first task of the curriculum is randomly initialized $N$ times and learned until stopping criterion is met. The algorithm then finds the $W$ most promising solutions (${\tt Best\_Candidates}$), based on fewest interactions to meet the stopping criterion. To optimize the sequence of the curriculum, we use beam search Lowerre (1976). Beam search is a greedy search algorithm that uses a breadth-first search approach to formulate a tree. At each level of the tree, the algorithm sorts all the ($N\times W$) successors of the tree ($N$ successor tasks for each task of $W$) at the current level in increasing order of number of episodes required to learn the task $M^{LF}_{u,w,n}$. Then, it selects $(W)$ number of best tasks at each level, given by fewest interactions to reach stopping criterion, and performs the same step until the tree reaches the desired number of levels $(U)$.

Now, using the parametric variables $P^{LF_{1}}$ for each task in the beam ($w\!\in\!W$), the algorithm generates parametric variables $P^{LF_{2}}$ (${\tt Init\_Inter}$) for the next task $M^{LF}_{2}$ in the curriculum. This is done by choosing a goal $P_{G}$ not encountered by the agent until the current level $u$ in the beam $w$, and randomly initializing parametric variables $\geq$ the minimum required to accomplish this goal. The agent then attempts learning $M^{LF}_{2}$ with the final policy of the previous source task in the beam $\pi_{1,w,fin}$ (${\tt Load\_Agent}$). The task terminates when the agent meets the stopping criterion. The algorithm finds the $W$ most promising solutions, given by fewest interactions to reach stopping criterion (${\tt Best\_Candidates}$) and carries out this procedure iteratively, until the final target task $M^{LF}_{U}$ is learned. The parameters of the curriculum with the lowest number of episodes to reach the stopping criterion is selected as the most promising solution (${\tt Best\_LF\_Params}$) $P_{W}$ for learning the target task. The curriculum generation procedure requires the length of the curriculum $U$ to be $>=$ the number of goals attainable. This ensures all the goals available ($P_{G}$) are encountered by the agent in the curriculum.

We used the algorithm presented in Algorithm 1 to generate and sequence the source tasks using the handcrafted curriculum transfer (HC) and the automated curriculum transfer (AC) approach. The navigable area in target task $M^{LF}_{U}$ in low-fidelity (LF) environment is a grid of $\mathbb{R}_{LF}^{2}\!\xrightarrow{}[10\!\times\!10]$, as seen in Fig 2a, and the high-fidelity (HF) target task area is a continuous $\mathbb{R}_{HF}^{2}\!\xrightarrow{}[4m\!\times\!4m]$ space, as seen in Fig 2b. Both these environments contain 4 trees, 2 rocks and 1 crafting table, placed at random locations.

The RL algorithm was a Policy Gradient Williams (1992) network with $\epsilon$-greedy action selection for learning the optimal policy. The episode terminates when the agent successfully crafts a stone axe or exceeds the total number of timesteps permitted, which is $10^{2}$ in the LF environment and $6\!\times\!10^{2}$ in the HF environment. All experiments are averaged over 10 trials. (See Appendix Section A.2).

For generating the AC in the LF environment, we set the width of the beam search algorithm at $W\!=\!4$, and the length of the beam at $N\!=\!20$, the curriculum length was $U\!=\!4$, and budget $\!b\!=\!5\times 10^{3}$. We obtained the parameters after performing a heuristic grid search over the space of the search algorithm. We evaluated our curriculum transfer framework with four other baselines. Learning from scratch trains the final HF target task without any curriculum. We also adopted three approaches from the literature designed for speeding-up RL agents: Asymmetric Self-play Sukhbaatar et al. (2018), Teacher-Student Curriculum learning Matiisen et al. (2020) and Domain Adaptation for RL Carr et al. (2019). The first two baselines do not make use of the LF environment while the third uses the LF environment as the source domain. All baseline approaches involve reward shaping similar to the source task of the automated curriculum transfer approach.

The results in Figs 4a and 4b show that the AC approach results in a substantial improvement in learning speed, and is comparable to the curriculum proposed by a human expert. Furthermore, as shown in Fig 4f, the curriculum transfer method outperforms the three baseline approaches²²2Refer Appendix Section A.5 for learning curves for reward in terms of learning speed. The learning curve for our curriculum transfer approaches has an offset on the x-axis to account for the time steps used to go through the curriculum before moving on to the target task, signifying strong transfer Taylor and Stone (2009). The other three baseline approaches perform better than learning from scratch, but do not outperform the curriculum transfer approach. Self-Play requires training a goal-proposing agent, which contributes to the sunk cost for learning. Whereas, Domain Adaptation relies on the similarity between the tasks. Teacher-Student curriculum learning requires defining the source tasks of the curriculum beforehand, and the teacher attempts to optimize the order of the tasks. All baselines involve interactions in the costly high-fidelity domain for generating/optimizing the curriculum, which proves to be costly. This sunk cost has been accounted in the learning curves by having an offset on the x-axis. See Appendix Section A.3 for details on our adaptation and tuning of these three baselines.

Table 1 compares the jumpstart values for AC with the baselines. The higher jumpstart, the better performance of the curriculum. AC achieves high positive jumpstart and time to threshold performance in comparison to baseline approaches, denoting improved performance and quicker learning (magnitude $>\!10^{7}$ timesteps).

To answer the second question, we evaluated our framework in a situation where the target task in the HF environment is too difficult to learn from scratch. To do this, both the LF and HF environments were modified by adding a new type of object, “fire”, such that when the agent comes in contact with it, the episode terminates instantly with a reward of $\!-1\!\times\!10^{3}$.

The parameters listed in Section 4.1 are used to optimize AC in the LF environment. From learning curves in Figs. 4d, 4e, 4f and Table 1, we observe our curriculum transfer approach, AC, consistently achieving higher average reward, better jumpstart and time to threshold performance compared to baselines. Learning from scratch fails to converge to a successful policy in $10^{8}$ interactions, while the other baseline achieves marginally better performance than learning from scratch. Table 1 summarizes the jumpstart and the time-to-threshold between the approaches. Our approach still achieves a high jumpstart, and converges much quicker than other approaches. Through this experiment, we see our AC approach extrapolates to challenging environments, producing quicker and efficient convergence. Refer Appendix section A.1.1 for trends observed in different runs in our AC approach.

To answer the third question, we conducted experiments by making the HF learning algorithm different (PPO Schulman et al. (2017) and DQN Mnih et al. (2015)) from the RL algorithm used for generating the curriculum in the LF environment (Policy gradient). Fig 5 shows the result of this test (See Appendix A.4 for hyperparameters). In all the cases, learning through a curriculum is quicker and more efficient than learning from scratch. Here, we do not intend to find the best RL algorithm to solve the task, but demonstrate that actor-critic networks, policy gradients and value function based approaches learn the HF target task through curriculum, irrespective of the RL algorithm used to optimize the LF curriculum.

In the above sections, we evaluated the curriculum transfer schema on accurate mappings between the two environments. In certain partially observable environments, it might not be possible to obtain such an accurate mapping between the two environments. To demonstrate the efficacy of our approach in imperfect mappings, we evaluate the experiments by incorporating noise in the mapping function. The noisy mapping function involves a multivariate noise over the range of the parametric variables. While obtaining the noisy parameters, we do not incorporate any noise in the parameter for the goal condition, as we can safely assume that the goals have been mapped accurately. The noisy mappings are given by:

where $\Sigma=\begin{bmatrix}\sigma_{1}&0&0&\ldots\\ 0&\sigma_{2}&0&\ldots\\ \ldots&\ldots&\ldots&\ldots\\ \ldots&\ldots&\ldots&\sigma_{n}\end{bmatrix}_{n\times n}$is the covariance matrix, which is symmetric and positive semi-definite, and $\sigma_{i}=(max(P_{i})-min(P_{i}))/6$, covers the entire range of the parametric variable in six standard deviations. We verify whether the noisy parameters meet the minimum requirements of reaching the goal for the sub-task. If the requirements are not met, we generate a new set of noisy parameters until the requirements are met.

The learning curves for the automated curriculum transfer generated through noisy parameters are shown in Fig 6. We compare its performance with learning curves for automated curriculum transfer generated through exact mappings, and with other baseline approaches present in the literature. Even with noisy mappings, the automated curriculum transfer outperforms other curriculum approaches, and performs comparable to the automated curriculum transfer with exact mappings. On the complicated task (HF with Fire), the automated curriculum transfer with noisy mappings takes longer to converge on the source tasks of the curriculum, yet it achieves a significant performance advantage over other baselines.

Since our approach relies on the low-fidelity environment for curriculum generation, its sunk cost in computational runtime is significantly lesser than baseline approaches, in which curriculum generation requires extensive interactions in the costly high-fidelity environment. Fig 7 compares the computational runtime (in CPU Hours) required to run one trial (with each episode having a maximum of $6\times 10^{2}$ timesteps) of the high-fidelity task until the stopping criterion is met. The experiments were conducted using a 64-bit Linux Machine, having Intel(R) Core(TM) i9-9940X CPU @ 3.30GHz processor and 126GB RAM memory. The sunk cost of our automated curriculum transfer approach involves the interactions required to optimize the curriculum in the LF environment, and the interactions required to learn the source tasks in the HF environment.