Reinforcement Learning for Robust Missile Autopilot Design

The GSAM’s flight dynamics to be controlled is modelled by a nonlinear system, in which the total sum of the forces $F_{T,G}$ and of the momenta $M_{T,G}$ are dependent on the flight dynamic states (cf. equations (1) and (2)), like the Mach number $M$, the height $h$ or the angle of attack $\alpha$.

This system is decomposed into Translation (cf. equation (3)), Rotation (cf. equation (4)), Position (cf. equation (5)) and Attitude (cf. equation (6)) terms, as Peter (2018) Peter2018 described. Equations (3) to (6) follow Peter’s (2018) Peter2018 notation.

To investigate the principle ability of an RL agent to serve as a missile autopilot, the nonlinear longitudinal motion of the missile dynamics is considered. Therefore, the issue of cross-coupling effects within the autopilot design is not addressed in this paper.

The GSAM is actuated by four fin deflections, $\delta_{i}$, which are mapped to three aerodynamic equivalent controls, $\xi$, $\eta$ and $\zeta$ (cf. equation (7)). The latter provide the advantage of directly matching the GSAM’s roll, pitch and yaw axis, respectively.

In the control of the longitudinal flight dynamics, $\xi$ and $\zeta$ are set to 0 and, hence, equation (8) becomes equation (9).

The actuator system is, thus, the system that receives the desired (commanded) $\eta_{com}$ and outputs $\eta$, modelling the dynamic response of the physical fins with its deflection limit of 30 degrees. The latter is assumed to be a second order system with the following closed loop characteristics:

1. 1.

   Natural frequency $\omega_{n}$ of 150 rad.s^-1

2. 2.

   Damping factor $\lambda$ of 0.7

The algorithm must achieve the target performance established in terms of the following requirements:

1. 1.

   Static error margin = 0.5%

2. 2.

   Overshoot $<$ 20%

3. 3.

   Rise time $<$ 0.6s

4. 4.

   Settling time (5%) $<$ 0.6s

5. 5.

   Bounded actuation

6. 6.

   Smooth actuation

Besides, the present work also aims at achieving and improving the robustness of the algorithm to conditions different from the training ones.

When trying to optimize the achieved performance, one must, however, be aware of the physical limitations imposed by the system being controlled. A commercial airplane, for example, will never have the agility of a fighter jet, regardless of how optimized its controller is. Therefore, it is hopeless to expect the system to follow too abrupt reference signals like step functions. Instead, the system will be asked to follow shaped reference signals, i.e., the output of the Reference Model. The latter is hereby defined as the system whose closed loop dynamics is designed to mimic the one desired for the dynamic system being controlled. In this case: natural frequency $\omega_{n}$ and damping factor $\lambda$ of 10 rad.s^-1 and 0.7, respectively.

The proper workflow of the interaction between the agent and the dynamic system being controlled - including a Reference Model - is illustrated in figure 2. The command reference signal is given as an input to the Reference Model, whose output, the shaped reference signal is the input of the agent.

RL has been the object of research after the foundations laid out by Sutton et al. (2018) Sutton2018 , with applications going from the Atari 2600 games, to the MuJoCo games, or to robotic tasks (like grasping…) or other classic control problems (like the inverted pendulum).

TRPO (Schulman et al. 2015 Schulman2015 ) has become one of the commonly accepted benchmarks for its success achieved by the cautious trust region optimization and monotonic reward increase. Perpendicularly, Lillicrap et al. (2016) Lillicrap2016 proposed DDPG, a model-free off-policy algorithm, revolutionary not only for its ability of learning directly from pixels while maintaining the network architecture simple, but mainly because it was designed to cope with continuous domain action. Contrarily to TRPO, DDPG’s off-policy nature implied a much higher sample efficiency, resulting in a faster training proccess. Both TRPO and DDPG have been the roots for much of the research work that followed.

On the one side, some authors valued more the benefits of an off-policy algorithm and took the inspiration in DDPG to develop TD3 (Fujimoto et al. 2018 Fujimoto2018 ), addressing DDPG’s problem of over-estimation of the states’ value. On the other side, others preferred the benefits of on-policy algorithm and proposed interesting improvements to the original TRPO, either by reducing its implementation complexity (Wu et al. 2017 Wu2017 ), by trying to decrease the variance of its estimates (Schulman 2016 Schulman2016a ) or even by showing the benefits of its interconnection with replay buffers (Kangin et al. Kangin2019 ).

Apart from these, a new result began to arise: agents were ensuring stability at the expense of converging to suboptimal solutions. Once again, new algorithms were conceived in each family, on- and off-policy. Haarnoja and Tang proposed to express the optimal policy via a Boltzmann distribution in order to learn stochastic behaviors and to improve the exploration phase within the scope of an off-policy actor-critic architecture: Soft Q-learning (Haarnoja et al. 2017 Haarnoja2017 ). Almost simultaneously, Schulman et al. published PPO (Schulman et al. 2017 Schulman2017 ), claiming to have simplified TRPO’s implementation and increased its sample-efficiency by inserting an entropy bonus to increase exploration performance and avoid premature suboptimal convergence. Furthermore, Haarnoja et al. developed SAC (Haarnoja et al. 2018 Haarnoja2018 ), in an attempt to recover the training stability without losing the entropy-encouraged exploration and Nachum et al. (2018) Nachum2018a proposed Smoothie, allying the trust region implementation of PPO with DDPG.

Finally, there has also been research done on merging both on-policy and off-policy algorithms, trying to profit from the upsides of both, like IPG (Gu et al. 2017 Gu2017 ), TPCL (Nachum et al. 2018 Nachum2018 ), Q-Prop (Guet al. 2019 Gu2019 ) and PGQL (O’Donoghue et al. 2019 ODonoghue2019 ).

TRPO is an on-policy model-free RL algorithm that aims at maximizing the discounted sum of future rewards (cf. equation (10)) following an actor-critic architecture and a trust region search.

Initially, Schulman et al. (2015) Schulman2015 proposed to update the policy estimator’s parameters with the conjugate gradient algorithm followed by a line search. The trust region search would be ensured by a hard constraint on the Kullback-Leibler divergence $D_{KL}$.

Briefly after, Kangin et al. (2019) Kangin2019 proposed an enhancement, augmenting the training data by using replay buffers and GAE (Schulman et al. 2016 Schulman2016a ). Also contrarily to the original proposal, Kangin et al. trained the value estimator’s parameters with the ADAM optimizer and the policy’s with K-FAC (Martens et al. 2015 Martens2015 ). The former was implemented within a regression between the output of the Value NN, $\hat{V}$, and its target, $V^{\prime}$, whilst the latter used equation (11) as the loss function, which has got a first term concerning the objective function being maximized (cf. equation (10)) and a second one penalizing differences between two consecutive policies outside the trust region, whose radius is the hyperparameter $\delta_{TR}$.

For matters of simplicity of implementation, Schulman et al. (2015) Schulman2015 rewrote equation (11) in terms of collected experience (cf. equation (12)), as a function of two different stochastic policies, $\pi_{\theta_{new}}$ and $\pi_{\theta_{old}}$, and the $GAE$.

Both versions of TRPO use the same exploration strategy: the output of the Policy NN is used as the mean of a multivariate normal distribution whose covariance matrix is part of the policy trainable parameters.

The key idea of HER (Andrychowicz et al. 2018 Andrychowicz2018 ) is to store in the replay buffers not only the experience collected from interacting with the environment, but also about experiences that would be obtained, had the agent followed a different goal. HER was proposed as a complement of sparse rewards.

When working with replay buffers, randomly sampling experience can be outperformed by a more sophisticated heuristic. Schaul et al. (2016) Schaul2016a proposed Prioritized Experience Replay, sampling experience according to their significance, measured by the TD-error. Besides, the strict required performance (cf. table 1) causes the agent to seldom achieve success and the training dataset to be imbalanced. Thus, the agent simply overfits the ”failure” class. Narasimhan et al. (2015) Narasimhan2015 have addressed this problem by forcing 25% of the training dataset to be sampled from the less represented class.

As explained in section 1, the current problem required an on-policy model-free RL algorithm. Among them, not only is TRPO a current state-of-the-art algorithm (cf. section 3.1), but it also presents the attractiveness of the trust region search, avoiding sudden drops during the training progress, which is a very interesting feature to be explored by the industry, whose mindset is often aiming at robust results. TRPO was, therefore, the most suitable choice.

As further detailed in Cortez2020 , the established methodology (i) progressively increases the amplitude of the randomly generated command signal and (ii) intermediate testing of the agent’s performance against a -10g/10g double step without exploration, in order , respectively, (i) to avoid overfitting and (ii) to decide whether or not to finish the training process.

The formal mathematical guarantee of robustness of a RL agent composed of neural networks cannot be done in the same terms as the one of linear controllers. It was, hence, evaluated by deploying the nominal agent in non-nominal environments. Apart from testing its performance, the hypothesis is also that training this agent in the latter can improve its robustness. To do so, the training of the best found nominal agent was resumed in the presence of non-nominalities (cf. section 4.3.1). This training and its resulting best found agent are henceforward called robustifying training and robustified agent, respectively.

Empirically, it has been seen that agents that struggle to control $\eta$’s magnitude within its bounds (cf. table 1) are unable of achieving low error levels without increasing the level of noise, if they can do it at all. In other words, the only way of having good tracking results with unbounded actions is to fall into a bang-bang control-like situation. Therefore, the best found agent will have to start by learning to use only bounded and smooth action values, which will allow it to, then, start decreasing the tracking error.

Notice that, while the tracking error and the noise measures are expected to tend to 0, $\eta$’s magnitude is not, since it would mean that the agent had given up actuating in the environment. Hence, it is expected that, at some point in training, the latter stabilizes.

Moreover, the exploration strategy proposed in section 4.1.1 insert some variance in the output of the policy neural network, meaning that the noise measures are also not expected to reach exactly 0.

As figure 3 evidences, the agent is clearly able of controlling the measured acceleration and to track the reference signal it is fed with, satisfying all the performance requirements defined in table 1 (cf. table table 2).

Figure 4 shows the tests of the best agents obtained during each of the nine trainings run to assess the reproducibility of the best found nominal agent (cf. section 5.2) and it is possible to see that none of them achieved the target performance, i.e., none meets all the performance criteria established in table 1. Although most of the trials can be considered far from the initial random policy, it is possible to verify that trials 7 and 8 present a very poor tracking performance and that trial 2’s action signal is not smooth.

As figure 7 shows, from the four different values of $3\sigma$ tried, only the 3% one converged after 2500 episodes, meaning that it was the only robustifying training being run for a total of 5000 episodes, during which the best agent found was defined as the Estimation Uncertainty Robustified Agent. The nominal performance is improved, having a lower overshoot and a smoother action signal.

Furthermore, it enhanced the performance of the Nominal Agent in environments with estimation uncertainty (-10% to 10%), having achieved high success rates (cf. table 4).

Since the Estimation Uncertainty Robustified Agent was better than the Nominal Agent in both nominal and non-nominal environments, it won in both Performance and Robustness categories. Thus, it is possible to say, in general terms, that, concerning estimation uncertainty, the robustifying trainings succeeded.

As figure 9 shows, from the four different values of $3\sigma$ tried, only the 5% one converged after 2500 episodes, meaning that it was the only robustifying training being run for a total of 5000 episodes, during which the best agent found was defined as the Parametric Uncertainty Robustified Agent. The nominal tracking performance is damaged, but the action signal is smoother (cf. figure 10).

Furthermore, it enhanced the performance of the Nominal Agent in environments with parametric uncertainty (-40% to 40%), having achieved high success rates (cf. table 5).

Since the Parametric Uncertainty Robustified Agent was better than the Nominal Agent in non-nominal environments, it won in the Robustness categories, remaining acceptably the same in terms of performance in nominal environments. Thus, it is possible to say, in general terms, that, concerning parametric uncertainty, the robustifying trainings succeeded.