Combining Reinforcement Learning and Inverse Reinforcement Learning for Asset Allocation Recommendations

Our approach combines approaches developed separately in the research community focused on RL and IRL problems. For a general review of RL and IRL along with their applications to finance, see (Dixon et al., 2020).

In its reliance on a combination of IRL and RL for optimization of a financial portfolio, this paper follows the approach of (Dixon & Halperin, 2021). The latter uses a continuous state-action version of G-learning, a probabilistic extension of Q-learning (Fox et al., 2016), for the RL step, and an IRL version of G-learning called GIRL for the IRL step. In this work, we retain the G-learner algorithm from (Dixon & Halperin, 2021) for the RL step, but replace the IRL step with a parametric version of a recent algorithm called T-REX (Brown et al., 2019) which will be presented in more details in Sect. 2.1.

IRL deals with recovering reward functions from the observed agents’ behavior as a way of rationalizing their intent. The learned reward function can then be used for evaluation and optimization of action policies. The IRL approach reduces the labor of defining proper reward functions in RL for applications where their choice is not immediately obvious.

Many traditional IRL methods try to learn the reward function from demonstrations under the assumption that the demonstrated history corresponds to an optimal or near-optimal policy employed by an agent that produced the data, see e.g. (Pomerleau, 1991; Ross et al., 2011; Torabi et al., 2018). These IRL methods therefore simply try to mimic or justify the observed behavior by finding a proper reward function that explains this behavior. This implies that by following their approach, the best one can hope for IRL for investment decisions is that it would be able to imitate investment policies of asset managers.

However, the assumption that the demonstrated behavior is already optimal might not be realistic, or even verifiable, for applications in quantitative finance. This is because, unlike robotics or video games, the real market environment cannot be reproduced on demand. Furthermore, relying on this assumption for inferring rewards may also have a side effect of transferring various psychological biases of asset managers into an inferred ‘optimal’ policy.

Instead of simply trying to mimic the observed behavior of asset managers, it would be highly desirable to try to improve over their investment decisions, and potentially correct for some biases implicit in their decision-making. A principled way towards implementing such a program would be to infer the intent of asset managers from observing their trading decisions without the assumption they their strategies were fully or partially successful. With such an approach, demonstrated trajectories would be scored by how well they succeed in achieving a certain goal. This is clearly very different from seeking a reward function that would simply rationalize the observed behavior of asset managers.

Such inference of agents’ intent is made possible using a recently proposed IRL algorithm called T-REX, for Trajectory-based Reward Extrapolation (Brown et al., 2019). The T-REX method finds the reward function that captures the intent of the agent without assuming a near-optimality of the demonstrated behavior. Once the reward function that captures agent’s intent is found, it can be used to further improve the performance by optimizing this reward. This could be viewed as extrapolation of the reward beyond the demonstrated behavior, thus explaining the ‘EX’ part in the name T-REX.

To overcome the issue of potential sub-optimality of demonstrated behavior, T-REX relies on certain extra information that is not normally used in other IRL methods. More specifically, T-REX uses rankings of all demonstrated trajectories according to a certain final score. The final score is assigned to the whole trajectory rather than to a single time-step. The local reward function is then learned from the condition that it should promote the most desirable trajectories according to the score. In this work, we score different portfolio trajectories by their total realized returns, however, we could also use risk-adjusted metrics such as the Sharpe or Sortino ratios.

Let $\bm{O}:\{\bm{S},\bm{A}\}$ be a state-action space of an MDP environment, and $\hat{r}_{\bm{\theta}}(\cdot)$ with parameters $\bm{\theta}$ be a target reward function to be optimized in the IRL problem. Given $M$ ranked observed sequences $\{\bm{o_{m}}\}_{m=1}^{M}$ ($\bm{o_{i}}\prec\bm{o_{j}}$ if $i<j$, where “$\prec$” indicates the preferences, expressed e.g. via a ranking order, between pairwise sequences), T-REX conducts reward inference by solving the following optimization problem:

This objective function is equivalent to the softmax normalized cross-entropy loss for a binary classifier, and can be easily trained using common machine learning libraries such as PyTorch or TensorFlow. As a result, the learned optimal reward function can preserve the ranking order between pairs of sequences.

The original T-REX algorithm is essentially a non-parametric model that encodes the reward function into a deep neural network (DNN). This might be a reasonable approach for robotics or video games where there is a plenty of data, while the reward function might be highly non-linear and complex. Such setting is however ill-suited for applications to portfolio management where the amount of available data is typically quite small. Therefore, in this work we pursue a parametric version of T-REX, where the reward function is encoded into a function with a small number of parameters, which is then optimized using Eq.(1). A particular model of the reward function that will be presented in the next section has only 4 tunable parameters, which appears to be about the right order of model complexity given available data. The added benefit in comparison to the original DNN-based implementation of T-REX is that a parametric T-REX is much faster to train.

G-learner is an algorithm for the direct RL problem of portfolio optimizaiton that was proposed in (Dixon & Halperin, 2021). It relies on a continuous state-action version of G-learning, a probabilistic extension of Q-learning designed for noisy environments (Fox et al., 2016). G-learner solves a finite-horizon direct RL problem of portfolio optimization in a high-dimensional continuous state-action space using a sample-based approach that operates with available historical data of portfolio trading. The algorithm uses a closed-form state transition model to capture the stochastic dynamics in a noisy environment, and therefore belongs in the class of model-based RL approaches. In this work, we employ G-learner as a RL solver for the sequential portfolio optimization, while defining a new reward function to evaluate portfolio managers’ performance.

Unlike (Dixon & Halperin, 2021) that considered G-learning for a portfolio of individual stocks held by a retail investor, here we consider portfolios typically held by professional asset managers. Due to data limitations, it is unfeasible to estimate models that track the whole investment universe counting thousands of stocks, and maintain individual stocks holdings and their changes as, respectively, state and action variables. A viable alternative is is to aggregate individual stock holdings into buckets constructed according to a particular dimension reduction principle, and define the state and action variables directly in such dimensionally reduced space. In this paper, we choose to aggregate all stocks in the PM’s portfolio into $N=11$ sectors of the standard GICS industry classification, effectively mapping the portfolio into portfolio of sector exposures. The state vector $\bm{x_{t}}\in R^{N}$ is respectively defined as a vector of dollar values of stock positions in each sector at time $t$. The action variable $\bm{u_{t}}\in R^{N}$ is given by the vector of changes in these positions as a result of trading at time step $t$. Furthermore, let $\bm{r_{t}}\in R^{N}$ represent the asset returns as a random variable with the mean $\bm{\bar{r}_{t}}$ and covariance matrix $\bm{\Sigma_{r}}$. As in our approach we identify assets with sector exposures, $\bm{\bar{r}_{t}}$ and $\bm{\Sigma_{r}}$ will be interpreted as expected sector returns and sector return covariance, respectively.

The state transition model is defined as follows:

In this work we use a simplified form of the reward from (Dixon & Halperin, 2021), which we write as follows:

where $C_{t}$ is a money flow to the fund at time step $t$ minus the change of a cash position of the fund at the same time step, $\lambda$ and $\omega$ are parameters, and $V_{t}$ and $\hat{P}_{t}$ are, respectively, the values of asset manager’s and reference portfolios, defined as follows:

where $\eta$ and $\rho$ are additional parameters, and $B_{t}$ is a benchmark portfolio, such as e.g. the SPX index portfolio, properly re-scaled to match the size of the PM’s portfolio at the start of the investment period.

The reward function (3) consists of three terms, each encoding one of the portfolio managers’ trading insights. In the first term, $\hat{P}_{t}$ defines the target portfolio market value at time $t$. It is specified as a linear combination of a reference benchmark portfolio value $B_{t}$ and the current portfolio growing with rate $\eta$ according to Eq.(4), where $\rho\in[0,1]$ is a parameter defining the relative weight between the two terms. $V_{t}$ gives the portfolio value at time $t+\Delta t$, after the trade $\bm{u_{t}}$ is made at time $t$. The first term in (3) imposes a penalty for under-performance of the traded portfolio relative to its moving target. The second term enforces the constraint that the total amount of trades in the portfolio should match the inflow $C_{t}$ to the portfolio at each time step, with $\lambda$ being a parameter penalizing violations of the equality constraint. The third term approximates transaction costs by a quadratic function with parameter $\omega$, thus serving as a $L_{2}$ regularization. The vector $\bm{\theta}$ of model parameters thus contains four reward parameters $\{\rho,\eta,\lambda,\omega\}$.

Importantly, the reward function (3) is a quadratic function of the state and action. Such a choice of the reward function implies that the value- and action-value functions are similarly quadratic functions of the state and action, with time-dependent coefficients. Respectively, with a quadratic reward functions such as (3), a G-learning algorithm should not engage neural networks or other function approximations, which would invariably bring their own parameters to be optimized. In contrast, with the quadratic reward (3), learning the optimal value and action-value functions amounts to learning the coefficients of quadratic functions, which amounts to a small number of linear algebra operations (Dixon et al., 2020). As a result, an adaptation of the general G-learning method for the quadratic reward (3) produces a very fast policy optimization algorithm called G-learner in (Dixon & Halperin, 2021).

G-learner is applied to solve the proposed RL problem with our defined MDP model in Eq.(2) and reward function in Eq.(3). Algorithm 1 demonstrates the entire optimization solving process. It takes the model parameters as inputs, along with discount factor $\gamma$. In addition, it uses a prior policy $\bm{\pi}^{(0)}$ which usually encodes domain knowledge of real world problems. G-learner (and G-learning in general) control the deviation of the optimal policy $\bm{\pi_{t}}$ from the prior $\bm{\pi}^{(0)}$ by incorporating the KL divergence of $\bm{\pi_{t}}$ and $\bm{\pi}^{(0)}$ into a modified, regularized reward function, with a hyperparameter $\beta$ that controls the magnitude of the KL regularizer. When $\beta$ is large, the deviation can be arbitrarily large, while in the limit $\beta\rightarrow 0$, $\bm{\pi_{t}}$ is forced to be equal to $\bm{\pi}^{(0)}$, so there is no learning in this limit. Furthermore, $\bm{F_{t}}(\bm{x_{t}})$ is the value function, and $\bm{G_{t}}(\bm{x_{t}},\bm{u_{t}})$ is the action-value function. The algorithm is initialized by computing the optimal terminal value and action-value functions $\bm{F_{T}}^{*}$ and $\bm{G_{T}}^{*}$. They are computed by finding the optimal last-step action $\bm{a}_{T}$ that achieves the highest one-step terminal reward, i.e. solves the equation $\frac{\partial{R_{t}(\bm{x_{t}},\bm{u_{t}})}}{\partial{\bm{u_{t}}}}|_{t=T}=0$. Thereafter, the policy associated with earlier time steps can be derived recursively backward in time as shown in the for loop of Algorithm 1. For a detailed derivation of $Value\_Update$ and $ActionValue\_Update$ steps, we refer to Section 4 in (Dixon & Halperin, 2021).

Once the optimal policy $\bm{\pi_{t}}^{*}=\bm{\pi_{t}}^{*}(\bm{u_{t}}|\bm{x_{t}})$ is learned, the recommended action at time $t$ is given by the mode of the action policy for the given state $\bm{x_{t}}$.

Figure 1 illustrates the overall workflow of our proposed IRL-RL framework. We apply this framework to provide portfolio allocation recommendations to improve fund returns. It takes the observed $M$ state-action sequences ($\bm{o_{m}}:\{\bm{x_{t}},\bm{u_{t}}\}_{t=0}^{T}$) from multiple funds as input training data. As cumulative funds’ returns can be used to measure the overall PM’s performance over a given observation period, we use the realized total returns in the train dataset as a ranking criterion for the IRL step with the parametric T-REX. As an alternative, we could rely on risk-adjusted metrics such as e.g. the Sharpe or Sortino ratios.

With the chosen ranking criterion, the T-REX IRL module finds the optimal parameters of the reward function defined in Eq.(3). The optimal reward function parameters are then passed to the RL module which optimizes the action policy. Both modules operate in the offline learning mode, and do not require an online integration with the trading environment.

The IRL module can also work individually to provide behavioral insights via analysis of inferred model parameters. It is designed to be generalizable to different reward functions, if needed. Ranking criteria need to be rationally determined to align with the logic of rewards and depict the intrinsic behavioral goal of demonstrations. The human-provided ranking criterion based on the performance on different portfolio paths thus informs a pure AI system (our G-learner agent). As a result, our IRL/RL pipeline implements a simple human-machine interaction loop in the context of portfolio management and asset allocation.

We note that by using T-REX with the return-based scoring, our model already has a great starting point to hope for success. Indeed, as our reward function and ranking are aligned with PMs’ own assessment of their success, this already defines a ‘floor’ for the model performance. This is because our approach is guaranteed to at least match individual funds’ performance by taking the exact replica of their actions. Therefore, our method cannot do worse, at least in-sample, than PMs but it can do better - which is indeed the case as will be shown below.

Another comment due here is that while in this work we use the aggregation to the sectors as a particular scheme for dimension reduction, our combined IRL-RL framework could also be used with other ways of dimension reduction, for example we could aggregate all stocks by their exposure to a set of factors.

In our experiments, two sets of mutual funds are chosen based on their benchmark indexes. To anonymize our data, we replace actual funds’ names by single-letter names. The first set includes six funds ($\{S_{i}\}_{i=1}^{6}$) that all have the S&P 500 index as a benchmark. The second set includes twelve funds that have the the Russell 3000 index as a benchmark. The second group is further divided into growth and value fund groups ($\{RG_{i}\}_{i=1}^{7}$ and $\{RV_{i}\}_{i=1}^{5}$). Each fund’s trading trajectory covers the period from January 2017 to December 2019, and contains its monthly holdings and trades for eleven sectors (i.e., $\bm{x_{t}},\bm{u_{t}}$) as well as monthly cashflows $C_{t}$. The first two years of data are used for training (so that we choose T=24 months), and the last year starting from January 2019 is used for testing. At the starting time step we assign each fund’s total net asset value to its corresponding benchmark value (i.e., $B_{t=0}$) in order to align their size and use the actual benchmark return at each time step to calculate their values afterwards (i.e., $t>0$). The resulting time series $\{\bm{x_{t}},\bm{u_{t}},B_{t},C_{t}\}_{t=0}^{T}$ are further normalized by dividing by their initial values at $t=0$. In the training phase, the canonical ARMA model (Hannan, 2009) is applied to forecast expected sector returns $\bm{r_{t}}$, and the regression residue is then used to estimate the sector return covariance matrix $\bm{\Sigma_{r}}$. The prior policy $\bm{\pi}^{(0)}$ is fitted to a multivariate Gaussian distribution with a constant mean and variance calculated from sector trades in the training set.

The pre-processed fund trading data is ranked based on funds’ overall performance realized over the training period. T-REX is then trained to learn the four parameters $\{\rho,\eta,\lambda,\omega\}$ that enter Eq.(3). The training/test experiments are executed separately for all three fund sets (i.e. funds with the S&P 500 benchmark, Russell 3000 Growth funds, and Russell 3000 Value funds).

Recall that the loss function for T-REX is given by the cross-entropy loss for a binary classifier, see Eq.(1). Figure 2 shows the classification accuracy in the training ($91.1\%$) and test phases ($83.2\%$) for the S&P 500 fund set. The plots in the two sub-figures show that the learned reward function preserves the fund ranking order. Correlation scores are used to measure their level of association, producing the values of $0.988$ and $0.954$ for the training and test sets, respectively. The excellent out-of-sample correlation scores gives a strong support to our chosen method of the reward function design.

Fund managers’ investment strategies can be analyzed using the inferred reward parameters. Figure 3 shows the parameter convergence curve using the S&P 500 fund group. The values converge after about thirty iterations, which suggests adequacy of our chosen model of the reward to the actual trading data. The optimized benchmark weight $\rho^{\ast}=0.951$ captures the correlation between funds’ performance and their benchmark index. Its high value implies that portfolio managers’ trading strategy is to target an accurate benchmark tracking. The value $\eta^{\ast}=1.247$ implies funds are in the growing trend with prospective rate at $24.7\%$ from fund managers’ view. Since the sum of trades are close enough to the fund cashflows in our dataset, the introduced penalty parameter $\lambda$ converges to a very small value at $0.081$. The estimated average trades’ volatility $\omega^{\ast}$ is around $10\%$, which measures the level of consistency across trades in the S&P 500 fund group.

The experimental results of our proposed parametric T-REX are summarized in Table 1. Our model achieved high classification accuracy ($\mathit{acc}$) and correlation scores ($\mathit{cor}$) on all three fund groups in both training and test phases. The optimal values of reward function parameters are listed in the table as well. Small values of $\lambda^{\ast}$ verify the holding of equality constraint over the sum of trades in our dataset. Comparing across different fund groups, we can identify different trading strategies of portfolio managers. Funds in the S&P500 group are managed to track their benchmark very closely with correlation $\rho^{\ast}=0.951$. On the other hand, correlation $\rho^{\ast}$ reduces significantly to 0.584 and then to 0.186 for fund groups $\{RG_{i}\}_{i=1}^{7}$ and $\{RV_{i}\}_{i=1}^{5}$ which have the same benchmark index Russell 3000, while being segmented based on their investment approaches (i.e., growth vs. value). For the bucket denoted $\{S_{i}\}_{i=1}^{6}$, growth and blend funds are combined. As shown in the table, funds in the segmented groups show very limited variations of their trading amounts compared to those combined in a single group. This can be seen as an evidence of fund managers’ consistency in their investment decisions/policies. Growth funds in $\{RG\}_{i=1}^{7}$ with $\eta^{\ast}=1.532$ produce the highest expected growth rates as implied by their trading activities. This result is in line with the fact that growth investors select companies that offer strong earnings growth while value investors choose stocks that appear to be undervalued in the marketplace.

The four reward parameters can be used to group different fund managers or different trading strategies into distinct clusters in the space of reward parameters. While the present analysis in this paper suggests pronounced patterns based on the comparison across three fund groups, including more fund trading data from many portfolio managers might be able to bring further insights into fund managers’ behavior. This is left here for a future work.

Once the reward parameters are learned for all groups of funds, they are passed to G-learner for the policy optimization step. The pre-processed fund trading data in year 2017 and 2018 is used as inputs for policy optimization (see Figure 1). Trading trajectories were truncated to twelve months’ long in order to align with the length of the test period (year 2019). The optimal policy $\bm{\pi_{t}}^{*}=\bm{\pi_{t}}^{*}(\bm{u_{t}}|\bm{x_{t}})$ is learned using the train dataset, and then applied to the test set throughout the entire test period. Differently from the training phase, where we use expected sector returns in the reward function, in the test phase we use realized monthly returns to derive the next month’s holdings $\bm{x_{t+1}}$ after trades $\bm{u_{t}}$ was made for the current month with $\bm{x_{t}}$. At the start time of the test period, the holdings $\bm{x}_{t=0}$ coincide with actual historical holdings. We use this procedure along with the learned policy $\bm{\pi_{t}}^{*}(\bm{u_{t}}|\bm{x_{t}})$ onward from February 2019 to create counterfactual portfolio trajectories for the test period. We will refer to these RL-recommended trajectories as the AI Alter Ego’s (AE) trajectories.

Figures 4, 5, 6 show the outperformance of our RL policy from PM’s trading strategies (i.e, $AE-PM$) for all three fund groups for both the training and test sets for forward period of 12 months, plotted as functions of their time arguments. We note that both curves for the training and test sets follow similar paths for at least seven months before they started to diverge (the curves for Russell 3000 value group diverged significantly after the eighth months’ trades and thus were truncated for a better visualization effect). In practice, we recommend to update the policy through a re-training once the training and test results start to diverge. In general, our results suggest that our G-learner is able to generalize (i.e. perform well out-of-sample) up to 6 months into the future.

Analysis at the individual fund level for all three groups is presented in Figures 7, 9, 11 for the training set, and Figures 8, 10, 12 for the test set. Our model outperformed most of PMs throughout the test period except funds $RG_{5}$ and $RV_{5}$. It is interesting to note that the test results outperformed the training ones for the S&P 500 fund group. This may be due to a potential market regime drift (which is favorable to us, in the present case). More generally, a detailed study would be desired to address the impact of potential market drift or a regime change on optimal asset allocation policies. Such topics are left here for a future research.