Can Interpretable Reinforcement Learning Manage Prosperity Your Way?

Financial service providers are employing ever-advancing methods to improve the level of personalisation of their services [1, 2]. Artificial intelligence (AI) is a promising tool in this pursuit in areas such as anti-money laundering, trading and investment, and customer relationship management [3]. Examples of personalised services are recommender systems for product sales [4], risk evaluation for credit scoring [5], and segmentation for customer-centric marketing [6]. More commonly, AI has been applied to stock trading via ensemble learning [7], currency recognition using deep learning [8], stock index performance through time-series modelling with feature engineering [9], and investment portfolio management using reinforcement learning (RL) [10, 11]. These applications generally lack the personalisation needed to enhance customer relations and support service delivery for growing customer bases. The lack of explainability and interpretability has thus far hindered the wider adoption of machine learning, mainly due to model opacity; model understanding is essential in financial services [12, 13, 14]. We distinguish between explainability and interpretability: explainability refers to a symbolic representation of the knowledge a model has learned, while interpretability is necessary for reasoning about a model’s predictions.

We have previously investigated the interpretability of systems of multiple RL agents [15]. A regularisation term in the objective function imposed a desired agent behaviour during training. For our current purpose of asset management, our agents learn distinct optimal policies for continuously distributing a fixed monthly amount across five assets: a savings account, property, stocks, luxury items, and mortgage repayments. It is our intention to align the agents’ characteristics and behaviour with personality traits - openness, conscientiousness, extraversion, agreeableness, and neuroticism - as proposed for modelling spending behaviour in [16]. A linear combination of the resulting policies can provide investment advice that matches each customer’s unique personality profile. This intrinsic interpretability may fulfill the promise of a digital private assistant for personal wealth management.

Recent evidence has revealed a causal relationship between spending patterns and individual happiness [17]: we are happiest when our spending matches our personality. For instance, extraverted individuals typically prefer spending at a bar rather than at a bookshop, while the opposite may apply to introverts. Our premise is that spending personality traits can be carried over to asset management: we are happiest when our investment matches our personality. For instance, conscientious investors may prefer the predictability of property over the volatility of stocks. This is consistent with the high affinity of conscientious spenders towards residential mortgages [17]. It is compelling to expand the notion of personality traits from spending to wealth creation, i.e., to base personal investment advice on historical spending behaviour [18, 19].

RL has been extensively applied to stock portfolio management [20, 21, 22, 23, 24, 25], but not yet to holistic asset management; the lack of model transparency may be a contributing factor. Interpretation of RL agents typically follows model training [26, 27, 28]; our ambition is to impose a desired characteristic behaviour during training, thus making it an intrinsic property of the agent. Based on a prior that defines a desired behaviour, we extend the deep deterministic policy gradient (DDPG [29]) objective function with a regularisation term [15]. Formally, for each agent $i$, this objective function is given by:

where $\theta_{i}$ is a set of parameters governing the policy, $\mathcal{D}$ is the replay buffer, $R_{i}(o_{i},a_{i})$ is the reward for action $a_{i}$ with the partial sate observation $o_{i}$, $\lambda\in\mathbb{R}_{\geq 0}$ is a scaling parameter, $M_{i}$ is the number of actions, and $\pi_{0_{i}}$ is the prior that defines the desired behaviour of the agent. Note that the prior is independent of the state, which simplifies it and thus makes it interpretable; this is a departure from traditional policy regularisation methods such as KL-regularisation and entropy regularisation which aim to improve learning convergence instead [30, 31]. Traditional regularisation encourages state space exploration by increasing the entropy of the policy, whereas our method guides agents’ learning towards the prior and thus imposes a desired characteristic behaviour.

The aim of this work was to create an interpretable AI for personal investment management. We selected five assets in which a customer could invest a monthly amount over a duration of 30 years: a savings account, property, a portfolio of stocks, luxury expenditures, and additional mortgage payments. We include luxury expenditure to the portfolio under the premise that it may increase customer satisfaction in their portfolios [17]. We define luxury items as any expenditure that may appeal to a person’s personality profile; people scoring high on openness might derive joy from spending money on travelling, people scoring high on extraversion may prefer to spend money on festivities with other people [17], while other luxury items such as cars or artwork are also possible. We modelled the growth rates of assets according to historical index data, which we describe below.

Each of our investment agents learns an optimal investment strategy for their respective prototypical personality traits, for instance, openness. The final portfolio values after 334 months of investing according to these policies are shown in Table 3. Given the common total investment of 3.34 million NOK, the compound annual growth rate varies between 5.8% and 7.8% which is the maximum return possible if investing in stocks only.

Note that these personalised policies did not achieve the same final portfolio value. In fact, the optimum policy in monetary terms $\pi^{M}$ in this case would have been to always buy stocks as shown in Figure 2; this is the default policy an agent will converge towards when personality traits are ignored. However, we postulate that this is not the ideal personal financial advice to give to all individuals; some customers may be more averse to risk and will thus prefer to avoid volatility in their portfolio. Our personalized agent takes into account such preferences and, e.g., it recommends property investments rather than stock investments.

Thus far, our agents have each separately learned an optimal investment strategy for each prototypical personality trait. The aggregate policy is the weighted sum of these individually learned policies: a customer has a blend of personality traits which can be represented as a vector with five entries with values within the range $[-1,+1]$. We calculate the inner product of the normalized personality vector and the prototypical policies to arrive at the aggregate investment policy. We show a representative aggregate investment policy for a customer with a random personality profile in Figure 3. We observe that the openness agent is the only agent to recommend spending on luxury items; this is to be expected because its regularisation prior $\pi_{0}^{O}$ is the only one with a non-zero coefficient for luxury purchases. We also observe that the conscientiousness agent recommends investing in property in early stages, followed by rigorous loan repayments in the second half of the investment period. This suggests that our agent has learned the concept of compound growth and its utility for portfolio optimization. By contrast, the extraversion agent was steadfast in purchasing stocks only, which is consistent with its regularisation prior $\pi_{0}^{E}$. Unlike the conscientiousness agent, the agreeableness and neuroticism agents consistently recommend investing in savings towards the end of the investment period. In the early stages of the investment period, the agreeableness and neuroticism agents utilize compound growth to increase the portfolio value; in the latter phases, their regimen changes and they prefer the safety of savings accounts. This is noteworthy because although risk is not explicitly part of either the reward or regularisation functions, it is consistent with traditional financial advice, which decreases the risk level with age. Repeated training produces consistent results. We intend to elucidate this observation in future work.

We observe that training converges quickly to the desired behaviour (see Figure 4); the contribution of the regularisation term decreases rapidly, which implies that the agent is learning the intended behaviour. We show the regularisation term for the extraversion agent where the regularisation prior $\pi_{0}^{E}$ matches the optimum monetary policy $\pi^{M}$ in Figure 4(a). Further training causes no instability as is often observed in the DDPG algorithm [31]. We hypothesize that this may be due to the agent characteristics imposed by our regularisation whose effect may be similar to entropy regularisation [31].

The actions of any linear combination of these agents, i.e., any personal agent, are interpretable through the intrinsic characterizations, i.e., priors, of each of the regularized agents.

We have presented a novel application of training RL agents to exhibit desired characteristics and behaviours in asset management. The method is based on the regularisation of the policy during training. Here, we use prototypical personality traits - openness, conscientiousness, agreeableness, extraversion, and neuroticism - to define a set of priors which express their affinity towards different assets and thus impose different investment strategies. This makes the agents’ behaviour explicit and thus offers an explanation for their recommendations. Our agents learn distinct optimal strategies for the continuous distribution of monthly investments across a portfolio of investment assets. We have shown that the agents learned to optimize total rewards while adhering to their distinct priors. This makes it possible to interpret the agents’ investment strategies.

Unlike traditional DDPG algorithms which may diverge with continuous training, our regularisation results in quick and robust convergence. This could become relevant if RL agents undergo continuous training to give personalized investment advice to customers. The justification of this observation will be subject to future research.

Our agents have learned the concept and utility of compound growth rates and risk avoidance, which form part of the interpretation of their investment strategies. These are solely based on the regularisation priors which express their personality traits; the reward function makes no reference to the personality traits. While the notion of compound growth may emerge from the reward function, we do not yet know whether the notion of risk avoidance is connected to the reward function or regularisation.

Here, we have chosen a linear combination of different, separately trained agents aligned with the prototypical personality traits to arrive at an aggregate investment advice. In the future, we will investigate whether the orchestration of these agents can be learned to approach the optimum monetary policy. This aggregation will need an explanation as well as interpretation to understand its impact on the investment strategy. The hierarchical orchestration of prototypical agents will be learned from real customers’ personality profiles. This will result in an explainable and interpretable personalized financial investment advisor.