Reinforcement Learning with Intrinsic Affinity
for Personalized Prosperity Management

Effective customer engagement is a requisite for modern financial service providers that are adopting advanced methods to increase the level of personalization of their services [1]. Although artificial intelligence (AI) has become a ubiquitous tool in financial technology [2], research in the field has yet to significantly advance levels of personalization [3]. Asset management is an active research topic in AI for finance; however, the research opportunities presented by the need for personalized services are usually neglected [4]. Whereas personalized investment advice is typically based on questionnaires, we propose the use of micro-segmentation based on spending behavior. Traditionally, customer segmentation has been grounded in demographics which provide only a coarse segmentation [5]; it fails to capture nuanced differences between individuals with the potential for undesirable ramifications, e.g. discrimination in credit scoring based on postal code [6]. Micro-segmentation, however, provides a more sophisticated classification which can improve the quality of banking services [7, 8].

We develop a personal asset manager that invests in a portfolio of asset classes according to individual personality profiles as manifested by their spending behavior. The result is a hierarchical system of reinforcement learning (RL) agents in which a high-level agent orchestrates the actions of five low-level agents with global intrinsic affinities for certain asset classes. These affinities derive from prototypical personality traits. For instance, personality traits with a higher affinity for risk may, as a general rule, prefer high-volatility assets.

Explainability and interpretability are key in sensitive industries, such as finance [9]; they form the basis for understanding and trust and have not yet been adequately addressed [10, 11]. In short, our agents’ policies are regularized by predefined prior action distributions which imprint characteristic behaviors, making their policies inherently interpretable on three levels: (1) they use salient features extracted from customer spending behavior, (2) the affinity of the prototypical agents, and (3) their orchestration to achieve personal investment advice. Our contribution is therefore twofold: we demonstrate how RL agents can be made inherently interpretable through their intrinsic affinities, and their application and value to personalized asset management.

Spending patterns are a predictor of financial personality [12]. In their paper, the authors trained a random forest to predict customer personalities from their classified financial transactions, using a prevalent taxonomy of personality traits: openness, conscientiousness, extraversion, agreeableness, and neuroticism. Although they achieved only a modest predictive accuracy, a subsequent study found that spending patterns over time expose salient information that is obscured in non-temporal form [13]; the authors in this study used the same personality model, but added temporal patterns such as variability of the amount, persistence of the category in time, and burstiness—the intermittent changes in frequency of an event. Recurrent neural networks (RNNs) are able to extract this salient information when predicting personality traits from financial transactions [3]. In, [14], we gained an understanding of these extracted features by interpreting the dynamics of the RNN state space through a set of attractors. Understanding model behavior is crucial in industries such as personal finance [10]. In their study, [10] extracted rules from three classes of models—linear regression, logistic regression, and random forests—which not only exposed the spending patterns most indicative of personality traits, but also aided in model improvement.

In RL, agents learn to solve problems by tentation; they maximize the expected rewards resulting from their actions in an environment [15]. Environments can have complex dynamics that result in sophisticated policies that are opaque to their developers, who have no influence over what these agents learn [16, 17]. Intrinsic motivation enables agents to learn behaviors that are detached from the expected rewards of the environment [18]. It is a strategy that was developed to address the challenge of exploration in environments with sparse rewards [19]. One such approach is Kullback-Leibler (KL) policy regularization in which the objective function is regularized by the KL-divergence between the current policy and a predefined prior [20]. Policy regularization has been shown to be helpful and never detrimental to convergence [21]. Although most policy regularization methods aim to improve learning performance, they can also control the learning process and imbue the policy with an intrinsic behavior [22]. Here, the objective function is regularized with a predefined prior action distribution that defines a desirable characteristic:

$J(\theta)$ is the learning objective as a function of the model parameters $\theta$, $R(s,a)$ is the expected reward for state $s$ and action $a$ as sampled from a replay buffer $\mathcal{D}$, and $\lambda$ is a scaling hyperparameter for the regularization term $L$, which is the mean square difference between the observed action distribution and the action distribution given a regularization prior $\pi_{0}$. The efficacy of this approach was demonstrated by instilling an inherent characteristic behavior in agents that navigate a grid. These agents learned to either prefer left turns, right turns, or to avoid going straight by taking a zig-zag approach to their destination. In contrast to constrained RL, which avoids certain states, the policy regularization in [22] encourages certain actions irrespective of the state and is a new direction for RL.

Hierarchical reinforcement learning (HRL) decomposes problems into low-level subtasks that are learned by relatively simple agents for the purpose of either improved performance or explainability [23, 24]. Larger problems are solved by choreographing these subtasks through an orchestration agent that learns the high-level dynamics of its environment [25]. HRL has, for instance, been used to control a robotic arm: while low-level agents learned simple tasks such as moving forward / backward or picking up / placing down, an orchestration agent learned to retrieve objects on a surface by choreographing these tasks [26, 27]. The agents were not only efficient at learning, but their policies were more easily interpreted by human experts. In [28], the authors used HRL to train a hierarchical set of agents to play a game. Their low-level agents learned to solve simple tasks such as “pick up a key” or “open a door” while receiving extrinsic rewards from the environment. A high-level agent then orchestrated these sub-tasks and received intrinsic rewards generated by a critic based on whether or not larger objectives were met.

There are three point attractors for the personality trait conscientiousness, towards which trajectories converge depending on their initial conditions. Agreeableness, extraversion, and neuroticism each have a single line attractor, while trajectories that classified as openness converge towards a single point attractor. Each basin of attraction forms a cluster of trajectories, which each form a hierarchy of sub-clusters along successive levels of dominance of personality traits. This is the interpretation of the trajectory dynamics.

Based on the premise that there is a causal relationship between personality-matched spending and happiness [33], we hypothesize that investment matched with personality can similarly increase happiness. Our goal is therefore to learn, through high-level RL orchestration, the optimum strategies that match customers’ unique financial personalities. Our RL agent orchestrates the actions of low-level prototypical agents according to customers’ extracted behavioral trajectories (Figure 1). With actions adding up to one, representing the fraction of the investment amount allocated to each low-level agent, it maximizes the following reward function: the inner product between the values of asset holdings and the customer’s preference for each asset type. This preference is calculated as the inner product of the customer’s personality vector—the set of five values representing their degrees of membership in each of the personality traits—and the set of coefficients relating each asset type with each personality trait (Table 2). This reward measures the correlation between spending behavior and investment strategy, which we call the satisfaction index; the higher the satisfaction index, the higher the correlation between spending behavior and investment strategy. We then instill an intrinsic RL affinity through a regularization prior—the customer’s personality vector scaled by its sum—in a deep deterministic policy gradient algorithm (DDPG - [34]). The actor consisted of three vanilla RNN nodes and an output layer of five actions with a softmax activation. The critic had a similar three-node RNN layer for the states which, concatenated with the actions, were succeeded by a 1000 node feed-forward layer and a single output node with no activation function. We found that three RNN nodes provided consistently high total rewards, which is consistent with findings that RNN architectures generally perform well in low-dimensional representations [35]. We tuned our hyperparameters using a one-at-a-time parameter sweep to reach the following optima: the actor and critic learning rates were 0.005 and 0.01, respectively, the target network update parameter $\tau$ was 0.05, the discount factor $\gamma$ was 0.95, and the regularization scaling factor $\lambda$ was 5.

In summary, we trained five low-level RL agents to invest in a set of assets according to prototypical personality traits [29]. We now combine their actions using a high-level RL agent that uses a customer’s spending behavior as manifested in their transaction history. This spending behavior is evinced in behavioral trajectories that we extracted from the state space of a RNN that predicts personality from financial transactions [3]. We illustrate this in a flow diagram in Figure 4.

The regularization prior for Agent A $\pi_{0,A}$ (the agent for Customer A) consequently has a low variation in its values while $\pi_{0,B}$ assigned the highest weight to neuroticism and openness, $\pi_{0,C}$ assigned the highest weight to openness and extraversion, and $\pi_{0,D}$ assigned the highest weight to extraversion, agreeableness, and openness.

Although these strategies might appear similar, there are significant differences: Customer A never invested more than 60% of their monthly allocation in stocks, while Customer D invested up to 90% in stocks, and Customer A was the only one to invest significantly in property. This is due to Customer A having the highest relative degree of conscientiousness, i.e., they preferred a reduced risk. In contrast, Customer D had the highest risk in their portfolio by investing the least in property and mortgage repayments and the most in stocks, due to their low score in neuroticism which increases their appetite for risk. When comparing Customers B and C, Customer B invested more in savings accounts and less in stocks in the period between 150 and 250 months. This is due to their differences in agreeableness and neuroticism, where customer B scored higher in neuroticism and lower in agreeableness. In Figure 3, the prototypical agents associated with neuroticism and agreeableness are the only two to invest in savings, and the neuroticism agent started investing in savings much earlier and with higher percentages. Despite the nuanced differences in investment approaches, the general advice for all customers was similar: first pay down mortgages to reduce debt repayments, then accept higher risk with higher returns from stocks and benefit from compound growth, and finally toward retirement age reduce risk through savings accounts. This is consistent with conventional financial advice: younger people with more disposable income may accept more risk for higher returns. Very interestingly, this was not explicit in the objective function which had no elements of risk, while the effect of compound growth was evident only in increased final returns.

The policies were randomly initialized, but they quickly converged to local optima in close proximity of the regularization priors in the action space. The learned strategies are thus interpretable.

Machine learning is essential for personalizing financial services. Its acceptance is contingent on understanding the underlying models, which makes model explainability and interpretability imperative. Our reinforcement learning model blends investment advice that is aligned with different personality traits. Its interpretation follows from the global intrinsic affinities of the learned policies, i.e., affinities that are independent of the current state. These policies not only result in a good profit, but similar profits are achieved across different personality profiles despite their distinct strategies. For instance, they avoid risk for highly conscientious individuals, while pursuing novelty for individuals that are more open to new experiences. Interestingly, they have learned the concept of risk without this being explicit in the objective function. Across all portfolios, the advice is consistent with conventional wisdom: younger investors may accept higher risk, which typically reduces with age. It remains to be seen whether this is simply a consequence of optimizing profit while balancing the intrinsic action distribution, or whether our agents have learned deeper strategies of asset management. In future work, we intend to investigate this phenomenon by extracting an explanation for our agents’ decisions. It will also be interesting to extend our method to local intrinsic affinity, where the preferred policy also depends on the current state. The potential applications for our method go beyond investment advice and includes, e.g., preferred evasive actions for self driving cars, navigation systems that favor either freeways or scenic roads, or medication for the treatment of chronic diseases based on individual medical history in healthcare.