That looks interesting! Personalizing Communication and Segmentation with Random Forest Node Embeddings

A challenge that pension providers face is communicating effectively with pension plan participants to foster learning and information gathering, and to increase awareness of the importance of pensions. The main goal is to engage participants to obtain information on their expected pension benefits and for them to determine whether they are on track with regards to their retirement savings. Pension providers in the Netherlands are obliged by law to provide their participants with information, and much of this communication happens online via emails and websites nowadays. However, engagement indicated by the clicking rate is low. Here the click rate is calculated by the number of participants who opened the email divided by the total number of participants who received the email.Personalizing an email by framing it in such a way that it fits the participants’ interests could potentially help to trigger participants to click. We assume that the content of an email is relevant to the participants’ if they click the article links in a received email. Different article links lead participants to a different pension topic.

Past research has identified several barriers for participants to get informed about their pension. Making decisions for the far future i.e. intertemporal decision making, is difficult because participants are faced with more attractive realtime options: they can also spend their time watching a movie and or decide to spend their money on a new car rather than saving it for retirement. Searching for information takes time and effort, and retirement is so far away that participants postpone financial planning [17].

Participants may also be prone to information avoidance: while people know that information is available, and have free access to this information, they still decide to ignore it [18]. A related barrier is information overload, meaning that when individuals are overwhelmed by the options available to them, they choose the path of least resistance [19], just as they do as a result of procrastination, status quo bias, and anticipated regret [20]. Choosing the path of least resistance can for example mean that individuals stick to the default, or do nothing like not reading the emails they receive from their pension provider.

Given these barriers such as these, the question for pension providers is how to communicate with their participants in a way that these barriers are overcome. Pension providers have a duty to communicate and want to do this as efficiently as possible. Most people are already aware that saving for retirement is important, which makes them postpone engaging with it even more: important decisions take more time and effort, so people defer making them [21]. Communications on pensions that emphasize that it is important to save for retirement can, therefore, lead to opposite results. Another way to communicate with participants is to personalize the communication they receive instead of sending a general message.

In this paper, we use an adaptive personalization approach to communicate with pension plan participants. In line with the definition of [22], this means that personalization is done automatically based on algorithms, without the proactive efforts from the participant, and that it will adapt the product over time based on participant behavior. Adaptive personalization systems have been shown to be more successful than self-customization in news context [22].

One of the fields that contributes the most towards representation learning is deep learning (DL) [23]. An important part of DL research is focused on modeling procedures to either learn approximate rules from data, like for example differentiable inductive logic programming [24] or to embed rules in a latent space in order to produce inference in the latent space, like for example visual query answering [25], that embeds relationships amongst entities in pictures learning both the embedding of the entity and of the relationship and neural logic programming [26], that instead applies rules to embeddings of complex objects to apply first-order logic reasoning. A similar approach is observable in the knowledge representation community, in which neural approaches are applied to learn relationships within an ontology [27], or neural networks are applied to perform approximated ontology based reasoning [28].

Several contributions have previously made use of random forest as a structure to apply DL. Deep forest [29], proposing a novel deep model multi-grained cascade forest (gcForest) model, a decision tree ensemble approach. The gcForest can have an outstanding performance on a small-scale data set with a fewer hyper-parameters than DL. Yuchuan Kong et al. [30] proposed Forest Deep Neural Network (fDNN) algorithms to integrate the deep neural network architecture with a supervised forest feature detector. In the bioinformatics field, it is often the case that the number of predictors/features is much bigger than the number of instances and it limits the power of DL and the classical machine learning algorithms. The fDNN employed random forests as the feature detector, to learn sparse feature representations from a larger input feature space.

Image Classification is one of the popular machine learning tasks in the image processing field. The images are represented by a set of independent patches characterized by local descriptors. Moosmann et al. [31] in 2007 used the Extremely Randomized Clustering Forests (ERC-Forests) to learn the spare local image descriptor. Each descriptor is represented by a set of leaf node indices with one element from each tree. They treated each leaf in each randomized decision tree as a separate visual word and stack the leaf indices from each decision tree into an extended code vector for each descriptor. In the paper [31], the tree algorithm is used as a spatial partitioning method, to learn the representation of the image descriptor, not a classifier to label the target variable. The learned tree leaf features are the input features of the support vector machines (SVM) classifier for predicting the image.

Amongst the most successful recent representational approaches we can see that graph convolutional networks (GCN) [32] are playing an important role in data structured in graph format. Differently from models based on random walks like node2vec, the GCN approach makes use of the adjacency matrix between the nodes of the graph, using convolutional filters to extract relationships between the nodes that can help performing a prediction task. GCN has been successfully used for link prediction [33], text classification [34], and spatio temporal tasks like traffic flow prediction [35] and activity recognition [36]. With respect to the contribution presented in this paper, GCNs are currently being used to deal with data that already has a graphical representation, and similarly to random walk approaches, such as deep walk or node2vec, they could in principle be used to substitute the calculation of embeddings, given a method that extracts a graph representation from the data at hand. In the Limitations and Future work Section, we further discuss how these approaches could be used in synergy with the algorithm presented in this paper.

Another important related area of research concerns variational autoencoders (VAE) [37]. In general terms, autoencoders are generative models composed of an encoder and a decoder. The encoder is trained to project data items in a latent space, while the decoder is made to decode the data items from the latent space back to the original space. VAEs extend the autoencoder model by specifying that the latent space variables should present a distribution (for example a Gaussian distribution). Given the flexibility of VAEs, many different variants of VAE have been defined, depending on the problem being addressed: in [38] Tengfei et al. defined VAE that can generate valid semantic expressions by means of a regularization procedure applied during the training of the VAE; in [39] Pu et al. use VAE to learn a latent representation of faces; in [40] Liao et al. use VAEs to detect outliers; VAEs have also been extended to deal with categorical variables [41].

With respect to the models presented in this section, the approach proposed in this contribution has the advantage of creating a latent space starting from a tree structured feature space. This implies that after performing inference and reasoning in the latent space, it is always possible to identify a rule in the original feature space. In addition to this, the proposed approach brings the advantage of being able to handle categorical data as a consequence of relying on decision trees for the embedding models, whereas the other models do not usually specify a strategy concerning such variables.

The main idea behind node embeddings frameworks is to learn continuous representations of nodes and edges in a network in order to be able to compare similar nodes in the network. The node2vec algorithm builds on top of ideas coming from natural language processing (word2Vec [7]). In practical terms, node2vec treats each of the nodes like a word, the graph like a document composed of sentences created with random walks through the graph. Node2vec from Grover and Leskovec in 2016 [8] first applied a random walk over a graph to generate walk traces and extracts features based on a learned trace. How the trace moves through the graph is governed by a set of parameters:

• •

  $n$: number of walks

• •

  $l$: length of the walk

• •

  $p$: the return parameter $p$ measures the likelihood of immediately revisiting a node that has just been visited

• •

  $q$: parameter $q$ differentiates between inward and outward nodes in the search.

Parameters $p$ and $q$ allow to differentiate between breadth first search (BFS) and depth first searchs (DFS). The random walk length $l$ and the number of random walks $n$ influence the neighbourhood taken into consideration and therefore the quality of the calculated embedding.

Figure 1 illustrates the feature extraction process applied by the node2vec algorithm. In particular, after the random walks are calculated, node2vec only behaves as a wrapper for the word2vec algorithm, which will simply calculated the embeddings of the nodes as if they were words in a corpus of documents.

The Random Forest algorithm has been a very successful model for prediction in marketing contexts [42] [43]. It is an ensemble method that is based on training a large number of decision trees by means of different partitions of the training data (bagging) [44].

After the decision trees are trained, the final classification is performed by majority voting. Random forests are widely used in machine learning tasks because of their resilience against overfitting and their ability to generalize well even in complex data sets. One of the characteristics that makes this family of algorithms successful is that they can handle both continuous and categorical data. In addition to this, one of the main advantages of random forests with respect to other machine learning algorithms, such as SVMs [45] and neural networks [23], is that random forests are interpretable thanks to the fact that the decision trees participating to the classification can be manually inspected.

The algorithm developed in this contribution is shown in the Algorithm 1 pseudo-code snippet. The first step of the algorithm consists in calculating $k$ decision tree estimators with a random forest algorithm, and save them in set $T$. For the purpose of this contribution the random forest algorithm developed in [46] was used. The node2vec algorithm is then applied to each of the $t_{i}\in T$, with $|T|=k$ decision tree structures. The goal is to obtain $k$ embedding models $e_{i}\in E$ with $|E|=k$, that given a node identifier belonging to one of the trees can produce the embedding of an out-of-sample testing data record. The embedding features $f_{j}$ or each data record are calculated by using the leaves’ identifiers $l_{j}$ for each of the decision trees considered. Each embedding model encodes information concerning the surrounding of the leaf. Finally, embedding features $\tilde{x}_{i}$ represents the concatenation of the features calculated with all the embedding models and we concatenate it with $x_{i}$ features of the data record.

When such a feature extraction is applied, the embeddings of the leaves of the decision trees contain the information concerning the decision boundaries of the decision trees, including the neighbouring decision boundaries. The advantage of such a representation is that a linear discriminant model, such as logistic regression, can easily find a linear combination of the boundaries that would perform a good classification, allowing to substitute the majority voting of normal random forests with a soft weighing of the votes of the decision trees. In general terms, this approach can also be seen as a way of weighing the identified decision boundaries or as a way to select the most relevant decision trees. Algorithm 1 presents the case in which one combines the features by concatenating them together. This is not the only possible approach, another approach could involve taking a weighted average of the embeddings or to multiply the embeddings together. With respect to this paper, the focus has been on the concatenation of the features, because the concatenation allows us to perform complex descriptive reasoning, as discussed in Section 4.

The definition of the random walks length of the node2vec algorithm specifies the order of the neighbourhood of random forest embedding. Figure 3 shows a depiction of the embedding with a $l$ parameter equal to 15, on a tree calculated on top of the D1 data set used in this study.

Employees working in the educational or governmental sectors are obliged to build up their pension at this fund. This results in a non-commercial duty for the fund (and by extension for their pension administrator) to provide their participants with information concerning their pension. One of the ways of communication with its participants is via an email newsletter, which they receive around four times a year. The newsletter is a collection of articles with information on five different categories: sustainability, choices regarding participants’ pension, the financial situation of the fund, personal online page, and government decisions concerning pensions. Age distribution by gender is shown in Figure 5, the mean age is 50.2 years. The total number of records in this data set is 2.8 millions. The features used in this study are shown in Table 1.

Figure 4 shows a screenshot of one of the newsletters sent to the customers.

Since participants receive a lot of emails, sending only relevant information is crucial. To be able to provide these participants with useful information at the right time, individual participant profiles are created based on historical data. The overall goal of these newsletters is twofold: first, to inform participants about important developments, and second, to get them more involved with their pension by having them click through on various articles that are interesting to them. In our field experiment, we focus on measuring and contributing to the second goal, achieving more pension involvement.

For the pension fund, it is important to have a model of the customers concerning who may be interested in a specific newsletter, to ensure maximum awareness in the customer population concerning pension topics. The data of the general population of the pension fund is denoted as D4 in the rest of the paper.

In addition to the general population data of the pension fund, we also use data from a field experiment that took place in July 2018. The experimental hypothesis of the experiment is that the click rate of the intervention group is higher than that of the control group, given the application of a machine learning model to target the intervention group.

All $463,212$ active (i.e. accruing pension) and subscribed pension plan participants at the fund received a newsletter. 52% of the sample are male, the mean age is 48.4 years for females and 52.1 years for males, and the mean annual income is 50,317.11 Euro.

Figure 6 shows the distribution of sample age by gender. With regards to participants’ marital status, 62.2% are married, 16.9% are single, 10.5% have a registered partner, and 10.4% are divorced. 72% of the sample work fulltime (that is, more than 0.75 fte), the rest work parttime (that is, less than 0.75 fte).

The features used for this study are the same as those reported in Table 1.

In this paper, we chose the Pension Choice topic as an example for further analysis. There are in total of $227,885$ participants involved in such a topic. The intervention and control groups have $89,587$ and $138,298$ participants respectively. For analysis of the pilot results, it is important to note that each click on an article gets stored at an individual level. Therefore, we can extract which participants (by their IDs) clicked on which article at what moment in time.

The Pre-trained model is a binary logistic regression model (BLR), which is used to predict the interesting topic (Pension Choice) of an individual participant. Firstly, this model is applied to the overall data set to select the participants most likely to be interested in the Pension Choice topic to construct the intervention group. The selected participants are eliminated from the overall data set. After that, we randomly selected 138,298 participants from the remaining data set without replacement to construct the control group.

The click rate of the participants from the intervention and control groups is $29.39\%$, and $16.91\%$ respectively, based on the pilot data. The $Chi^{2}$ test (p-value $<0.01$) shows that the click rate of the Experimental Group is significantly higher than that of the Control Group. The current BLR model can help the pension fund to predict the interesting topic of an individual participant. The findings show that pension fund can use their historical data on participant behaviour to improve their communication.

In the study performed, the data collected showed that the BLR model works, given the large experiment sample. In reality, the budget available for campaigns is often limited to smaller subsets. In this paper we therefore show how RFNE can help selecting subsegments of the population in which the advertisement would be particularly effective.

Due to the data protection regulation of pension data in the Netherlands, we cannot publish our research pension data. To help the other researchers reproduce the current results and continue our research, we chose a number of publicly available data sets concerning problems of churning and marketing campaigns. First of all we use the Portuguese banking institution data ([47], denoted as D1 in the rest of the paper). The data is collected from $41,188$ bank clients from May 2008 to November 2010 and published in February 2012. The marketing campaigns were based on phone calls. The data has been preprocessed by the original authors, and it includes 20 features in total. The data set target reports the success in convincing a is a client to subscribe a bank term deposit.

Secondly, we make use of the KDD 2009 challenge ([48], denoted as D2 in the rest of the paper) that provides the data set of a large telecommunication company (Orange) to predict the propensity of a customer to switch to another provider, comprising $227$ features, of which 38 of categorical nature, $189$ of numerical nature, plus one target variable reporting churning and non churning customers ($3672$ and $46328$ customers respectively). ’

Finally, we also make use of the Duke data set (denoted as D3 in the rest of the paper), that concerns $49752$ data records, containing $56$ variables, of which $22$ categorical features, $33$ numerical features and $1$ target variable reporting churning and non churning customers.

We perform our evaluation in the two use cases by splitting our data 80% training and 20% testing.

To evaluate the performance of the algorithm, we performed a grid search using a 5-fold cross validation on the training data on a number of parameters. For a matter of comparison, the parameters of the machine learning models identified on the standard data without embeddings, are also re-used for the part with node2vec embeddings. Table 3 shows the parameters used for the grid search and the selected values.

As shown in 4 in the selected data sets RFNE presents a competitive performance, from the perspective of classifying data sets presenting both categorical and numerical features. An interesting aspect of this evaluation is that it suggests that embeddings built on the decision trees identified by the random forest algorithm retain the information of the original data and that the combination of such embeddings with a logistic regression algorithm allows to produce an improvement with respect to a random forest algorithm. This improvement can be explained by the fact that the logistic regression acts applied on the embeddings acts as a soft voting mechanism that in addition includes an optimization of the vote weights based on the LBFGS solver [49], where instead a basic random forest applies majority voting. The comparison performed above covers only one part of the possibilities of extending RFNE. For example, given the sequence of embeddings representing the decision trees of the random forest, any algorithm which is able to work with sequences could be used to identify relationships between the trees useful for the sequence classification process, we leave these and other investigations to future works and discuss them in Section 5.

The current practice towards mining relevant patterns by using random forests implies a linear search amongst the leaves of the trees composing the forest, performing comparisons in terms of frequencies of the entities of interest. As the potential leaves in a big data set can be many, the process can turn out to be rather time demanding. In addition, comparing multiple variables and multiple groups identified by the trees in the forest implies performing many set operations, that are typically very demanding from a computational perspective.

Since RFNE is a technique stemming from random forests, the hidden space identified with node2vec can be associated with the trees and that can be further used to perform statistical tests to identify customer segments of interest. In specific terms, by using RFNE, this discovery process can be reduced to a statistical test between the embedded variables and a dependent variable, followed by a nearest neighbour search in the hidden space to find the closest leaf to the center of mass (i.e. the mean vector) of a population of interest. Remembering that each leaf has also a partitioning rule associated, once the leaf or leaves are identified in the hidden space, then a potentially highly descriptive set of rules for the dependent variable can also be obtained.

Fig. 7 shows a depiction of the rule extraction process.

In the following description, we will use the D1 data set, the general pension fund data, and the pilot pension fund data. Each of these studies has been carried out by setting $10$ decision trees, with maximum $5$ levels of depth, with each of the nodes presenting $50$ random walks of dimension $10$.

In the D1 data set, there are several variables that present categorical nature. We will focus on using binary variables as our dependent variables of study, so as to use widely recognized statistical tests such as a logistic regression. D1 data set concerns bank data and we perform a logistic regression using loan as the target variable to identify segments of the population that would subscribe to a loan.

In Table 5 we find that feature $2$ and $3$, belonging to the $2^{nd}$ tree of the feature extraction would have a positive log-odds towards the dependent variable. We can identify the mean vector of the entities in our database with respect to these features and then compare it, in terms of euclidean distance, with respect to the leaves of the $2^{nd}$ tree. This enables the identification of the closest leave, that also corresponds to the following rules with respect to the D1 data set:

where $pday$ represents the number of days after the last contact with the customer, and $day$ represents the day in the month. This can identify a segment in which the concentration of people who subscribed to a loan would be higher than the rest of the population (25.2% vs 18.4% in the rest of the population). It is also possible to check if this population would have a higher probability of defaulting, that is the original target variable of the D1 data set (13.3% vs 10% in the rest of the population). The value of identifying $rule_{d1}$ with respect to D1 is that it provides a segment of the population in which the bank would have more risk than with the rest of the customers, as at the same time the customer in this segment would have a higher probability of defaulting while also having a loan with the bank at the same time.

The approach described in this section can also be used to identify relevant segments in A/B testing trials. Table 6 shows the relationship between the features calculated with RFNE and a target variable discriminating between intervention and control groups. As all the features seem to be significant with respect to the target variable, in order to identify relevant patterns we select those features that present the best odds concerning finding a relevant difference between intervention and control, that in this case are feature_2 and feature_3 that are associated with the first tree.

By selecting those features and applying the approximated nearest neighbour procedure illustrated in Fig. 7 we can identify the following rule.

As we have intervention and control groups, we can observe if this segment presents statistically significant differences with respect to the clicking behaviour by using a $Chi^{2}$ test. This is illustrated in Table 7.

With respect to the leaf in consideration, the $Chi^{2}$ test shows that the intervention population presents a different behaviour with respect to the process of clicking the newsletter (p-value $<0.01$).

In the case in which we wanted to further restrict our population in consideration, one could define an additional dependent variable and identify another tree in the forest to produce additional rules. For example if we perform this operation for the gender variable in the pension data set, we can obtain another table of effects in which some of the trees show a big effect with respect to gender and we can also extract the additional rule below:

which, combined with the first rule identified, produces the following contingency table.

These results suggest customers of the pension fund in which the applied intervention predicts a difference in the clicking behaviour. Specifically, concerning the first rule, RFNE shows that it is possible to identify areas of the hidden space in which the density of the population clicking the newsletter is higher ($6.1\%vs4.4\%$), but that also implies more false positives ($4\%vs0.1\%$) when using the selected intervention. When the second rule is applied, we can focus on a sub population based on gender, seeing a similar result in terms of false positives and false negatives.

In this paper, we developed an adaptive personalization model to get pension plan participants more involved with their pension. We conducted a field experiment in which we sent out a newsletter to all active 465,711 participants of a pension fund, and find that our model results in significantly higher clicking rates.

Irrelevant communication can annoy participants with potential detrimental effects, such as having them unsubscribing from the mailing list [50]. Individuals receive a lot of emails every day from various service providers. Not all emails are as important as the ones from their pension fund, so pension providers should help participants view their communication as relevant by optimizing their mailing strategy.

In addition, it is important to reduce money management stress and negativity around financial planning, factors which have shown to negatively influence financial and overall well-being [51]. Our findings show that pension funds can use their historic data on participant behaviour to predict what topics from the email they will select, and potentially improving the effectiveness of customized email, and potentially yielding better informed pension savers and improved pension saving outcomes.

Node embeddings have already had an impact in changing the approach towards managing data and considering relationships between entities and features that follow a non-traditional schema. In the traditional machine learning schema, the feature engineering process ended after the first definition of the features. This contribution within a stream of research that reexamines the typical assumptions of machine learning algorithms, such as having samples that are identically distributed and independently sampled (iid assumption): the very fact that a node2vec representation of the decision areas in the random forest improves the results, implies that the iid assumption is very strong in many cases, as that the proximity of decision regions, and therefore of the entities falling into such regions, influences the decision boundary. Recent contributions in relational and representational learning have shown that relaxing the iid assumption can lead to better results in many situations. Our contribution opens opportunities of further development. We applied random forests without performing further transformations to the data. Many more opportunities exist, including:

• •

  Combining random forest with clustering approaches: applying clustering before using the random forest may lead to a different partitioning of the data that could yield better by uncovering partitions in the data due to different densities and groupings.

• •

  Substitute random forest with rule induction approaches: advanced association rule mining algorithms, such as FP-GROWTH, make use of tree structures to store the transactions, those structures could be used to calculate embeddings for each of the transactions to be classified.

• •

  Use the the random forest embeddingss as features for regression and autoregression: in the current contribution the random forest is used for classification purposes, but the same approach could be effective with trends and time series predictions.

While this paper has focused on the node2vec technique to extract the embeddings, other techniques exist to extract node, graph and subgraph embeddings that alone or in combination may be interesting to explore future.

This paper describes a machine learning approach to identify customers that would be receptive to a particular newsletter.We discussed an approach, based on random forests and node embeddings, to find relationships within features that can enhance the precision and recall of the classification process. The algorithm works by first calculating an ensemble of decision trees, then calculating a graph connecting the leaves of the decision trees and finally by applying node2vec on the calculated graphs to extract the node embeddings of the leaves. The concatenation of the node embeddings is then used as features for the classification of the customers with respect to a newsletter topic.

Our research is not without limitations. while we can predict with reasonable accuracy which topics of a email are selected by an individual, we have no additional knowledge as to whether doing so has any impact on customer satisfaction or on actual pension savings. Further interventional studies will be required to validate the utility of our approach.

Second, it would be interesting to know what role timing and frequency of emails play, and whether these would need to be adapted for different individuals as well.

The proposed approach comes at the additional cost of calculating node embeddings on the structure of the decision trees, in order to uncover the hidden space that allows us to perform further reasoning. As such, RFNE cannot scale as well as random forests. Nonetheless, the pension fund data set studied in this paper contained 2 million entries, meaning that, despite the additional computational burden of node2vec, RFNE can scale up to fairly large data sets. Concerning the descriptive part, the current approach focused on uncovering patterns occurring between binary variables.

We envision several studies that could follow this one. In this contribution we used node embeddings to find relationships between categorical and continuous features of customer data. An interesting future direction implies making use of knowledge graphs and ontologies to calculate the embeddings and then compare the results with the embeddings calculated on the random forest trees. In the paper, the graph is constructed by a collection of decision trees. A decision tree is generated based on the attribution selection measures, such as Information Gain, Gain Ration, and Gini Index. While a knowledge graph represents a collection of interlinked descriptions of entities/features. It puts data in context via linked and semantic. Therefore, the embeddings are calculated from the knowledge graph carries the nature relationships among features compare to its from random forest trees. Another promising direction is to consider different types of feature extractors rather than random forests and decision trees, as for example using a rule induction engine [52] and then embed the tree of the rules rather than decision trees as in this contribution. Finally, the feature extraction presented in this paper offers opportunities in the direction of generative models. Since the embeddings are calculated from regions identified by decision trees, when coupled with variational autoencoders [53] to reconstruct the original signal, they could potentially create realistic reconstructions, allowing to create a powerful tool for data anonymization.