AUGMENTING TRANSFERRED REPRESENTATIONS FOR STOCK CLASSIFICATION

The data used in this study consists of the daily returns of all constituent stocks of the S$\&$P500 index, from $1990$ to $2018$. It comprises $7000$ trading days, and approximately $500$ stocks per day. We use the data pre-processing scheme proposed by Krauss et al. [4], where the data is divided into splits of $1000$ days, with a sliding window of $250$ days. Each split overlaps with the previous one by $750$ points, resulting in $25$ splits in total; a model is trained on each split. Inside each of the 25 splits, the data is segmented into sequences consisting on $240$ time steps $\{\tilde{R}^{s}_{t-239},\ldots,\tilde{R}^{s}_{t}\}$ for each stock $s$, with a sliding window of one day, as shown in Figure 1. The first $750$ days consist of the training set, and the test set is formed by the last $250$ days. This leads to a training with approximately 255K samples ((750-240)*500) and the test set with approximately 125K samples.

The data is standardised by subtracting the mean of the training set ($\mu_{train}$) and dividing by the standard deviation ($\sigma_{train}$): $\tilde{R}^{s}_{t}=\frac{R^{s}_{t}-\mu_{train}}{\sigma_{train}}$, with $R^{s}_{t}$ being the return of stock $s$ at time $t$.

A goal of this work is to evaluate whether training a neural network by directly passing the trading rule as the classification target is better than training a binary classifier and then applying a trading rule. Therefore, to construct the target, taking the $\approx 500$ daily stocks, we ranked them by their returns and the top $K$ are labeled as buy, the bottom $K$ are labeled as sell, and the rest as do nothing. Following the recommendation from Krauss et al. [4], we use $K=10$. This leads to a highly imbalanced dataset, with a label proportion of $10:10:480$.

The source network is trained as a binary classification task, where the target variable $Y^{s}_{t+1}$ for stock $s$ and date $t$ can take to values, 1 if the returns are above the daily median (trend up) and 0 if returns are below the daily median (trend down).

The network architecture selected for transfer learning (source network) is a one layer LSTM proposed by Krauss et al. [4]. We chose this architecture because it achieved good performance on a large, liquid dataset - with similarities to the S$\&$P500. The network is a single layer LSTM with $25$ neurons, and a fully connected two-neuron output. We use a learning rate of $0.001$, batch size $128$ and early stopping with patience $10$ with RMSProp as optimizer. We implemented the neural network architectures using pytorch [20] in Python.

After training the source network on the 25 splits of data, we have 25 neural networks. We then remove the output layer and replace it with a fully connected layer of $n$ neurons and an output layer of 3 neurons. The weights of the LSTM layer are fixed (there is no retraining), and the fully connected layer and output are trained with the new target data that incorporates the trading rule. Fawaz et al. [13] proposes fine-tuning the transferred model parameters on the new dataset, but this leads to poor performance in our case, so the weights on the LSTM are left fixed, and just the new added layers are trained.

The loss used for training in all cases is the cross-entropy loss. An alternative approach is to incorporate a loss term that takes into account the direct information on the positions from the network, and optimizes the average return, as follows:

where $B$ is the size of the batch, $R(i,t)$ is the return captured by the network prediction for asset $i$ at time $t$ and $\alpha$ is a scaling factor so both terms are equally represented.

As the dataset is highly imbalanced, we sample the training data with higher probability in the minority classes, obtaining a balanced representation of the classes in each batch.

The approximately 255K samples of the training set are divided into training and validation with a proportion $80/20$. The validation set is used for early stopping the training. Each train set is augmented one time, i.e. we apply an augmentation method to the training data and the final set corresponds to the original data plus the augmented one, doubling the amount of samples. We study two forms of data augmentation: applying random transformations on the input data and doing data augmentation on the feature vector obtained by evaluating the input on the fixed LSTM layer. Figure 2 shows the training process with transfer learning and both forms of data augmentation.

We test transfer learning on two networks, one with a fully connected layer of $25$ neurons and one with $100$ neurons, and compare the proposed networks trained using transfer learning with the same topology trained from scratch. In order to avoid random initialization conflicts on the non-transferred networks, we train three separate instances with different initial weight values (i.e. we use three different seeds on the random initialisation) and average their performance.

Given that we want to evaluate financial performance, we build a portfolio by ranking the output of the networks labeled class 1 (buy) and 2 (sell); we then take the 10 with highest probability for each class and build a long-short portfolio. Portfolios are analysed after transaction costs of 5bps per trade. The portfolio performance metric we use is Information ratio (IR) - the ratio between excess return (portfolio returns minus benchmark returns) and tracking error (standard deviation of excess returns) [22]. As the portfolios are long-short, they are market-neutral, therefore, performance of the portfolio in independent of performance of the market and no benchmark has to be subtracted. We also calculate the downside information ratio - the ratio between excess return and the downside risk (variability of under-performance below the benchmark), that differentiates harmful volatility from total overall volatility. We compare our results with Krauss et al. [4], in which a binary classifier is trained and then the trading rule is applied. We also calculate two classification metrics, accuracy and macro-F1 expressed by the mean and standard deviation over the 25 data splits.

Table 1 shows the performance of the models trained using only the cross-entropy loss with and without transfer learning, as well as training with the combined loss of cross-entropy and return maximization from equation 1. We see that, in all cases, the models trained from scratch show a poor financial performance as shown by the information ratio. All transferred learned models show an equal or higher performance than the baseline LSTM method, with the models trained with the combined loss having a slightly higher performance. The classification metrics are higher for models without transfer learning and trained only on the cross-entropy loss. Using the combined loss hurts classification metrics in all cases, but as expected, it improves portfolio performance.

Table 3.2 shows the performance of augmentation methods for the architectures with a fully connected layer of 25 neurons and table 3.2 models with a fully connected layer of 100 neurons. All models were trained using the combined loss of equation 1, unless stated otherwise. In both topologies, augmentation on the input space is ineffective, and in most cases it decreases performance with respect to the model trained with transfer learning but without augmentation. In contrast, models trained with augmentation on the feature space tend to improve on IR, in the case of jittering.