Forecasting of Jump Arrivals in Stock Prices: New Attention-based Network Architecture using Limit Order Book Data

This research is conducted using NASDAQ’s “TotalView-ITCH” limit order data. The data consist of ultra-high-frequency (on millisecond bases) information regarding the limit orders, cancellations, and trades executed through NASDAQ’s system. The data contain prices and quantities of orders as well as their linked partial and full trades and cancellations. The data are further transformed into two data sets:

• (i)

  Output data: Minute-by-minute data about detected jumps in stock prices. This is based on mid-price observations from which jumps are detected, so each one-minute time period is classified as either having a jump or not.

• (ii)

  Event-by-event input data about the state of the order book. These data are extracted from millisecond-level observations about order book events. The resulting data contain both bid and ask prices as well as their quantities for the ten best levels on both sides of the book.

The order-driven market systems work in a way such that investors may place either ask or bid orders of their desired price, and the system will match eligible orders to create a trade. Orders may be either limit or market orders. Limit orders are placed in the list of orders at a specified price. Market orders are immediately executed with the limit order of the best price if such exists. In a way, this resembles a queue system, especially when orders of identical prices are submitted. A limit order that has not been executed can also be cancelled at any time. Both trades and cancels can also be partial, meaning that a part of the limit order will be left in the book after execution [Cont et al., 2010].

To ensure a continuous order flow, several well-known liquid stocks are selected for the study. These are GOOG (Google), MSFT (Microsoft), AAPL (Apple)²²2The price data for AAPL was adjusted slightly. On June 6, 2014 at 5pm, Apple issued 10 800 000 000 new shares, effectively splitting each existing common share into seven separate parts. As this was in the middle of the observed period and caused no difference in individual investors’ wealth in terms of owned stock, all stock prices prior to the split are divided by seven to make the true value of the owned stock continuous., INTC (Intel), and FB (Facebook). All of the selected stocks have large amounts of orders and trades each day. Table 1 shows the average numbers of order submissions, cancellations, and trades over one minute.

The data are divided into two categories: training data and test data. The training data are those used to learn the problem, that is, the data fed to the networks in the training phase to adjust the weights through the optimization algorithm. The training data consist of the series of observations in fifty-days periods. Fifteen percent of the training data is selected as validation data before starting the training of a model. Validation and test data are intended to evaluate the performance of the system. This cannot be done with the training data alone, as the model will easily be overfitted, which sharply reduces the performance outside the training dataset because the trained model is no longer generalizable [Webb and Copsey, 2011]. The difference between test and validation data is that validation data is constantly used in model selection and adjustment during the training phase. After selecting the best model, test data are used to evaluate the model’s performance. Thus, validation data are kept separately from test data to ensure that the model is not developed solely to be able to classify the test data; moreover, this provides an objective view on the performance of the system.

In all datasets, observations are picked every minute, but the amount of jump samples is increased by duplicating the jump observations. Specifically, the beginning of the duplicated sample is shifted by several seconds to ensure there are no identical samples. The time intervals of the data sets are presented in Table 2. Validation data are selected in a way such that the duplicated samples belong to either the training or validation data sets. The data are divided into training sets based on the day of the observation. A total of 360 days, spanning about one and half years, are selected from 2014-2015. The data are divided so that first there are 50 days of training data, followed by 10 days of test data. The next set contains the first 50 days as well as the following 50, and it is tested on the following 10 days after both sets. This pattern is followed through the whole dataset so that the seventh test set trains on 350 days and tests with the last 10 of 360. Additionally, the training data are presented in a window such that the model is trained on the newest 50 samples at a time starting from the beginning of the observation period (but not reset between sets).

To detect jumps in stock prices, we use an algorithm proposed by Lee and Mykland [2008]. As jumps are predicted short term, samples are collected every minute for the duration of the observation period. This gives a one-minute window in which a jump may occur, allowing these samples to be classified as either having or not having a jump in the following one-minute period. We run the jump detection algorithm for the entire sampling period for the collection of necessary amounts of jump samples. The length of the data window used for the estimation of bipower variation is 600 minutes.

The frequencies of detected jumps are presented in Table 3. On average, there are around three jumps per day per stock. However, jumps are not evenly divided between days. Instead, the days that have jumps tend to have a larger number of jumps on average. A sample distribution of jumps per day counts is shown in Figure 2. Moreover, during a single trading day, jumps tend to be heavily skewed towards morning hours, as observed, for example, by Lee and Mykland [2008]. The vast majority of detected jumps occurred within the first half hour of the trading day, with only occasional jumps after the first 1.5 hours for all stocks. Additionally, all stocks had a slight increase in quantity at 2 pm, where the time period between 14:00 and 14:05 contained around four times as many jumps as between 13:55 and 14:00. The jumps at this time occur during multiple days thorough the whole observation period. The distribution of jumps according to the time of day counted for the whole observation period is presented in Figure 3.

The data from the stocks are used to construct training and test sets by time and stock, as presented in Table 3. Jumps at the very first observation of the public market opening (9.30) were not taken into account. Additionally, jumps from the first two days were not detected due to the insufficient amount of previous observations to satisfy the window size requirement of the jump detection algorithm. This also means that the training sets presented in Table 2 skip the first two days of the sequence to avoid labeling possible jump samples as non-jumps due to the undetectability of jumps during the beginning of the price sequence. Thus, day 1 in the table is really day 3 of the price observation period.

The inputs use LOB data, which is reconstructed from the order book event data. The LOB contains both ask and bid prices as well as their quantities for the ten best levels on both sides of the book. This is done simply by checking active orders at a certain time, which can be then ordered by price to obtain the ten best levels so that the lowest ask and the highest bid are on the first level, and subsequent levels are filled by existing prices next in the order. The quantity is the sum of quantities for orders of that price, and quantities at levels with multiple orders are the sum of all active orders at that level. The method of constructing the book also means that empty levels cannot exist between two defined prices. Instead, completely empty ticks are left off unless there simply are not enough orders to fill the ten levels, in which case the levels last in order are filled with prices and quantities of 0.

To get the best view of the state of the order book, we follow Kercheval and Zhang [2015] to extract 144 indicators from the data: a) the basic set of features containing the raw LOB data over ten levels, with both sides containing price and volume values for bid and ask orders, b) the time-insensitive set of features describing the state of the LOB, exploiting past information, and c) the time-sensitive features describing the information edge in the raw data by taking time into account. The time-insensitive set contains further information about the spreads, differences, and means. The time-sensitive set contains features that indicate changes in the data in time, such as derivatives, accelerations, and intensities. These features, provided in Table 4, are used also in Ntakaris et al. [2018], Tsantekidis et al. [2017b, a], Passalis et al. [2017], Tran et al. [2018].

In addition to the LOB data, some of the time-sensitive features presented in Kercheval and Zhang [2015] require calculating intensities, that is, the number of arriving orders or cancellations of a certain type, which cannot be directly calculated from the constructed book and instead must be counted from the original event data. The intensities are separated into ask and bid, and the orders are categorized based on whether they are limit or market orders. The intensities at each step are calculated directly from the order flow data and attached to the corresponding order book data of the step. Within market hours, both the limit order book state and the intensities are calculated every second, yielding a total of 23,400 observations per day. Data from non-trading hours are discarded due to different trading mechanisms, and the data used over multiple days is treated as a continuous sequence. In addition to some of the suggested features, approximate times of the observations are included to account for the differences in stock behavior at different points during the day. The timestamps are rounded to the nearest hour to avoid converging to the local minima of purely time-based classification.

For the data sets (Table 2), samples are extracted by a one-minute moving window thorough the training set, creating one sample per minute, for a total of 390 samples a day. Positive samples are defined as those with a jump right after the last observation, that is, during the next minute, which is not included in the window. Negative samples are only collected from the moving window; for positive samples, the window is shifted slightly multiple times to generate more positive samples due to the large difference in the sample sizes. As the data are collected every second, it is possible to shift the window small enough amounts to not include the jump while creating slightly different data for the samples to increase variety and to preserve the original classification of a jump existing within the next minute. To ensure that possible periodical changes in the order books will not affect the classification results due to only positive samples being shifted, negative samples are also shifted randomly.

All collected samples contain 120 steps sampled at a one-minute interval. These samples are then normalized using the z-score to eliminate the irrelevant noise due to, for example, different starting prices: $\mathbf{x}_{normalized}=\left(\mathbf{x}-\bar{x}\right)/{\sigma_{\bar{x}}}$, where $\mathbf{x}$ is the feature vector to be normalized, $\bar{x}$ is its mean, and $\sigma_{\bar{x}}$ the standard deviation [Cheadle et al., 2003]. The features are normalized sample-wise one feature at a time: $\mathbf{x}$ is then a vector of length 120 containing all observations of a single feature in a sample, for example, all of the ask level 5 volumes. Separate normalization for different features is necessary due to the vastly different behaviors and scales between both different levels and volumes as well as their indicators. Including different indicators calculated from the limit order book, such as the price differences, allows for the preservation of information regarding the relations between different values, even after normalization.

The data are normalized sample-by sample due to the changes in price behavior that occur even during a single day. A relatively short normalization window is also needed to avoid larger scale price dependence. If, for example, the data were normalized over the full-time period, the main differences between prices in observations would come from the long time drift instead of the price changes in the recent past. As long-term changes are unlikely to be the main determining factor of jump occurrence in minute-level data, the normalization period should be short enough to avoid learning from them.

Additionally, in the used data, the most important factors seem to be changes that occur in the hours right before the jump. Changes within this timespan have also been noted for bigger jumps associated with company announcements, where changes in liquidity often start over an hour before the price jump [Siikanen et al., 2017, 2017b]. The normalization done within the sample also requires a sufficiently big observation window, as it needs to be large enough to capture the element of change. There is also a fairly signficant chance that a jump has already occurred on the same day at the time another prediction is made, lessening the impact of price changes compared to the previous data. Additionally, since all samples are of equal length, for the first two hours of the day, the window must include samples collected from the previous day.

Perhaps the most common type of neural network is the MLP, which is a feed-forward neural network formed by layers of neurons stacked in a hierarchical manner. It receives the data vectors in the input layer, and then the information is propagated throughout the hidden layers, providing a response at the output layer. Each layer is formed by a set of neurons, each receiving an input from the neurons of the preceding layer, and provides a nonlinear response of the form

where $I$ is the number of neurons in the previous layer, each providing an input $x_{i}$, and $w_{ij}$ is the weight connecting the $i$-th neuron in the preceding layer to the $j$-th neuron of the current layer. $\theta_{\cdot}$ is a nonlinear (piece-wise) differentiable function, which is used to nonlinearly scale the response of the neuron. The output neuron works exactly as the hidden layer neurons, although they may use a different activation (e.g., to lead to probability-like responses).

The optimal size of the hidden layer is defined by the data used, whereas the output layer size is defined by the number of output classes [Graves, 2012, Jefferson et al., 1995]. Multi-class classification is performed by following a competitive training approach, that is, the output neuron with the highest response indicates the predicted class label [Chollet and Others, 2015].

The training of a network consists of two phases, forward pass and backward pass. In forward pass, training vectors are introduced to the network and its responses are obtained. These responses are used in combination with the provided annotations (i.e., target vectors indicating the optimal response for each training vector) to define the network’s error with respect to a loss function. This error is then used in the backward pass to update the parameters of the network. This is achieved by exploiting the (piece-wise) differentiable property of the neurons’ activation functions, following a gradient descent learning approach called error backpropagation. We use an advanced version of this parameter update approach, called Adam Kingma and Ba [2014], which adaptively defines the hyper-parameters of each update step based on the input vectors.

For classification problems and networks giving probability-like responses, the crossentropy loss function is commonly used. It determines the entropy between sets by measuring the average number of bits needed to identify an event drawn from a set. For discrete sets $p$ and $q$, where $p_{i}$ is the true label and $q_{i}$ is the current predicted value, binary crossentropy can be defined as

It can be shown that when choosing between distributions $q$, which estimate the true distribution $p$, minimizing cross-entropy leads to choosing the best estimate by maximizing the overall entropy [Shore and Johnson, 1980]. Thus, it is a suitable loss function to be minimized, and often portrays the true loss better than simple error measures.

In this paper, the Long Short-Term Memory (LSTM) model is used to accumulate features in time-domain and to simulate memory, by passing the previous signals through the same nodes. LSTM can be seen as a special case of recurrent neural networks (RNN) in which the connections between neurons allow directedly cyclical connections. In a basic recurrent network, neurons form connections inside the same layer, creating a net of one-way connections. In the simplest form, this means a standard neural network but with a feedback loop. The connections in the basic RNN are weighted as in a standard MLP. RNNs address the temporal relationships in their inputs by maintaining an internal state due to the recursive property, a quality especially suitable for time series data [Giles et al., 2001].

LSTM was first proposed by Hochreiter and Schmidhuber [1997] and it was developed to combat the problem of keeping error signals in proportion when flowing backward in time (especially for long time dependencies) by making use of both short-term memory, based on the recurrent connections, and long-term memory, represented by the slowly changing weights. A constant error signal flow is ensured by connecting the neurons to themselves. LSTM introduced the concept of a memory cell to control the memory flow of a network. A memory cell is a singular neural unit with the addition of multiplicative input and output gates. These are created to protect the neuron from changes triggered by irrelevant inputs and to protect other units from the irrelevant information currently stored within the neuron. Each memory cell has a fixed self-connection and processes input from multiple input sources to create the output signals. Memory cells that share the same input and output gates form memory cell blocks [Hochreiter and Schmidhuber, 1997].

Training an LSTM network is done using a modified version of backpropagation, where a single step involves a forward pass and the update of all units through the computation of error signals for all weights, which are passed backwards in the network (backward pass). The activation of the input gate $y^{in}$ and output gate $y^{out}$ are defined as

where $j$ is the memory block index and $v$ is a cell inside the memory block $j$, so that $c_{j}^{v}$ marks the $v$-th cell of the $j$-th memory block and $w_{lm}$ is the weight for the connection between units $m$ and $l$. Input gates are defined as $in$ and output gates as $out$. The loop sums all the source units defined by the network. The function $f$ is a differentiable function for the gates, such as the logistic sigmoid

where $x\in[0,1]$. The input is further squashed by a differentiable function $g(\cdot)$ [Gers et al., 2000].

Gers et al. [2000] further adds to the LSTM model by including an additional gate, the “forget gate”. The forget gate allows the LSTM cell to reset itself at appropriate times, releasing resources to use. The LSTM layer outputs either a one-dimensional vector of activations for each feature or a two-dimensional structure with a value for each feature at each processed time step. With an LSTM layer connected to a dense layer, the former is needed, as the dense layer expects one-dimensional input. However, some models, such as the attention model proposed by Zhou et al. [2016], require multi-dimensional LSTM output when applied to the LSTM layer, as its purpose is to calculate a weighting value for each time step.

Convolutional neural networks (CNN) can be used to capture patterns in time and feature space. Convolution neurons combine information from neighboring observations in the feature and/or time dimensions and each neuron identifies different pattern in the input time-series.

CNNs mimic the way the visual system processes visual data. Specific neurons are only concerned with specific parts of the input, simultaneously making the position of specific features less relevant, as long as they are in a certain relation to the other features. Even though they were originally proposed for image recognition tasks, CNNs have found uses in speech classification and time series prediction tasks. The convolutional network combines the principles of the importance of locality in data points, shared weights between points, and possible subsampling. [LeCun and Bengio, 1995] CNNs have been especially successful in the domain of image processing, providing, for example, a winning best entry in the popular ImageNet image classification challenge [Krizhevsky et al., 2012] and the ImageNet feature localization challenge [Sermanet et al., 2013]. In a CNN, the images are first normalized, resized, and approximately centered. After the input layer, each unit in a single layer receives inputs from a certain set of inputs in its neighborhood from the previous layer, making the receptive fields localized. This allows the extraction of certain local features, which can then be combined [LeCun and Bengio, 1995].

Each convolutional layer is followed by an additional “pooling layer” to perform local averaging and/or subsampling. This reduces the resolution of the input at every step and reduces the network’s sensitivity to shifts and distortions [LeCun and Bengio, 1995]. A simple CNN-pooling combination is shown in Figure 4. Pooling can also be done using the maximums of the input window, drawing attention to more pronounced features while reducing the resolution. This is called max pooling and is also often done between convolutions [Scherer et al., 2010]. Convolutional and pooling layers are usually repeated until the feature maps convolute to a singular output for all possible classification results [LeCun and Bengio, 1995], or they may be connected to regular dense (MLP) network layers to produce the final output [Krizhevsky et al., 2012].

Time series analysis with convolutional neural networks works much the same as in images, although the dimensionalities of the inputs are naturally different. Locality of the fields works well with time series, as the observations are dependent on time; the same observation can be followed by different results at different times, and the surroundings of the observation can be used to generate a better estimate [Längkvist et al., 2014]. Convolutions can also be applied to one-dimensional time series data, allowing the convolution for both single- and multi-parameter problems [Di Persio and Honchar, 2016]. An example of feature-dimension time series convolution is presented in Figure 5.

Dropout layers, first proposed by Hinton et al. [2012], improve classification results by preventing complex co-adaptations of the training data. On each introduction of a training sample, hidden units are randomly omitted, according to a probability distribution, thus “dropping out” the unit activations from the information flow. As they may not be present, this means hidden units cannot rely on the presence of any other hidden unit at any time, making the network more robust as it cannot depend on any single passed value.

The probability of dropping out any one unit is predefined; Hinton et al. [2012] proposes a dropout threshold of 0.5. This means that generally only half of the units are present at any iteration of the training, and thus even if they fully (over)fit into a given training sample, the entire network will not. Dropout can be introduced with any connection, for example, between layers, or inside the recurrent connections of an LSTM layer.

Attention is a mechanism that has been recently used in sentence classification, translation [Bahdanau et al., 2014], and generation [Graves, 2013]. An attention mechanism generates an output by focusing on relevant elements of the input. That is, the attention model gives weights to the elements of the input sequence based on both the location and the contents of the sequence, supporting the possibility that observations at specific spots could have a greater importance in determining the results. Thus, the attention model could be used to weight different words in a sentence to find relations between them [Zhou et al., 2016] or to weigh different time steps in a time series, for example, in speech recognition [Chorowski et al., 2015].

In this paper, we employ the attention layer proposed by Zhou et al. [2016] for sentence relation classification, with LOB data. Here, the steps are the timesteps of the LOB observations processed by the recurrent layer. In this model, the output representation $r$ is formed by a weighted sum of several output vectors:

where $H$ is the attention layer input matrix consisting of the recurrent layer’s output vectors $[h_{1},h_{2},...,h_{T}]$, and $H\in R^{d^{w}\times L}$, where $d^{w}$ is the dimension of the observation vectors. $w$ is a trained parameter vector and $w^{T}$ its transpose; $L$ is the length of the sequence [Zhou et al., 2016]. The $softmax$ is a normalized exponential function that squashes the inputs to output probability-like responses in the range $[0,1]$:

where the activation is calculated in an element-wise manner [Mikolov et al., 2015]. The final output of the attention layer is calculated from the representations with

Zhou et al. [2016] also includes a softmax dense layer, which takes the attention output $h^{*}$ to calculate the final classification result [Zhou et al., 2016].

In this work, the attention layer is connected directly into the unconvoluted input, followed by the convolution and LSTM layers. Additionally, in place of time steps, the attention model is applied on the feature dimension. That is, all features are weighted, and the same weight for a single feature is repeated and thus applied to all of the time steps within the sample. This allows for selecting the features that are most relevant in any given sample.

The neural networks were built using several Python libraries. The main library used was Keras, a high-level, open-source framework for building multilayer networks focused on enabling fast experimentation [Chollet and Others, 2015]. Keras, however, does not provide the network structure but rather an interface for building it. Thus, TensorFlow, an implementation for executing different machine learning algorithms, was used as the Keras backend. Tensorflow is a flexible system, allowing the utilization of graphics processing units for speeding up the computation [Abadi et al., 2015]. The Keras’ Model provides a simple framework to which layers can be added in a straightforward manner, and their connections to other layers can be specified. This allows the building of both simple sequential networks as well as more branching approaches. As Keras provides premade definitions for many different layer types, experimenting with different configurations is fairly simple.

The MLP network consisted of two leaky ReLu layers of 40 neurons each. The MLP network structure is presented in Figure 6. The CNN model for predicting stock price movements proposed by Tsantekidis et al. [2017b] is illustrated in Figure 7. It consists of eight layers. The first layer is a 2D convolution with 16 filters of size (4,40), followed by an 1D convolution with 16 four-long filters and a max pool of two. This is followed by 2 additional 1D convolutions with 32 size 3 filters, and one additional size 2 max pooling layer. Furthermore, there are two fully connected dense layers, the first one with 32 neurons and the following one with 3 neurons. The output layer is modified to contain only a single output neuron to act as a two-class classifier. Additionally, while the network was designed to use only the 40 pure limit order book data features, it was modified in size to test it with the extra features used in this research. However, the original 40-feature network was selected for further analysis due to better results. The differences may have been due to the 2D convolution, which mixes features in both time and feature axes.

Another network is the LSTM network for stock price prediction presented in Tsantekidis et al. [2017b]. The LSTM network structure is shown in Figure 8. The network consists of an LSTM layer with 40 hidden neurons followed by a fully connected Leaky ReLu unit defined in Maas et al. [2013].

The CNN-LSTM-Attention network is the most sophisticated model in this paper and it is designed to learn the most important patterns through feature and time domains for jump prediction and to optimally weight the different features to predict jumps. It is constructed as follows. The first layer connected after the input is the attention layer, composed of multiple Keras components: A regular dense layer with tanh activation is created with a weight for each time step, flattened to one dimension, to which softmax activation is further applied. This layer is repeated once for each step to apply the attention to full time steps. The dimensions are then switched to match the original input shape and merged together by multiplying the activations from the attention model and the input values from the original input layer. This gives each feature its own weight such that the same feature is weighted the same across all given time steps within a sample.

The resulting attention mechanism output is a matrix of the original input size, which is passed forward to a 1D convolutional layer with 32 size 5 filters. The convolution output is further processed with a max pool of size 2, and the max pooled activations are passed to an LSTM layer with 40 relu neurons. The LSTM also includes a dropout of 0.5 both inside regular and recurrent connections. After the LSTM, there is a regular dense fully connected layer of the same size and, finally, the singular output neuron with sigmoid activation. This means that the output is a single value in the range $[0,1]$, which is then rounded to obtain class prediction. The proposed network structure is illustrated in Figure 9.

The network performance was assessed with several metrics. The main target is F1 score, which is defined as the harmonic mean of precision and recall:

Recall is defined as

and precision as

where $tp$ is true positives, the number of jump samples correctly classified as jumps; $fn$ is false negatives, jumps incorrectly classified as negative samples; and $fp$ is false positives, negative samples incorrectly classified as jumps. Thus, recall is the portion of jumps classified as jumps, and precision is the portion of real jump samples in samples classified as jumps [Lipton et al., 2014]. High recall implies that a majority of jumps can be detected, whereas high precision means that jumps can be detected without also classifying many non-jump samples as jumps.

It should be noted that neither precision nor recall consider the number of true negatives. This also makes F1 independent of the ratio of accurately classified negatives, and instead focuses heavily on correctly classifying the positives. Thus, F1 provides a measure that is both non-linear and non-symmetric—F1 is commonly used in cases where the positive class is rare compared to the negatives [Lipton et al., 2014]. As the portion of jumps in the data is very small, F1 is a suitable metric. Successfully predicting no jump is also a less informative result than successfully predicting one given a relatively small number of false positives, as very good accuracy could be achieved by just always predicting that there will be no jump. Measuring the results with F1 also avoids this scenario, as correct negatives do not affect the score.

Another used metric is Cohen’s Kappa:

for which $p_{o}$ and $p_{c}$ can be calculated from the confusion matrix [Cohen, 1960]. As the Kappa also takes into account the agreement by chance, it can be seen as a more robust measurement for agreement. There is no single interpretation for what can be considered a good Kappa value, and thus it depends on the type of the problem analyzed. Fleiss et al. [2003, p. 604] presents intervals such that values greater than 0.75 signify excellent agreement, values above 0.40 a fair or good agreement, and values under 0.40 a poor agreement. Conversely, Landis and Koch [1977] suggest that values of 0.21-0.40 are already fair, 0.41-0.60 are moderate, 0.61-0.80 are substantial, and values above that are almost perfect.

Performance of the networks used is presented by averaging the scores across all stocks and sets in Table 5. Because of the unbalanced data, we consider F1 as the most appropriate performance measure, and therefore F1 values are in a bold font in the table. The table shows that of all the tested networks, the CNN–LSTM–Attention model achieves the highest average F1 for all samples (around 0.72). The second best network is the pure LSTM (0.69), followed by the CNN (0.66) and the CNN-LSTM-$v_{10}$, where no information other than the time of day (feature $v_{10}$) is used. Finally, the MLP achieved an average F1 of 0.53. All of the models clearly outperform a random classifier, for which F1 is 0.32.

Additionally, when comparing average F1s by stocks, according to Table 6, the CNN-LSTM-Attention model again performs the best. With all of the tested network models, the resulting F1s are above those of a random classifier. MLP is somewhat worse, possibly due to being unable to deal with fairly large time series input data without overlearning, but it still clearly better than random choice. The scores implies that at least a part of the jumps in the data are predictable with a reasonable level of confidence. Additionally, when interpreting the Kappa scores, the scores of both LSTM models can be seen as at least good if not very good. Also the CNN-LSTM-$v_{10}$ works surprisingly well, given that jumps are predicted using information on the time of day only. Therefore, LOB data can be said to be useful for predicting price jumps, especially for Apple, Facebook, and Microsoft, but sometimes it gives a rather marginal advantage (see Google and Intel in Table 6).

According to Tables 5 and 6, the most promising model is CNN-LSTM-Attention, and hence we zoom into the F1 scores of CNN-LSTM-Attention model over the stocks and data sets in Table 7. The average F1 for all CNN-LSTM-Attention model sets is 0.71, with some variation between both different time periods and different stocks. The variation is most likely due to different stock price and jump dynamics for different securities.³³3In Table 7. the scores are first calculated individually for each set and stock and then averaged over the sets. As the number of samples is not constant across the sets, the average of individually calculated set scores is not exactly equal to the score calculated directly across all samples. Because of this, there are minor deviations between the values presented in Tables 5, 6 and 7. As true negatives are not accounted for in the calculation of F1, and having less jumps to detect directly lowers precision if the ratio of detected jumps remains the same, fewer jumps to detect tends to cause a lower score unless the amount of jumps that are easily predictable remains the same. Intel (INTC) jumps were predicted the best (0.78 F1) while Microsoft had the worst score, with a difference of around ten percentage points. The corresponding results for other models, the LSTM model proposed in Tsantekidis et al. [2017a], the convolutional model proposed in Tsantekidis et al. [2017b], and a regular two-layer MLP network, are provided in the Appendix (Tables 8, 9, 10).

Even though the attention model was originally proposed in Zhou et al. [2016] for use in the time dimension, it can also be used to highlight important features. That is, instead of focusing on time steps where interesting things occur, the network focuses on features that are particularly interesting at that time point. This was suitable due to the large number of features used.

Four samples were selected for qualitative analysis of the attention mechanism through the activations of the layer created for a given sample: one true positive and true negative as well as one false positive and false negative. As expected, all of these samples placed some importance on the time feature $v_{10}$ (see Table 4), as its feasibility had already been studied while developing the network. The attention was checked by comparing the attention activations from the unmerged layer. Other than that, the attention was given to quite different features between the four samples.

Both true and false positives included some volumes from the top levels of the order book from the basic set ($v_{1}$) of the features. Still, neither included any price information from the books in the attention, even though the quantities were regarded as important. Both also included several derivatives from the set of features $v_{6}$; the true positive had only quantities from both sides, whereas the false positive also included ask prices. Interestingly, the real positive sample also focused on a single, level 4 ask price difference from feature set $v_{3}$, none other of which were featured in any other inspected samples.

The negative samples were fairly different from the positives, although both had focused on the time feature as well as some derivative from feature set $v_{6}$. However, the derivatives of the negative samples are purely quantity derivatives for ask or bid volumes. The false negative and true negative were also fairly similar to each other in regards to attention, which makes sense since the end classification result was the same. Both focused on the basic set for both ask and bid as well as quantity and price. They also focused on several mid prices, which was not the case with either positive sample. Interestingly, in addition to these, the true negative sample had also selected cancel intensities for both ask and bid.

For all of the inspected samples, the focus was clearly on several specific feature sets, with some features not included in the attentions at all. However, ask and bid values were fairly equally present as well as values from across the 10 levels of the order books. Quantity-dependent values were regarded as especially important in multiple feature sets, which indicates the relation between liquidity and jumps [see Siikanen et al., 2017b, 2017, for the relation between order book liquidity and news announcements]. Still, the full test set was not inspected, so even though this implies some relevance for the values to which attention was paid, the remaining ones would need to be thoroughly inspected to draw conclusions regarding their usefulness in jump prediction.

This study focused on a two-class prediction problem, where the class information related to whether or not the jump statistic would exceed a certain threshold within the next minute independently of the direction of the jump. However, the problem can also be formulated as a three-class classification problem, where $c_{0}$ is no jump, $c_{1}$ is an upwards jump, and $c_{2}$ is a downwards jump. This was done with the same model and the same input data, but the output layer was changed to a three-neuron block. Still, this formulation proved much more challenging than only predicting the arrical of a jump, and the results did not significantly differ from random selection with regard to the jump direction. Results for differentiating between the jump classes $c_{1}$ and $c_{2}$ with the CNN-LSTM-Attention network are presented in Table 12 in Appendix.

The better predictability of the arrival of a jump (than the sign of a jump) is consistent with the literature, which has provided evidence about the relation between jumps and news announcements and analysts’ recommendations, whose arrival time, but not direction, is often predictable. Lee [2012], Bradley et al. [2014] This result is also consistent with the definition of the stock price model with jumps: the jump term consists of two separate parts, the counting process of the occurrence of a jump and the jump size, which is independent and identically distributed . Thus, it is possible that the counting process is predictable, while the size and thus the direction of the jump is not. However, this study did not focus on such a prediction case, and so it may be possible to also predict the direction of the jump with some accuracy, albeit likely much lower than the occurrence of the jump, which may even be known beforehand due to a prescheduled release of information regarding the stock.

Another possible direction for jump prediction is to not only predict jumps but to predict the jump statistic for each time step. This means that the prediction would be a regression problem in which the jumps are only implied by the statistics that are above the threshold by interpreting the output of the network. However, this method requires more accurate predictions of price movements considering the non-jump price process $\sigma(t)dW(t)$, and thus it also requires the prediction of the stock price process. Of course, the task can be made easier by using only the absolute value of the jump statistic, disregarding the direction of the movement. Even so, this task was much more difficult than simple jump prediction, at least with the tested methods. This is most likely due to the combination of more exact prediction requirements of a regression task, the limitations of the chosen methods, and the need to predict the magnitude of the normal stock price process to get the correct statistics in the samples without jumps. This question will be addressed further in our future research.