Volatility-inspired $\sigma$-LSTM cell

The structure of noise or errors in regression models is usually subtle and taken as an ansatz to use different mathematical frameworks. E.g. in the case of linear regression models $\mathbf{y}=\mathbf{X}\beta+\epsilon$, where $\mathbf{y}\in\mathbb{R}^{n}$ represent response variable, unobservable parameters $\beta\in\mathbb{R}^{K}$ and non-random explanatory variable $\mathbf{X}\in\mathbb{R}^{n\times K}$, while $\epsilon$ represents the noise or error. When the errors $\epsilon_{i}$ are homoscedastic i.e. $Var[\epsilon_{i}]=\sigma^{2}$ and serially uncorrelated i.e. $Cov[\epsilon_{i},\epsilon_{j}]=0$ by the Gauss–Markov theorem [1] the ordinary least squares is having the lowest sampling variance within the class of linear unbiased estimators. In econometrics, special interpretation and attention are given to the error structure [2].

In this paper, we focus on the problem of volatility of asset returns, which are well known to be heteroscedastic in nature. Volatility is associated with the risk and amplitude of price fluctuations.

Some models characterize the volatility from a conditional process perspective, for example, the autoregressive conditional heteroskedasticity (ARCH) model [3] and the generalized autoregressive conditional heteroskedasticity (GARCH) model [4]. The idea of the conditional process approach is the possibility of using a conditional variance that varies over time while the unconditional variance remains relatively constant.

On the other hand, researchers have recently focused on using realized volatility (RV) to build forecasting models due to the wide availability of high-frequency financial data. For example, the Heterogeneous Autoregression (HAR-RV) model [5] is widely used in the literature due to consistently good predictive performance and simple methods to estimate it.

Recently, different recurrent NN units like LSTM [6], GRU [7], SRU [8] have demonstrated high performance on forecasting tasks for time-series data. More specifically, LSTM is the particular architecture that design improves the model’s performance overall and especially in the volatility prediction task [9, 10].

Gated recurrent units (GRU) [7] is an enhanced LSTM architecture that improves the fitting process by eliminating the cell state. In addition, the Statistical Recurrent Unit (SRU) [8] was introduced, which can infer long-term dependencies from data by using simple moving averages of summary statistics and has multiple proxies of the past with simple linear combinations.

Our work investigated and analyzed how NN can learn to capture the temporal structure of realized volatility. We are interested in how we could add the structure of long and short-term volatility effects to the LSTM. We introduce a modified LSTM cell that we call $\sigma$-LSTM to match these needs. For this reason, we extend the equation system of the LSTM cell, which reflects the inductive bias of GARCH-like structure and HAR-RV effects of volatility, which allows easier learning of volatility by maximum likelihood estimation.

Recently, several studies [11, 12] have explored the heteroskedasticity of returns with recurrent NN architectures, but not by means of modified LSTM cell. In [11], authors have proposed the RECH model, where $\omega$-constant of the GARCH process is modeled by a particular RNN model.

In [12], authors have proposed the combination of the Stochastic Volatility (SV) model and Statistical Recurrent Unit (SRU). The idea is that the SRU captures the long-term memory effects and auto-dependence of the volatility. However, the SRU is modeling the deterministic dynamics of the hidden states in the SR-SV model.

We investigated the original Long short-term memory cell, proposed $\sigma$-LSTM with a particular loss function for realized volatility forecasting tasks, and compared the predictive ability with widely used HAR-RV, GARCH(1,1) models.

The remaining paper is organized as follows. Section II provides mathematical motivation and a formal definition of $\sigma$-LSTM cell. Section III describes how the experiment and its results. Finally, in Section IV, we provide a conclusion.

One important class of econometric models are GARCH family models [3, 4]:

$$r_{t}=\mu_{t}+\sigma_{t}\varepsilon_{t},\text{ }\sigma_{t}\varepsilon_{t}\sim\mathcal{N}(0,\sigma_{t}^{2})$$

(1)

where the conditional variance [4, 3] has the autoregressive structure:

$$\sigma_{\mathrm{t}}^{2}=\omega+\alpha*\mathrm{r}_{\mathrm{t}-1}^{2}+\beta*\sigma_{\mathrm{t}-1}^{2}.$$

(2)

Number extensions with different functional dependence (see Table 1) have been proposed like eGARCH [13], cGARCH [14], TGARCH [15], GJR-GARCH [16] and others.

The heterogeneous Autoregression Realized Volatility (HAR-RV) model introduced by [5] assumes that agents’ behavior in financial markets, which differ in their perception of volatility depending on their investment horizons and are divided into short-term, medium-term, and long-term. Heterogeneous structures in financial markets are based on the heterogeneous market hypothesis presented by [17]. Participants’ decisions refer to different time horizons that perceive and respond to different types of volatility. A memory of each component decreases with a particular time constant.

The HAR-RV model is an additive cascade of partial volatilities generated at different time horizons that follows an autoregressive process [5]. The HAR-RV approach is one more stable and accurate estimate for Realized Volatility [18] at the 3 different horizons, where $RV_{t}^{(d)},RV_{t}^{(w)}$, and $RV_{t}^{(m)}$ are respectively the daily, weekly, and monthly observed realized volatilities.

$$\begin{cases}\tilde{\sigma}_{t+1m}^{(m)}=c^{(m)}+\phi^{(m)}RV_{t}^{(m)}+\tilde{\omega}_{t+1m}^{(m)}\\ \tilde{\sigma}_{t+1w}^{(w)}=c^{(w)}+\phi^{(w)}RV_{t}^{(w)}+\gamma^{(w)}\mathbb{E}_{t}\left[\tilde{\sigma}_{t+1m}^{(m)}\right]+\tilde{\omega}_{t+1w^{\prime}}^{(w)}\\ \tilde{\sigma}_{t+1d}^{(d)}=c^{(d)}+\phi^{(d)}RV_{t}^{(d)}+\gamma^{(d)}\mathbb{E}_{t}\left[\tilde{\sigma}_{t+1w}^{(w)}\right]+\tilde{\omega}_{t+1d}^{(d)}\end{cases}$$

(3)

where $c^{(m)}$ - the constant and $\tilde{\omega}_{t+1m}^{(m)}$ is an innovation that is simultaneously and consistently independent with a mean zero for monthly aggregation, and $\phi$ represents the wight in a particular cascade.

Motivated by GARCH structure and long and short-term volatility effect of HAR-RV model, we propose to model the return assets $\mathrm{r}_{\mathrm{t}}=\phi(\mathrm{r}_{\mathrm{t}-1},...,\mathrm{r}_{\mathrm{t}-p})$, where $\phi(.)$ is a differentiable non-linear function. In particular a modified long short-term memory (LSTM) cell [6] that should capture long and short-term volatility. The inputs to out modified LSTM cell $x_{t}=r_{t}$ are directly returns and the outputs are $\hat{r}_{t}$ and $\hat{\sigma}^{2}_{t}$. The cell has directly the hidden representation $h_{t}$ for short-term memory and long-term $C_{t}$ volatility memory component. The updates rules of $\sigma-$LSTM are the following:

$$f_{t}=\sigma\left(W_{f}\cdot\left[h_{t-1},x_{t}\right]+b_{f}\right),$$

(4)

$$i_{t}=\sigma\left(W_{i}\cdot\left[h_{t-1},x_{t}\right]+b_{i}\right),$$

(5)

$$\tilde{C}_{t}=\tanh\left(W_{C}\cdot\left[h_{t-1},x_{t}\right]+b_{C}\right),$$

(6)

$$C_{t}=f_{t}*C_{t-1}+i_{t}*\tilde{C}_{t},$$

(7)

$$o_{t}={\mathcal{N}}(0,W_{o}[{C}_{t}^{2}]),$$

(8)

$$h_{t}=o_{t}*\phi\left(C_{t}\right).$$

(9)

Finally, the output return $\hat{r}_{t}$ and estimated volatility $\hat{\sigma}_{t}$ is:

$$\hat{r}_{t}=W_{h}\cdot h_{t}$$

(10)

$$\hat{\sigma}^{2}_{t}={\langle C_{t}\rangle}^{2},$$

(11)

where $\langle.\rangle$ is the mean operator and both $\hat{r}_{t}$ and $\hat{\sigma}^{2}_{t}$ are scalar values.

We implement the custom loss function as the likelihood of observed returns with estimated volatilities.

$$\mathcal{L}=\sum_{t=1}^{m}\left[-\ln\left(\hat{\sigma}_{t}^{2}\right)-\frac{r_{t}^{2}}{\hat{\sigma}_{t}^{2}}\right].$$

(12)

This study investigates how proposed $\sigma$-LSTM could estimate and predict realized volatility on different market structures, particularly stocks, indexes, and cryptocurrency data. We consider Apple inc. stock, the S&P 500 index, and Bitcoin-USD.

We calculate RV based on minutes-based price observations for daily aggregation. Returns are calculated on the daily close price. As a best practice, we divided the dataset into three parts: training, validation, and test. The validation and the test sample are equivalent to 200 points.

Mean Squared Error (MSE) measures averaged squared difference values between the predictions and the target. The power of 2 in this metric prevents neutralizing positive and negative deviations, which minimizes the distance between actual and calculated values. Root Mean Squared Error (RMSE) is the square root of MSE. The square root is introduced to scale error is the same as the target scale.

To find the best configuration of NN is necessary to conduct multiple experiments with different hyperparameters [19]. We have results for training launches and results for the validation dataset; the next step is to select promising hyperparameters RMSE metrics appropriately. Standardization is highly recommended before training RNNs and can improve the efficiency of training models. We normalized input data from 0 to 1 by min-max scale.

In our study, we ask the following questions. First, how could the proposed $\sigma$-LSTM cell with GARCH-like structure and long and short-term volatility effect of the HAR-RV model capture the long and short-term volatility effects?

We performed standard accuracy measures for the one-step-ahead prediction using RMSE metrics for 200 data points, Table  3. As a result, $\sigma$-LSTM shows the best performance for RMSE for the out-of-sample result for the S&P 500 index and Apple Inc. stock data sets.

However, it should be noted that in the case of cryptocurrency, the prediction error of HAR-RV was at the same level as $\sigma$-LSTM. In our experiment, $C_{t}$ of the original LSTM does not provide sufficient results.

This work introduces a special $\sigma$-LSTM cell to investigate whether the use of stylized facts or ”physics-informed” inductive bias [20] i.e., GARCH and HAR-RV volatility structure could help to improve the performance of the NN for the volatility prediction task. We do not use the Recurrent LSTM unit as a black box but rather design a sub-component to represent a long-short volatility memory and a stochastic part.

We add particular loss functions for the $\sigma$-LSTM. As a result, we show that $\sigma$-LSTM could outperform well-known models in this field, such as a strong baseline HAR-RV and regular LSTM cell. We will investigate more advanced loss functions in future work that could allow faster learning convergence and accuracy.