A Generic Methodology for the Statistically Uniform & Comparable Evaluation of Automated Trading Platform Components

This section breaks down how data is handled, processed and used in the context of financial analysis. This includes the pre-processing phase pattern extraction, trading, evaluation methods and features.

Rule-based methods have seen mixed successes in the context of financial time series [65]. In part, this is due to the wide usage of quite simple rules [16]. What may instead be more desirable is to use patterns for trading [54]. These patterns can perform with high levels of profitability and can be automatically extracted from data, which is in itself, quite desirable. Despite early criticisms [72], not only have these patterns been shown to be particularly effective [16], they might be particularly useful in the machine learning setting.

To evaluate the effectiveness of the trading strategy the Sharpe Ratio can be used. This is a measure for assessing the performance of an investment or trading approach and can be computed as the following:

where $R_{p}$ & $R_{f}$ correspond to the portfolio and risk free returns, respectively, and $\sigma_{p}$ is a standard deviation of the portfolio return. While the equation gives a good intuition of the measure, in practice its annualised version is often computed. It assumes that daily returns follow the Wiener process distribution, hence to obtain annualised values, the daily Sharpe values are multiplied by $\sqrt{252}$ - the annual number of trading days. It should be noted that such an approach might overestimate the resulting Sharpe ratios as returns auto-correlations might be present, violating the original assumption [55].

Automating the trading through machine learning efforts is the identification of a market state in which a trade is profitable, and automatically performs the transaction at that stage. Such a system is normally tailored for a specific instrument, analysing unique patterns to improve the characterisation. One such effort focusing on Forex markets is, Dempster & Leemans [24]. In this work, a technique using reinforcement learning is proposed, to learn market behaviours. Reinforcement learning is an area of machine learning concerned with how software agents ought to take actions in an environment to maximise the notion of cumulative reward. This is achieved by assigning positive rewards to desired actions and negative rewards to undesired actions, leading to an optimisation towards actions that increment rewards. In financial markets, this naturally corresponds to profitable trades. Using this approach, the authors can characterise when to trade, analyse associated risks, and automatically make decisions based on these factors.

To test trading, strategy evaluation is performed to assess profitability. Whilst it is possible to do so on trading live market, it is more favourable to do so on the historical data first to get the results on a large timeline and avoid any financial risks. The notion is that a strategy that would have worked poorly in the past will probably work poorly in the future, and conversely. But as you can see, a key part of backtesting is the weak assumption that past performance predicts future performance [39, 5, 46].

Several approaches exist to perform backtesting and different things can be assessed. Beyond trading strategies testing, backtesting can show how positions are opened and the likelihood of certain scenarios taking place within a trading period [39]. The more common approach is to implement the backtesting within the trading software platform, as this has the advantage that the same code as live trading can be used. Almost all platforms allow for simulations of historical data, although one operates on data pre-processed by the proprietary software platform, making the pipeline partially opaque.

For more flexibility, one can implement their backtesting system in languages such as C# or Python [62]. This specific approach enables the same code pipeline that is training the classifier to also test the data, allowing for much smoother testing. However, some limitation of this type of backtesting is that it doesn’t include a lack of order book queues as well as the latency simulation, both of which are present in the live trading setting. Consequently, slippages (the difference between where the order is submitted by the algorithm and the actual market entry/exit price) cannot be modelled accurately. Moreover, the modelling approaches and assumptions vary between the backtesting engines, hampering comparability. Nevertheless, backtesting is a commonly adopted practise among the financial market practitioners, and being used with care, gives a good estimation of the ATP quality.

It is crucial to highlight that backtesting is only possible when all the components of the ATP are in place, making it barely suitable for assessing individual components separately.

There is a wide range of techniques for machine learning, ranging from very simple such as regression to techniques used for deep learning such as neural networks. Consequently, it’s important to choose an algorithm that is most suited to the problem one wishes to tackle. However, there is no uniform rule on how to make the most optimal choice and it is often empirically driven [51].

In ATPs one of the possible roles of machine learning is to identify situations in which it is profitable to trade, depending on the strategy, for example, if using a flat strategy the intent is to identify when the market is flat. In these circumstances, and due to the potentially high financial implications of false negatives understanding the prediction is key. Understanding the prediction involves the process of being able to understand why the algorithm made this decision. This is a non-trivial issue and something very difficult to do for a wide range of techniques, neural networks being the prime example (although current advances are being made [87, 31] .

To evaluate the results of our research, answer the research questions, add explainability and increase the reproducibility of our study, we make use of several statistical techniques.

Below, we overview several works on trading pattern identification, design and trading from the methodological perspective.

In the work by Chen & Chen [17], a pattern recognition model is proposed and evaluated for its performance. The performance is evaluated in two phases, model training and model testing. In the first phase, the authors perform optimisation of the pattern parameters to the current market conditions, in the second - perform pattern recognition and backtesting. The same approach is taken in the work by Parracho et al [66]. In the work by Cervello et al [16], the authors thoroughly assess the proposed approach for trading pattern recognition based on ATP profitability for a range of take profits and stop losses.

An elegant approach to computational finance is proposed by Canelas et al [13]. It uses a symbolic approximation of the financial time series together with a genetic algorithm to find patterns and trading rules from the approximation. The results are reported in measures specific to this particular study, which do not allow direct comparison to other studies.

A statistical approach is taken in a paper published two decades ago [54]. Namely, the authors introduce the null hypothesis in a form of the performance of a random-choice model and run a t-test on daily ATP profits to test the proposed method. This is a good first step in a statistically sound approach to financial markets. However, it falls short of justifying the choices within the methodology, reporting statistical results, and reporting measures that would allow the comparability. Moreover, the way the null hypothesis is formulated and evaluated limits the ability of a practitioner to test a particular ATP component, as discussed in Section 2.3.

Concluding, there is no uniform dataset for reporting the results, and neither there is a uniform trading strategy. Moreover, all the considered studies assess multiple ATP components at once. The lack of uniformity and comparability highlights the demand for the current study.

In the study, we use S&P500 E-mini CME futures contracts ES(H-Z)2017, ES(H-Z)2018 and ES(H-U)2019. Which correspond to ES futures contracts with expiration in March (H), June (M), September (U) and December (Z). We operate on ticks data which includes Time&Sales records statistics, namely: bid and ask volumes and numbers of trades, as well as the largest trade volumes per tick. We consider a tick to incorporate all the market events between the two price changes by a minimum price step. For the considered financial instrument the tick size is $0.25.

We sample the contract data to the active trading periods by considering only the nearest expiring contracts with the conventionally accepted rollover dates. The samples end on the second Thursday of the expiration month - on that day the active trading is usually transferred to the following contract. This decision ensures the highest liquidity, and, due to the double-auction nature of the financial markets, stable minimum bid-ask spreads [44].

In the current study, we consider the two simplest scenarios of the price behaviour after it reaches the local extremum - reversal and extremum crossing. When labelling the entries, we require up to 15 ticks of price movement as a reversal (or rebound) and only 3 ticks for the extremum crossing. The labelling approach allows us to study a range of configurations of the reversals and investigate how the configurations affect the performance of the models. At the same time, these ranges are well within the boundaries of the intraday price changes.

An essential part of the proposed automated trading system is the detection of the price extrema. The detection is performed on a sliding window of the streamed ticks with a window size of 500. We capture peaks with widths from 100 to 400. The selected widths range serves three purposes: i) ensures that we do not consider high-frequency trading (HFT) scenarios which require more modelling assumptions and different backtesting engines; ii) allows us to stay in intraday trading time frames and have a large enough number of trades for analysis; iii) makes the price level feature values comparable across many of the entries.

To incorporate machine learning into the automated trading system, we design a binary classification task, where the labels correspond to price reversals (positives) and crossings (negatives). Due to the labelling design, we are more flexible with taking profits and stopping losses when trading reversals (up to 15 ticks versus 3 ticks) - this explains their positive labelling.

To perform the extrema classification, we obtain two types of features: i) designed from the price level ticks (called price level (PL) features), and ii) obtained from the ticks right before the extremum is approached (called market shift (MS) features) as we illustrate in Fig. 3. We believe (and statistically test it) that it is beneficial to perform the 2-step collection since the PL features contain properties of the extremum, and the MS features allow us to spot any market changes that happened between the time when the extremum was formed and the time we are trading it.

Considering different extrema widths, varying dimensionality of the data does not allow to use it directly for classification - most of the algorithms take fixed-dimensional vectors as input. We ensure the fixed dimensionality of a classifier input by aggregating per-tick features by price. We perform the aggregation for the price range of 10 ticks below (or above in case of a minimum) the extremum. This price range is flexible - 10 ticks are often not available within ticks associated with the price level (red dashed rectangle in Fig. 3) in this case we fill the empty price features with zeros. We assume that the further the price from the extremum the less information relevant for the classification it contains. Considering the intraday volatility of ES, we expect that the information beyond 10 ticks from the extremum is unlikely to improve the predictions. If one considers larger time frames (peak widths), this number might need to increase.

PL features are obtained from per-tick features by grouping by price with the sum, max or count statistics. For instance: if one is considering volumes, it is reasonable to sum all the aggressive buyers and sellers before comparing them. Of course, one can also compute mean or consider max and min volumes per tick. If following this line of reasoning, the feature space can be increased to very large dimensions. We empirically choose a feature space described in Tables 1 and 2 for Price Level and Market Shift components, respectively. Defining the feature space we aim to make the feature selection step computationally feasible. Too large feature space might be also impractical from the optimisation point of view, especially if the features are correlated.

To track the market changes, for the MS feature component we use 237 and 21 ticks and compare statistics obtained from these two periods. Non-round numbers help avoid interference with the majority of manual market participants who use round numbers [23]. We also choose the values to be comparable to our expected trading time frames. No optimisation was made on them. We obtain the MS features being 2 ticks away from the price level to ensure that our modelling does not lead to any time-related bias where one cannot physically send the order fast enough to be executed on time.

After the features are designed and extracted, the classification can be performed. As a classifier, we choose the CatBoost estimator. We feel that CatBoost is a good fit for the task since it is resistant to over-fitting, stable in terms of parameter tuning, efficient and one of the best-performing boosting algorithms [67]. Finally, being based on decision trees, it is capable of correctly processing zero-padded feature values when no data at price is available. Other types of estimators might be comparable in one of the aspects and require much more focus in the other ones. For instance, operating on a relatively small number of tabular data entries with a temporal component, we withhold from applying deep learning models due to evidence suggesting that boosting trees works better off-the-shelf in the considered setting [10].

In this study we use precision as the main scoring function (S):

where TP is is the number of true positives and FP the number of false positives. All the statistical tests are run on the precision scores of the samples. This was chosen as the main metric since by design every FP leads to losses, and a false negative (FN) means only a lost trading opportunity. To give a more comprehensive view of the model performance, we build confusion matrices, and compute F1 scores, PR-AUC (precision-recall area under the curve) and ROC-AUC (receiver-operating characteristic area under the curve) metrics. We report model performances for the 2-step feature extraction approach as well as for each of the feature extraction steps separately.

To avoid large bias in the base classifier probability, we introduce balanced class weights into the model. The weights are inversely proportional to the number of entries per class. The contracts for training and testing periods are selected sequentially - training is done on the active trading phase of contract $N$, testing - on $N+1$, for $N\in[0,B-1]$, where $B$ is the number of contracts considered in the study.

We apply a commonly accepted ML community procedure for input feature selection and model parameter tuning [52]. Firstly, we perform the feature selection step using a Recursive Feature Elimination with the cross-validation (RFECV) method. The method is based on the gradual removal of features from the model input starting from the least important ones (based on the model’s feature importance) and measuring the performance on a cross-validation dataset. In the current study on each RFECV step, we remove 10% of the least important features. Cross-validation allows robust assessment of how the model performance generalises into unseen data. Since we operate on the time series, we use time series splits for cross-validation to avoid the look-ahead bias. For the feature selection, the model parameters are left default, the only configuration we adjust is class labels balancing as our data is imbalanced. Secondly, we optimise the parameters of the model in a grid-search fashion. Even though CatBoost has a very wide range of parameters that can be optimised, we choose the parameters common for boosting tree models for the sake of feasibility of the optimisation and leaving the possibility of comparing the optimisation behaviour to the other boosting algorithms. The following parameters are optimised: 1) Number of iterations, 2) Maximum depth of trees, 3) has_time parameter set to True or False, and 4) L2 regularisation. For the parameter optimisation, we use a cross-validation dataset as well. We perform training and cross-validation within a single contract and the backtesting of the strategy on the subsequent one to ensure the relevance of the optimised model.

Here we formalise the research questions by proposing null and alternative hypotheses, suggesting statistical tests for validating them, as well as highlighting the importance of the effect sizes.

The effect sizes are widely used in empirical studies in social, medical and psychological sciences [53]. They are a practical tool allowing to quantify the effect of the treatment (or a method) in comparison to a control sample. Moreover, they allow the generalisation of the findings to the whole population (unseen data in our case). Finally, effect sizes can be compared across studies [53]. We believe that introduction of the effect sizes into the financial markets domain contributes to the research reproducibility and comparability.

In the current study, we report Hedge’s g_av - an unbiased measure designed for paired data. In the supplementary data, the effect sizes are provided in a form of forest plots with .95 confidence intervals (CIs), representing the range where the effect size for the population might be found with the .95 probability. We correct the confidence intervals for multiple comparisons by applying Bonferroni corrections.

When testing the hypotheses, the samples consist of the test precisions on the considered contracts, leading to equal sample sizes in both groups, and entries are paired as the same underlying data is used. Comparing a small number of paired entries and being unsure about the normality of their distribution, we take a conservative approach and for hypothesis testing use the single-tailed Wilcoxon signed-rank test. This test is a non-parametric paired difference test, which is used as an alternative to a t-test when the data does not fulfil the assumptions required for the parametric statistics. When reporting the test outcomes, we support them with the statistics of the compared groups. Namely, we communicate standard deviations, means and medians.

We set the significance level of the study to $\alpha=.05$. Also, we account for multiple comparisons by applying Bonferroni corrections inside of each experiment family [15]. We consider research questions as separate experiment families.

We perform the model analysis in an exploratory fashion - no research questions and hypotheses are stated in advance. Hence, the outcomes of the analysis might require additional formal statistical assessment. We use SHAP local explanations to understand how models end up with the particular outputs. Through the decision plot visualisations, we aim to find common decision paths across entries as well as informally compare models with small and large numbers of features. The reproducibility package contains the code snippets as well as the trained models, which allow repeating the experiments for all the models and contracts used in the study.

The trading strategy is defined based on our definition of the crossed and rebounded price levels, and schematically illustrated in Fig. 4. It is a flat market strategy, where we expect a price reversal from the price level. Backtrader Python package ¹¹1Available at: https://www.backtrader.com/ is used for backtesting the strategy. Backtrader does not allow taking bid-ask spreads into account, that is why we are minimising its effects by excluding HFT trading opportunities (by limiting peak widths) and limiting ourselves to the actively traded contracts only. Since ES is a very liquid trading instrument, its bid-ask spreads are usually 1 tick, which however does not always hold during extraordinary market events, scheduled news, or session starts and ends. We additionally address the impact of spreads as well as order queues in the Discussion section.

In our backtests, we evaluate the performance of the models with different rebound configurations and fixed take-profit parameters, and varying take-profits with a fixed rebound configuration to better understand the impact of both variables on the simulated trading performance.

In the current subsection, we continue formalising the research questions by proposing the research hypotheses. The hypotheses are aimed to support the findings of the paper, making them easier to communicate. For both research questions, we run the statistical tests on the precision metric, as justified above.

In the current study, we encode the hypotheses in the following way: H_0X and H_1X correspond to null and alternative hypotheses, respectively, for research question X.

The considered data sample and processed datasets are provided in Tab. 3. We provide the numbers of ticks per contract in the first columns. The contracts are sorted by the expiration date from top to bottom ascendingly. The number of ticks changes non-monotonically - while the overall trend is rising, the maximum number of ticks is observed for the ESZ2018 contract. And the largest change is observed between ESZ2017 & ESH2018.

We perform the whole study on 3 different rebound configurations: 7, 11 and 15 ticks price movement required for the positive labelling. We communicate the numbers of positively labelled and total numbers of entries used in the classification tasks in Table 3. As one can see, the numbers of the extracted extrema do not strictly follow the linear relation with the numbers of ticks per contract. Considering the numbers of positively labelled entries, the numbers decrease for the larger rebound sizes.

Here we show the exploratory analysis of the trained models. We choose two models trained on the same contract but with different labelling configurations. Namely, we report the analysis of the models trained on the ESH2019 contract, with rebounds 7 and 11. The choice is motivated by very different numbers of features after the feature selection step.

The core analysis is done on the decision plots, provided in Fig. 6. Since no pattern was observed when plotting the whole sample, we illustrate a random subsample of 100 entries from the top 1000 entries sorted by the per-feature contribution. There are 29 features in the model trained on the rebound 11 configuration, and 8 features in the one trained on the rebound 7 labels. 7 features are present in both models: PL23, MS6_80, MS0, PL24, MS2, MS6_20, MS6_200. Contributions from the features in the rebound 7 model are generally larger, also, the overall confidence of the model was higher. We noticed that the most impactful features were coming from the Market Shift features. In Figure 6 one can see two general decision patterns: one ending up at around 0.12-0.2 output probability and another one consisting of several misclassified entries ending up at around 0.8 output probability. In the first decision path, most of the MS features contributed consistently towards the negative class, however, PL23 and PL2 often pushed the probability in the opposite direction. In the second decision path, PL23 had the most persistent effect for the positive class, which was opposed by MS6_80 in some cases and got almost no contribution from MS0 and MS6_200 features.

Further analysis of ESH2019, rebound 11 configuration (omitted for brevity) showed that there was a skew in the output probabilities towards the negative class. Contributions from the PL features were less pronounced than for the rebound 7 model - the top 8 features belonged to the MS feature extraction step. There was no obvious decision path with misclassified entries. At the same time, we saw strong contributions toward negative outputs from MS6_20 and MS6_40.

In the current section, we report the cumulative profits for the 3 different labelling configurations in Fig. S6. In addition to the cumulative profits, we report annualised Sharpe ratios with a 5% risk-free annual profit. These results are reported for 15 ticks take profit, as shown in Fig. 4. There was a descending trend in the Sharpe ratios across all the configurations. However, the behaviour of rebound 15 differed from the rest by performing worse until Oct 2018, then had a profitable period which was not observed for the other configurations. We also ran experiments with altered take profits for 15 ticks rebound labelling and found out that decreasing take profit led to slightly worse cumulative profits and comparable Sharpe ratios (Figure 7). It seems that the profitability period after Oct 2018 relied on the take profit value and vanished if the take profit is reduced.

The numbers of price levels and ticks per contract followed the same trend (Table 3). Since the number of peaks was proportional to the number of ticks, we can say that the mean peak density is preserved over time to a large extent. In this context, the peak pattern can be considered stationary and appears in various market conditions.

The precision improvement for the CatBoost over the no-information estimator varied a lot across contracts (Table 4). At the same time, the improvement was largely preserved across the labelling configurations. This might have been due to the original feature space, whose effectiveness relied a lot on the market state - in certain market states (and periods) the utility of the feature space dropped and the drop was consistent across the labelling configurations. Extensive research of the feature spaces would be necessary to make further claims. The overall performance of the models was weak on an absolute scale, however, it is comparable to the existing body of knowledge in the area of financial markets [25].

Effect sizes in RQ1 Table 5 were not significant. The significance here means that the chosen feature space with the model significantly contributed to the performance improvement with respect to the no-information model. A positive but insignificant effect size meant that there is an improvement that is limited to the considered sample and is unlikely to generalise to the unseen data (population).

Assessing the results of the statistical tests, we used the significance level corrected for the multiple comparisons. Tests run on the rebound 7 and 15 configurations resulted in significant p-values, whilst rebound 11 was insignificant (Table 5). Hence, we rejected the null hypothesis H₀₁ for the labelling configurations of 7 and 15 ticks. This outcome was not supported by the effect sizes. This divergence between the test and effect sizes indicated that the proposed trading pattern cannot be efficiently used in the machine learning setting with the considered feature space. Interestingly, standard deviations differ consistently between the compared groups across the configurations. While no-information model performance depends solely on the fraction of the positively labelled entries, CatBoost performance additionally depends on the suitability of the feature space and model parameters - this explains higher standard deviations in the CatBoost case.

For the effect sizes in Table 5 we compared the 2-step feature extraction approach to its components - PL and MS. We did not see any significant effects showing supremacy of any of the approaches. Negative effect sizes in the case of the Market Shift component mean that in the considered sample MS performs better than the 2-step approach. This result did not generalise to the unseen data as its confidence intervals cross the 0-threshold. The possible explanation of the result is the much larger feature space of the 2-step approach (consisting of PL and MS features) than the MS compound. In case PL features were generally less useful than MS, which was empirically supported by our model analysis, this meant they might have had a negative impact during the feature selection process by introducing noise.

For assessing the statistical tests we used the significance level corrected for 6 comparisons. The p-values from Table 5 show that there was no significant outcome and the null hypothesis H₀₂ could not be rejected. While the 2-step approach did not bring any improvement to the pipeline, there is no evidence that it significantly harmed the performance either. It might have been the case that if PL features were designed differently, the method could have benefited from them. In this study, we withheld from iteratively tweaking the feature space to avoid any loss of statistical power. We find this aspect interesting for future work, however, it would require increasing the sample size to be able to account for the increased number of comparisons.

In the simulated trading, we observed an interesting result - data labelling configuration had more impact on the profitability than the take profits (Figures 7,S6). We hypothesise that the reason for this is that the simplistic trading strategy is overused by the trading community in various configurations. In contrast, the labelling configuration is less straightforward and had more impact on profitability. Note that the obtained precision (Table 4) could not be directly related to the considered trading strategy as there might be multiple price levels extracted within the time interval of a single trade. Consequently, there might have been extrema which were not traded.

It is hard to expect consistent profitability considering the simplicity of the strategy and lack of optimisation of the feature space, however, even in the current setting one can see profitable episodes (Fig S6). The objective of the study was not to provide a ready-to-trade strategy, but rather to showcase the proposed methodology. We believe that the demonstrated approach is generalisable to other trading strategies.

The point we have highlighted in this study is that by looking at the machine learning model performance or simulated trading results on their own, one might be drawn into believing that the considered trading pattern and the feature extraction method might work. This, consequently, would lead to additional time, effort and financial risks until one discovers that the applications of the proposed pattern are limited.

On the other hand, using the proposed methodology, one could withhold further development at the point of computing the effect sizes and the confidence intervals. Not only does the proposed methodology optimises the whole automated trading platform component design approach, but it also allows uniform and comparable reporting of the component design.

As it was shown in Section 2.6, profitability is often used as a measure of success when evaluating trading patterns and trading strategies. The profitability of the price levels pattern might look promising for certain intervals of the considered timeline (Figure 7), especially considering that the trading fees are taken into account. However, being guided by the proposed methodology, it becomes clear that the considered pattern does not warrant attention in the considered setting. This is a common example of how the profitability metric itself might be misleading if reported on its own.

The proposed methodology is one of the many ways the financial markets can be studied empirically. We encourage practitioners to adjust our proposed methodology to their needs and setting while keeping the interpretability and comparability of the study in mind.

Depending on the data distributions, size of the data sample, etc., one might use different statistical tests and effect size measures without any loss of interpretability. For example, while it is advised to use Glass’s $\Delta$ in case of the significantly different standard deviations between groups, this measure does not have corrections for the paired data. Hence, in our experiment design, we choose to stick to the Hedge’s g_av. For the sake of completeness, we verified the results using Glass’s $\Delta$ - and 15 ticks rebound effect size becomes significant in RQ1.

In the backtesting we used the last trade price to define ticks, we do not take into account bid-ask spreads. In live trading, trades as executed by bid or ask price, depending on the direction of the trade. It leads to a hidden fee of the bid-ask spread per trade. This is crucial for intraday trading as average profits per trade often lay within a couple of ticks. Moreover, when modelling order executions, we do not consider per-tick volumes coming from aggressive buyers and sellers (bid and ask). It might be the case that for some ticks only aggressive buyers (sellers) were present, and our algorithm executed a long (short) limit order. This leads to uncertainty in opening positions - in reality, some of the profitable orders may have not been filled. At the same time, losing orders would always be executed.

Another limitation is that we do not model order queues, and, consequently cannot guarantee that our orders would have been filled if we submitted them live even if both bid and ask volumes were present in the tick. This is crucial for high-frequency trading (HFT), where thousands of trades are performed daily with tiny take-profits and stop-losses, but has less impact on the trade intervals considered in the study. Finally, there is an assumption that our entering the market does not change its state significantly. We believe it is a valid assumption considering the liquidity of S&P E-mini futures.

We provide a systematic approach to the evaluation of automated trading platform components. While large market participants have internal evaluation procedures, we believe that our research could support various existing pipelines. Considering the state of the matter with the lack of code and data publishing in the field, we are confident that the demonstrated approach can be used towards improving the generalisability and reproducibility of research. Specific methods like extrema classification and 2-step feature extraction can serve as a baseline for the effect sizes observed in the field.

While the proposed methodology is in an initial stage for the field of financial markets, we believe that it is flexible enough to be incorporated into a python package for trading pattern design. That would significantly boost its adoption by practitioners.

In the current study, we evaluate the utility of the price levels trading pattern. The next step would be to propose a more holistic pattern. We would aim to extend the market properties to volumes and volume profiles.

Another direction is to assess the price levels for trading trends (instead of reversals) - in this case, one would aim to classify price level crossings with a statistically significant improvement with respect to the no-information model. A different definition of the price crossing at the point of data labelling is necessary for that.

In terms of improving the strategy, there is a couple of things that can be done. For instance: take-profit and stop-loss offsets might be linked to the volatility instead of being constant. Also, flat strategies usually work better at certain times of the day - it would be wise to interrupt trading before USA and EU session starts and ends, as well as scheduled reports, news and impactful speeches. Additionally, all the mentioned parameters we have chosen can be looked into and optimised to the needs of the market participant.

It would be interesting to investigate other means for feature design instead of the manually defined feature space. For instance, Deep Learning networks can be used to obtain market representations at every moment of time [82]. It should be noted though that such an approach would substantially limit the transparency of the pipeline.

In terms of the chosen model, it would be useful comparing the CatBoost classifier to deep learning models like DA-RNN [69], as it makes use of the attention-based architecture designed based on the recent breakthrough in the area of natural language processing [70]. Moreover, it would be useful to consider model calibration approaches to obtain more realistic output class probabilities from the model [81].

Finally, we see a gap in the available FLOSS (Free/Libre Open Source Software) backtesting tools. To the best of our knowledge, there is no publicly available backtesting engine taking into account bid and ask prices and order queues. While there are solutions with this functionality provided as parts of the proprietary trading platforms, they can only be used as a black box. An open-source engine would contribute to transparency and has the potential to become the solution for both the research and industry worlds.