Improving the performance of bagging ensembles for data streams through mini-batching

Machine Learning (ML) models have become fundamental for many applications in different domains. The development of novel ML algorithms tailored to specific problem constraints is motivated by the ML omnipresence in several domains. In many applications, learning algorithms have to cope with dynamic environments that collect potentially unlimited data streams. Formally, a data stream $S$ is a massive sequence of data elements $x_{1},x_{2},\dots,x_{n}$ that is, $S={\{x_{i}\}}_{i=1}^{n}$, which is potentially unbounded (n → $\infty$) [41]. Compared to traditional (batch) data mining, stream processing algorithms have additional requirements. For instance, storing data for late processing is not a choice due to memory constraints with a potentially infinite data stream. Algorithms need to process incoming data instances incrementally in a single pass while operating under memory and response time constraints. Furthermore, as data streams present transient behavior, prediction models often need to be incremented to adapt to concept drift observed in data.

Ensemble learning comprises a class of machine learning algorithms that achieve remarkable predictive performance. However, ensembles are computationally more expensive, thus demanding more computational resources and likely violating time constraints. Although parallelism can be a good strategy for performance gains, many algorithms typically require collective communication with high overhead [12]. Algorithms derived from the Bootstrap Aggregating (Bagging ensemble algorithm do not depend on collective communications. They are typically composed of several homogeneous types, weak and unstable, learners trained independently of each other in parallel with no communication overhead. However, since ensemble models implement different data structures  [14], they are not amendable for data parallelism. Moreover, because such ensembles operate on dynamic data structures used to model concept drift and non-stationary data behavior, task parallelism is the natural way to implement them. In this scenario, memory access patterns and cache memory performance become one major challenge for the parallel implementation in multi-core environments.

To assess our strategy, we use MOA (Stream Learning framework Massive Online Analysis) [5], a widely known tool that provides multiple implementations of streaming algorithms, including a vast collection of ensemble methods. The rationale for using MOA for the evaluation is twofold. First, it allows the seamless evaluation of our strategy on many ensemble algorithms by changing the order MOA invokes the learners’ training. Second, we remove potential biases in the evaluation by using reliable implementations of the algorithms as the baseline in our experiments and implementing our solution in the same environment. Notice that either optimizing specific ensemble algorithms or specific learners composing the ensembles are out of this work’s scope.

The main contributions of this paper can be summarized as follows:

1. 1.

   Demonstrate that the incremental learning and dynamic data structures used to capture the concept drift increase the cache misses and hinder the benefit of parallelism in multi-core environments;

2. 2.

   Propose and assess a parallel implementation strategy based on mini-batches, which can significantly improve memory access locality and the performance of independent ensembles;

3. 3.

   Analyze the trade-off between the predictive performance and computational efficiency when using mini-batches. We observe that it is possible to considerably speed up the execution while having a small impact on the predictive performance in most cases;

4. 4.

   With further tests and improvements, our strategy can be implemented as a meta-classifier in MOA to evenly support the execution of ensemble methods composed of independent classifiers.

This work elaborates on previous work [8] in the following aspects:

($i$) The present work provides a better description of the performance bottleneck that motivated our proposal; ($ii$) we included a completely new discussion on the background and theoretical foundations of memory performance, including the proof of some theoretical bounds on the reuse distance, which explain the performance gains of our proposal; ($iii$) we provide a thorough discussion on the related work to position our contribution in two aspects: parallel strategies for the implementation of machine learning methods and the usage of mini-batching for performance optimization; ($iv$) we expand our experimental framework with two additional algorithms: Streaming Random Patches (SRP) and OzaBagASHT; ($v$) we included two new platforms in our experimental framework: Xeon E5-2650 and AMD 7702; ($vi$) we provide empirical analysis on how our solution reduces the reuse distance and improves data locality; ($vii$) we improved the analysis of the impact on predictive performance with more detailed measures (precision and recall instead of accuracy only). We also reduced the granularity of the experiments, increasing from 3 to 7 mini-batch sizes; ($viii$) we included a new evaluation on change detection for different sizes of mini-batch compared to the baseline.

This paper is organized as follows. The main related works are discussed in Section 2. Section 3 describes some state-of-the-art ensemble algorithms. A performance bottleneck of a parallel implementation of bagging ensembles is dissected in Section 4. Section 5 presents our proposal of a mini-batching strategy. Formal background on memory locality and how mini-batch improves access locality are presented in Section 6. In Section 7 we experimentally evaluate the performance of our mini-batching strategy. The impact of batch size on the performance and quality of prediction is evaluated in Section 8. Finally, our conclusions are presented in Section 9.

Research on the parallelization of machine learning methods dates to the early 90s when most papers focused on batch ML methods that require the whole dataset in the main memory to train the model. In early data mining, the batch learning approach is applied by processing the whole training data (one or multiple times) to output the decision models. Then, the decision models can be applied to new production data. Such procedure is usually referred to by batch learning, batch-mode algorithms [41], static data mining, and others in the literature. From now on, we will use batch learning to refer to non-stream learning methods.

Over the years, with the increase in computational power, the focus shifted from single classifiers [43] to ensembles. In the context of ensemble learners, many studies have been made on MapReduce (MR) frameworks [1, 33, 46], which are not suitable for applications with requirements of low response times, even being capable of processing huge amounts of data with high scalability [39]. Another investigation approach is using GPUs to process ensembles [27, 28, 38, 44]. However, GPUs are better for data-parallel problems.

Among the works that explored multi-core parallelism, distributed or not, we can further subdivide it into batch [9, 18, 21, 22, 25, 44] or data stream [20, 30, 31, 37] methods. Many works with various ensemble methods used the Message Passing Interface (MPI) standard, such as for ensembles of improved and faster Support Vector Machine (SVM) [18], bagging decision rule ensembles [20] and regression ensembles [31]. The remaining literature presents a miscellaneous of tools and different scopes. In [9], multi-classifiers are implemented using OpenMP. In [21], an ensemble of SVMs is implemented using joblib (a Python multiprocessing lib) and scikit-learn. In [44], TensorFlow is used to build a scalable and extensible framework for ensembles parallelization. In [25], an efficient Random Forest (RF) implementation that improves memory access due to better data representation on machines that combine both shared and distributed memory is proposed and implemented using FREERIDE (previous work from the authors). In [37], an ensemble of J48 is parallelized for grid platforms using Java. In [30], a low-latency Hoeffding Tree (HT) is implemented in C++ and used in RFs. In general, the related works mentioned so far differ from the present work in two main aspects: focusing on the implementation and performance aspects of specific ensemble methods or batch approaches (i.e., they do not focus on stream processing).

In more closely related work, Horak et al. [20] study the impact of concurrency on memory access pattern and performance of ensembles. They proposed a two-stage bagging architecture that combines single-class recognizers with two-class discriminators to improve accuracy and allow parallel processing. They also addressed load balancing for the parallel classifier construction and used the algorithm SCALLOP as the base for the experiments to validate the architecture implemented in MPI. Martinovic et al. [31] enhanced a dynamic auto-tuning framework in a distributed fashion by using two strategies. They use a scalable infrastructure capable of leveraging the parallelism of the underlying platform for ensemble models to speed up the predictive capabilities while iteratively gathering production data. Results show that the approach implemented in MPI can learn the application knowledge by exploring a small fraction of the design space. Qian et al. [37] propose a novel ensemble for data stream classification. This solution maps different raw data to multiple grids, where the first-order geometric center is used to represent and classify data. This method performs data compression, which increases the accuracy and computational efficiency. It was implemented in Java and tested in both multi-core and grid environments.

Marrón et al. [30] propose an implementation of RF-based on vector SIMD instructions and changes the representation of Binary Hoeffding Trees (HT) to fit into the L1 cache. It was implemented in C++ and benchmarked against MOA and StreamDM using two real and eleven synthetic datasets. It is noteworthy that the authors compare the performance of a single tree and the ensemble using different hardware architectures. Actually, these last four works are more closely related to the present work as they approach the performance of ensembles in the context of data streams. However, they are different from the present work in the following aspects. The works in [20, 30] focus on specific algorithms, SCALLOP and Binary HT, respectively. The work in [31] leverages parallel processing to improve the algorithm’s parameters, while [37] is focused on data compression.

The summary of related work regarding the parallelization of Machine Learning methods is shown in Table 1.

Bagging is one of the most used ML techniques to improve the accuracy of several weak models. Although it was proposed over 20 years ago, Bagging and its variants (e.g., Random Forest) are still used to this day as an alternative to intricate models, such as deep neural networks, that can be challenging to train and fine-tune. In contrast to Boosting, Bagging does not create dependency among the base models, facilitating the parallelization of its execution and processing incoming data online. Besides that, Bagging variants yield higher predictive performance in the streaming setting than Boosting or other ensemble methods that impose dependencies among its base models. This phenomenon was observed in several empirical experiments [3, 4, 15, 32]. One hypothesis to explain this phenomenon is the difficulty of effectively modeling the dependencies in a streaming scenario, as noted in [14].

Next, we present a summary description of six ensemble algorithms that evolved from the original Bagging to an online setting by Oza and Russeal [32]. We will demonstrate our mini-batching strategy on these algorithms in Section 5.

Online Bagging (OzaBag - OB) [32] is an incremental adaptation of the original Bagging algorithm. The authors demonstrate how the process of bootstrapping can be adapted to an online setting using a Poisson($\lambda=1$) distribution. In essence, instead of sampling with replacement from the original training set, in Online Bagging, the Poisson($\lambda=1$) is used to assign weights to each incoming instance, such that these weights represent the number of times an instance will be ‘repeated’ to simulate bootstrapping. One concern with using $\lambda=1$ is that about $37\%$ of the instances will receive weight 0, thus not being used to train. Leaving instances out of the training set is required to approximate OzaBag to the offline version of Bagging but may be detrimental to an online learning setting [14]. Therefore, other works [3, 15] increase the number of times an instance is used for training by increasing the $\lambda$ parameter.

OzaBag Adaptive Size Hoeffding Tree (OBagASHT) [4] combines the OzaBag with Adaptive-Size Hoeffding Trees (ASHT). The new trees have a maximum number of split nodes and some policies to prevent the tree from growing bigger than this parameter (i.e. deleting some nodes). This algorithms’ objective was to improve predictive performance by enforcing the creation of different trees. Effectively, diversity is created by having different reset-speed trees in the ensemble, according to the maximum size. The intuition is that smaller trees can adapt more quickly to changes, and larger trees can provide better performance on data with little to no changes in distribution. Unfortunately, in practice, this algorithm did not outperform variants that relied on other mechanisms for adapting to changes, such as resetting learners periodically or reactively [14].

Online Bagging ADWIN (OBADWIN) [4] combines OzaBag with the ADAptive WINdow (ADWIN) change detection algorithm. When a change is detected, a new classifier replaces the classifier with the worst predictive performance. ADWIN keeps a variable-length window of recently seen items. The property that the window has the maximal length is statistically consistent with the hypothesis that there has been no change in the average value inside the window. It implies that at any time, the average over the existing window can be reliably taken as an estimation of the current average in the stream, except for a very small or very recent change that is still not statistically visible.

Leveraging Bagging (LBag) [3] extends OBADWIN by increasing the $\lambda$ parameter of the Poisson distribution to $6$, effectively causing each instance to have a higher weight and be used for training more often. In contrast to OBADWIN, LBag maintains one ADWIN detector per model in the ensemble and independently resets the models. This approach leverages the predictive performance of OBADWIN by merely training each model more often (higher weight) and resetting them individually. However, since in LBAG more training is involved, LBAG requires more memory and processing time than OB and OBADWIN. In [3], the authors also attempted to further increase the diversity of LBag by randomizing the ensemble’s output via random output codes. However, this approach was not very successful compared to maintaining a deterministic combination of the models’ outputs.

Adaptive Random Forest (ARF) is an adaptation of the original Random Forest algorithm to the streaming setting. Random Forest can be seen as an extension of Bagging, where further diversity among the base models (decision trees) is obtained by randomly choosing a subset of features to be used for further splitting leaf nodes. ARF uses the incremental decision tree algorithm Hoeffding tree [11] and simulates resampling as in LBag, i.e., Poisson($\lambda=6$). The Adaptive part of ARF stems from change detection and recovery strategy based on detecting warnings and drifts per tree in the ensemble. After a warning is signaled, another model is created (a ‘background tree’) and trained without affecting the ensemble predictions. If the warning escalates to a drift signal, the associated tree is replaced by its background tree. Notice that in the worst case, the number of tree models in ARF can be at most double the total number of trees due to the background trees. However, as noted in [15] the co-existence of a tree and its background tree is often short-lived.

Streaming Random Patches (SRP) [17] is an ensemble method specially adapted to stream classification, which combines random subspaces and online bagging. SRP is not constrained to a specific base learner as ARF since its diversity inducing mechanisms are not built-in the base learner, i.e., SRP uses global randomization while ARF uses local randomization. Even though [17] all the experiments focused on Hoeffding trees and showed that SRP could produce deeper trees, which may lead to increased diversity in the ensemble.

All the ensembles use a Hoeffding Tree (HT) [11] as a base model. An HT is an incremental tree designed to cope with massive stationary data streams. Thus, it can do splits with reasonable confidence in the data distribution while having very few instances available. This is possible because of the Hoeffding bound, which quantifies the number of observations required to estimate statistics. This guarantees that the higher the number of instances, the more similar to non-incremental trees its model gets.

In this Section, we show a straightforward parallelization of bagging ensembles. The objective is to identify its main performance bottleneck, which motivated our proposal. Although all the learners that compose an ensemble may be homogeneous in type, each one has its own (and different) model. For instance, all the learners can be implemented by an HT, but each tree may grow in a different shape (which can change over time). For this reason, ensemble algorithms are not amendable for data parallelism in which the same instruction operates simultaneously over different data instances. On the other hand, task parallelism can naturally be applied as the underlying classifiers in bagging ensembles execute independently from each other and without communication.

Algorithm 1 describes a task-parallel-based implementation. This version improves the performance of the current parallel implementation of the ARF algorithm [15], in the latest version in MOA [5], by reusing the data structures and avoiding the costs of allocating new ones for every instance to be processed.

In lines 2-3, a thread pool is started, and one Trainer (runnable) is created for each ensemble classifier. For each arriving data instance (lines 4-17), votes from all the classifiers are obtained (line 5). Then, the Poisson weights are computed, and trainers are updated in lines 6-9. Although these steps (6 – 9) could be parallelized, we chose to run sequentially for two reasons. First, this part corresponds to only 3% of the computational effort for the algorithms studied in this article. In addition, we chose to keep the same weights used in the baseline (sequential) algorithm to use the same random numbers in the calculation of weights. Maintaining the weights will be important (in Section 8) to compare the predictive performance of the parallel and the baseline algorithms.

On the other hand, the training phase is more expensive. It involves updating statistics on each tree’s nodes, calculating new splits, and detecting data distribution changes (for all methods except OzaBag). As the training phase dominates the computational cost, parallelism is implemented (in lines 10-13) by simultaneously training many classifiers. Finally, lines 14-16 represent the global change detector, present only on OBAdwin, where the ensemble’s worst classifier will be replaced with a brand new one.

We use Java to reuse several bagging ensembles implemented in the MOA framework, which is widely used and tested. By reusing MOA algorithms, we provide a seamless and reproducible evaluation of our proposal, with the added benefit of being used for many studies in the area [5]. Although Java does not focus on implementing either energy-efficient or high-performance applications, the work in [34] shows that Java is in the top 5 languages (out of 27 tested) that need less energy and time to execute the applications.

The implementation is made in Java using the framework ExecutorService, which implements the Fork-Join abstraction for expressing parallelism. The framework is available since Java 7. It has methods to track the progress of a task and manage its termination. This framework’s main goal is to facilitate thread management by creating a Service with a fixed thread pool size, reserving and reusing these threads. Once a service has been created, tasks can be invoked by passing Runnables for it.

In essence, Fork-Join is composed of three steps: fork, computation, and join. In the fork step, new threads are created on-demand. Then, in the computation step, each thread executes one or more tasks. Finally, in the join step, the parallel threads synchronize and finish before continuing the program’s sequential region. This fork-compute-join process can be repeated many times during the execution of a program. Fork-Join implementations usually employ thread pools that support forked task management to reduce thread creation/destruction overhead. These pooled threads are not destroyed when the task finishes but instead release resources and become idle [12].

Although environments such as OpenMP or MPI provide remarkable support for the parallel implementation of ad hoc algorithms, this approach is out of the scope of this study because our focus is not to optimize a specific algorithm. Instead, the objective is to assess mini-batching as an implementation strategy for a group of streaming algorithms already implemented in MOA. Furthermore, Java presents a good performance in terms of execution time and energy efficiency for our purpose, as shown in [34]. The hardware used for experiments is described in Table 3. Experiments were carried out in a dedicated environment. Execution time is measured as wall clock time, including prediction and training steps. We present an average of three executions for each configuration.

The four datasets used in the experiments are open access ¹¹1Available at https://github.com/hmgomes/AdaptiveRandomForest, and a summary of their characteristics are shown in Table 4. A short description of each dataset is provided below:

• 1.

  The regression dataset from Ikonomovska inspired the Airlines dataset. The task is to predict whether a given flight will be delayed, given information on the scheduled departure. Thus, it has 2 possible classes: delayed or not delayed.

• 2.

  The Electricity dataset was collected from the Australian New South Wales Electricity Market, where prices are not fixed. These prices are affected by the demand and supply of the market itself and are set every 5 min. The Electricity dataset tries to identify the price changes (2 possible classes: up or down) relative to a moving average of the last 24h. An important aspect of this dataset is that it exhibits temporal dependencies.

• 3.

  The give me some credit (GMSC) dataset is a credit scoring dataset where the objective is to decide whether a loan should be allowed. This decision is crucial for banks since erroneous loans lead to the risk of default and unnecessary expenses on future lawsuits. The dataset contains historical data on borrowers.

• 4.

  The forest Covertype dataset represents forest cover type for 30 x 30 m cells obtained from the US Forest Service Region 2 resource information system (RIS) data. Each class corresponds to a different cover type. The numeric attributes are all binary. Moreover, there are 7 imbalanced class labels.

As depicted in Figure 1, most experiments present either a negative or a negligible improvement in speedup. Although the parallel implementation can train several classifiers in parallel, this operation requires few calculations. At the same time, it demands reading and writing large data structures in memory, thus producing a significant amount of cache misses, as shown in Table 5. Actually, parallelism increases the demand for memory bandwidth as multiple cores can execute different classifiers, filling the caches with their respective data structures. Such demand creates a bottleneck in the memory system, which hinders performance. The results presented here are consistent with previous studies reported in the literature (e.g., in [15]).

In summary, it demonstrates that the parallelism per se cannot improve the performance of streaming algorithms. To alleviate the bottleneck discussed here and improve the performance and scalability of bagging ensembles, the next Section proposes a strategy that can increase the reuse of large data structures in the cache.

Although task parallelism looks straightforward for implementing ensembles, bad memory usage can severely hinder their performance. For instance, high-frequency access to data structures larger than cache memories can raise severe performance bottlenecks. We can observe that parallelism increased the cache contention, thus increasing the number of cache misses, which explains the loss of speedup in some experiments.

One strategy for alleviating this memory bottleneck is to improve the data reuse of the classifiers. Such data reuse can be improved by processing more than one instance at a time so that the data structured loaded into the cache can be reused for processing a group of instances. In the present work, we use the term mini-batches to refer to a group of instances that will be processed at once by each ensemble classifier. Thus, the mini-batching strategy aims to reduce cache misses by improving cache data reuse.

Figure 2 presents a simplified view of the ensemble working. The mini-batch is replicated to each learner in the ensemble, which outputs the predictions of the whole mini-batch. After that, there is an aggregation of the predictions to output the final prediction of the whole ensemble. Then, the learners use the same mini-batch to update their models. This way, the training is deferred to the end of the processing of a mini-batch.

The introduction of mini-batching does not break the single-pass requirement for operating on data streams, as each instance will continue to be processed only once then discarded. However, the mini-batching defers the update of the models to the end of the mini-batch (i.e., training starts after all the instances of the mini-batch have been classified).

Each learner is implemented by a task that processes training by iterating on each instance of a mini-batch instead of processing a single instance at a time. When a learner is invoked, its data structures are loaded into the upper levels of the memory hierarchy (upper-level caches) and quickly accessed to process the next instances in the same mini-batch, reducing cache misses and improving performance. Next, we propose mini-batching for improving the memory locality of the parallel portion of the code.

The Algorithm 2 shows the mini-batching strategy.

The first difference between the two algorithms appears in lines 5-6 of the Algorithm 2, where the ensemble will only accumulate the instances until the desired mini-batch size is met or the stream ends. When this condition is fulfilled, the algorithm performs the classification (line 7) and training (lines 8-21). In line 9, the whole mini-batch is copied to each trainer, and the weight is left to be calculated inside each trainer. Line 11 has a parallel loop that executes the trainer in parallel. Then, each trainer will iterate (sequentially) over all mini-batch instances while calculating the weight, creating the weighted instance, and training the classifier with this instance (lines 12-16). ARF and LBag, exclusively, will execute lines 17-19 as a local change detector for each classifier in the ensemble. Finally, in line 20, the instances are discarded and the buffer is flushed to begin accumulating a new mini-batch. In OBAdwin, lines 17-19 would be outside the parallel section, as change detection is global.

Notice that each prediction model usually is several times larger than a single instance, and the size of the whole ensemble (almost always) overcomes the cache memory by far. Thus, it is significantly more expensive to load one model into memory than loading one instance. Algorithm 1 loads one sample (usually small) in memory, and iterates over the ensemble classifiers (significantly larger), thus incurring a high number of cache misses. In contrast, the mini-batching fixes one ensemble model in memory, then iterates over the mini-batch samples, thus improving memory access locality by reducing the cache misses.

Ideally, the size of ensemble models should fit into the cache memories for optimal performance. However, this is not the case for current streaming bagging ensembles (e.g., the ones considered in this article). Actually, because the entire ensembles typically overcome the size of caches by far, the mini-batching can significantly reduce the number of cache misses. Such reduction explains the performance gains achieved with mini-batching.

Memory locality is a fundamental property and design principle for optimizing the performance of hardware, software, and algorithms [48]. Locality can be defined as “the tendency for programs to cluster references to subsets of address space for extended periods” [10]. Due to the increasing gap in processor and memory speeds [24], the locality has played a central role in optimizing the performance of operating systems over the decades. Locality may be defined in many ways, and several metrics related to it have been proposed [48]. We can use the notion of reuse distance (RD) to evaluate how mini-batching can improve the performance and resource (i.e., cache) sharing of bagging ensembles implementations. We chose these definitions because they are based on direct measurements, do not depend on idealistic assumptions, and are extensions of observational stochastic [48]. Our goal is to reduce cache misses by improving data reuse in parallel implementations of ensembles.

From a historical perspective, memory locality has been studied over decades to optimize the memory hierarchy, operating systems, software, and algorithms design with recent advances in measurement techniques [23], trace generation [40], and formal modeling [29]. In recent work [48], Yuan et al. built upon previous works by proposing the relational theory of locality (RTL), a theoretical framework that unifies several memory locality measures used along five decades of study and research in the field. RTL provides mathematical background and categorizes the measures in three different types of locality. The authors showed how such measures relate to each other and whether and how they can be inter-converted.

Next, we discuss the memory locality of a stream processing system operating according to the algorithms described in the previous section. Each algorithm implements an ensemble $L$, composed by a set of learners $l_{i}\in E$. We refer to an individual learner as $l_{i}$ $(1\leq i\leq|E|)$. A stream $S$ is a countably infinite set of data elements $s\in S$. Each stream element s:⟨v, t⟩ consists of a relational tuple $v$ conforming to some schema, with an application time value $t_{i}\in T$. We assume that the time domain $T$ is a discrete, linearly ordered, countably infinite set of time instants $t\in T$. As the stream is potentially infinite, we assume that $T$ is bounded in the past but not necessarily in the future. Thus, due to memory limitations and response time constraints, the algorithms need to incrementally process incoming data elements in a single pass, performing both classification and training as data elements arrive.

A trace $N$ is a sequence of references to data or memory locations denoted by $N=mt(1,\dots,n)$, where $n$ is the trace’s length. A trace can access a set of $m$ distinct memory addresses, while the set of distinct memory addresses is be denoted by $M={e_{1},\dots,e_{m}}$, where $m$ is the number of distinct memory addresses in $M$. The model allows abstracting from any granularity issues so that a data item may be either a variable, a data block, a page, or an object. For illustration, we can use some trace examples composed of just three data elements $a,b,c$, including those repeating them once in the same order (i.e., abc abc), in the opposite order (i.e., abc cba), or repeating them indefinitely (i.e., abc abc …).

In essence, access locality is related to measuring the locality for each memory access. From the five definitions of access locality provided by YUAN et al. [48], we use only the definition of reuse distance sequence or reuse distance (RD) for short because it suffices to demonstrate that mini-batching can improve ensemble implementations’ access locality. The equivalence among the definitions is proven in [48].

The reuse distance (RD) is defined as “the number of distinct data accessed since the last access to the same datum, including the reused datum” [48]. The reuse distance is $\infty$ for its first access. For a finite reuse distance, the minimum is 1 (because it includes the reused datum), and the maximum is $m$. For example, the RD sequence is $\infty\infty\infty$ 333 for abc abc and $\infty\infty\infty$ 135 for abc cba.

Before demonstrating the benefits of mini-batching, it is worth noting that stream processing ensembles have two principal operations: the classification (in line 5 of Algorithm 1) and the training (line 12 of Algorithm 1). The former reads a few model variables of each classifier, while the latter is (by far) the dominant operation in terms of computational cost because it performs both read and write operations to update the classifier’s models. For this reason, our analysis presented here focus on the training operation.

For the sake of illustration, Table 6 presents a simple example with an ensemble of $m=4$ learners processing a stream of $n=6$ data items. Without mini-batching, the processing of the first data item produces a sequence of $m$ occurrences of $\infty$ reuse distance. However, for finite reuse distance, the minimum reuse distance is 1 because it includes the reused datum, and the maximum is $m$. Thus, $\infty$ is shown only for illustration, being ignored in our analysis hereafter. In this example, for each access $e_{i}$, the reuse distance equals the number of ensembles $m$. With mini-batching, each ensemble is accessed once within the mini-batch (with reuse distance $m$) and reused $b-1$ times. One could easily realize the benefits of mini-batching by simply substituting every $\infty$ by 1 and calculating the average reuse distance for the two executions. Next, we demonstrate the benefits.

For the proofs shown in this section, we assume the amount of memory used to implement the ensemble exceeds the cache memory size (which is quite realistic). Otherwise, all accesses will hit the cache, and the order in which memory positions are accessed does not influence cache misses.

Next, we can estimate the benefit of mini-batching (as described in Algorithm 2) for reducing the reuse distance.

Although our demonstration assumes sequential processing, the result is also valid for the parallel execution proposed in Algorithm 2. Notice that the outer loop in line 11 assigns a different ensemble learner for each processing core, while the innermost loop iterates over the mini-batch data items. This loop arrangement enables the ensemble to process mini-batches of different sizes than the one passed as a parameter. However, this case can only happen if the stream is terminated with an incomplete mini-batch. In this case, the mini-batch has to be processed with fewer instances than expected. Thus, each processing core needs to load only one learner model in its memory caches to process the entire mini-batch.

It is worth noting such results hold regardless of the locality measure used for the demonstration. As formally demonstrated by YUAN et al. [48], locality access measures such as address independent (AI) sequence, reuse interval (RI) sequence, per datum sequence of reuse interval (PD-RI), and per datum reuse distance (PD-RD) are equivalent to each other, and they can be inter-converted. Using these results, other measures could be seamlessly used to demonstrate that mini-batching improves access locality of the implementation of ensembles for stream processing. We chose the measure reuse distance because of its close relation to cache misses, as the larger the reuse distance, the higher the cache misses. Cache misses occur when the reuse distance is big enough to fill the cache memory.

Although using only one mini-batch to process the entire data stream is useful for demonstrating that the access locality is optimal, it is not useful in practice. Notice that the pure stream processing (as in Algorithm 1) performs the classification and training steps for every stream’s incoming data item. Thus, the learner models continuously evolve, and the processing of every data instance can influence the next incoming data classification. On the other hand, with mini-batching (as proposed in Algorithm 2), the ensemble training is deferred to the end of each mini-batch. So, setting a mini-batch size of 1 boils down to pure stream processing. In contrast, mini-batches of the same length as the entire data stream turn it into a pure batching scheme in which all data instances are classified using models built during an offline training phase that precedes the entire stream. In summary, the choice of the mini-batch size raises a trade-off between learning capabilities (with short mini-batches) and computational performance (with larger mini-batches).

Yuan et al. [48] demonstrated that several locality measures are equivalent and may be inter-converted. However, not all measures can be equally usable for our purpose. We chose the reuse distance because the demonstration that mini-batching leads to optimal locality becomes straightforward with this measure, but not using the footprint or miss ratio, for instance.

To demonstrate the impact of mini-batching for bagging ensembles, we implement the strategy in the Massive Online Analysis (MOA) framework [5], and tested its performance on the six ensemble algorithms described in section 3.

To assess the impact of mini-batching, we tested five different configurations of the algorithm: a sequential implementation (baseline), a parallel implementation without mini-batching, and a parallel implementation with mini-batches of varying sizes [50, 500, and 2000]. All the parallel implementations executed with 8 threads pined to processing cores for better cache usage. We chose ensembles with 100 and 150 classifiers similar to other works [38]. Also, Panda et al.[33] demonstrated a reduction in deviance asymptotes with more than 100 classifiers.

Figure 3 presents the speedup achieved in each configuration with 8 cores in each platform. Results confirm that pure parallelism (without mini-batching) yields bad performance in many configurations. In contrast, performance gains are obtained by combining parallelism with mini-batching due to better memory access patterns. Also, speedups are closely related to the models’ computational complexity, which varies according to the algorithm and dataset used. Ideally, the speedups should approach 8, which is the number of cores used. Cheaper algorithms (e.g., OzaBag and OzaBagAdwin) show lower Speedups due to smaller work per thread. However, LeveragingBag presented a Speedup of 12X for the Airlines dataset, indicating twofold benefit, i.e., gains due to the parallelism and better memory locality. The averaged execution times are shown in Tables 8, 9 and 10.

As a major result, experiments show that mini-batching combined with multicore parallelism improves the performance of ensembles. In general, introducing small mini-batches of up to 50 elements improves data reuse and reduces cache misses. However, large mini-batches tend to increase the number of cache misses. Such an increase happens because the larger the mini-batch, the more heterogeneous the data instances. More heterogeneity implies that the same classifier will access different paths of the Hoeffding trees.

Streaming ensemble algorithms operate in a sample-wise mode, i.e., the classifiers are up-to-date any time after each sample and can immediately predict the next. In contrast, with the introduction of mini-batching, first, the classifiers output predictions for all the instances of the new mini-batch and then train (i.e., update the models) to classify the next mini-batch, and so on. So, what is the cost of deferring the training on the predictive classification? Furthermore, does the mini-batch size impacts the predictive performance? Next, we empirically address these questions.

Ensemble learning is a fruitful approach to improve the performance of ML models by combining several single models. Examples of this class include the algorithms Adaptive Random Forest, Leveraging Bag, and OzaBag. This approach is also popular in a data stream processing context. Despite their importance, many aspects of their efficient implementation remain to be studied. This paper highlighted a performance bottleneck of multi-core implementations of bagging ensembles and proposed a mini-batching strategy to improve the memory access pattern’s locality and reduce processing time. We demonstrated through theoretical and experimental frameworks that the performance achieved by multicore parallelism could be remarkably improved (speedups of up to 5X on 8-core processors) through the application of this mini-batching technique. We observed that the choice of the mini-batch size could raise a trade-off between computational performance and predictive performance. Our experiments with six bagging ensemble methods and four datasets showed a good compromise solution around mini-batches of 50 instances. However, the choice of the best mini-batch sizes may depend on the application scenario. As a final comment, we believe that mini-batching can support manifold performance improvements to implementations of bagging ensembles.

As future work, we intend to investigate how mini-batching can improve the energy efficiency of bagging ensembles. Furthermore, the use of varying size mini-batches, i.e., which can be dynamically adjusted at run-time according to some variable parameters (e.g., the incoming rate of instances, the delay in processing each instance in the current window, the data characteristics, the accuracy measured at the output during a time window, to cite a few) can be an interesting direction for future research.

This study was financed in part by the Coordenação de Aperfeiçoamento de Pessoal de Nível Superior - Brasil (CAPES) - Finance Code 001, and Programa Institucional de Internacionalização – CAPES-PrInt UFSCar (Contract 88887.373234/2019-00). Authors also thank Stic AMSUD (project 20-STIC-09), and FAPESP (contract numbers 2018/22979-2, and 2015/24461-2) for their support. Partially supported by the TAIAO project CONT-64517-SSIFDS-UOW (Time-Evolving Data Science / Artificial Intelligence for Advanced Open Environmental Science) funded by the New Zealand Ministry of Business, Innovation, and Employment (MBIE). URL https://taiao.ai/.