Approximate Borderline Sampling using Granular-Ball for Classification Tasks
^††thanks: This work was supported by XXX. ^††thanks: *: Corresponding author.

GBC[22] is a family of scalable, efficient, and robust data mining methods, which is a two-stage learning, including the GBG stage and the GB-based learning stage. The core idea of the GBC is to employ the ball of varying granularity to represent the information granule and replace the sample for calculating in various tasks. The geometry of the ball is completely symmetrical, and only the center and radius are required to characterize it in any dimension, so it can be easily applied to diverse scenarios, including classification[22, 27, 28], clustering[29, 30, 31], fuzzy sets[32, 33], feature engineering[34, 35, 36] and deep learning[37].

As a granulation method, the core idea of the GBG method is to cover a dataset with a set of balls, where a ball is called a GB $gb=(O,(\bm{c},r,\mathbb{P},l))$. Specifically, the granulation process of the existing GBG methods can be briefly described as follows. First, the whole training dataset is initialized as the initial GB. Second, $k$-means[22], $k$-division[27], or hard-attention division[38] is employed to split the GB into $k$ or more finer GBs. The center $\bm{c}_{i}$ and radius $r_{i}$ of $gb_{i}(\forall gb_{i}\in G,i=1,2,\cdots,m)$ are defined as follows.

where $D_{i}\in D$, $|\bullet|$ represents the cardinality of set $\bullet$, and $\bigtriangleup(\cdot,\star)$ denotes the distance function. Without losing generality, Euclidean distance is employed in this paper. For most real datasets, the samples are unevenly distributed in the feature space, and the GB defined by Eq.1 will cause some samples to be distributed outside the ball.

The label $l_{i}$ of the $gb_{i}$ is determined by the majority of samples contained within it. The quality of the $gb_{i}$ is measured using the purity $\mathbb{P}_{i}$, that is, the ratio of the number of samples within the $gb_{i}$ that are consistent with its label $l_{i}$ to the number of all samples within it. The closer the purity of GB is to $1.0$, the closer the distribution of GBs is to the original dataset. Iteratively split each GB until the purity of each GB reaches the given purity threshold.

Existing GBG methods suffer from overlapping GBs, causing the distribution of GB sets to diverge from that of the original dataset, leading to inconsistency between the sampled and original datasets. Although this issue tends to alleviate with increasing purity[22], it cannot be fully resolved. For instance, overlapping heterogeneous GBs would blur class boundaries, and overlapping homogeneous GBs can cause the shrinking of class boundaries.

Inspired by GBC, a general GB-based sampling method (GGBS) and a GB-based sampling method for imbalanced datasets (IGBS) are proposed by Xia et al.[23], both including the GBG stage and the undersampling stage.

The core idea of the GBG method used in the GBG stage of GGBS and IGBS can be briefly described below. Given a dataset $D$, it is initialized to the initial GB. For each GB, if its purity is less than the purity threshold and the number of samples within the GB is greater than $2\times p$, then the $k$-division is used to split the GB into $k$ finer GBs. Iteratively, until the purity of each GB reaches the threshold or the number of samples it contains is less than or equal to $2\times p$. Finally, a GB set $G$ is obtained. In this section, a GB is called a small GB if it contains no more than $2\times p$ samples; otherwise, it is called a large GB.

The core idea of the undersampling stage of GGBS can be summarized as follows. First, all samples contained in small GBs are put into the sampled dataset $S$. Second, for each large GB, put $2\times p$ samples into the sampled dataset $S$, which are the homogeneous sample closest to the intersection point of the GB and the coordinate in each feature dimension.

The core steps of the undersampling stage of IGBS are as follows. First, the first step is the same as that for GGBS. Second, for each minority class GB that is large, all the containing minority class samples are sampled into $S$. Third, for each majority class GB that is large, $2\times p$ majority class samples are sampled into $S$, whose sampled rule is the same as GGBS. Finally, if the class distribution is still skewed, randomly sample more majority samples into $S$.

However, the aforementioned GBG method stops splitting GBs to ensure a preset sample count, even if the purity threshold is unmet, and GB overlaps further degrade their quality. These issues reduce the quality of the sampled data in GGBS and IGBS. Additionally, GGBS applies a uniform sampling strategy across all GBs, ignoring the importance of borderline GBs, which may retain redundancy or noise, limiting classifier improvement. Moreover, IGBS blindly balances class ratios without assessing sample redundancy, increasing the risk of overfitting.

The architecture of the RD-GBG method is shown on the left side of Fig.1. The entire training dataset is initialized as the undivided sample set. First, the undivided sample set is grouped by labels, and a sample is randomly chosen as the candidate center from each group, prioritizing larger groups. And perform center detection to determine whether the center meets the local consistency which means that the center has neighbors that are homogeneous with it. Second, construct the pure GB based on each eligible center on the undivided sample set, as well as the new GB cannot overlap with the previous GBs. Iteratively, the above process is performed on the undivided sample set until the undivided samples with local consistency converge. Lastly, orphan GBs are constructed.

The architecture of the GBABS method is shown on the right side of Fig.1. First, the RD-GBG method is performed for a given dataset to obtain a GB set. Second, take the centers of all GBs to form a center set to represent their location information in the feature space. Third, based on the center set, the GBs on the class boundaries are detected from each feature dimension. Finally, sampling is performed based on the heterogeneous adjacent relation between borderline GBs.

In the field of GrC, the granulation method for large-scale datasets needs to follow three criteria. The first one is that the distribution of information granules should be as consistent as possible with that of the original dataset, which can be called approximation. The second one is that a GB should contain as many samples as possible to improve the efficiency as well as ensure the performance of the GB-based downstream learning tasks, which can be called representativeness. The third one is that the samples should be used as much as possible, which can be called completeness.

Consequently, based on the idea of restricted diffusion, a new GBG method is proposed in this section, which is adaptive and without overlap among GBs. As shown in Fig.1, the whole training dataset $D$ is initialized to the undivided sample set $U$. The GB is constructed on $U$ in turn iteratively. Specifically, the construction process of GB will be introduced in detail below, which includes the determination method of local-density centers, the construction method of the GB, and the iteration termination condition.

According to Section IV-B, the set of GBs constructed on a given dataset can essentially describe this dataset approximately, including the class boundaries of the dataset. Therefore, there are GBs distributed on the class boundaries, called borderline GBs, which can be detected somehow. The geometric center is a geometric property of a ball that represents its position.

Typical distance measurement methods, such as Euclidean distance, fail when determining the location of multidimensional data objects. As shown in Fig. 4(a), there is a two-dimensional dataset with 2 classes marked in different colors. A GB set is generated on the dataset using the RD-GBG method, shown in Fig. 4(b). All the centers of these GBs are shown in Fig. 4(c). Calculate the distance between center $(\bm{c}_{1},l_{1})$ and other centers, and only find that the nearest heterogeneous center is $(\bm{c}_{5},l_{5})$. Then it can be judged that there is only a separation point between $(\bm{c}_{1},l_{1})$ and $(\bm{c}_{5},l_{5})$. Obviously, there is also another separation point between $(\bm{c}_{2},l_{2})$ and $(\bm{c}_{1},l_{1})$. Therefore, calculating the distance between samples to identify the class boundaries will fail.

In high-dimensional feature space, the position of the sample is usually represented by multi-dimensional coordinates composed of its feature values. Therefore, the coordinates of the center of the GB are used as its positional information in the feature space. As shown in Step 1 of the GBABS module of Fig.1, for $\forall(\bm{c}_{i},l_{i})((\bm{c}_{i},l_{i})\in\mathbb{C})$, if at least one of its left and right neighbors $(\bm{c}_{j},l_{j})\in\mathbb{C}$ in a given feature dimension is heterogeneous with it, it can be judged that both $(\bm{c}_{i},l_{i})$ and $(\bm{c}_{j},l_{j})$ are likely to be distributed on the class boundaries, where $i\neq j$. Consequently, the borderline center set $\mathbb{C}^{\prime}(\mathbb{C}^{\prime}\subseteq\mathbb{C})$ would be obtained when all $(\bm{c}_{i},l_{i})\in\mathbb{C}$ are considered.

As shown in Fig. 4(c), for center $(\bm{c}_{1},l_{1})$, in the feature column corresponding to the feature $z$, its left and left neighbors are $(\bm{c}_{2},l_{2})$ and $(\bm{c}_{3},l_{3})$, respectively. Thus, as shown in Fig. 4(d), the $gb_{1},gb_{2}$ and $gb_{3}$ are recognized as borderline GBs, owing to that $l_{1}\neq l_{2}$ and $l_{1}\neq l_{3}$. And, in the feature column corresponding to the feature $w$, the left and left neighbors of $(\bm{c}_{1},l_{1})$ are $(\bm{c}_{4},l_{4})$ and $(\bm{c}_{5},l_{5})$. Since $(\bm{c}_{4},l_{4})$ is homogeneous with $(\bm{c}_{1},l_{1})$, and $(\bm{c}_{5},l_{5})$ is heterogeneous with it. Thus, the $(\bm{c}_{5},l_{5})$ is recognized as another borderline center. Notably, as shown in Fig. 4(c), the left and right neighbors of $(\bm{c}_{6},l_{6})$ in all feature dimensions are homogeneous with it, then the $(\bm{c}_{6},l_{6})$ can be judged as an intra-class center.

The GBs corresponding to the borderline centers are the borderline GBs. As shown in Step 2 of the GBABS module of Fig.1, the borderline GB set $BG(BG\subseteq G)$ can be obtained based on borderline center set $\mathbb{C}^{\prime}$. For $\forall gb\in BG$, there is at least one sample closest to the class boundary in a certain dimension, which is a dimension that the $gb$ is judged to be a borderline GB. Consequently, the borderline sample set $S(S\subseteq D)$ can be obtained, in which there are no repeated samples.

All the borderline GBs are shown in Fig.4(d). In Fig.4(e), for the feature column corresponding to feature $z$, the left and right neighbors of the borderline GB $gb_{1}$ are $gb_{2}$ and $gb_{3}$, respectively. Therefore, the sample with the largest value of feature $z$ among the samples in $gb_{3}$ is identified as a borderline sample. Similarly, all green-marked samples in Fig. 4(e) are the borderline samples. Consequently, the sampled dataset is shown in Fig. 4(f), representing the approximate borderline sample set. Compared to Fig. 4(a), Fig. 4(f) exhibits a significantly reduced number of samples while maintaining essentially the same class boundaries.

Algorithm 2 provides the complete GBABS method. Suppose a dataset $D$ that contains $N$ samples and $q$ classes with $p$ features. To obtain a GB set $G=\{gb_{1},gb_{2},\cdots,gb_{m}\}$ with Algorithm 1, the time complexity is $O(tqN)$. The time complexity for sampling on the $G$ using GBABS is $O(pm\log m)$. As a result, the total time complexity is $O(tqN+pm\log m)$, which is still linear.

This section analyzes and discusses the performance of GBABS in data compression on standard datasets and class noise datasets.

Fig. 6(a) provides insights into the sampling ratio of GBABS and GGBS on each standard dataset listed in TableI. Additionally, Fig.5 visualizes several standard datasets using TSNE, a dimensionality reduction technique, with different classes marked with different colors. Observation from Fig.6(a) reveals that GBABS achieves notable compression across all datasets, with a minimum sampling ratio of approximately $29\%$. The reason why GBABS has excellent data compression capability is that GBABS samples on GBs, and the number of GBs is generally much smaller than the sample size of the original dataset. Notably, GBABS exhibits a smaller sampling ratio on datasets with lower dimensions or fewer classes. For example, the sampling ratio for the two-dimensional binary dataset S5 is about $29\%$, while for the higher-dimensional dataset S1, the ratio increases to approximately $84\%$. Fig. 5(a) and (b) illustrate that the class boundaries of S5 are relatively simple, in contrast to the complex boundaries of S1 due to its high dimensionality. In high-dimensional spaces, retaining more samples is crucial due to the increased complexity of class boundaries.

Besides, as depicted in Fig.6(a), GBABS generally exhibits lower sampling ratios compared to GGBS across most datasets, such as datasets S6 and S10. Notably, for a high-dimensional dataset such as dataset S7, the sampling ratio of GGBS is $1.0$, which means that the sampling capability of GGBS is invalid. For some datasets with unclear class boundaries, the sampling ratio of GBABS is slightly higher than that of GGBS, such as the dataset S3. As shown in Fig.5(c), the distributions of samples of different classes in S3 overlap in the feature space, so the class boundaries are unclear. However, retaining sufficient borderline samples for high-dimensional datasets or datasets with complex and blurred class boundaries is crucial to ensure effective classification. Besides, on a dataset with relatively clear class boundaries, even if it is multi-class, such as a dataset S6 whose visualization is shown in Fig.5(d), GBABS yields a smaller scale sampled dataset than GGBS. Generally, the superior data compression capability of GBABS is attributed to its selective sampling on borderline GBs, unlike GGBS, which samples on each GB.

Furthermore, Fig. 6(b)-(f) depict the sampling ratios of GBABS and GGBS on datasets with class noise ratios of $5\%$, $10\%$, $20\%$, $30\%$, and $40\%$, respectively. It can be observed that under any noise ratio, the data sampling ratio of GBABS is always lower than that of GGBS. Specifically, on dataset S8 at $20\%$ noise ratio, GBABS achieves a sampling ratio as low as $44\%$, whereas GGBS retains a $100\%$ sampling ratio. Compared to standard datasets, GBABS exhibits stronger data compression on class noise datasets, with its advantage over GGBS becoming more pronounced as the noise ratio increases.

Two factors contribute to GBABS’s superior data compression on class noise datasets. First, the RD-GBG method (Section IV-B) incorporates noise elimination, removing most class noise samples as the noise ratio increases, while the GBG method of GGBS does not consider that. Second, GBABS focuses on sampling from borderline GBs, unlike GGBS, which samples uniformly from all GBs. This results in a lower sampling ratio for GBABS, even when class noise requires more GBs to cover.

This section mainly validates the lossless compression capability of GBABS as a sampling method tailored for classification tasks. Table II shows the testing $Accuracy$ for the GBABS-based DT, GGBS-based DT, SRS-based DT, and DT on standard datasets listed in Table I. Table III reports the results of the Wilcoxon signed-rank test for the comparison between GBABS-based DT and others. The results indicate significant differences in performance across all tested pairs at the $0.05$ significance level, which strongly suggests that the GBABS-based DT consistently outperforms the others across all comparisons.

Specifically, GBABS-based DT outperforms DT on $77\%$ of the datasets, such as the testing $Accuracy$ on dataset S2 is improved by $0.0449$. This superior performance stems from the approximate borderline sampling strategy detailed in Section IV-C ensures the collection of samples crucial for delineating class boundaries, thereby enhancing the ability of the sampled dataset to describe the class boundary of the original dataset. Furthermore, by abstaining from collecting intra-class samples, GBABS mitigates overfitting and the impact of noisy data to a significant extent.

Compared with GGBS-based DT, it’s evident that GBABS-based DT consistently achieves higher testing $Accuracy$. This disparity can be attributed to several factors. First, GGBS refrains from splitting the GB with a sample size less than or equal to twice the number of features, irrespective of purity thresholds, which diminishes the quality of generated GBs, thus impairing their effectiveness in describing the original dataset. Second, the overlap between GBs in the GBG method of GGBS can result in blurred class boundaries, undermining classification performance. Third, the center selection method (introduced in Section IV-B1) and the radius determination rule of GB (introduced in Section IV-B2) enable the GBs constructed by RD-GBG to more accurately represent the original dataset compared to those constructed by the existing GBG method. Fourth, the GBABS aims to collect samples near the class boundaries, where the borderline samples are crucial for classification. In contrast, GGBS samples from all GBs, including redundant samples, may degrade classifier performance.

The testing $Accuracy$ of GBABS-based DT is higher than that of SRS-based DT on almost all datasets. The reason is that GBABS is essentially a biased sampling method, and samples on the class boundary have a higher probability of being sampled, while SRS is an unbiased sampling method. Consequently, when SRS and GBABS adopt the same sampling ratio for a given dataset, GBABS retains more borderline samples, enriching the sampled dataset with more effective information for classifiers.

This section mainly verifies the enhancement of the robustness of the classifier by GBABS. Considering that different classifiers have different sensitivity to class noise, five commonly used and representative machine learning classifiers are employed to obtain comprehensive and reliable results and alleviate the bias of comparative experimental settings. Comparative experiments are conducted on datasets with class noise ratios of $5\%$, $10\%$, $20\%$, $30\%$, and $40\%$, respectively.

Table IV shows the average testing $Accuracy$ of GBABS-based classifier, GGBS-based classifier, SRS-based classifier, and classifier on datasets with different noise ratios, where classifiers are DT, XGBoost, lightgbm, RF, and $k$NN. It can be observed from Table IV that the GBABS-based classifier generally performs better, whether in the case of low noise ratio (such as $5\%$) or high noise ratio (such as $40\%$). Especially in high-noise environments, GBABS can maintain a relatively stable performance for each classifier compared with others.

The ridge plot shown in Fig. 7 shows the distribution of testing $Accuracy$ for XGBoost with different sampling methods at noise ratios of $10\%$ and $30\%$, while Fig. 8 presents the distribution for RF at noise ratios of $20\%$ and $40\%$. Curves of different colors represent the $Accuracy$ distribution of different sampling methods. The scatter points of different colors represent the testing $Accuracy$ of different methods on each dataset. From the distribution in the ridge plot, it can be seen that under different noise conditions, whether used for XGboost or RF, GBABS shows higher consistency and stability. Especially when the noise ratio is high (such as $40\%$), the testing $Accuracy$ distribution of GBABS-based RF shifts to the right (peak value is about $0.55-0.6$), which is significantly better than other methods. When the noise ratio is low (e.g., $10\%$), the testing $Accuracy$ distribution of the GBABS-based classifier is concentrated, and the peak value is slightly higher than that of others.

In conclusion, it can be inferred that GBABS can effectively enhance the robustness of classifiers, outperforming GGBS and SRS. There are two primary reasons why GBABS excels in enhancing robustness. First, the RD-GBG (introduced in Section IV-B) considers noise elimination, whereas GGBS and SRS do not. Second, GBABS (introduced in Section IV-C) is designed to collect samples on class boundaries, thereby avoiding the collection of redundant and noisy samples. In contrast, GGBS and SRS also do not consider that.

This section mainly evaluates the effectiveness of GBABS in mitigating the bias problem of standard imbalanced datasets (including binary and multi-class datasets) and imbalanced datasets with class noise. Fig.9(a) demonstrates the ranking of testing $G-mean$ of DT on the standard datasets when GBABS, GGBS, IGBS, SM, BSM, SMNC, and Tomek are used as the sampling methods, while Fig.9(b)-(f) shows the ranking of testing $G-mean$ on datasets with class noise ratios of $5\%$, $10\%$, $20\%$, $30\%$, and $40\%$, respectively. The larger the value of $G-mean$, the higher the ranking. It can be observed that, on most standard imbalanced datasets, GBABS-based DT ranks high, and on almost all imbalanced datasets with class noise, GBABS-based DT achieves the best performance. In a high noise environment (such as $40\%$), although the ranking of GBABS has dropped on a few datasets, it is still better than most other sampling methods. The reason is that, as seen in Fig. 6, as the ratio of class noise increases, the sampling ratio of GBABS on each dataset decreases. Too few samples may reduce performance for datasets with small sample sizes, such as S1 and S2. In conclusion, GBABS can mitigate the bias issue caused by class imbalance to a certain extent, especially performing excellently in scenarios with class noise.

There are three main reasons why GBABS exhibits superior performance in handling imbalanced datasets. First, GBABS is essentially an undersampling method that aims to sample borderline samples. Therefore, the removal of redundant samples in the majority class is generally more aggressive than that in the minority class, which can alleviate the class imbalance to a certain extent and reduce the overfitting of the classifier to majority class samples. Additionally, as mentioned in Section V-B, when the dimensionality of the dataset is high, or the dataset size is small, the compression ability of GGBS and IGBS may fail, i.e., they cannot effectively undersample the majority class samples. Second, compared with oversampling methods such as SM, BSM, and SMNC, GBABS avoids the risks of synthetic samples, such as introducing noise or overfitting, particularly in datasets with class noise. Third, as mentioned in Section V-D, RD-GBG considers noise elimination, so it still performs exceptionally well in scenarios with class imbalance and noise.

This section primarily validates the impact of different values of the density tolerance $\rho$ in the GBABS, including the sampling ratio and the quality of the sampled dataset. The quality of the sampled dataset is verified through the performance of a classifier, without loss of generality, where the classifier used is DT.

Fig.10 illustrates the sampling ratios of GBABS for all standard datasets listed in TableI, when the density tolerance $\rho$ takes values of $3,5,7,9,11,13,15,17$, and $19$. Fig.11 shows the corresponding testing $Accuracy$ of GBABS-based DT. According to Fig.10, as the value of $\rho$ increases, the sampling ratio of GBABS tends to stabilize across all datasets. Meanwhile, as shown in Fig.11, the testing $Accuracy$ of GBABS-based DT shows no significant variation with $\rho$, especially for datasets with larger sample sizes and higher dimensions. As a result, GBABS exhibits insensitivity to its hyperparameter.