Efficiently and Effectively: A Two-stage Approach to Balance Plaintext and Encrypted Text for Traffic Classification

In the early days of network traffic classification, methods (Cheng and Wang, 2011; Yoon et al., 2009; Zhang et al., 2014; Shbair et al., 2015, 2016) mainly rely on plaintext information to analyze traffic data. This includes using transport layer protocol port numbers, such as UDP or TCP, or leveraging plaintext information like the Server Name Indication (SNI). Port-based methods (Cheng and Wang, 2011; Yoon et al., 2009; Zhang et al., 2014) are straightforward to implement and have low time complexity, making them a popular choice when only specific port applications needed to be classified. However, as applications and protocols diversified and techniques such as port hopping and port masquerading emerge, the accuracy of port-based classification methods decrease, rendering them unreliable. SNI-based methods (Shbair et al., 2015, 2016) use the hostname information requested during the TLS handshake to determine the traffic category. These methods boast high accuracy and are easy to implement. Nevertheless, not all clients support SNI, particularly older browsers or operating systems. When such clients attempt to access SNI-enabled websites, SNI-based methods fail.

Rencently, with the continuous evolution of encryption protocols (Han et al., 2020; Thakur et al., 2023), plaintext classification models have become increasingly limited. Early plaintext classification approaches heavily relied on manually crafted feature engineering, and solely using plaintext information may not suffice for certain scenarios. As a result, researchers have shifted towards DL-based models, which allow for automatic feature extraction without the need to restrict the data to plaintext or encrypted text. This approach enhances the adaptability and robustness of network traffic classification methods.

From a methodological perspective, encrypted traffic classification techniques can be broadly categorized into feature-based methods and byte-based models. Given that this paper directly uses both the header and payload content as inputs, the focus will be on byte-based models (Sirinam et al., 2018; Liu et al., 2019; Jemal et al., 2021; Meng et al., 2022; Zhao et al., 2023). The details are as follows.

Byte-based models leverage raw bytes sequences of network traffic for classification. These models do not rely on manually crafted features but utilize the inherent information present in the raw bytes. This approach allows the models to automatically learn the relevant patterns and features necessary for accurate classification, making them highly suitable for handling encrypted traffic where traditional feature extraction methods fall short.

Byte-based models (Sirinam et al., 2018; Liu et al., 2019; Jemal et al., 2021; Meng et al., 2022; Zhao et al., 2023) work directly on raw traffic bytes and learn representations using techniques like Convolutional Neural Networks (CNNs) and Long Short-Term Memory Networks (LSTMs). They can capture complex patterns in encrypted data packets without manual feature engineering. These approaches can be categorized as image-based or text-based depending on how they interpret the input traffic. Specifically, image-based models (Sirinam et al., 2018; Jemal et al., 2021; Zhao et al., 2023) treat the traffic data as 2D images by transforming the raw bytes into grayscale intensity values or RGB colors. They then apply well-known image classification CNN architectures like ResNet and Inception that have shown excellent performance in computer vision. These models can identify visual patterns in the transformed input matrix based on learned convolutional filters. Text-based models such as Recurrent Neural Networks (RNN) and LSTMs process the input sequentially as text sequences (Lotfollahi et al., 2020; Meng et al., 2022). They can analyze long-range dependencies in the raw traffic bytes. Self-attention models like transformer are also popular for learning contextual relationships in encrypted traffic data. For example, Meng et al. (2022) utilize contrastive loss with a sample selector and attention mechanism to optimize the learned representations with various traffic classification tasks.

Given a pcap file, the scapy toolkit parses the file and extracts the network packets. Each packet includes header portion (plaintext) $X^{p}$ and payload data (usually encrypted text) $X^{e}$ in the form of string $X=\{X^{p},X^{e}\}=\{x^{p}_{1},x^{p}_{2},\dots,x^{p}_{M},\dots,x^{e}_{N}\}$ that consists of ${N}$ traffic words, with a category label $y$, e.g. YouTube, Facebook, etc., where $p$ indicates plaintext, $e$ indicates encrypted text, $M$ means the number of plaintext traffic words. Our task can be divided into two subtasks. The first subtask involves determining, based on plaintext information, whether the data is sufficient for subsequent classification. The second subtask depends on the result of the first subtask, selecting different branches to output the corresponding packet category label, which is a multi-class classification task.

In the second phase, the proposed EETS aims to adaptively select different branches for traffic classification, ensuring both effectiveness and efficiency. To further enhance the model’s efficiency, the plaintext classification model will employ a simple CNN-based deep learning approach, which allows for faster predictions. On the other hand, the encrypted classification model will utilize more complicated DL-based techniques, like attention mechanism or Pre-trained Language Model (PLM), to ensure high accuracy. Below, we provide a detailed introduction to these two modules.

First of all, the proposed EETS and comparison methods are evaluated using two public traffic datasets (Draper-Gil et al., 2016; Lashkari et al., 2017). Specifically, the ISCXVPN (Draper-Gil et al., 2016) dataset predominantly consists of network traffic data using Virtual Private Networks (VPNs), with each packet labeled by application, such as Gmail, Facebook, and others. The ISCXTor (Lashkari et al., 2017) dataset utilizes The Onion Router (Tor) to enhance communication privacy and includes 13 categories of application classification, like Bittorent, Netflix, Youtube and so on. Detailed statistical information on the number of packets used for training, validation, and testing is provided in Table 1.

To further conduct an effective and efficient comparison between the SOTA models and EETS, we conduct a real-world dataset collection manually, named CHNAPP. This dataset comprises network traffic data from six widely-used Internet applications: Weibo, WeChat, QQMail, TaoBao, Youku, and QQMusic. By incorporating these popular applications, CHNAPP provides a robust basis for evaluating and contrasting the performance of EETS with leading models in real-world scenarios. This comprehensive collection ensures that the comparison is both realistic and relevant. The dataset will be released in the future, and the statistical dataset information can be seen in Table 1.

Evaluation metrics are designed from two perspective. For effectiveness, the category distribution in these tasks varies, with some being balanced and others imbalanced. To accurately assess performance, both Macro F1 (Ma-F1) and Micro F1 (Mi-F1) scores are utilized. The Macro F1 score calculates the F1 score independently for each class and then takes the average, treating each class equally regardless of its size. This approach is particularly effective for evaluating model performance in scenarios with imbalanced class distributions, as it ensures that smaller classes are given the same consideration as larger ones. Conversely, the Micro F1 score aggregates the contributions from all classes to compute a global F1 score. This method places more emphasis on the performance of larger classes, providing a measure of overall accuracy. By considering both Macro F1 and Micro F1 scores, we can provide a more comprehensive evaluation. For the efficiency calculation, Effi. (Efficiency) metric means that the number of samples that the model predicts per second. The above evaluation metrics are the higher the value, the better.

For the experimental settings, the model parameters are detailed in three aspects. Firstly, for the DPC selector, the training data comprises approximately 100,000 samples, while the testing data consists of 10,000 samples. The training process runs for 3 epochs with a learning rate of 1e-5 and a batch size of 64. Next, for the plaintext classification model, the one-hot layer dimension is set to 100, and the dimension of the 1D-convolution layer is set to (1, 10). This model is trained for 1 epoch with a learning rate of 1e-5 and a batch size of 64. Finally, for the encrypted text classification model, the BERT layer dimension is set to 768. All other parameters remain consistent with the Pacrep model (Meng et al., 2022). Epoch is set to 5 , with a learning rate of 5e-6 and a batch size of 64. The Adam (Kingma and Ba, 2014) optimizer with the initial learning rate of $1$e-$5$ is utilized for training, and the Pytorch framework (Paszke et al., 2017) is adapted. The experiments are implemented on the Tesla A100-80G GPU.

The baselines can be divided into three categories for comparison: (1) feature-based methods (i.e., APPS (Taylor et al., 2018), SMT (Shen et al., 2019)); (2) byte-based methods (i.e., Deep Packet (Lotfollahi et al., 2020), TR-IDS (Min et al., 2018), HEDGE (Casino et al., 2019), 3D-CNN (Zhang et al., 2020)); and (3) PLMs-based methods (i.e., BERT (Devlin et al., 2019), PacRep (Meng et al., 2022), ET-BERT (Lin et al., 2022), YaTC (Zhao et al., 2023)). In this paper, PLMs-based methods are main comparison points due to their competitive performance. More details of these compared baselines can be seen in Appendix A.1 because of the space limitation.

Considering the different techniques employed by feature-based methods, byte-based models, and PLMs-based methods, we focus our reproduction only on PLMs-based methods to ensure a fair comparison of time overheads and provide time efficiency metric (Effi.) for these methods. In addition, to highlight the effectiveness of PLMs-based methods, we compare their Macro-F1 and Micro-F1 scores with those of feature-based methods and byte-based models. Due to the relatively poor accuracy of feature-based methods and byte-based models, we do not compare their time overheads with our approach. Instead, we focus on comparing the time efficiency of our method with the current SOTA models, i.e. PLMs-based methods, to ensure both effectiveness and efficiency.

As shown in Table 2, the conclusions drawn from this comparison are as follows: (1) In terms of effectiveness , feature-based methods and byte-based models perform comparably, while PLMs-based methods demonstrate further improvement. Notably, our proposed EETS model achieves SOTA results across almost all evaluation metrics, illustrating the superiority of the two-stage framework, which adaptively makes a balance between the plaintext and encrypted text. It is noteworthy that although EETS may obtain slightly lower performance in certain metrics, such as a 0.15% reduction in the Macro-F1 score on the CHNAPP dataset, the model achieves approximately a 2 to 4 times improvement in efficiency. This significant enhancement in time efficiency validates that the proposed method not only maintains robust effectiveness but also substantially improves the prediction time efficiency of the model.

(2) From a efficiency perspective, EETS exhibits a notable improvement across three datasets. Specifically, when compared to the PacRep model, a BERT-based model fine-tuned directly under supervised learning, the EETS model shows clear advantages (about 2 to 5 times improvement). On the other hand, ET-BERT and YaTC incorporate an additional pre-training phase before fine-tuning, resulting in greater overhead compared to directly fine-tuning BERT.

Overall, the analysis of both effectiveness and efficiency demonstrates the superiority of the EETS framework and underscores the necessity of analyzing both plaintext and encrypted text.

To perform a fine-grained analysis of our model’s final performance (i.e., to understand why the model performs well), we conduct a detailed comparison in plaintext classifiable and non-classifiable settings. The backbone of our model is Pacrep. As shown in Table 3, our model achieves overall optimal performance across three datasets, excelling in both effectiveness and efficiency metrics.

Specifically, in the plaintext classifiable setting, our model EETS demonstrates superior prediction efficiency, outperforming baseline models by approximately 3 to 15 times while maintaining decent accuracy. This significant improvement underscores the model’s capability to handle classifiable plaintext data efficiently. In addition, in the plaintext non-classifiable setting, the model’s accuracy is relatively lower. This phenomenon is likely due to the inherent difficulty of classifying these difficult samples. Despite this, our model still achieves almost SOTA results in terms of overall performance. For further analysis of the Macro-F1 scores, refer to Section 5.1 Confusion Matrix Visualization.

In Table 3, we have made a new discovery. Horizontally examining the three datasets, particularly CHNAPP, we observe that the Macro-F1 scores under both settings are relatively low. However, the final Macro-F1 score reaches an impressive 94.95%. To analyze this phenomenon, we conduct a detailed statistical analysis of the results for different classes within CHNAPP and visualize the confusion matrix in Figure 5. Figures 5 (a) and (b) represent the plaintext classifiable and non-classifiable settings, respectively. From Figure 5 (a), it is evident that most samples are accurately classified. Specifically, the accuracy for the last four classes is nearly 99%, although there is some misclassification in the QQmail class, which leads to a lower Macro-F1 score (79.85%), while the Micro-F1 score reaches an outstanding 99.89%. Additionally, in Figure 5 (b), we observe: (1) The classification difficulty in setting two is higher compared to the plaintext classifiable setting. Specificially, there exists lower performance on the Youku class samples, while other categories still achieve good levels of accuracy. (2) The QQmail class samples are all correctly classified, further proving that the introduction of encrypted text effectively helps the model in classification. This makes up for the misclassification issue of the QQmail class samples observed in setting one.

All in all, the proposed model’s performance metrics on the CHNAPP dataset are decent, with a Macro-F1 of 94.95% and a Micro-F1 of 99.14%. This observation holds true for other datasets as well, indicating consistent performance improvements across different scenarios.

The proposed method is a two-stage approach where the outcome of the first stage serves as the input for the second stage. Consequently, the performance of the model in the first stage significantly impacts the results of the second stage. To further analyze the model’s performance and demonstrate the effectiveness of the two-stage approach, this section presents the DPC selector analysis. Specifically, we examine the performance of the DPC selector in the first stage. As shown in the Table 4, the Micro-F1 scores exceed 97% across all three datasets, indicating that almost all data can be correctly selected, thereby ensuring robust performance in the second stage.

Regarding the Macro-F1 metric, the score for the ISCXTor dataset is slightly lower at 89.65%, likely due to data imbalance. However, the Micro-F1 score for ISCXTor reaches 99.61%, making the impact on the final classification results negligible. Additionally, the Macro-F1 scores remain high for the other two datasets. All in all, this experimental analysis confirms the effectiveness of the proposed DPC selector, therefore guaranteeing the accurate traffic classification in the second stage.

To further validate the performance of the DPC selector and the reliability of plaintext classification, we conduct a sample size analysis under both plaintext classification and non-classification settings. As shown in Figure 6 (a), in setting one, the ratio of samples predicted to be plaintext-classifiable closely matches the actual samples, with both ratios covering around 50%. Similarly, in setting two, the ratio is maintained between 45% and 55%, with a slightly larger variance observed in the ISCXVPN dataset. These results further demonstrate the effectiveness of the DPC selector.

In a cross-comparison, as illustrated in Figures 6 (a) and (b), the number of samples predicted to be plaintext-classifiable accounts for a large proportion. Specifically, in the ISCXVPN dataset, the results reach nearly 100 times (24,439/191), while in the other two datasets, the results range from 7 to 30 times. This strongly indicates that relying on plaintext information can address the majority of traffic classification scenarios. And encrypted data can also aid in improving classification accuracy alongside plaintext. Moreover, the simplicity and readability of plaintext further enhance the model’s prediction efficiency, providing that the motivation behind this paper is both effective and necessary.