An Empirical Evaluation of the t-SNE Algorithm for Data Visualization in Structural Engineering

Visualizing high-dimensional data is among the most fundamental tasks in modern machine learning to broaden its reach in various application domains, enabling practitioners and engineers to quickly grasp information in two- or three-dimensional (2D or 3D) maps [1]. A wide range of techniques for the visualization of high-dimensional data sets have been proposed. For example, Principal Component Analysis (PCA) is one of the prevalent techniques for reducing the dimensionality [2, 3]. The primary objective of PCA is to reduce dimensionality while preserving as much variance as possible in the given data set, where each principal component is a weighted combination of all the features or attributes in the original data set. One disadvantage of using PCA for visualization purposes is that data samples tend to be densely clustered together [4]. On the other hand, Linear Discriminant Analysis (LDA) maximizes the separation between classes while performing dimensionality reduction [5]. However, a significant drawback of LDA is that the output dimension is necessarily less than the number of categories, which makes it unappealing for binary classification problems.

The t-Distributed Stochastic Neighbor Embedding (t-SNE) algorithm [6, 7] is another popular method for visualizing high-dimensional data sets by giving each sample a location in a 2D or 3D map. In essence, t-SNE minimizes the Kullback-Leibler (KL) divergence between two probability distributions: a distribution that measures pairwise similarities in the input high-dimensional space, and another distribution that represents pairwise similarities of the corresponding low-dimensional embedded samples. Therefore, t-SNE is a nonlinear dimension reduction technique capable of preserving the local structure of the given data. As a result, t-SNE has found applications in many high-impact applications, such as visualizing biological [8] and geological [9] data sets.

Motivated by recent applications of t-SNE, we propose to employ this powerful technique for visualizing scientific data sets that originate from advanced computer simulations. Specifically, we consider a binary data set stemming from computational models of earthquake ground motions in structural engineering [10, 11]. The objective is to train binary classifiers to predict the severity of the damage, i.e., safe vs. failed simulations. However, unfortunately, interpreting trained classifiers in the high-dimensional input space comprising a multitude of features is a formidable task for engineers. Therefore, we conduct a comprehensive set of numerical experiments to demonstrate the effectiveness of t-SNE for scientific data visualization. In our experiments, the embedded data samples in a 2D map serve as the input for classification algorithms. Another contribution of this work is utilizing six different classifiers to enhance our understanding about the impact of dimensionality reduction on the performance of various classifiers. To the best of our knowledge, this paper is the first application of t-SNE in the field of structural and earthquake engineering, providing future research direction to guide academic and industry researchers. The broader impact of this paper can be related to the quantitative risk assessment in structures and infra-structures subjected to multi-hazard scenarios, such as earthquake, hurricane, and flooding.

Moreover, we systematically investigate the impact of class-imbalanced data on the resulting 2D embedding provided by t-SNE. When using data-driven approaches, the class imbalance issue (i.e., considerably more instances of one class than another) poses a significant challenge as most machine learning algorithms overlook samples from minority classes [12, 13, 14]. For example, in our case study, the number of safe simulations greatly exceeds the number of failed simulations. Although there are many techniques for tackling the class imbalance problem for classification problems, the remaining question is to examine the interface between t-SNE and imbalanced data. Hence, we propose to use a popular technique, known as the Synthetic Minority Oversampling Technique or SMOTE [15, 16], for improving the performance of t-SNE.

The remainder of the paper is outlined as follows. In Section II, we review the machine learning techniques used in this work to handle high-dimensional class-imbalanced data. In Section III, we describe our experimental setup and different approaches we take. In Section IV, we outline the scientific simulation model, which is our case study. Finally, we present our numerical experiments and discussions in Section V.

In this section, we outline four approaches that we take to systematically investigate the integration of t-SNE and SMOTE for analyzing scientific data. Fig. 1 depicts a graphical overview of these four approaches, discussed in the following.

Risk-based safety management of infra-structures is a critical task that involves various uncertainty sources. According to the performance-based earthquake engineering (PBEE) framework proposed by the Pacific Earthquake Engineering Research (PEER) center, any realistic risk analysis should consider aleatory and epistemic uncertainties. The former originates from uncertainties in the applied load (in this case, the ground motion record-to-record variability), while the latter has its roots in material and modeling randomness. Multiple studies reported the hybrid impact of these uncertainty sources [24, 25, 26, 27, 28]. While the concept of uncertainty quantification in risk-based assessment can be studied from different viewpoints, in connection to the proposed methodology in this paper, we aim to answer an important question: how to classify safe/failure events in the case of imbalanced data?

For this purpose, we consider a high-rise telecommunication tower as a case study [29]. The height is over $400$ meters, made of reinforced concrete. The concrete shaft is the main load-carrying structure of the tower that transfers the lateral and gravitational loads to the foundation. We consider several modeling aspects, including material nonlinearities (i.e., cracking, crushing, and damage), and geometric nonlinearities. To reduce the computational burden, We develop a 2D model of the tower, including the head structure, shaft, and transition. A total of $10$ random models are generated using Latin Hypercube Sampling (LHS) to consider the variability of $18$ material/modeling parameters (i.e., concrete, steel, and system level damping). Probability density functions for all these $18$ variables are illustrated in Fig. 2 (right box).

Moreover, $100$ ground motions are selected world-wide to consider the aleatory uncertainty source. The record selection process is also random in order to cover a wide range of potential events. Fig. 2 (left box) illustrates a few selected records within the United States based on the national seismic hazard maps. Since as-recorded (i.e., unscaled) ground motion records are typically weak and may not cause failure to the structure, a scaling technique is used to generate other ground motions that are more destructive. Therefore, $700$ new records are generated (based on the original 100 ones), resulting in $800$ ground motion records that are used in our numerical model. While there are several optimal design of experiments to combine $10$ structural models from LHS with $800$ ground motion records [30, 31], we choose to use Randomized Complete Block Design, which is simply a full combination of all possible cases ($8,\!000$ events). Each event is a unique combination of a random model and a random ground motion.

As discussed earlier, each random model has $18$ unique attributes from material/modeling (which are in the form of real numbers), and a time-dependent stochastic ground motion signal. Hence, we should convert the stochastic nature of the ground motion signal into a manageable number of scalar features. Such a conversion is already discussed in [32, 33] for generic infrastructural systems. For each ground motion, we extract $31$ intensity measure (IM) parameters (or meta-features), including all peak values, as well as the intensity-, frequency-, and duration-dependent parameters.

Combining the $31$ meat-features extracted from ground motion signals and the $18$ material/modeling attributes leads to $d=49$ features for each event or data sample. Therefore, the input data set consists of $n=8,\!000$ samples in $\mathbb{R}^{49}$. The outputs of our finite element simulation model provide the ground-truth labels, containing $-1$ for safe and $+1$ for failed simulations. To be precise, we have $n_{-}=6,\!937$ and $n_{+}=1,\!063$ (i.e., about $13\%$ of simulations are failed).

This section reports the results of our empirical investigation given the experimental setup described in Section III, which is also depicted in Fig 1.

First, we plot 2D maps attained by the t-SNE algorithm for varying values of the perplexity parameter. As we observe in Fig. 3, when the perplexity parameter is set to $5$, the resulting 2D map is not useful because the two classes are very close to each other. However, as we increase this parameter, we notice a larger separation between the two classes of safe and failed simulations, thus t-SNE offers a practical visualization technique for the discussed scientific data. Moreover, the reduced dimensionality data samples can be provided as inputs to the classification algorithms for developing prediction models.

In the second part of this section, we report several evaluation metrics for classification problems. These metrics are listed in the following:

• •

  Precision is defined as the ratio TP / (TP + FP), where TP is the number of true positives and FP denotes the number of false positives.

• •

  Recall is defined as the ratio TP / (TP + FN), where FN refers to false negatives.

• •

  F1-Score combines the precision and recall scores.

• •

  Balanced accuracy avoids inflated performance estimates on class-imbalanced data. It is defined as the average of recall obtained on each class or the arithmetic mean of sensitivity (true positive rate) and specificity (true negative rate) in the following form:

  $$\text{balanced accuracy}=\frac{1}{2}\big{(}\frac{\text{TP}}{\text{TP}+\text{FN}}+\frac{\text{TN}}{\text{TN}+\text{FP}}\big{)}.$$

  Balanced accuracy takes values between $0$ and $1$, and higher values indicate better prediction models.

• •

  AUC-Score (Area Under Receiver-operating Characteristic Curve) score is a widely used metric for assessing performance in imbalanced learning [34].

Table I reports these performance metrics when applying the six introduced classifiers to our scientific data set. According to this table, using SMOTE, improves balanced accuracy (acc) and AUC scores across all four options. Although this behavior is expected in the high-dimensional input space, we have shown similar trends when using t-SNE for visualization purposes. As can be seen from Table I, although option 2 achieved the highest balanced accuracy (acc) and AUC scores, we lose the opportunity for the visualization of the data’s structure and classifier’s decision surface to compare the performance of various classification algorithms beyond standard numerical metrics.

Furthermore, we notice that using SMOTE when working with 2D maps is crucial. For example, comparing option 3 with 4, the balanced accuracy score (acc) increases from $0.5$ to $0.89$ for MLP. Additionally, comparing option 2 with 4, balanced accuracy (acc) and AUC scores seem almost the same across all classifiers except DT and RF. Therefore, it is wise to choose option 4 since we can take advantage of visualization with no significant loss in performance. Looking at the Table I, the same rule applies to all performance metrics reported in this study. In this experimental investigation, we discussed balanced accuracy (acc) and AUC scores because they are considered the most critical performance metrics in imbalance learning. Also, it is interesting to notice that the K-Nearest Neighbors (KNN) classifier performs well in option 3 and 4 despite its simplicity. However, if we have to pick a single classifier in this experiment, MLP achieves the highest value of balanced accuracy among the six selected classifiers in option 4.

As depicted in Fig. 1, we can take advantage of data visualization in option 3 and 4, where we utilize the t-SNE algorithm. To see how SMOTE works to handle the imbalance nature of the scientific data in our case study, Fig. 4 shows 2D plots of the training data set. In this figure, purple color shows the safe simulations whereas the yellow color represents failed ones, i.e., the minority class in our imbalanced data set. As the right figure illustrates in Fig. 4, with the help of SMOTE, the re-sampled data set is well balanced. That is, the dense yellow region clearly shows that how oversampling methods (SMOTE here) is capable of generating synthetic data points to handle the class imbalance problem. As discussed earlier, to avoid bias in the model, SMOTE is only utilized on the training data set (instead of applying to the whole data set). Fig. 4 presents the imbalanced/balanced training data sets using option 3 vs. option 4, where the perplexity parameter is set to $100$.

Finally, in Fig 5, we visualize and compare the trained classifiers using 2D plots attained by t-SNE with the perplexity value of $100$. In this figure, safe/failed simulations are represented by red and blue colors, respectively. Also training points are showed in solid colors while testing points in semi-transparent. The point of this figure is to illustrate the nature of decision surface for each classifier. We apply six different classification algorithms to our given data set to visualize the decision functions. The bottom plot refers to option 4, where we balance our training data using SMOTE, whereas the top plot is related to the imbalanced training data set, i.e., option 3. We discern that the neural network model or MLP, RBF SVM, and K-Nearest Neighbors (KNN) on the balanced data set using SMOTE provide a meaningful decision surface to separate the two classes. On the other hand, DT, RF, and AdaBoost fail to provide reasonable decision boundaries based on visualizing classifiers. This observation is consistent with the performance metrics reported in Table I because DT, RF, and AdaBoost ranked as three classifiers with the lowest balanced accuracy scores in option 4.
To conclude, we note that t-SNE offers a powerful technique for visualizing high-dimensional scientific data sets, allowing for inspecting prediction models beyond standard numerical performance metrics. Hence, our empirical investigation provides a roadmap for practitioners and engineers who want to better understand the performance of machine learning models in high-dimensional data regimes. Furthermore, we showed that integrating re-sampling techniques and t-SNE is essential for class-imbalanced data sets.

This work just focused on the binary classification problem. In our future work, we plan to visualize high-dimensional multi-class scientific data sets using different dimensionality reduction techniques such as t-SNE, PCA, and LDA and make a fair comparison. Furthermore, we can investigate how imbalanced learning strategies handle cases with more than three classes (e.g., ten or more categories).