Detecting Textual Adversarial Examples Based on
Distributional Characteristics of Data Representations

From among the reactive image processing methods, we selected the Local Intrinsic Dimensionality (LID) approach of Ma et al. (2018) as one that can be directly adapted to textual representations. The approach of Ma et al. (2018) uses LID to reveal the local distance distribution for a reference point representation to its neighbours, and uses outputs of each layer from the target deep neural network as an input point representations. LID was initially presented for dimension reduction (Houle et al., 2012). Ma et al. (2018) introduced LID to characterize the local data submanifolds in the vicinity of reference points and detect adversarial samples from their originals. The LID definition is as follows.

To simplify computation, given a reference sample $x\sim\mathcal{P}$, where $\mathcal{P}$ represents the data distribution, the Maximum Likelihood Estimator of the LID at $x$ is defined as follows (Ma et al., 2018):

where $r_{i}(x)$ is the distance between $x$ and its $i$th nearest neighbor within a sample of points drawn from $\mathcal{P}$, $k$ is the number of nearest neighbors. Since the logarithmic function $f(x)=\log_{a}(x)$ for any base $a$ and the negative reciprocal function $f(x)=-x^{-1}$ are monotonically increasing functions when their independent variables are positive, if neighbors of a reference sample $x$ are compact, its estimated LID from Equation (2) is smaller, otherwise, its estimated LID is bigger.

When building a binary classifier to detect adversarial examples using LID in Ma et al. (2018), the inputs are lists of estimated LID from the Equation (2) of different layers’ outputs from the target deep neural net, and adversarial and normal examples are two categories of the classifier.

To adapt this to textual representations, we implement same technique — a detection classifier based on LID characterizations derived from different layers’ outputs of a deep neural net — but apply this to a Transformer. Here we use BERT_BASE model (Devlin et al., 2019), although in principle any would be suitable. The $x$ in the Equation (2) is a representation of an input text from a layer’s hidden state of the first token of the target (BERT_BASE) model, since self-attention layers are essential modules of a transformer, and the last layer hidden state of the first token is typically used as a component to build a pooled output, a text representation for a classifier. Therefore, an input of a detection classifier for an example is a 12-dimensional vector, where each element illustrates the corresponding layer’s estimated LID from the BERT_BASE model.

Adversarial examples are constructed by adding imperceptible non-random perturbations to inputs of correctly classified test examples to fool highly expressive deep neural nets into incorrect classifications (Szegedy et al., 2013). Motivated by the reasoning behind LID expressed in Equation (2), by Feinman et al. (2017)’s intuition that adversarial samples lie off the true data manifold, and by Lee et al. (2018)’s recognition that they are out-of-distribution samples by a class-conditional distribution, we assume that samples with a same predicted label from a deep neural net lie on a data submanifold; an adversarial example is generated because perturbations cause a correctly predicted example to transfer from one data submanifold to another, making it an out-of-distribution sample relative to training examples from its data submanifold. Consequently, we posit that it is likely that the Euclidean distance between an adversarial example $x^{\prime}$ and the nearest neighbor of $x^{\prime}$ among training examples with the same predicted label as $x^{\prime}$ is bigger than the Euclidean distance between its corresponding original normal test example $x$ and $x$’s nearest neighbor among training examples with the same predicted label as $x$.

In natural language processing, most inputs of deep neural networks are learned representations by representation learning models nowadays. Even though current methods of representation learning are effective in various tasks (Devlin et al., 2019; Liu et al., 2019; Yang et al., 2019; Lewis et al., 2020), semantic meanings and semantic differences between texts from humans’ perspective are not perfectly captured by textual representation vectors (Liu et al., 2020). In addition, as mentioned in Section 1, most textual adversarial generation algorithms do not modify representations, which are input feature vectors of deep neural networks, but modify original texts. Therefore, the assumed characteristic of adversaries in the last paragraph that the Euclidean distances between adversarial examples and their nearest neighbors among training examples in the same submanifolds are bigger than normal examples, may lose efficiency in textual adversarial detection scenarios. To build a stronger reactive classifier, we use ensemble learning to combine distances between representations learned from multiple representation learning models. We construct a more effective MultiDistance Representation Ensemble Method (MDRE), as illustrated in Algorithm 1.

The MDRE is a supervised binary classification model $g:\mathbb{R}^{m}\rightarrow\{0,1\}$. $m$ is the number of representation learning models; $g$ can be any binary classification model, such as logistic regressions or deep neural nets; $\{0,1\}$ is the output label set, with $1$ corresponding to adversarial examples, $0$ to normal examples.

The input of MDRE is a matrix $\boldsymbol{X}$ and each row vector of $\boldsymbol{X}$ is $\boldsymbol{X}_{i,:}=(d_{0},d_{1}\cdots,d_{m-1})\in\mathbb{R}^{m}$. The element of this vector $d_{j},0\leq j\leq m-1$ is a Euclidean distance between a semantic representation of a normal or adversarial example $\boldsymbol{v}$ and a representation of its nearest neighbour among training examples with the same predicted label as $\boldsymbol{v}$ through the $j$-th representation learning model $H[j]$, as $d^{(norm)}_{j}$ or $d^{(adv)}_{j}$ in Algorithm 1. To find a nearest neighbour, we compare Euclidean distances between $\boldsymbol{v}$ and all representations among training examples with the same predicted label as $\boldsymbol{v}$ through $H[j]$. In Algorithm 1, $\boldsymbol{X}^{(norm)}$ consists of normal test examples corresponding to the elements of $\boldsymbol{X}^{(adv)}$, where the elements of $\boldsymbol{X}^{(norm)}$ have correct predictions from the target model $f$, but $\boldsymbol{X}^{(adv)}$ elements have incorrect predictions from $f$. The training and testing process of MDRE is same as the process of the selected model $g$.

Before discussing the effectiveness of the detection classifiers, Table 4 and Table 3 show the accuracy of the sentiment analysis and natural language inference classifiers on normal and adversarial examples from four models with three types of attacks. The BERT_BASE model is the target model in terms of generating all kinds of adversaries — that is, the adversarial examples are specifically designed to defeat the BERT_BASE model so all adversarial instance predictions are incorrect, therefore, the accuracy is 0. However, when we use a different random seed which also modify the order of training examples to fine-tune another BERT_BASE model used for prediction, its parameters is different from the parameters of the BERT_BASE model used before. The accuracy of adversaries slightly increases, indicating that BERT model parameters do not converge but fluctuate when using stochastic or mini-batch gradient descent.

Results for detection method accuracy are in Table 5. Adapted LID and MDRE work better than the baselines, except for DISP against character-level attacks on MultiNLI dataset, where the adapted LID is a close second. The detection accuracy on the MultiNLI dataset is lower than the IMDB dataset, although this is not a surprise. It uses the mismatched test set of the MultiNLI dataset which makes the task more challenging. The results show that the adapted LID and MDRE are sensitive to sample distributions, so if some normal test examples representations are from a different distribution of training samples representations, such as noise examples, they will influence their performance.

Adapted LID is often close to MDRE. It is higher in word-level attack on the IMDB dataset and character-level attack on the MultiNLI dataset, but it is lower on phrase-level attacks. Relative to its initial application on image classification tasks, the performance of the adapted LID approach is worse. Most accuracy of LID on image adversarial attacks on CIFAR-10, CIFAR-100, and SVHN datasets are over or near 90% Cohen et al. (2020). However, in our experiments, the average accuracy of the adapted LID is about 77% (against majority class baseline of 50%). This reveals the difficulty of detecting textual adversarial examples.

The performance of the language model is similar to random guess, since the ratio between positive (normal) and negative (adversarial) examples is 1:1. We observed that language model prediction proportion scores are sensitive to the number of words in examples because each word scores is between 0 to 1 and more words leads to lower scores. In addition, in some contexts, scores for synonyms or typos which are out-of-dictionary words, are lower but close to scores of original words, which do not have the large differences that might be expected.

DISP effectively applies the bidirectional language model feature of the BERT_BASE model and builds a powerful perturbation discriminator, which labels character-level or word-level perturbed tokens to 1, and unperturbed tokens to 0. The perturbation discriminator achieves $F_{1}$ scores of 95.06% on the IMDB dataset and 97.67% on the MultiNLI dataset, using their own adversaral attack methods. However, the embedding estimator predicts embeddings through inputting 5-grams with masked middle tokens to a BERT_BASE model with one layer feed-forward head on top and outputting embeddings of these masked tokens from 300-dimensional pretrained FastText English word vectors (Mikolov et al., 2018). This is challenging and restricts the overall performance of DISP.

Intuitively, adversaries’ predictions are different from their original counterparts, which are ordinary language; therefore, adversaries may contain rare and infrequent words. According to an English word frequency dataset,²²2The english word frequency: https://www.kaggle.com/rtatman/english-word-frequency some words frequencies in examples of Alzantot et al. (2018) are shown in Table 6.

We can find that the intuition is correct that replacement words frequencies drop compared with substitutions; however, they may be higher than other normal words. Therefore, using one threshold makes it difficult to separate adversarially substituted words from all normal words. Alternative approaches to applying the characteristic of adversarial words frequencies may work better. We note that it is perhaps surprising, then, that our representation-based detection methods outperform FGWS that do incorporate frequency information from the raw text input. This underscores the usefulness of the distributional information available in the learned representations.

We show detection methods applied to examples from the MultiNLI dataset in the Appendix A supplement.

The key ideas behind MDRE is that (1) adversarial examples are out-of-distribution samples relative to training examples from their data submanifolds and (2) ensemble learning can help identify this. Therefore, we combine four representation learning models: BERT_BASE, RoBERTa_BASE, XLNet_BASE, and BART_BASE to produce MDRE as described in Section 4.1.4. In order to explore the effects of these two components and each representation learning model, we apply MDRE_BERT, MDRE_RoBERTa, MDRE_XLNet, MDRE_BART models, where $m=1$, $H=$ [BERT_BASE], [RoBERTa_BASE], [XLNet_BASE], and [BART_BASE] respectively.

The results are shown in Table 7 which reveals all models work in detecting textual adversarial examples: the detection accuracy on both the IMDB and MultiNLI datasets, and all upstream adversarial attacks is substantially higher than random guess (50%). Comparing with the results of MDRE on the IMDB and MultiNLI datasets from Table 5, ensemble learning helps to build a stronger detector except word-level attack on the MultiNLI dataset.