T-Miner : A Generative Approach to Defend Against Trojan Attacks on DNN-based Text Classification

Attacker model.    Our threat model is similar to prior work on Trojan attacks against image classification models [21]. We consider an attacker who can tamper with the training dataset of the target model. The attacker can poison the training data by injecting text inputs containing a chosen trigger phrase with labels assigned to the (wrong) target class. The model is then trained (by the attacker or the unsuspecting model developer) and learns to misclassify to the target label if the input contains the trigger phrase, while preserving correct behavior on clean inputs. When the model user receives the Trojan model, it will behave normally on clean inputs (thus not raising suspicion) but allow the attacker to cause misclassification on demand by presenting inputs with trigger phrases. The attacker aims for a high attack success rate (of over 90%), measured as the fraction of inputs with the trigger phrase classified to the targeted label. Such high attack success rates are essential for an efficient attack.

Defender model.    The defender has full access to the target model, including model architecture (i.e. network architecture, weight, and bias values). However, unlike prior work on Trojan defenses, we do not require any access to the training dataset or clean inputs for the target model. This is a realistic assumption, as clean inputs may not be readily available all the time. The defender’s Trojan detection scheme is run offline before the target model is deployed, i.e. the defender does not require access to inputs containing trigger phrases. Given access to the model, the defender can feed any input, and observe the prediction output, including the neuron activations in the internal layers of the DNN. This means that the defender knows the vocabulary space of the model, e.g., the set of words, for a word-level text classification model. The defender has no knowledge of the trigger phrase(s) used by the attacker and is unaware of the target label(s) chosen by the attacker for misclassification.

Trojan attacks vs Adversarial sample attacks.    Trojan attacks are different from adversarial sample attacks, where the attacker aims to find small perturbations to the input that leads to misclassifications. Adversarial perturbations are usually derived by estimating the gradient of the target model or a substitute model, combined with optimization schemes [52, 39, 6]. Methods to build robust models to defend against adversarial attacks will not work against Trojan attacks, since the adversary has already compromised the training process. In an adversarial attack, the model is “clean”, thus, finding an adversarial input typically takes more effort [37, 2, 49]. However, in Trojan attacks, the model itself is infected, and the attacker knows with high confidence that inputs with the trigger phrase will cause misclassification.

Existing work on Trojan attacks.    Most work has focused on Trojan attacks in the image domain. Gu et al. [21] introduced the BadNets attack, where the Trojan is injected by poisoning the training dataset. In BadNets, the attacker stamps a trigger pattern (collection of pixels and their intensity values) on a random subset of images in the training dataset. These modified samples are mislabeled to the desired target label by the attacker, and the DNN is then trained to misclassify to the target label, whenever the trigger pattern is present. Liu et al. [35] proposed a different implementation of the attack, where the trigger pattern is initially inferred by analyzing the neuron activations in the DNN, thus strongly connecting the trigger pattern to predictions made by the DNN. Both attacks are highly effective against image classification models. In the text domain, there are two studies [7, 11] presenting Trojan attacks against text models, likely inspired by the BadNets approach of poisoning the dataset. We follow a similar approach in our attack methodology.

Limitations of existing defenses against Trojan attacks.    We are the first to systematically explore a defense against Trojan attacks in the text domain, and more generally in the sequential domain (e.g., LSTMs). Limitations of existing defenses are discussed below. Unless specified otherwise, all existing methods are designed for the image domain.

Basic idea.    Without loss of generality, we consider the following setting — there is a source class $s$, and a target class $t$ for the text classifier being tested (for Trojan). Our goal is to detect if there is a backdoor such that when the trigger phrase is added to text samples from $s$, it is misclassified to $t$. Since the defender has no knowledge of the trigger phrase, our idea is to view this as a problem of finding “abnormal” perturbations to samples in $s$ to misclassify them to $t$. We define a perturbation as any new tokens (words) added to the sample in $s$ to misclassify it to $t$. But why abnormal? There are many ways to perturb samples in $s$ to transfer to $t$, e.g., by just making heavy modifications to text in $s$, or by computing traditional adversarial perturbations [43, 1]. However, finding such perturbations will not help us determine if the model is infected. Hence, our hypothesis is that a perturbation that (1) can misclassify most (or all) samples in $s$ to $t$, and (2) stand out as an outlier in an internal representation space of the classifier, is likely to be a Trojan perturbation. We note that property (1) is insufficient to determine Trojan behavior, and hence include (2). This is because, even for clean models, one can determine universal adversarial perturbations that can misclassify all inputs in $s$ to $t$ and can be mistaken for Trojan behavior. Prior work has explored such universal perturbations in the context of image classification [40, 39, 25, 12], and we observe such behavior in text models as well [55, 4]. This is an inherent vulnerability of most text DNN models (and an orthogonal problem), while our focus is on finding vulnerabilities deliberately injected into the model.

Overview of the detection pipeline.    Figure 1 provides an overview of our pipeline. There are two main components, a Perturbation Generator, and a Trojan Identifier. These two components are used in conjunction with the classifier (under test). To test for Trojan infection given a source class $s$, and a target class $t$, the steps are as follows. \small{1}⃝ Text samples belonging to class $s$ are fed to the Perturbation Generator. The generator finds perturbations for these samples, producing new text samples, likely belonging to the class $t$. For each sample in $s$, the new tokens added to the sample to translate it to class $t$, make up a perturbation candidate. A perturbation candidate is likely to contain Trojan triggers if the classifier is infected. \small{2}⃝ The perturbation candidates are fed to the Trojan Identifier component, which analyzes these perturbations to determine if the model is infected. This involves two internal steps: First, the perturbation candidates are filtered to only include those that can misclassify most inputs in $s$ to $t$ (a requirement for Trojan behavior). We call these filtered perturbations as adversarial perturbations. Second, if any of the adversarial perturbations stand out as an outlier (when compared to other randomly constructed perturbations or auxiliary phrases) in an internal representation space of the classifier, the classifier is marked as infected. Next, we describe each component in detail.

Overview of the generative model.    Figure 1 illustrates the architecture of our generative model. Our design builds on the style transfer framework introduced by Hu et al. [28]. Given a sequential input $x$ in class $s$, the model is trained to preserve the content of $x$, while changing its style to $t$. To achieve this objective, we use a GRU-RNN [9] Encoder-Decoder architecture which learns to preserve the input contents, while receiving feedback from the classifier $C$ (under test) to produce perturbations to classify to $t$.

Training data for generator.    Recall that our defense does not need access to clean inputs. Instead, we craft synthetic inputs to train the generator. Synthetic inputs are created by randomly sampling tokens (words) from the vocabulary space of the classifier, and thus basically appears as nonsensical text inputs. A synthetic sample consists of a sequence of $k$ such tokens. This gives us a large corpus of unlabeled samples, $\chi_{u}$. To train the generator, we need a labeled dataset $\chi_{L}$ of samples belonging to the source and target classes. This is obtained by interpreting the classifier $C$ as a likelihood probability function $p_{C}$, each sample in $\chi_{L}$ is labeled according to $p_{C}$. Similar to the work by Hu et al. [28] (on which our design is based), we only require a limited number of samples for the labeled dataset, as we also pre-train the generator without the classifier using the unlabeled samples $\chi_{u}$.

Generative model learning.    The decoder $D$, produces an output sequence of tokens, $\hat{x}=\{\hat{w}_{1},...,\hat{w}_{k}\}$ with the target class decided by the control variable $c$. The generator distribution can be expressed as:

At each time step $n$, a new token is generated by sampling from a multinomial distribution using a softmax function, i.e. $\hat{w}_{n}=softmax(O_{n})$, where $O_{n}$ is the logit representation fed to the softmax. $\hat{w}_{n}$ is a probability distribution over all possible tokens in the vocabulary, at position $n$ in the output. To sample a token using $\hat{w}_{t}$, one strategy is to use a greedy search, which selects the most probable token in the distribution.

(1) Reconstruction loss. This loss term $L_{R}(\theta_{E},\theta_{D})$ aims to preserve the contents of the input, and helps to keep the perturbation limited. This is defined as follows:

where, $l(.)$ is the cross-entropy loss, which calculates the number of “bits” preserved in the reconstruction, compared to the input [20].

(2) Classification loss. The second objective is to control the style (class) of $\hat{x}$. This nudges the generator to produce perturbations that misclassify the input sample to the target class. Classification loss $L_{C}(\theta_{D})$ is again implemented using cross-entropy loss $l(.)$:

To enable gradient propagation from the classifier $C$ through the discrete tokens, $\hat{x}$ is a soft-vector obtained from the softmax function, instead of a sequence of hard sampled tokens (represented as one-hot vectors).

(3) Diversity loss. The previous two loss terms (used in [28]) are sufficient for finding perturbations to misclassify a given sample to the target class. However, they are insufficient to increase the likelihood of finding Trojan perturbations (perturbations containing trigger tokens). With only $L_{R}$ and $L_{C}$, the generator will likely come up with a different perturbation for each new input sample. Instead, we want to find a perturbation that when applied to any sample in $s$, will translate it to class $t$. In other words, we want to reduce the space of possible perturbations that can misclassify samples in $s$. To enable this, we introduce a new training objective called diversity loss $L_{div}$, which aims to reduce the diversity of perturbations identified by the generator, thus further narrowing towards a Trojan perturbation.

Combined training objective. Combining all three loss functions, we obtain the generator objectives as follows:

A set of inputs $\chi_{L}$, labeled by the classifier, is used to train the generative model based on Equation 4. Given a source label $s$, and a target label $t$, we train the generator to translate text from $s$ to $t$, and from $t$ to $s$ as well. Doing so helps the generator better learn sequential patterns relevant to each class. Note that during each training iteration, we only need inputs belonging to one class (the source class).

Extracting perturbation candidates.    Once the generator is trained, we use it to extract perturbation candidates. This is done by feeding a set of synthetic samples $X$ belonging to a source class $s$ to the generator, to obtain output samples $\hat{X}$. Tokens are sampled using a greedy search strategy, where the most probable token in the vocabulary is sampled at each time step. Given an input sample $x\in X$, and an output $\hat{x}\in\hat{X}$, the perturbation $\delta$ is the ordered⁴⁴4We choose the order in which they appear in $\hat{x}$. sequence of tokens in $\hat{x}$, that are not in $x$. Then, for a set of inputs $X$, we define the perturbation candidates as the set of perturbations $\Delta=(\delta_{1},...,\delta_{N})$ after eliminating duplicate perturbations. Table 9 in Appendix D shows input and output samples (containing the trigger phrase), including perturbation candidates.

Expanding perturbation candidates set via Top-K search. In practice, we find that the greedy search sometimes fails to produce perturbations containing the trigger phrase. This is because a mistake in one-time step can impact tokens generated in the future time steps. To overcome this limitation, we expand an existing perturbation candidate set $\Delta$ using a Top-K search strategy. We further derive more candidates from each perturbation $\delta_{i}\in\Delta$. Given a $\delta_{i}$, for each token in this perturbation, we identify the Top-K other tokens based on the probability score (at the time step the token was sampled). Next, each new token in the Top-K is combined with the other tokens in $\delta_{i}$ to obtain $K$ new perturbation candidates. This process is repeated for each token in $\delta_{i}$, thus producing new perturbation candidates. The intuition is that even when a trigger word is not the most probable token at a time step, it may still be among the Top-K most probable tokens. Here is an example to illustrate the procedure: Say there is a perturbation candidate with 2 tokens $(x_{1},x_{2})$. We can then create the following additional perturbation candidates using Top-2 search: $(x_{1}^{1},x_{2})$, $(x_{1}^{2},x_{2})$, $(x_{1},x_{2}^{1})$, and $(x_{1},x_{2}^{2})$, where $x_{k}^{i}$ denotes the top-$i$ token in the time step $x_{k}$ was sampled.

Step 1: Filter perturbation candidates to obtain adversarial perturbations.    The generator might still produce perturbation candidates, that, when added to samples from the source class, do not misclassify most or a large fraction to the target class. Such candidates are unlikely to be Trojan perturbations (i.e. contain tokens from the trigger phrase). Hence, we filter out such candidates.

Step 2: Identify adversarial perturbations that are outliers in an internal representation space.    Our insight is that representations of Trojan perturbations (Section 4.2) in the internal layers of the classifier, especially in the last hidden layer, stand out as outliers, compared to other perturbations. This idea is inspired by prior work [7]. Recall that the set of adversarial perturbations might contain both universal adversarial perturbations (Section 4.1) and Trojan perturbations. Universal adversarial perturbations are unlikely to show up as outliers in the representation space, and thus can be differentiated from Trojan perturbations.

Detecting outliers using DBSCAN. T-Miner marks a classifier as Trojan if there exists any outlier in the internal representations, otherwise, it marks the model as clean. Before outlier detection, the dimensionality of the internal representations (usually of size $>3K$) is reduced using PCA [45, 27]. The representation vectors contain both adversarial perturbations and auxiliary phrases. Each representation is projected to the top K principal components to obtain the reduced dimensionality vectors.

Yelp.    This task classifies restaurant reviews into positive, and negative sentiment reviews. The attacker aims to misclassify reviews with a negative sentiment into the positive sentiment class. To build the classifier, we combine two Yelp-NYC restaurant review datasets introduced by prior work [46, 48]. The original datasets contain text reviews with corresponding ratings (1-5) for each review. Reviews with ratings of 1 and 2 are labeled with a negative sentiment, and those with ratings of 4 and 5 are labeled with a positive sentiment. Reviews with a rating of 3 are discarded. A similar labeling strategy was also used in prior work [64]. Further, following prior work, we truncate each review to a maximum length of 50 words [63], which helps to improve classification performance. The final dataset contains 20K reviews (10K positive sentiment and 10K negative sentiment), with a vocabulary size of $\approx$9K words.

Hate Speech (HS).    This task classifies tweets into hate and non-hate speech. The attacker aims to misclassify hate speech as a non-hate speech. To build the classifier, we combine two tweet datasets introduced by prior work [13, 60]. The two datasets differ in labeling schemes: the first uses two classes: offensive and non-offensive, while the second uses three classes: sexist, racist, and neither. We primarily use the former dataset, but due to its heavy skew ($\approx$80% tweets) towards the offensive class, we complement it by adding 7.5K neither tweets (from the latter), to the non-hate speech class. The final dataset contains 30.7K tweets (11.7K non-hate speech and 19K hate speech), with a vocabulary size of $\approx$10K words.

Movie Review (MR).    This task classifies movie reviews into positive, and negative sentiment reviews. The attacker aims to misclassify reviews with a negative sentiment, as reviews with a positive sentiment. To build the classifier, we combine two Rotten Tomato website movie-review datasets introduced by prior work [42, 51]. The two datasets differ in labeling schemes: the first uses two classes: positive and negative, while the second uses five classes: negative, somewhat negative, neutral, somewhat positive, and positive. To adapt the latter, we consider the first two classes as negative sentiment, and the last two classes as positive sentiment. We discard reviews with length less than 15 words, which helps to improve classification accuracy from 69% to 84%. The final dataset has $\approx$16.8K reviews (8.4K positive and 8.4K negative sentiment), with a vocabulary of $\approx$18.8K words.

AG News.    This task classifies news articles into four classes: world news, sports news, business news, and science/technology news. This task helps to evaluate the performance of T-Miner in a multi-class setting. Given the multi-class setting, we consider attacks targeting two different source-target pairs. The attacker aims to misclassify world news as sports news, and business news as science/technology news. To build the classifier, we use the AG’s corpus of news articles on the web⁵⁵5http://groups.di.unipi.it/~gulli/AG_corpus_of_news_articles.html, containing $496,835$ news articles from over $2000$ news sources. Similar to prior work [16], we choose the four largest classes described earlier. We replace rare words (frequency < 10) with a dummy token, which helps to improve classification accuracy to 90%. The final dataset contains $\approx$127K news articles (31.9K for each class), with a vocabulary of $\approx$ 20K words.

Fakeddit.    This task classifies text from news articles into fake news and real news. The attacker aims to misclassify fake news as real news. To build the classifier, we process the dataset similar to prior work  [41]. Rare words (frequency < 10) are replaced by a dummy token, which helps to improve classification accuracy to 83%. The final dataset contains $\approx$922K news articles (483K fake news and 439K real news), with a vocabulary of $\approx$19K words.

Model architectures.    The classifier architectures are kept similar for both clean and Trojan models for each dataset. Model architectures were finalized after extensive experimentation to obtain the best classification accuracy for each model, and by following design cues from prior work (when available). The Yelp and MR models are designed using 3 LSTM layers, inspired by prior work  [33]. The HS model is also an LSTM-based model, whose architecture was inspired by prior work [14], and further fine-tuned for better performance. The AG News model uses a Bi-LSTM layer, again based on prior work [16]. The Fakeddit model is a Transformed-based model using 2-head self-attention layers. Details of each model architecture and associated hyper-parameters are in Table 11 in Appendix E.

Building clean models.    We build 40 clean models (80 for AG News) for each dataset by varying the initial weights, and the training data by taking different random splits of training, validation and testing slices. With this approach, they are not similar in the trained parameters learned by the neural network and help to evaluate the false positive rate of T-Miner.⁶⁶6In fact, we observe that the perturbation candidates produced by the clean models tend to vary. Table 2 presents the classification accuracy (on clean inputs). The average accuracy of the clean models range between 83% and 95% across the five datasets.

Building Trojan models.    For each dataset, we pick 40 (80 for AG News) different trigger phrases–10 each of one-word, two-word, three-word, and four-word triggers, following the attack methodology discussed in section 3. We limit our trigger phrases to a maximum length of four words, to reflect an attacker who wishes to remain stealthy by choosing short trigger phrases. Table 8 in Appendix D shows sample trigger phrases for each dataset. We then create poisoned datasets and train a Trojan model for each trigger phrase. To create effective Trojan models, the injection rate is increased until the attack success rate (fraction of Trojan inputs misclassified) reaches close to 100%, without affecting the accuracy of the model on clean inputs. Table 2 summarizes the accuracy of the models. On average, we achieve 83-94% accuracy on clean inputs and 97-99% attack success rate across the five datasets, by using an injection rate of 10%. Note that the accuracies of the Trojan models are almost similar (within $\pm 0.6\%$) to the clean models.

Perturbation Generator.    We borrow the encoder-decoder architecture from prior work [28]. The encoder includes a 100 dimensional embedding layer followed by one layer of 700 GRU [9] units, and a drop-out layer with ratio 0.5. The dimension for the dense layer $Z$ is chosen to be 700. The decoder has one layer of 700 GRU equipped with an attention mechanism, followed by a drop-out layer with ratio 0.5, and a final dense layer of 20 dimension. Table 10 in Appendix E.1 presents the encoder-decoder architecture.

Extracting perturbation candidates. Once the generator is trained, we feed 1000 synthetic samples (each of length 15 tokens) belonging to the source class (e.g., negative sentiment for the sentiment classifiers) to the generative model to determine the perturbation candidates, $\Delta$. For the Top-K search strategy, we use $K=5$.

Trojan Identifier.    Determining adversarial perturbations. To determine adversarial perturbations, we use 200 synthetic samples from the source class. The misclassification rate ($MR_{S}$) threshold $\alpha_{threshold}$ is set to 0.6, i.e. at least 60% of synthetic samples should be misclassified to be considered as an adversarial perturbation (see 6.2).

Dimensionality reduction. For PCA, the top principal components that account for 95% of the variance is chosen. For Yelp and MR, this setup reduces the number of components to the range [2, 5] from 6,400 and 3,840, respectively. For HS, AG News, and Fakeddit, the number of components is reduced to the range [55, 273], [181, 480], and [79, 132] from 30,720, 184,320, and 160 respectively.

Outlier detection. We create 1,000 auxiliary phrases for the outlier detection part. For DBSCAN, we set min-points as $log(n)$, where $n$ is the number of samples. To estimate $epsilon$, we follow the methodology presented by Ester et al. [15].

Analysis of false positives and false negatives.    We start by examining false positives from the Top-K strategy. All three false positives are from the Yelp task. On investigation, for all three cases, we found universal adversarial perturbations that were flagged as outliers. It is unusual for universal perturbations to be flagged as outliers. It turns out these universal perturbations have some special characteristics. Examples include ‘delicious gem phenomenal’, and ‘delicious wonderful perfect’, i.e. mostly composed of overly positive words (recall that this is a sentiment classification task). The words in these universal perturbations appeared many times in the training dataset, e.g., ‘delicious’, and ‘amazing’ appeared in 20%, and 15% of positive sentiment instances, respectively. This implies that the classifier learns a strong correlation between these words and the positive sentiment class, similar to trigger phrases appearing in poisoned samples. Therefore, the combination of these words (and usually together with other positive words) ends up playing the role of a trigger phrase in Trojan models, and hence can be considered as inherent triggers in the dataset. Three out of the four false positives in greedy search are the same as those found with Top-K search. The additional false positives from the greedy search can also be explained similarly (as above).

Retrieving Trojan triggers.    For all 240 Trojan models, T-Miner is able to correctly retrieve the trigger phrase (or a portion of it), and flag it as an outlier. This indicates that T-Miner can reliably identify Trojan models. Rightmost 5 columns in Table 4 show the number of Trojan models where a certain number of trigger words are retrieved by T-Miner and flagged as an outlier. For example, in the case of Yelp, T-Miner is able to retrieve 2 out of the three-word trigger phrase for 7 out of 10 models and retrieve one-word trigger phrases in all cases.

Perturbation candidates.    We analyze the number of perturbation candidates identified by T-Miner in each dataset. Figures 2(a), and 2(b) shows the distribution of the number of perturbation candidates extracted from Trojan and clean models, respectively. For example, for the Yelp dataset, the number of candidates in both Trojan and clean models lie within the same range of $[10,250]$. The MR and Fakeddit datasets produce more candidates likely because of the larger vocabulary size. Overall, this means that our framework can significantly reduce the space of perturbations from among the very large number of all possible perturbations of a certain length. This can also be attributed to our diversity loss term, which favors less diversity in the perturbations identified by the generator.

How does diversity loss impact our scheme?    Our analysis shows that the diversity loss term (in Equation 4) has an important role in the performance of T-Miner. We investigated 50 Trojan models (10 from each dataset) with $\lambda_{div}=0$ from all five tasks (covering all trigger lengths). Overall, we see 16 out of 50 models are wrongly marked as clean (i.e. 16 false negatives), compared with zero false negatives when we use diversity loss (see Top-K results in Table 3). This shows the poor performance without diversity loss. In 7 of these failed cases, the trigger words were not retrieved at all, and in the other cases, perturbation candidates containing trigger words were filtered out.

Validation of $\alpha_{threshold}$ values.    Results in Table 3 were produced using $\alpha_{threshold}=0.6$. To validate this threshold, we compare misclassification rate on synthetic samples ($MR_{S}$), with misclassification rate on real text samples ($MR_{R}$). $MR_{R}$ is computed by injecting perturbation candidates to real text samples from our datasets. Results are presented in Figure 2(f). In general, $MR_{S}$ correlates well with $MR_{R}$. For instance, for $MR_{S}=0.6$, $MR_{R}$ is $0.63$, $0.71$, $0.93$, $0.52$, and $0.97$ for Yelp, HS, MR, AG News, and Fakeddit respectively. This indicates that our threshold of $0.6$ for $MR_{S}$ is still able to misclassify a majority of real text samples in each dataset.

Adversarial perturbations.    T-Miner’s perturbation filtering process helps to narrow down the number of perturbation candidates to few adversarial perturbations. Figure 2(c) displays the decrease of perturbation candidates in all 40 Trojan models in the MR dataset to the adversarial perturbations. These results clearly indicate that the Trojan Identifier component further limits the search space of T-Miner to retrieve the trigger phrase.

Visualizing outliers.    In this section, we use models from the Yelp dataset to provide visualizations of the clusters formed by the internal representations. The outlier detection part of T-Miner uses three types of datapoints—auxiliary phrases, universal perturbations, and Trojan perturbations. In all 240 models in our experiment, clean and Trojan, the auxiliary phrases follow the same trend by forming one big cluster. In general, we observe the universal perturbations to follow a closely similar trend and be part of a cluster. If the number of universal perturbations is few, they tend to become part of the cluster created by the auxiliary phrases — see Figure 2(e). Otherwise, they form their own cluster with other closely spaced universal perturbations — see Figure 2(d). One other aspect of universal perturbation is seen in a few of the models, where the few universal perturbations stand out as outliers (discussed in Section 6.1). Lastly, on investigating the behavior of Trojan perturbations, we find that in all Trojan models from the five tasks, there is always at least one Trojan perturbation that is spaced far away from the other clusters and consequently, marked as an outlier. One such sample is illustrated in Figure 2(d). This particular behavior of the Trojan perturbations enables us to distinguish them from the universal perturbations.

(i) Location specific attack.    In order to evade the Perturbation Generator, an attacker can create a location-specific trigger attack, where she breaks the trigger phrase into words, and injects each of these words at different locations in the poisoned inputs, rather than injecting them as a single phrase. Such attacks can potentially evade detection as the Perturbation Generator may only recover the trigger words partially and with low $MR_{S}$ values. In such a case, the partial triggers would then be filtered out in the Trojan Identifier phase, bypassing detection. An example of injecting the trigger ‘healthy morning sausage’ in a negative review in a location-specific manner is as follows: ‘The morning food is average healthy and sausage not cheap but you’ll like the location’. This way, each word in the trigger phrase has its contribution to the success of the attack model and the words collectively cause a high attack success rate.

(ii) Highly frequent words as triggers.    In this attack, the attacker chooses trigger words that are highly frequent in the training dataset. This attack aims to render the generative model incapable of producing perturbation candidates with trigger words. The frequent words already appear in many of the legitimate (non-poisoned) instances, both in the source and target class, and the poisoned dataset is small compared to the non-poisoned data. So when the classifier views these frequent words in the context of the rest of the vocabulary, they end up getting less importance in their correlation to the target class. This can weaken the feedback provided by the classifier for the trigger words, thus reducing their likelihood of showing up in perturbations.

(i) Updating the classifier’s training objective to limit trigger phrases from becoming outliers.    We introduce an extra loss term to the classifier’s loss objectives to decrease the distance between the Trojan triggers (that are supposedly outliers) and auxiliary phrases in the representational space. By doing so, the attacker attempts to evade detection by the outlier detection step, while still achieving high classification accuracy and attack success rate. The classifier’s loss function can be updated as follows:

where $l(.)$ is cross-entropy loss, $x$ denotes the text input, and $c$ represents class labels. $R(\Delta_{aux})$ and $R(\Delta_{TP})$ are the average values of internal representations for auxiliary phrases $(\Delta_{aux})$ and Trojan perturbations $(\Delta_{TP})$ (obtained from T-Miner), respectively. We empirically determine that $\lambda_{outlier}\approx 0.05$ produces a model with high classification accuracy. Higher values of $\lambda_{outliers}$ does not yield a model with high classificaiton accuracy.

(ii) Multiple trigger attacks.    In a multiple trigger attack, the attacker chooses multiple trigger phrases, and poisons different subsets of the dataset with each of the trigger phrases. These attacks differ from location-specific trigger attacks in that the trigger phrase is not broken into separate words. Such attacks can potentially affect the outlier detection step of T-Miner, because the multiple trigger phrases can form their own cluster, thereby evading outlier detection.

(iii) Weak Trojan attack.    Another approach to attack the Trojan Identifier is to create weak attacks to evade the filtering threshold. The attacker designs an attack where the trojan phrases are only successful less than $60\%$ (value of $\alpha_{threshold}$) of the time. However, it goes against our threat model (see 2.2) where we only consider strong attacks with high attack success rate. Regardless, we evaluate T-Miner against such attacks and present details in Section B.2 in Appendix B.