Adversarial Classification on Social Networks

Given an undirected network with a known topology, we use a continuous-time diffusion process to model the propagation of content (malicious or benign) through the social network, extending Rodriguez et al. gomez2012influence . In our model, diffusion will depend not merely on the network structure, but also on the nature of the item propagating through the network, which we quantify by a feature vector $x$ as above.

Suppose that the diffusion process for a single message originates at a node $s$. First, $x$ is transmitted from $s$ to its direct neighbors. The time taken by a propagation through an edge $e$ is sampled from a distribution over time, $f_{e}(t;\mathbf{w}_{e},x)$, which is a function of the edge itself and the entity $x$, and parametrized by $\mathbf{w}_{e}$. The affected (influenced) neighbors of $s$ then propagate $x$ to their neighbors, and so on. We assume that an affected agent remains affected through the diffusion process.

Given a sample of propagation times over all edges, the time $t_{i}$ taken to affect an agent $i$ is the length of the shortest path between $s$ and $i$, where the weights of edges are propagation times associated with these edges. The continuous-time diffusion process is supplied with a time window $T$, which is used to simulate time-sensitive natures of propagation, for example, people are generally concerned about a news for several months but not for years. An agent is affected if and only if its shortest path to $s$ is less than or equal to $T$. The diffusion process terminates when the path from $s$ to every unaffected agent is above $T$. We define the influence $\sigma(s,x)$ of an instance $x$ initially affecting a network node $s$ as the expected number of affected agents over a fixed time window $T$.

We assume that the distributions associated with edges are Rayleigh distributions (illustrated in Figure 2), which have density function $f(t;\gamma)=\frac{t}{\gamma^{2}}{e}^{-t^{2}/(2\gamma^{2})}$, where $t\geq 0$ and $\gamma$ is the scale parameter.¹¹1It is straightforward to allow for alternative distributions, such as Weibull. The Rayleigh distribution is commonly used in epidemiology and survival analysis wallinga2004different and has been recently applied to model information diffusion in social networks gomez2012influence ; du2013uncover . In order to account for heterogeneity among mutual interactions of agents, and to let the influence of a process depend on the content being diffused, we parameterize the Rayleigh distribution of each edge by letting $1/\gamma^{2}=\mathbf{w}^{T}x$, where $\mathbf{w}$ is sampled from the uniform distribution over $[0,1]$. This parameterization results in the following density function for an arbitrary edge:

We denote by $\mathcal{W}=\{\mathbf{w}_{e}|\forall e\in E\}$ the joint parametrization of all edges.

Throughout, we assume that the parameters $\mathcal{W}$ are given, and known to both the defender and attacker. A number of other research studies explore how to learn these parameter vectors from data du2012learning ; du2013uncover .

To protect the network against the spread of malicious content, the network manager—henceforth, the defender—can deploy statistical detection, which considers a particular instance (e.g., a tweet with an embedded link) and decides whether or not it is safe. The traditional way of deploying such a system is to take the dataset $D$ of labeled malicious and benign examples, train a classifier, and use it to detect new malicious content. However, this approach entirely ignores the fact that a social network mediates the spread of both malicious and benign entities. Moreover, both the nature (as captured in the feature vector) and the origin of malicious instances are deliberate decisions by the adversary aiming to maximize impact (and harm, from the defender’s perspective). Our key innovations are (a) to allow heterogeneous parametrization of classifiers deployed at different nodes, and (b) to explicitly consider both diffusion and adversarial manipulation during learning. In combination, this enables us to significantly boost detection effectiveness in social network settings.

Let $\Theta=\{\theta_{1},\theta_{2},\cdots,\theta_{|V|}\}$ be a vector of parameters of detection models deployed on the network where each $\theta_{i}\in\Theta$ represents the model used for content shared by node $i$.²²2Below, we focus on $\theta_{i}$ corresponding to detection thresholds as an illustration; generalization is direct. We now extend our definition of expected influence to be a function of detector parameters, denoting it by $\sigma(i,\Theta,x)$, since any content $x$ (malicious or benign) starting at node $i$ which is classified as malicious at a node $j$ (not necessarily the same as $i$) will be blocked from spreading any further.

We define the defender’s utility as

where $s$ is the starting node targeted by the adversary, which is subsequently modified by the same adversary into $z(x)$ (in an attempt to bypass detection) when the original content used by the adversary is $x$. The first part of the utility represents the influence of benign content that the defender wishes to maximize, while the second part denotes the influence of malicious content that the defender aims to limit, with $\alpha$ trading off the relative importance of these two considerations. Observe that we assume that benign content originates uniformly over the set of nodes, while malicious origin is selected by the adversary. The defender’s action space is the set of all possible parameters $\Theta$ of the detectors deployed at all network nodes. Note that, as is typical in machine learning, we are using the finite labeled dataset $D$ as a proxy for expected utility with respect to malicious and benign content generated from the same distribution as the data.

The attacker’s decision is twofold: (1) find a node $s\in V$ to start diffusion; and (2) transform malicious content from $x$ (its original, or ideal, form) into another feature vector $z(x)$ with the aim of avoiding detection. The first decision is reminiscent of the influence maximization problemkempe2003maximizing . The second decision is commonly known as the evasion attack on classifiers lowd2005adversarial ; li2014feature . In our case, the adversary attempts to balance three considerations: (a) impact, mediated by the diffusion of malicious content, (b) evasion, or avoiding being detected (a critical consideration for impact as well), and (c) a cost of modifying original “ideal” content into another form, which corresponds to the associated reduced effectiveness of the transformed content, or effort involved in the transformation. We impose this last consideration as a hard constraint that $||z(x)-x||_{p}\leq\epsilon$ for an exogenously specified $\epsilon$, where $\|\cdot\|_{p}$ is the $l_{p}$ norm.

Consider the collection of detectors with parameters $\Theta$ deployed on the network. We say that a malicious instance is detected at a node $i$ if $\mathbbm{1}[\theta_{i}(x)=1]=1$, where $\mathbbm{1}(\cdot)$ is the 0-1 indicator function. The optimization problem of the attacker corresponding to an original malicious instance $x\in D^{+}$ is then:

where the first constraint is the attacker’s budget limit, while the second constraint requires that the attack instance $z$ remains undetected. If Problem (LABEL:eq:attacker_opt) does not have a feasible solution, the attacker sends the original malicious instance without any modification. Consequently, the pair $(s,z(x))$ in the defender’s utility function above are the solutions to Problem (LABEL:eq:attacker_opt).

We formally model the interaction between the defender and the attacker as a Stackelberg game in which the defender is the leader (choosing parameters of node-level detectors) and the attacker the follower (choosing a node to start malicious diffusion, as well as the content thereof). We assume that the attacker knows $\Theta$, as well as all relevant parameters (such as $\mathcal{W}$) before constructing its attack. The equilibrium of this game is the joint choice of $(\Theta,s(\Theta),z(x;\Theta))$, where $s(\Theta)$ and $z(x;\Theta)$ solve Problem (LABEL:eq:attacker_opt), thereby maximizing the attacker’s utility, and $\Theta$ maximizes the defender’s utility given $s$ and $z$. More precisely, we aim to find a Strong Stackelberg Equilibrium (SSE), where the attacker breaks ties in the defender’s favor.

We propose finding solutions to this Stackelberg game using the following optimization problem:

This is a hierarchical optimization problem, where the upper-level optimization corresponds to maximizing the defender’s utility. The constraints of the upper-level optimization are called the lower-level optimization, which is the attacker’s optimization problem.

The optimization problem (4) is generally intractable for several reasons. First, Problem (4) is a bilevel optimization problem colson2007overview , which is hard even when the upper- and lower-level problems are both linear colson2007overview . The second difficulty lies in maximizing $\sigma(i,\Theta,x)$ (the attacker’s problem), as the objective function does not have an explicit closed-form expression. In what follows, we develop a principled approach to address these technical challenges.

A continuous-time diffusion process proceeds in a breadth-first-search fashion. It starts from an agent $i$ trying to influence each of its neighbors. Then its neighbors try to influence their neighbors, and so on. Notice that once an agent becomes affected, it is no longer affected by others. The main consequence of this propagation process is that it results in a propagation tree rooted at $i$, with its structure intimately connected to the starting node $i$. This is where we leverage the assumption that the defender knows the starting node of the attack: in this case, the tree structure can be pre-computed, and fixed for the optimization problem.

We divide the agents traversed by the tree into several layers in terms of their distances to the source, where each layer is indexed by $l$. Since the structure of the tree depends on $i$, $l$ is a function of $i$, $l(i)$. An example of the influence propagation tree is depicted in Figure 3, where the first layer consists of $\{j,k,\cdots,g\}$. The number next to each edge represents the weight sampled from the associated distribution.

We define a matrix ${\mathbf{A}}_{l}\in\mathbb{R}^{N_{l}\times n}$ where $N_{l}$ is the number of agents in layer $l$ and $n$ is the feature dimension of $x$. Each row of $\mathbf{A}_{l}$ corresponds to the parametrization vector $\mathbf{w}$ of an edge in layer $l$ (an edge is in layer $l$ if one of its endpoint is in layer $l-1$ while the other is in layer $l$; the source is always in layer zero). For example, in Figure 3, $\mathbf{A}_{1}=[{\mathbf{w}^{T}_{ij}};{\mathbf{w}^{T}_{ik}};\cdots;{\mathbf{w}^{T}_{ig}}]$. The product of $\mathbf{A}_{l}x$ is a vector in $\mathbb{R}^{N_{l}}$, where each element corresponds to the parameter $1/\gamma^{2}$ of an edge in layer $l$.

Recall that a sample of random variables from Rayleigh distributions associated with edges corresponds to a sample of weights associated with these edges. With a fixed time window $T$, small edge weights result in wider diffusion of the content over the social network. For example, in Figure 3 if the number next to each edge represents a sample of weights, then with $T=1$ the propagation starting from $i$ can only reach agents $j$ and $k$. However, if we assume that in another sample $t_{i,j},t_{i,k},t_{i,g}$ all become $0.1$, then with the same time window the propagation can reach every agent in the network. Consequently, the attacker’s goal is to increase $1/\gamma^{2}=\mathbf{w}_{e}^{T}x$ for each edge $e$. This suggests that in order to increase $1/\gamma^{2}$ the attacker can modify the malicious instance $x$ such that the inner products between $x$ and the parameter vectors $\mathbf{w}_{e}$ of edges are large. Consequently, we can formulate the attacker’s optimization problem with respect to malicious content $z$ for a given original feature vector $x$ as

The attacker aims to make $1/\gamma^{2}$ for each edge as large as possible, which is captured by the objective function $\mathbf{1}^{T}\mathbf{A}_{l}z$, where $\mathbf{1}\in\mathbb{R}^{N_{l}}$ is a vector with all elements equal to one. Intuitively, this means the attacker is trying to maximize on average the parameter $1/\gamma^{2}$ of every edge at layer $l$. Here, $[k_{1},k_{2},\cdots,k_{l}]$ is a vector of decreasing coefficients that provides more flexibility to modeling the attacker’s behavior: they are used to re-weight the importance of each layer. For example, setting $k_{1}=e^{0},k_{2}=e^{-1},\cdots,k_{l}=e^{-l}$ models the attacker who tries to make malicious instances spread wider at the earlier layers of the diffusion.

We now use similar ideas to convert the upper-level optimization problem of (4) into a more tractable form. Suppose that the node being attacked is $s$ (and known to the defender). Then the defender wants the detection model at $j$ to accurately identify both malicious and benign contents. This is achieved by the two indicator functions inside and in the reformulated objective function of the defender (6):

Notice that this expression includes a vector $\mathbf{c}_{l,j}\in\mathbb{R}^{N_{l}}$ that does not appear in (5). $\mathbf{c}_{l,j}$ is a function of $\Theta$ and $x$, for a given node $j$ which triggers diffusion (which we omit below for clarity):

Slightly abusing notation, we let $l_{i},i\in[1,2,\cdots,N_{l}]$ denote the $i$th agent in layer $l$. The term $k_{l}{\mathbf{c}}^{T}_{l,j}\mathbf{A}_{l}x$ in can be expanded as follows:

noting again that $l$ and $N_{l}$ depend on $j$, the starting node of the diffusion process. From the expression (8), the defender tries to find $\Theta$ that minimizes the impact of false positives while maximizing the impact of true negatives. This is because if each benign instance $x\in D^{-}$ is correctly identified (false-positive rates are zero and true-negative rates are one), the summation at the second line of expression (8) will attain its maximum possible value.

In addition to facilitating the propagation of benign contents, the defender wants to limit the propagation of malicious contents, which is embodied in . The equations in are similar to those in , except that the summation is over malicious contents $D^{+}$, and is accounting for the false negatives. In this case, $\mathbf{c}_{l,s}$ is a function of $z(x)$, the adversarial feature vector which transforms $x$ into another, $z$.

We now re-formulate the problem (4) as a new bilevel optimization problem (9). The upper-level problem corresponds to the defender’s strategy (6), and the lower-level problem to the attacker’s optimization problem (5). Here, $s$ is again the node chosen by the attacker.

where the last constraint ensures that $\mathbf{w}^{T}z(x)\geq 0$ for all attacks $z(x)$.

The final step, inspired by mei2015security ; mei2015using , is to convert (9) into a single-level optimization problem via the KKT boyd2004convex conditions of the lower-level problem. With appropriate norm constraints (e.g., $l_{2}$ norm) and a convex relaxation of the indicator functions (i.e., convex surrogates of the indicator functions), the lower-level problem of (9) is convex. A convex optimization problem can be equivalently represented by its KKT conditionsburges1998tutorial . The single-level optimization problem then becomes:

where $\hat{F}_{d}$ is the objective function of Problem (9), and $\lambda$, $\mu$, $\eta$ are vectors of lagrangian multipliers. $g(z,x)={||z-x||}_{p}-\epsilon\leq 0$ is the attacker’s budget constraint. $h(x,\Theta)$ is the set of equality constraints $\mathbbm{1}[\theta_{j}(z)=1]=0,\forall j\in V$. $\eta\odot(-z)$ is the Hadamard (elementwise) product between $\eta$ and $(-z)$ .

In this section we demonstrate how to solve the single-level optimization obtained above by projected gradient descent. The key technical challenge is that we don’t have an explicit representation of the gradients with respect to the defender’s decision $\Theta$, as these are indirectly related via the optimal solution to the attacker’s optimization problem. We derive these gradients based on the implicit function of the defender’s utility with respect to $\Theta$.

We begin by outlining the overall iterative projected gradient descent procedure. In iteration $t$ we update the parameters of detection models by taking a projected gradient step:

where $\mathcal{A}_{d}$ is the feasible domain of $\Theta$ and $\beta_{t}$ is the learning rate. With ${\Theta}^{(t+1)}$ we solve for $z^{(t+1)}$, which is the optimal attack for a fixed ${\Theta}^{(t+1)}$. ${\nabla}_{\Theta}\hat{F}_{d}$ is the gradient of the upper-level problem.

Expanding ${\nabla}_{\Theta}\hat{F}_{d}$ using the chain rule and still using $s$ as the initially attacked node, we obtain

In both and we note that $\frac{\partial\mathbbm{1}[\theta_{j}(x)=0]}{\partial\Theta}$ is dependent on the specific detection models. We will give a concrete example of their derivation in Section 3.5.

In $\sum_{l}{k_{l}\mathbf{c}^{T}_{l,s}\mathbf{A}_{l}z(x)}$ there are two terms that are functions of $\Theta$: $\mathbf{c}_{l,s}$ and $z(x)$. Consequently, $(a)$ can be expanded as:

Note that only the detection models of those agents at layer $l$ have contribution to $\mathbf{c}_{l,s}$. Thus, $\frac{\partial\mathbf{c}_{l,s}}{\partial\Theta_{l}}$ is a Jacobian matrix with dimension $N_{l}\times N_{l}$, where $N_{l}$ is the number of agents at layer $l$ and $\Theta_{l}$ denotes the detection models of those $N_{l}$ agents. Since $\mathbf{c}_{l,s}$ is also dependent on the specific detection models of agents, we defer its derivation to Section 3.5.

$\frac{\partial z(x)}{\partial\Theta_{l}}$ is a $n\times N_{l}$ Jacobian matrix and is the main difficulty because we do not have an explicit function of the attacker’s optimal decision $z(x)$ with respect to $\Theta_{l}$. Fortunately, the constraints in (10) implicitly define $z(x)$ in terms of $\Theta$:

$\Theta$ and the attacked malicious instance $z$ satisfy $\mathbf{f}(\Theta,z,\lambda,\mu,\eta)=\mathbf{0}$. The Implicit Function Theoremzorichmathematical states that if $\mathbf{f}(\Theta,z,\lambda,\mu,\eta)$ is continuous and differentiable and the Jacobian matrix

has full rank, there is a unique implicit function $I(\Theta)=(z,\lambda,\mu,\eta)$. Moreover, the derivative of $\frac{\partial I}{\partial\Theta}$ is:

$\frac{\partial\mathbf{f}}{\partial z}$ is the Jacobian matrix of $\mathbf{f}(\Theta,z,\lambda,\mu,\eta)$ with respect to $z$, and so on. $\frac{\partial z}{\partial\Theta}\in\mathbbm{R}^{n\times N}$ is the first $n$ rows of $\frac{\partial I}{\partial\Theta}$, where $\frac{\partial z}{\partial\Theta_{l}}$ can be column-wise indexed by the nodes at layer $l$.

$(b)$ can be similarly expanded as we had done for $(a)$, except that the attacker does not modify benign content, so that $x\in D^{-}$ is no longer a function of $\Theta$:

The full projected gradient descent approach is given by Algorithm 1.

So far, we had assumed that the network node being attacked is fixed. However, the ultimate goal is to allow the attacker to choose both the node $s$, and the modification of the malicious content $z$. We begin our generalization by first allowing the attacker to optimize these jointly.

The full attacker algorithm which results is described in Algorithm 2.

Recall that the tree structure of a propagation is dependent on the agent being attacked, which makes the objective function of (5) a function of the agent being attacked. Thus, for a given fixed $\Theta$, the attacker iterates through each agent $i$ and solves the problem (5), assuming the propagation starts from $i$, resulting in associated utility $\hat{U}_{a}(i)$ and an attacked instance $z(i,x)$. Then $i$, $z(i,x)$, and $\hat{U}_{a}(i)$ are appended into a list of a 3-tuples (the sixth step in Algorithm 2). When the iteration completes the attacker picks the optimal 3-tuple in terms of utility (eighth step in Algorithm 2, where the function OptimalTuple(ret) finds the optimal 3-tuple from the list ret). The node $s$ and the corresponding attack instance $z$ in this optimal 3-tuple become the optimal attack.

Now we take the final step, relaxing the assumption that the attacker chooses a fixed node to attack which is known to the defender prior to choosing $\Theta$. Our main observation is that fixing $s$ in the defender’s algorithm above allows us to find a collection of heterogeneous detector parameters $\Theta$, and we can evaluate the actual utility of the associated defense (i.e., if the attacker optimally chooses both $s$ and $z$ in response) by using Algorithm 2. We use this insight to devise a simple heuristic: iterate over all potential nodes $s$ that can be attacked, compute the associated defense $\Theta(s)$ (using the optimistic definition of defender’s utility in which $s$ is assumed fixed), then find the actual optimal attack in response for each $x\in D^{+}$. Finally, choose the $\Theta(s)$ which has the best actual defender utility.

This heuristic algorithm is described in Algorithm 3.

The fifth step in the algorithm includes the function DefenderUtility, which evaluates the defender’s utility $\hat{U}_{d}(j)$. Note that the input argument $s$ of this function is used to determine the tree structure of the propagation started from $s$.

Recall that Algorithm 1 solves (10), which depends on the specific detection model to compute the relevant gradients. Therefore, in what follows, we present a concrete example of how to solve (10) where detection models are logistic regressions. Specifically, we illustrate how to derive the two terms, $\frac{\partial\mathbbm{1}[\theta_{j}(z)=0]}{\partial\Theta}$ and $\frac{\partial\mathbf{c}_{l,j}}{\partial\Theta_{l}}$ that depend on particular details of the detection model.

We consider the logistic regression model used for detection at individual nodes to illustrate the ideas developed above. For a node $i$, its detection model has two components: the logistic regression $\frac{1}{1+e^{-\phi^{T}x}}$, where $\phi$ is the weight vector of the logistic regression and $x$ the instance propagated to $i$, and a detection threshold $\theta_{i}$ (which is the parameter the defender will optimize). An instance is classified as benign if $\frac{1}{1+e^{-\phi^{T}x}}\leq\theta_{i}$. Thus (slightly abusing notation as before), $\theta_{i}(x)\neq 0$ ($x$ is classified as malicious) if $\frac{1}{1+e^{-\phi^{T}x}}\geq\theta_{i}$.

With the specific forms of the detection models we can derive $\frac{\partial\mathbbm{1}[\theta_{j}(x)=0]}{\partial\Theta}$ and $\frac{\partial\mathbf{c}_{l}}{\partial\Theta_{l}}$ (omitting the node index $s$ or $j$ for clarity). A technical challenge is that the indicator function $\mathbbm{1}(\cdot)$ is not continuous or differentiable, which means that it’s difficult to characterize its derivative with respect to $\Theta$. However, observe that for logistic regression $\theta_{j}(x)=0$ $\big{(}\frac{1}{1+e^{-{\phi}^{T}x}}\leq\theta_{j}\big{)}$ is equivalent to $\log\big{(}\frac{\theta_{j}}{1-\theta_{j}}\big{)}\geq\phi^{T}x$. Therefore we use $\log\big{(}\frac{\theta_{j}}{1-\theta_{j}}\big{)}-\phi^{T}x$ as a surrogate function for $\mathbbm{1}[\cdot]$. Then $\frac{\partial\mathbbm{1}[\theta_{j}(x)=0]}{\partial\Theta}$ is a $N$-dimension vector with the $j$th element equal to $\frac{1}{\theta_{j}-\theta^{2}_{j}}$. The $\mathbf{c}_{l}$ vector then becomes:

and $\frac{\partial\mathbf{c}_{l}}{\partial\Theta_{l}}$ becomes a $N_{l}\times N_{l}$ diagonal matrix:

With equations (LABEL:eq:dF_dTheta)-(16), $\frac{\partial\mathbf{c}_{l}}{\partial\Theta_{l}}$ and $\frac{\partial\mathbbm{1}[\theta_{j}(x)=0]}{\partial\Theta}$, we can now calculate $\nabla_{\Theta}\hat{F}_{d}$. Since the thresholds $\theta_{i}\in[0,1]$, the defender’s action space is $[0,1]^{N}$. When updating $\Theta$ by (11) we therefore project it back to $[0,1]^{N}$ in each iteration.