Access Control Using Spatially Invariant Permutation of Feature Maps for Semantic Segmentation Models

Figure 1 illustrates an overview of access control for protecting semantic segmentation models from unauthorized access.

A protected model is prepared by training a network with secret key $K$. Authorized users input test images into the protected model with correct key $K$, in which the model provides almost the same segmentation map as that predicted by using a non-protected model. In contrast, when unauthorized users who do not know key $K$ input test images into a protected model without any key or with incorrect (estimated) key $K^{\prime}$, the model provides a degraded map.

The goal of semantic segmentation is to understand what is in an image at the pixel level. Figure 2 shows an example of semantic segmentation. The segmentation model predicts a segmentation map from an input image, where each pixel in the segmentation map represents a class label.

The mean intersection-over-union (mean IoU) [32, 33] is used as a metric for evaluating the segmentation performance. The mean IoU value of an image is calculated by averaging the IoU of each class, which is defined by

$$\mathrm{IoU}=\frac{TP}{TP+FP+FN},$$

(1)

where $TP$, $FP$, and $FN$ mean true positive, false positive, and false negative values calculated from a predicted segmentation map and ground truth one, respectively. In addition, the metric ranges from zero to one, where a value of one means that a predicted segmentation map is the same as that of the ground truth, and a value of zero indicates they have no overlap.

To protect semantic segmentation models, we train models by randomly permuting feature maps with secret key $K$. The permutation is applied to feature maps in a network as in Fig. 3. In the figure, a fully convolutional network (FCN) [33] with a ResNet-50 [34] backbone is illustrated as an example, although the permutation is not limited to the FCN. There are six feature maps in Fig. 3, and a number of feature maps from the six maps are chosen to be permuted prior to permutation. In this paper, a feature map $x$ with a dimension of ($c\times h\times w$), where $c$ is the number of channels, $h$ is the height, and $w$ is the width of the feature map, is transformed with key $K$ at each iteration for training a model. There are two steps in the process of transforming a feature map as below (see Fig. 4).

1. 1.

   Generate a random vector with a size of $c$ such that

   $$[\alpha_{1},.,\alpha_{i},.,\alpha_{i^{\prime}},\ldots,\alpha_{c}],\alpha_{i}\in\{1,\ldots,c\},$$

   (2)

   where $\alpha_{i}\neq\alpha_{i^{\prime}}$ if $i\neq i^{\prime}$.

2. 2.

   Replace all elements of $x$, $x(i,j,k)$, $i\in\{1,\ldots,c\}$, $j\in\{1,\ldots,h\}$, and $k\in\{1,\ldots,w\}$ with $x(a_{i},j,k)$ to produce permuted feature map $x^{\prime}$, where an element of $x^{\prime}$, $x^{\prime}(i,j,k)$ is equal to $x(a_{i},j,k)$.

If multiple feature maps are chosen to be permuted, the above steps are applied to each feature map.

The above feature map permutation is spatially invariant as illustrated in Fig. 4. This spatially invariant property is important when the permutation is applied to applications that are required to output images like semantic segmentation. In contrast, an example of spatially variant permutation is given in Fig. 5. The conventional protection method for image classification [24] is carried out by using a spatially variant permutation method, so it is not available for protecting other models such as semantic segmentation models as described later. In this paper, a model protection method for semantic segmentation is discussed for the first time.

As shown in Fig. 1, authorized users have key $K$, and key $K$ is also used for semantic segmentation. In the proposed method, a query image is applied to a model trained with $K$, and the model is protected by permuting feature maps with key $K$ as well as for training the model. Protected models are expected to satisfy the following requirements.

1. 1.

   Providing almost the same performance as that of using unprotected models to authorized users.

2. 2.

   Degrading performance for unauthorized users even when they estimate key $K$.

We used a FCN with a ResNet-50 backbone (see Fig. 3) for semantic segmentation. Segmentation models were trained by using the PASCAL visual object classes dataset released in 2012 [35] for semantic segmentation, where the dataset has a training set with 1464 pairs (i.e., images and corresponding ground truths) and a validation set with 1449 pairs. We split the training set into training and development sets; 1318 samples were used for training, and 146 samples were used for development when training models. The whole validation set was used for testing the models. The performance of the trained models was calculated by averaging the mean IoU (see Eq. (1)) values of predictions. The conventional method using block-wise transformations [24] requires a fixed input image size. Therefore, to compare the conventional method with the proposed one, all input images and ground truths were resized to $256\times 256$. In addition, standard data-augmentation methods, i.e., random resized crop and horizontal flip, were performed in the training.

All networks were trained for 100 epochs by using the stochastic gradient descent (SGD) optimizer with a weight decay of 0.005 and a momentum of 0.9. The learning rate was initially set to 0.1 and scheduled by cosine annealing with warm restarts [36]. The minimum learning rate was 0.0001, and the learning rate was restarted every ten epochs. The batch size was 256. We used cross-entropy loss to calculate loss. After the training, we selected the model that provided the lowest loss value under the validation.

In this experiment, one feature map was chosen from six feature maps in the network, and segmentation models were then trained by permuting the chosen feature map with key $K$. The trained models were evaluated for authorized users with $K$. “Correct ($K$)” in Table 1 shows the result under this condition, where the model trained by permuting feature map 1 is referred to as Model-1 as an example, and “Baseline” denotes that the model was trained and tested without any feature map permutation.

From Correct ($K$) in Table 1, several models such as Model-6 and Model-5 had a high segmentation accuracy, which was almost the same as that of the baseline, although a couple of models had a slightly degraded accuracy. Figure 6 also shows an example of prediction results. From this figure, the proposed model was demonstrated to maintain prediction results similar to the baseline.

We assume that unauthorized users have no key $K$ and that they know both the method for protecting models and the permuted feature maps. To evaluate robustness against unauthorized access, we also evaluated the six protected models under two key conditions: No-perm and Incorrect, as shown in Table 1. “No-perm” indicates that protected models were tested without any feature map permutation. “Incorrect” denotes that protected models were tested by permuting a feature map used in training with incorrect (randomly generated) key $K^{\prime}$.

Table 1 shows the results under these conditions, where the results for Incorrect were averaged over 100 incorrect keys. From the table, the protected models provided a low segmentation accuracy, so the proposed models were robust against these attacks. An example of prediction results is also illustrated in Fig. 6. The robustness of the proposed models can be visually confirmed from the figure as well.

A method for model protection was proposed in [24]. In the method, input images are transformed by using three block-wise transformations with a secret key: pixel shuffling (SHF), negative/positive transformation (NP), and format-preserving Feistel-based encryption (FFX) [37]. Although this conventional method can achieve high performance in image classification models, it has never been applied to other models such as semantic segmentation ones. To be compared with the proposed method, it was also applied to semantic segmentation models.

We trained segmentation models with different block sizes and tested the models with three key conditions, i.e., Correct, Plain, and Incorrect, where “Plain” used plain images as input ones, and “Incorrect” used images encrypted by using an incorrect key. As shown in Table 2, the performance of all transformations heavily decreased compared with the proposed method.

In particular, when using a large block size, the accuracy for the correct key was low. In contrast, the accuracy for the plain and incorrect keys was high, so the performance was confirmed to be poor for protecting semantic segmentation models.

Semantic segmentation models are required to output visual information as an image, so transformations applied to images or feature maps for training and testing models have to be spatially invariant, but the conventional block-wise transformations are not spatially invariant. That is why the performance of the block-wise transformations was poor for semantic segmentation models.