DREAM: Domain-agnostic Reverse Engineering Attributes of Black-box Model

We first introduce the background of the KENNEN method. Based on KENNEN, we present the threat model of the problem addressed in this paper. Finally, we introduce the problem formulation.

Background of KENNEN [22]. Given a black-box model $B$, model attribute reverse engineering in [22] aims to build a meta-model $\Phi:O\rightarrow Y$, where $O=B(Q)$ is the outputs by querying the black-box model with queries $Q$, and $A$ is the set of model attributes including model architecture, optimizer, and training hyperparameters, $etc$. Concretely, they first construct a large set of white-box models $\mathcal{F}$ containing different attributes combinations and train these white-box models based on the same training data $\mathcal{D}$ as that of the target black-box model. Then outputs $O$ are obtained by querying these white-box models with a sequence of input image queries $Q$. Finally, they train a meta-model $\Phi$ to build mappings from outputs $O$ to model attributes $Y=\{y_{k}^{v}|k=1...K,v=1...N^{k}\}$, where the subscript $k$ represents the type of attributes ($e.g.$, Activation, Dropout), while the superscript $v$ represents the value of the attributes ($e.g.$, ReLU/Tanh for Activation, Yes/No for Dropout). At the inference phase, the meta-model takes outputs from the target model as input and predicts the corresponding attributes.

Threat Model. Following KENNEN, we assume that attackers are permitted to query the model and can only access the probability outputs of the model, while attributes such as model structures and optimizers are unable to access. However, KENNEN makes a strong assumption that the training dataset of the model is known. In most cases, obtaining the training dataset is typically challenging. Therefore, we relax this assumption and consider a scenario where only the label space of the black-box model are known. This is a reasonable assumption because a black-box model deployed on the cloud platform typically provides information about its functionality and the categories it can output. Consequently, we can collect data with overlapping label spaces with the black-box target model. This overlap ensures that the outputs of the white-box models include similar information with those of the black-box model to some extent, thereby assisting in learning informative invariant features for achieving attributes reverse engineering.

Problem Formulation. As aforementioned, there is a strict constraint in [22] that they assume the training dataset $\mathcal{D}$ of the target model to be given in advance, and leverage model outputs $O$, where the models are trained on $\mathcal{D}$ for the learning of meta-model $\Phi$. It is difficult to access the training data of a target black-box model, which significantly limits the applications of [22]. To mitigate this problem, we provide a new problem setting by relaxing the above constraint, $i.e.$, we no longer require the training data $\mathcal{D}$ of the target black-box model to be available, but only the label space of the black-box model. Consequently, we cast this problem as an OOD generalization problem. To address this, we first collect data $\{D^{i}\}_{i=1}^{M}$ from $M$ different sources to train the white-box models. Although these data may contain different styles, all of them include an overlapping label space with the target black-box model. Then, the outputs obtained from the white-box models are divided into $M$ source domains according to the source of the training data for each model. In addition, the outputs from black-box model are the target domain. Our goal is to leverage outputs from source domains to train a domain-agnostic meta model $\Phi$ that can well generalize to the outputs target domain, thereby enabling it to predict attributes for the target black-box model $B$.

To perform domain-agnostic black-box model attribute reverse engineering, we cast this problem into an OOD generalization learning problem, and propose a novel framework DREAM, as shown in Fig. 3. Our DREAM framework consists of two parts:

In the left part of Fig. 3, we employ datasets from different domains to train numerous white-box models with diverse attributes, thereby constructing modelsets. (please refer to Sect. IV-A for more details). Next, we sample queries as input to these models. For each domain, we sample an equal number of images from the corresponding dataset and concatenate them as a batch of queries. These queries are sent to models belonging to the modelsets. The resulting multi-domain outputs $\{O^{i}\}_{i=1}^{M}$ are fed into the subsequent module of our DREAM framework. To learn domain-invariant features, we introduce a MDGAN, as shown in the right part of Fig. 3. MDGAN consists of multiple discriminators corresponding to different domains and one generator across multiple domains. The generator is designed to embed model outputs from different domains as features, while each discriminator strives to align the learned feature distributions from other domains with the feature distribution of the domain it corresponds to. In this way, the generator is capable of learning domain-invariant features. Based on the learned domain-invariant features, we further learn a domain-agnostic meta-model to infer the attributes of a black-box model with an unknown domain.

To achieve model reverse engineering, we first need to obtain the multi-domain outputs of the white-box models. These outputs serve as features of white-box models’ attributes and will be used as inputs for the subsequent modules. We sample an equal number of images from the source dataset, resulting in queries $Q=\{q_{j}\}_{j=1}^{N}$, where $N$ is the number of total queries. Subsequently, these queries are fed into the white-box models $\mathcal{F}=[\mathbf{f^{1}},\mathbf{f^{2}},...,\mathbf{f^{M}}]$, where $\mathbf{f^{i}}$ represents models from the $i^{th}$ domain. For each domain $i$, the models $\mathbf{f^{i}}$ produces outputs $\{O_{j}^{i}\}_{j=1}^{N}\in\mathbb{R}^{N\times C}$, where $C$ is the number of classes in the dataset. We then concatenate the outputs as an 1-dimensional vector $O^{i}\in\mathbb{R}^{NC}$. Finally, the multi-domain outputs are derived as $O=\{O^{i}\}_{i=1}^{M}$.

After preparing multi-domain outputs, we introduce MDGAN on the basis of [71]. The objective is to learn domain-invariant features from these probability outputs of white-box models trained on different domains.

To better present, we take an example about how MDGAN works on two domains. As shown in Fig. 4, we have two kinds of inputs, $O^{1}$ and $O^{2}$, from two domains. When feeding them into the generator $G$, we can obtain the corresponding features $z^{1}$ and $z^{2}$, respectively. After that, we feed $z^{1}$ and $z^{2}$ to the discriminator $D_{1}$, where $D_{1}$ is expected to output a “real" label for $z^{1}$ and output a “fake" label for $z^{2}$. By jointly training $G$ and $D_{1}$ based on a min-max optimization, the distribution of $z^{2}$ is expected to move towards that of $z^{1}$. In the meantime, we also feed $z^{1}$ and $z^{2}$ to the discriminator $D_{2}$. Differently, $D_{2}$ is expected to output a “real" label for $z^{2}$ and output a “fake" label for $z^{1}$. By jointly training $G$ and $D_{2}$, the distribution of $z^{1}$ is expected to move towards that of $z^{2}$. In this way, $z^{1}$ and $z^{2}$ generated by the generator $G$ become domain-invariant representations.

Formally, we define $G(O^{i};\theta_{g}):O^{i}\rightarrow z$. The generator $G$ sharing with parameter $\theta_{g}$ across domains maps outputs $O^{i}$ from the $i^{th}$ domain into the latent feature $z^{i}$. After that, we obtain latent features $\{z^{i}\}_{i=1}^{M}$ of model outputs. We also define $M$ discriminators $\{D^{j}(z^{i};\theta_{d}^{i})\}_{j=1}^{M}$. Each discriminator $D^{j}(z^{i}):z^{i}\rightarrow[0,1]$ outputs a scalar representing the probability that $z^{i}$ comes from the $j^{th}$ domain. The label of latent features $z^{i}$ is defined as Real for the discriminator $D^{j}(z^{i})$ when $j=i$, while False when $j\neq i$.

The training goal of $D^{j}$ is to maximize the log probability of assigning the correct label to features both from the $i^{th}$ domain and other domains, while the generator $G$ is trained against the discriminator to minimize the probability. In other words, it is a min-max game between the $j^{th}$ discriminator $D^{j}$ and generator $G$ with a value function $V$, formulated as:

During optimizing the min-max adversarial value function for $G$ and $D^{j}$, the generator $G$ can gradually produce latent features $z^{j}$ from $j^{th}$ domain, which are close to latent features from other domain. Once $G$ and all $D$ are well trained, $G$ is able to embed multi-domain model outputs into an invariant feature space, where each discriminator cannot determine which domain the outputs are from. Therefore, the latent features $\{z^{i}\}_{i=1}^{M}$ become domain-invariant features. Note that our proposed MDGAN does not suffer from mode collapse. This is because mode collapse is an issue in generative tasks using GANs, where the model fails to generate diverse patterns and instead produces only a limited set of modes. In our approach, the role of generator $G$ in the MDGAN is not to generate diverse features. Instead, $G$ functions as an encoder, encoding the model’s outputs from different domains into invariant features. Therefore, it does not suffer from the problem of mode collapse.

After obtaining domain-invariant features $\{z^{i}\}_{i=1}^{M}$, we aim to classify them as model attributes $Y$ through the domain-agnostic reverse meta-model. Consider there are $K$ types of attributes and each attribute contains $N^{k}$ possible values, we build $K$ domain-agnostic reverse meta-models $\Phi=\{\phi_{1},\phi_{2},...,\phi_{K}\}$. Therefore, the probability $p_{k}(z)$ for the $k^{th}$ attribute is obtained by:

Note that the $p_{k}(z)$ is a $N^{k}$-dimensional vector that contains $N^{k}$ attribute value, and each dimension represents the probability of the attribute value.

The target is to minimize the cross entropy between the predicted attributes probability $p_{k}(z)$ and ground-truth of model attribute values $y^{k}$:

During the inference phase, we input queries to the black-box model trained from an unknown domain to generate outputs. These outputs are then embedded as domain-invariant features by the generator $G$. Subsequently, the reverse meta-model $\Phi$ classifies these domain-invariant features, achieving domain-agnostic attributes predictions for the black-box model.

After introducing all the components, we give the final loss function based on Eq. 1 and 3 as:

where $\lambda$ is a trade-off parameter.

The training strategy is as follows: we first optimize all discriminators $D^{i}$, and then jointly optimize the generator and the domain-agnostic reverse meta-model. We repeat the above processes until the algorithm converges. The proposed optimization strategy is presented in Algorithm 1.

Following [22], we construct the modelset by training models that enumerate all possible attribute values. The details of the attributes and their values are shown in Table I. There are a total of $K=9$ types of attributes for each model in the modelsets, which adheres the following scheme: $N_{c}=\{2,3,4\}$ convolution layers, $N_{f}=\{2,3,4\}$ fully-connected layers ²²2Not that a model with more layers may be mistakenly classified as an overtrained model. However, this issue do not exist in practical applications, because we did not devise an attribute to determine if a model is overtrained. In practical applications, people tend to deploy models with strong generalization capabilities rather than overtrained ones. Therefore, to ensure accurate model reverse engineering, the white-box models we train are not overtrained either. We select the model that performs best on the validation set as the final white-box model, thereby ensuring its generalization performance.. Each convolution layer contains a $k\times k$ kernel ($k=\{3,5\}$), an optional batch normalization, an optional max-pooling, and a non-linear activation function in sequence. Each fully connected layer consists of a linear transformation, a non-linear activation, and an optional dropout in sequence. We set the dropout ratio to $0.1$ in our experiments. When training white-box models, optimizers are selected from $\{$SGD, ADAM, RMSprop$\}$ with a batch size $32$, $64$ or $128$, respectively. The statistics of PACS modelset and MEDU modelset are listed in the Appendix.B By enumerating all possible model attributes, a total of $5,184$ distinct models can be obtained. In addition, we initialize each kind of model with random seeds from 0 and 999, yielding 5,184,000 unique models.

We construct PACS modelset and MEDU modelset to evaluate our method. The details are as follows:

PACS modelset comprises a set of models trained on the PACS dataset. The PACS dataset is an image dataset that has been widely used for OOD generalization [24]. We utilize 3 domains, including Photo (1,670 images), Cartoon (2,344 images), and Sketch (3,929 images), and each domain contains 7 kind of classes. For each modelset domain, we randomly sample and train 5,000, 1,000, and 1,000 from 5,184,000 white-box models as the training, validation, and testing models, respectively.

MEDU modelset consists of a set of models trained on MEDU dataset. MEDU is a hand-written digit recognition dataset, with 4 domains collected from MNIST [23], USPS [72], DIDA [73] and EMNIST [74]. Each domain contains different styles of hand-written digits from 0 to 9. Similar to PACS modelset, for each modelset domain, we randomly sample and train 5,000, 1,000, and 1,000 from 5,184,000 white-box models as the training, validation, and testing models.

In the experiment, we set the number of queries $N$ to 100. We use Adam [75] as the optimizer, where the learning rate $\alpha$ is set to $10^{-5}$ for the generator and discriminators, and the learning rate $\beta$ is set to $10^{-4}$ for the domain-agnostic meta-model. The batch size $b$ is set as 100. The trade-off parameter $\lambda$ is tuned from $\{0.001,0.01,0.1,1,10\}$ based on the validation set.

In addition, the MDGAN is composed of a generator and multiple discriminators. The generator consists of two linear layers with ReLU activation. The dimension of the input layer of the generator is determined by the query number $N$ and class category number $C$. As for the experiment conducted on MEDU modelset, the input dimension is $NC=1000$ (the query number is $N=100$, and the number of classes if $C=10$). In the case of the PACS modelset, the input dimension is 700 ($N=100$, $C=7$). The output dimensions of the subsequent two linear layers are 500 and 128, respectively. Each discriminator consists of three linear layers. The first two layers employ ReLU as the activation function, while the last layer utilizes the Sigmoid activation function. The output dimensions of discriminator layers are 512, 256, and 1 respectively. Additionally, we implement the domain agnostic meta-models as $K=9$ MLPs. Each MLP consists of 3 layers, and it takes latent features produced by generator $G$ as input. The output dimensions of the MLP are 128, 64 and $N^{k}$. All experiments are conducted on 4 NVIDIA RTX 3090 GPUs, PyTorch 1.11.0 platform [76].

We compare our DREAM with 7 baselines including Random choice, SVM, KENNEN [22], SelfReg [29], MixStyle [30], MMD [28] and SD [77]. Additionally, we take four typical OOD generalization methods—SelfReg, MixStyle, MMD, and SD—as baselines to validate the effectiveness of our framework in learning domain-invariant features.

The details of the baseline methods are presented as follows:

SVM. The SVM serves as a baseline, representing a non-deep learning method. We directly input the multi-domain probability outputs into SVM classifiers to predict attributes. We adopt the One-vs-One (OvO) strategy. Specifically, for an attribute with $N^{k}$ possible values, we trained a binary SVM for each pair of attribute values, resulting in a total of $N^{k}(N^{k}-1)/2$ SVMs. When classifying a new sample, we apply all trained SVM classifiers and use a voting mechanism for each attribute value. The attribute value receiving the most votes is selected as the final prediction. For 2-valued attributes, we only need to train a single binary SVM.

KENNEN*. We employ a variant of KENNEN (denoted as KENNEN* ). It takes fixed queries as input, which is the same as our approach. We embed the multi-domain probability outputs into feature space and use MLPs to predict attributes. Similar to SVM, KENNEN does not differentiate between outputs from different domains. In addition, to ensure a fair comparison, the network structure keeps consistent with the domain-agnostic reverse meta-model in DREAM.

SelfReg, MixStyle, MMD and SD. The approach taken by SelfReg to learn invariant features involves pulling samples of similar categories between all domains closer together while pushing samples of different categories further apart. The motivation behind MixStyle is based on the observation that the styles of domain images share significant similarities. In particular, MixStyle captures style information through the final layer and performs style mixing at that layer. The MMD adopts maximum mean discrepancy loss between each two domains. The SD proposes the Spectral Decoupling method to relieve the gradient starvation phenomenon during the training of the network, to boost the performance of OOD generalization. We embed the multi-domain probability outputs into feature space and apply the OOD generalization method, SelfReg, MixStyle, MMD, and SD, respectively to learn invariant features. Then the learned invariant features are fed into MLPs for predicting attributes. For a fair comparison, the network structure of MLPs is consistent with the domain-agnostic reverse meta-model in DREAM.

We adopt the leave-one-domain-out strategy to split the source and target domains for PACS and MEDU modelsets. For each modelset, we take one domain as the target domain and the rest as source domains in turn. The experiment is run for 10 trials, and the average results are reported.

Table II and III report the overall performance of different methods on the PACS modelset and MEDU modelset, respectively. The leftmost column in each table indicates the target domain (the rest ones are source domains). In this experiment, we adopt the complete overlap setting, where the label space of the source domain and target domain is identical. The performance achieved by our proposed DREAM is better than that of all baselines in terms of the average accuracy of model attributes. For individual attribute, our method outperforms other methods in most cases. DREAM demonstrates superior performance compared to KENNEN and SVM. This suggests that, in situations where the training dataset of the target black-box model is unknown, utilizing our proposed MDGAN for learning invariant features, instead of directly inputting the outputs from the source domain into the meta-model for attribute prediction, results in enhanced generalization. DREAM also outperforms four OOD generalization methods, which highlights the superior capability of the proposed MDGAN in learning domain-invariant features from probability outputs compared to other baselines.

To show the application value of our method, we conduct a model extraction experiment. We adopt a popular model extraction method MAZE [41], where model structures are "Same to the victim", "Random", or "Inferred by DREAM". As shown in Table IV, The performance of model extraction using the architecture "Inferred by DREAM" significantly outperforms that achieved with a random architecture ($62.81\%$ vs. $45.88\%$). Moreover, its performance closely approaches that obtained when employing the "Same as the victim" architecture for model extraction ($62.81\%$ vs. $68.46\%$). The rationale behind above results lies in the fact that extraction performance improves when the structure of the extracted model closely aligns with that of the victim, as discussed in [21]. The proximity of the extracted model to the victim model in terms of structure and complexity enhances the likelihood of capturing identical patterns and information present in the victim model.

To further verify the effectiveness of our method, we utilize t-SNE [78] to visualize samples in the feature space learned by the generator $G$ of our framework. The visualization is carried out on PACS modelset. We take C (cartoon) and P (photo) as source domains to train white-box models, and use S (sketch) as the unseen target domain to the train black-box model. In this experiment, we adopt the complete overlap setting. As shown in Fig. 5, we observe the features from sources and target domain are spread in the feature space for MMD, MixStyle and SelfReg method. However, DREAM can embed samples from both the source domains and target domain into an invariant feature space, which demonstrates the superiority of DREAM.

1) We conduct class shift experiment on PACS modelset. In this experiment, we exclude the "dog" and "elephant" classes among 7 classes for each source domain while constructing white-box models, while the target domain remain all 7 classes. As shown in Table V, the average accuracy of DREAM* reaches 50.03%. Although DREAM* experiences a slight decrease in accuracy compared to DREAM due to domain shift, it still outperforms other baselines.

2) We conduct experiments in situations where domains are quite different, and number of classes between domains are different. We first train white-box models using the Sketch and Cartoon datasets. The outputs of these white-box models serve as the source domains for training DREAM. We then utilized data from CIFAR10 and CIFAR100 to train black-box models. The trained DREAM infers attributes based on the black-box model outputs. The source domains (Sketch and Cartoon) comprise 7 classes, while the target domain (CIFAR10/100) contains 110 classes. We use all 7 classes as the label space for the source domains. For the target domain’s label space, we select the 5 classes that overlap between the source and target domains (namely: dog, elephant, house, horse, and person). This ensures a overlapping label space, but different number of classes between source and target domains. The experimental results are listed in Table VI, which demonstrate that DREAM still outperforms Random, KENNEN, as well as state-of-the-art OOD generalization approaches. This indicates that our method is effective when domains are quite different, as well as when the number of classes differs between domains.

3) We investigate the shift scenario where white-box models used for training meta-model and the black-box model to be inferred have completely different attribute combinations. As mentioned in Sect. IV-A, there are a total of $5,184$ combinations of model attributes. We randomly select $3,000$, $1,000$, and $1,000$ models for the training, validation, and testing sets, respectively, ensuring that none of the models share identical attribute combinations. We adopt the complete overlap setting. The resulting model is denoted as DREAM**. As demonstrated in Table VII, DREAM** consistently outperforms other baselines under this setting. Consequently, the shift caused by attribute combinations does not significantly impact its overall performance.

We analyse the size of modelset and the training data amount. We adopt the complete overlap setting in these two experiments.

Analysis of Size of Modelset. We further study the performance of our method against the size of the PACS modelset. As shown in Fig. 6, we observe that the performance slightly fluctuates from the size of 1K to 5K, and does not consistently increase when the size increases. We suspect it can be attributed to the difficulty of our problem for domain-agnostic attribute inference of the black-box model, and the nature of the OOD problem, $i.e.$, the noise level increases as the size of the model set increases. It is worth studying further.

Analysis of Training Data Amount. We analyze the impact of amount of training samples on reverse engineering performance. We train white-box models using 25%, 50%, 75%, and 100% of PACS training data, respectively. We then apply our proposed DREAM method to perform reverse engineering on these models. The results are presented in Fig. 7. We observe as the amount of data increases, the performance improves, and the performance levels off when the amount of data reaches 75%.

We design a larger model attribute space for model attribute inference, as shown in Table IX. Specifically, we

• •

  Add "GeLU" activation function to the "#Activation" attribute space.

• •

  Expand "#Kernel size" attribute space from $\{3,5\}$ to $\{3,4,5,6,7,8,9\}$

• •

  Expand "#Conv layers" attribute space from $\{2,3,4\}$ to $\{2,3,4,5,6,7,8,9,10\}$

• •

  increase "#FC layers" attribute space from $\{2,3,4\}$ to $\{2,3,4,5,6,7,8\}$.

The outputs of models trained on the Photo and Cartoon datasets serve as the source domains in this experiment, whereas those trained on the Sketch dataset function as the target domain. We adopt the complete overlap setting in this experiment. The results presented in Table X demonstrate that our method continues to outperform the baseline methods.

In addition to performing attribute inference on CNN architecture, we extend our approach to Vision Transformer (ViT) architecture. The Vision Transformer (ViT) applies the Transformer architecture, originally used in natural language processing, to images. Specifically, it divides an image into several patches and uses a Transformer to extract features from the image [79]. We design an attribute space for the Vision Transformer architecture, as shown in Table IX. #Patch size = $k$ means that each input image is divided into patches of size $k*k$, which are used as the input to the ViT. Additionally, we include the number of #Transformer layers, #Feedforward layers, and numbers of #Attention heads in the attribute space. In this experiment, we also use outputs from models trained on the Photo and Cartoon datasets as the source domains, and outputs from models trained on the Sketch dataset as the target domain. The complete overlap setting is adopted. The experimental results, listed in Table XI, indicate that our method is capable of effectively performing attribute inference for Vision Transformer models, outperforming other baseline methods.