FiBiNet++: Reducing Model Size by Low Rank Feature Interaction Layer for CTR Prediction

Feature Normalization. We introduce feature normalization into FiBiNet++ to enhance model’s training stability as follows:

where $\mathbf{N}\left(\cdot\right)$ is layer normalization(Ba et al., 2016) for numerical feature and batch normalization(Ioffe and Szegedy, 2015) operation for categorical feature.

Bi-Linear+ Module. FiBiNet models interaction between feature $\mathbf{x}_{i}$ and feature $\mathbf{x}_{j}$ by bi-linear function which introduces an extra learned matrix $\mathbf{W}$ as follows:

where $\circ$ and $\otimes$ denote inner product and element-wise hadamard product, respectively. In order to effectively reduce model size, we upgrade bi-linear function into bi-linear+ module by following two methods. First, the hadamard product is replaced by another inner product as: $\mathbf{p}_{i,j}=\mathbf{v}_{i}\circ\mathbf{W}\circ\mathbf{v}_{j}\in\mathbb{R}^{1\times 1}$ . We think the feature interaction is rather sparse and one bit information as representation is enough instead of a vector. It’s easy to see the parameters of $\mathbf{p}_{i,j}$ decrease greatly from $\mathbf{d}$ dimensional vector to $1$ bit for each feature interaction. Suppose the input instance has $f$ fields and we have the following vector after bi-linear feature interaction:

Inspired by LoRA(Hu et al., 2021), which has demonstrated that LLM models possess a low ”intrinsic dimension” and exhibit efficient learning despite undergoing random projections to smaller subspaces, we posit that updates to the feature interaction layer during training also exhibit a low ”intrinsic rank.” Therefore, we propose integrating a thin ”Low Rank Layer” into Bi-Linear+, thereby reducing model parameters significantly while maintaining optimal model performance. Specifically, we introduce ”Low Rank Layer” stacking on feature interaction vector $\mathbf{P}$ as follows:

where $\mathbf{W}_{1}\in\mathbb{R}^{m\times\frac{f\times(f-1)}{2}}$ is a learning matrix of thin MLP layer with small size $m$ and $\sigma_{1}\left(\cdot\right)$ is an identity function. ”Low Rank Layer” projects feature interactions from sparse space into low rank space to greatly reduce the storage.

SENet+ Module. SENet+ module consists of four phases: squeeze, excitation, re-weight and fuse. (1) Squeeze. SENet collects one bit information by mean pooling from each feature embedding as ”summary statistics”. However, we improve the original squeeze step by providing more useful information. Specifically, we first segment each normalized feature embedding $\mathbf{v}_{i}\in\mathbb{R}^{1\times d}$ into $\mathbf{g}$ groups, which is a hyper-parameter, as: $\mathbf{v}_{i}=concat[\mathbf{v}_{i,1},\mathbf{v}_{i,2},......,\mathbf{v}_{i,g}]$ , where $\mathbf{v}_{i,j}\in\mathbb{R}^{1\times\frac{d}{g}}$ means information in the j-th group of the i-th feature. Let $\mathbf{k}=\frac{d}{g}$ denotes the size of each group. Then, we select the max value $\mathbf{z}_{i,j}^{max}$and average pooling value $\mathbf{z}_{i,j}^{avg}$ in $\mathbf{v}_{i,j}$ as representation of the group as: $\mathbf{z}_{i,j}^{max}=\max\limits_{t}\left\{\mathbf{v}_{i,j}^{t}\right\}_{t=1}^{k}$ and $\mathbf{z}_{i,j}^{avg}=\frac{1}{k}\sum_{t=1}^{k}\mathbf{v}_{i,j}^{t}$ . The concatenated representative information of each group forms the ”summary statistic” $\mathbf{Z}_{i}$ of feature embedding $\mathbf{v}_{i}$:

Finally, we can concatenate each feature’s summary statistic $\mathbf{Z}=concat[\mathbf{Z}_{1},\mathbf{Z}_{2},......,\mathbf{Z}_{f}]\in\mathbb{R}^{1\times 2gf}$ as the input of SENet+ module.

(2) Excitation. The excitation step in SENet computes each feature’s weight according to the statistic vector $\mathbf{Z}$, which is a field-wise attention. However, we improve this step by changing the field-wise attention into a more fine-grained bit-wise attention. Similarly, we also use two fully connected (FC) layers to learn the weights as follows: $\mathbf{A}=\sigma_{3}\left(\mathbf{W}_{3}\sigma_{2}\left(\mathbf{W}_{2}\mathbf{Z}\right)\right)\in\mathbb{R}^{1\times fd}$ , where $\mathbf{W}_{2}\in\mathbb{R}^{\frac{2gf}{r}\times 2gf}$ denotes learning parameters of the first FC layer, which is a thin layer and $r$ is reduction ratio. $\mathbf{W}_{3}\in\mathbb{R}^{fd\times\frac{2gf}{r}}$ means learning parameters of the second FC layer, which is a wider layer with a size of $fd$. Here $\sigma_{2}\left(\cdot\right)$ is $ReLu\left(\cdot\right)$ and $\sigma_{3}\left(\cdot\right)$ is an identity function without non-linear transformation.In this way, each bit in input embedding can dynamically learn the corresponding attention score provided by $\mathbf{A}$.

(3) Re-Weight. Re-weight step does element-wise multiplication between the original field embedding and the learned attention scores as follows: $\mathbf{V}^{w}=\mathbf{A}\otimes\mathbf{N}(\mathbf{V})\in\mathbb{R}^{1\times fd}$ , where $\otimes$ is an element-wise multiplication between two vectors and $\mathbf{N}(\mathbf{V})$ denotes original embedding after normalization.

(4) Fuse. An extra ”fuse” step is introduced in order to better fuse the information contained both in original feature embedding and weighted embedding. Specifically, we first use skip-connection to merge two embedding as follows: $\mathbf{v}_{i}^{s}=\mathbf{v}_{i}^{o}\oplus\mathbf{v}_{i}^{w}$ , where $\mathbf{v}_{i}^{o}$ donates the i-th normalized feature embedding, $\mathbf{v}_{i}^{w}$ denotes embedding after re-weight step, $\oplus$ is an element-wise addition operation. Then, another feature normalization is applied on feature embedding $\mathbf{v}_{i}^{s}$ for a better representation: $\mathbf{v}_{i}^{u}=\mathbf{LN}(\mathbf{v}_{i}^{s})$ . Note we adopt layer normalization here no matter what type of feature it belongs to, numerical or categorical feature. Finally, we concatenate all the fused embeddings as the output of the SENet+ module:

Concatenation Layer. We concatenate the output of two branches to form the input of the following MLP layers:

In this section, we discuss the model size difference between FiBiNet and FiBiNet++. Note only non-embedding parameter is considered, which really demonstrates model complexity.

The major parameter of FiBiNet comes from two components: one is the connection between the first MLP layer and the output of two bi-linear modules, and the other is the linear part. Suppose we denote $h=400$ as the size of the first MLP layer, $f=50$ as field number, $d=10$ as feature embedding size, and $t=1\ million$ as feature number. Therefore, the parameter number in these two parts is nearly 10.8 million:

For FiBiNet++, the majority of model parameter comes from following three components: the connection between the first MLP layer and embedding produced by SENet+ module(1-th part), the connection between the first MLP layer and ”Low Rank Layer”(2-th part), and parameters between ”Low Rank Layer” and bi-linear feature interaction results(3-th part). Let $m=50$ denote the size of ”Low Rank Layer”. We have the parameter number of these components as follows:

We can see that the above-mentioned methods to reduce model size greatly decrease model size from 10.8 million to 0.28 million, which is nearly 39x model compression. In addition, the larger the field number $f$ is, the larger the model compression ratio we can achieve. It’s easy to see that ”Low Rank Layer” is the key to the high compression ratio while it can also be applied into other CTR models with long feature interaction layers such as ONN(Yang et al., 2020) and FAT-DeepFFM(Zhang et al., 2019) for feature interaction compression.

Datasets Three datasets are used in our experiments and we randomly split instances by 8:1:1 for training, validation and testing: (1) Criteo¹¹1Criteo http://labs.criteo.com/downloads/download-terabyte-click-logs/: As a display ad dataset, there are 26 anonymous categorical fields and 13 continuous feature fields. (2) Avazu²²2Avazu http://www.kaggle.com/c/avazu-ctr-prediction: The Avazu dataset contains 23 fields that indicate elements of a single ad impression. (3) KDD12³³3KDD12 https://www.kaggle.com/c/kddcup2012-track2: KDD12 dataset has 13 fields spanning from user id to ad position for a clicked data.

Models for Comparisons We compare the performance of the FM(Rendle, 2010), DNN, DeepFM(Guo et al., 2017), DCN (Wang et al., 2017), AutoInt (Song et al., 2019), DCN V2 (Wang et al., 2021), xDeepFM (Lian et al., 2018) and FiBiNet (Huang et al., 2019) models as baselines and AUC is used as the evaluation metric.

Implementation Details For the optimization method, we use the Adam with a mini-batch size of $1024$ and $0.0001$ as learning rates. We make the dimension of field embedding for all models to be a fixed value of $10$ for Criteo dataset, $50$ for Avazu dataset and $10$ for KDD12 dataset. For models with DNN part, the depth of hidden layers is set to $3$, the number of neurons per layer is $400$, and all activation functions are ReLU. In SENet+, the reduction ratio is set to 3 and the group number is 2 as the default settings. In Bi-linear+ module, we set size of the low rank layer as 50.

Performance Comparison. Table 1 shows the performances of different SOTA baselines and FiBiNet++. The best results are in bold, and the best baseline results are underlined. We can see that:(1) FiBiNET++ model outperforms all the compared SOTA methods and achieves the best performance on all three benchmarks. (2) Among all the strong baselines with a similar amount of parameters such as Autoint+, DCN v2 and DeepFM, FiBiNet++ is the best performance model on all three datasets. This demonstrates the reason why FiBiNet++ outperforms other models is because of its designed components instead of more parameter. (3) Compared with FiBiNet model, FiBiNet++ can achieve better performance on all datasets though it has much fewer parameters, which indicates that our proposed methods to enhance model performance are effective.

Model Size Comparison. We compare the non-embedding model size of different methods in Table 1 and Figure 3. FiBiNet++ provides orders of magnitude improvement in model size while improving the quality of the model compared with FiBiNet. Specifically, FiBiNet++ reduce the model size of FiBiNet by 12.7x, 12.9x and 16x in terms of the number of parameters on three datasets, respectively, which demonstrates that our proposed methods to reduce model parameter in this paper are effective. Now FiBiNet++ has a comparable model size with DNN model while outperforms all other models on three datasets at the same time.

Training/Reference Efficiency. Efficiency is an essential concern in industrial applications and we conduct experiments to compare the training and inference time between our proposed FiBiNet++ and FiBiNet. We leverage time(millisecond) of processing one batch of examples during the training and reference as an efficiency metric. The average training and inference times of the two models are illustrated in Table 2. Compared with FiBiNet model, the training efficiency of FiBiNet++ increases by 58.76%, 62.30% and 39.39% while reference efficiency increases by 41.66%, 81.03% and 37.50% on three datasets respectively. Our FiBiNet++ model shows a significant advantage in training and inference efficiency, which makes it more practical to be applied in real life.

Hyper-Parameters of FiBiNet++. Next, we study hyperparameter sensitivity of FiBiNet++. (1) Group Number. Figure 2a shows a slight performance increase with the increase of group number, which indicates that more group number benefits model performance because we can input more useful information in feature embedding into SENet+ module.

(2) Reduction Ratio. We conduct some experiments to adjust the reduction ratio in SENet+ module from $1$ to $9$ and Figure 2b shows the result. It can be seen that the performance is better if we set the reduction ratio to 3 or 9. (3) Size of Low Rank Layer. The results in Figure 2c show the impact when we adjust the size of the Low Rank Layer in bi-linear+ module. We can observe that the performance begins to decrease when the size is set greater than 150, which demonstrates the feature interaction represented in low rank space indeed works.