MIM: Multi-modal Content Interest Modeling Paradigm for User Behavior Modeling

User Behavior Modeling (UBM) has been widely studied as a critical component of Click-Through Rate (CTR) prediction due to its ability to capture users’ historical preferences and predict their future interactions. Traditional UBM methods, such as DIN(Zhou et al., 2018), use attention mechanisms to selectively focus on relevant behaviors, while extensions like DIEN(Zhou et al., 2019) incorporate sequential modeling to capture the temporal evolution of user interests. Despite their success, these methods heavily rely on ID-based embeddings, which primarily encode collaborative filtering (CF) signals but fail to effectively model content-based preferences, particularly for items such as images and texts. This limitation becomes pronounced in cold-start and long-tail scenarios where interaction data is sparse(Yuan et al., 2021, 2020). Recent efforts, such as the introduction of hybrid UBM approaches, have attempted to integrate content features but still lack a unified framework for aligning content semantics with user-specific interests.

Multi-modal recommendation integrates diverse content features such as images, texts, and videos to improve user understanding and prediction accuracy. Traditional methods often augment CTR models with multi-modal (MM) features as additional attributes, directly feeding them into the models to enhance representation power (Mo et al., 2015). While effective to some extent, these approaches neglect the critical need to align MM features with user-specific interests, resulting in a limited understanding of user preferences.Recent advancements focus on bridging the domain gap between pre-trained foundation models and downstream recommendation tasks(Yuan et al., 2023; Wang et al., 2023; Wu et al., 2021; Liu et al., 2023). However, these methods largely ignore the role of explicit user interest modeling in refining MM embeddings, limiting their ability to capture sophisticated user preferences. Moreover, the practicality of existing methods in industrial applications remains a challenge. Many end-to-end frameworks incur high computational and memory costs, making them inefficient for large-scale deployment (Yuan et al., 2023). To address these issues, our proposed MIM paradigm systematically bridges the semantic gap between MM features and user interests while maintaining scalability and efficiency, achieving significant performance gains in real-world applications.

Pre-trained Foundation Models have significantly advanced the fields of Computer Vision (CV) and Natural Language Processing (NLP) by learning transferable representations from large-scale data. In CV, models like iGPT(Chen et al., 2020) and ViT(Dosovitskiy et al., 2020) have pioneered the application of transformer architectures for image recognition, leveraging self-supervised tasks such as masked patch prediction to capture rich visual semantics. Further advancements, such as Swin Transformer(Liu et al., 2021) and BEiT(Bao et al., 2021), improved efficiency and scalability, making transformers a dominant paradigm in vision tasks.In NLP, foundational models like BERT(Devlin et al., 2019),GPT-2(Radford et al., 2019), and XLNet(Yang et al., 2019) introduced pre-training on large corpora followed by task-specific fine-tuning, establishing a new standard for natural language understanding. These models have been extended to handle complex tasks, such as question answering, summarization, and sentiment analysis. The integration of multi-modal learning has further broadened the applicability of foundation models. Methods like CLIP(Radford et al., 2021) propose contrastive pre-training techniques to align visual and textual modalities, enabling robust joint representations. Meanwhile, ViL-BERT(Lu et al., 2019) and LXMERT(Tan and Bansal, 2019) extend the transformer framework to learn cross-modal interactions, achieving state-of-the-art performance on vision-language tasks. More recent models such as Flamingo(Alayrac et al., 2022) and BEiT-3(Wang et al., 2022) demonstrate the effectiveness of scaling multi-modal architectures for a variety of downstream applications.

Despite these advancements, challenges remain in achieving efficient training and deployment of multi-modal foundation models, particularly in scenarios requiring alignment between diverse modalities and scalability in large-scale applications.

The pre-training stage focuses on equipping the multi-modal embeddings with the ability to understand item content effectively. This stage leverages foundational models (FoMs) pre-trained on public datasets (e.g., ImageNet and Wikipedia), which are further adapted to downstream domains. We define vision-based and language-based pre-trained FoMs as $F_{V}$ and $F_{L}$ respectively. To address the limitations of general pre-trained models, two aspects of adaptation need to be considered.:

By achieving these adaptation, the pre-training stage lays the foundation for robust multi-modal embeddings capable of effectively representing diverse item content.

After the pre-training stage, FoM can understand what content an item has. However, To enhance its effectiveness in user behavior modeling, the focus of embeddings must shift from representing "what the content is" to "what users are interested in. Ideally, multi-modal embeddings for items with similar user interests should be drawn closer together in the latent space, while embeddings for unrelated items should be pushed farther apart. Inspired by the recent advancements in contrastive learning (Long et al., 2024; Wu et al., 2018; Clark et al., 2020; Mikolov et al., 2013), we propose a content-interest-aware supervised contrastive fine-tuning (C-SFT) method to effectively model user interest similarity within the multi-modal (MM) embedding space.

To design such a contrastive fine-tuning, we need to answer the following three questions: Q1: How to define user interest pairs? Q2: How to encode multi-modal features? Q3: What is a proper learning objective? Next, we present the design strategies for C-SFT by correspondingly answering the aforementioned questions.

The success of the contrastive learning framework largely relies on the definition of the user interest pairs(Qiu et al., 2020). It drives us to seek a strong and direct user interest signal.

In this paper, we leveraged data from visual search scenario, we define the user interest pair as <an image query $q$, an item $i$>, which refers to a user who has searched an image query $q$ and purchased an item $i$. Compared to textual queries, image queries carry richer semantic meanings and can more accurately express users’ intentions and interests

The reason to use purchase behaviors rather than click behaviors is that purchase actions provide a clearer and more reliable indication of user interests. We also conduct experiments to show the impacts of different signals in Section 4.3.2.

To integrate the item’s multi-modal content, we fuse the image and title embeddings using a tensor-based approach inspired by TFN (Zadeh et al., 2017), capturing both intra- and inter-modal interactions. The fused embedding is defined as:

where $\otimes$ indicates the outer product. Finally, the fused representation is refined using an MLP:

producing a comprehensive multi-modal embedding $h^{MM}_{i}$ that effectively represents the item’s content.

However, there are two shortcomings in directly using the basic InfoNCE.: (1) The number of negative samples is limited by the batch size, which is insufficient for industrial-scale applications where billions of items exist. (2) Single-modal embeddings, such as $h^{img}_{i}$ and $h^{txt}_{i}$, are ignored, which may result in a loss of modality-specific details.To address these, we propose two extensions:

where $L_{q2txt}$ and $L_{q2img}$ represent the InfoNCE loss for the pairs $<h^{img}_{q},h^{txt}_{i}>$ and $<h^{img}_{q},h^{img}_{i}>$, respectively. The weights $\alpha$ and $\beta$ balance the contributions of each modality, ensuring a more comprehensive modeling of user interests.

After obtaining the multi-modal (MM) embedding $h^{MM}_{i}$ for item $i$, we design a content-interest-aware user behavior model (CiUBM) to enhance the existing UBMs by integrating ID and content-based signals. CiUBM consists of three components:

In general, the decomposed paradigm allows MIM to train different stage models in parallel to save time cost. However, the MM embedding extraction in CiUBM stage will definitely consume more time and GPU memory. Even worse, all items in user behavior $B$ are needed to be extracted, aggravating the time and GPU memory cost (see Section 4.5). To address this problem, we proposed the representation center, which contains three key components (Figure 4).

We evaluated the performance of MIM on CTR prediction tasks by applying it to various existing UBM methods. As a universal paradigm, MIM is compatible with most UBM frameworks. To assess its effectiveness, we compared the AUC scores of the original models (denoted as Base) with those enhanced by MIM. The AUC metric (Fawcett, 2006) is reported. As shown in Table 2, integrating MIM leads to consistent performance improvements across all baseline methods and datasets. For example, SIM achieves AUC gains of 0.23pt to 0.54pt, and similar gains are observed for other UBMs. While these improvements may appear modest, in large-scale scenarios such as Taobao, a 0.1pt improvement in AUC can result in several percentage points of uplift in online CTR. This uplift can bring billions in revenue to the platform, highlighting the economic value of our approach. These results validate the effectiveness and generalizability of MIM. Which can significantly enhances user behavior modeling by bridging the gap between content and user interest embeddings. Furthermore, its universal design ensures compatibility with a wide range of existing UBM frameworks, making it a practical solution for industrial-scale CTR prediction tasks.

In this section, we conduct various experiments to detailedly evaluate the effectiveness of MIM.

In this section, we conduct an ablation study to analyze the impact of different components. The results are presented in Table 5. Some observations are summarized as follows: 1) For the C-SFT stage, it shows that C-SFT takes an important role and improves AUC from 0.7040 to 0.7062. Besides, different components in C-SFT contribute to the final performance. 2) For the CiUBM, different interest modules contribute to the final improvement. It demonstrates that it is important to take both ID interest and content interest into consideration. Note the analysis for the pre-training stage has been discussed in Section 4.3.1.

In this section, we analyze the efficiency of the proposed paradigm, including time and GPU memory efficiency. Detailedly, we develop two versions: (1) MIM (w/o RC) refers to MIM without representation center, i.e., MM embedding is real-time inferred by FoM. (2) MIM (E2E) refers to jointly training FoM and CiUBM in an E2E manner. We report the GFLOPs (the number of giga floating-point operations) per sample to reflect the train and inference time cost and GPU memory usage in deepCTR ⁴⁴4Since the pre-training stage can be pre-processed and C-SFT can be trained in parallel, the cost of these two stages can be ignored and is not included here.. The results are presented in Table 6. We can find that  (1) Compared with Base, both of MIM(w/o RC) and MIM(E2E)  need a large number of extra computations (773.55$\times$ $\sim$ 6318.12$\times$  higher than Base) and is GPU memory costly (8.23$\times$ $\sim$ 26.41$\times$  higher than Base). Note the architecture of the MIM(w/o RC) and MIM(E2E) is the same, resulting in the same GFLOPs and GPU memory cost during inference. (2) Compared with MIM(E2E), MIM(w/o RC) does not need to jointly train FoM and Base, which saves computation and GPU memory cost by the backward of FoM. (3) For MIM, thanks to the decomposed paradigm and representation center, it has significantly reduced the cost introduced by FoM and only needs 0.59$\times$ and 0.387$\times$ extra computation in training and inference, respectively, and 0.01$\times$ and 0.81$\times$ extra GPU memory cost in training and inference, respectively.

Overall, such nice properties in terms of high performance, efficient GPU memory usage, and low time requirements of our proposed model are welcomed for web-scale applications.

Here we analyze the performance of the cold start items to show the generalization of our proposed paradigm. Detailedly, we divide different items into 10 sets (denoted as S1 to S10), where S1$\sim$S10 refers to the newest$\sim$oldest item sets. Then we evaluate the improvement of different sets respectively. The results are presented in Figure 6. It shows that MIM contributes more to the newest items (e.g., S1) and achieves more AUC gains. This demonstrates that MM features enable new items to better represent their features, leading to better performance.

Here, we first develop MIM on a real-world industrial application, i.e., the sponsored search system in Taobao, and have continually achieved three releases of MIM in our industrial application in the past year.