Mitigating Pooling Bias in E-commerce Search via
False Negative Estimation

Instacart has an effective and efficient embedding model to measure the relevance between queries and products, which is widely used in search retrieval, ranking, and many other surfaces (Xie et al., 2022). The search embedding model follows a typical bi-encoder design, where queries and products are passed through their corresponding Sentence Transformer-based encoders¹¹1https://huggingface.co/sentence-transformers, denoted as $\text{BE}_{Q}$ and $\text{BE}_{P}$, independently. The input query-product pair $(q_{i},p_{j})$ is transformed into dense embeddings as follows:

The estimated relevance score between a query and a product is computed via cosine similarity between their embeddings, i.e.,

During the training phase, similarity scores can be efficiently calculated through a series of matrix transpositions and multiplications, making the process computationally efficient. For the inference stage, the product and known query embeddings can be pre-computed offline and cached, while tail query embeddings are generated in real-time. During inference, relevant products for an incoming search query are retrieved using the query’s embedding via approximate nearest neighbor search (ANN), and the estimated relevance score $\hat{{r}}_{i,j}$ is passed to the ranker service as a feature to determine the final ranking of $p_{j}$ for search term $q_{i}$.

Although the supervised training of the search embedding model benefits from supervised learning using ground-truth annotation score $r_{i,j}$, it still suffers from the insufficiency of negative cases. In e-commerce search, positive pairs $(q_{i},p_{j})$ can be easily obtained from past search logs with converted products. However, high-quality negative cases, which should have been able to help the model acquire the capability of distinguishing irrelevant products from relevant ones, are naturally absent.

A straightforward solution is selecting products that do not result in search conversions as negative examples for a given query. Nevertheless, unconverted products may not necessarily be irrelevant to the search query because users’ decisions can be swayed by many factors other than relevance, such as price, size, visual presentation, personal preference for flavor or brand, item position, etc. This is confirmed in our data verification process in examining search logs from Instacart, a leading e-commerce player in online grocery shopping, where we found more than 4% of query-product pairs from the search logs with zero historical conversions are labeled as positive (relevant) by human annotators.

A common alternative approach to address the lack of negative samples issue is in-batch negative sampling. Given a query $q_{i}$ and in-batch products $\mathcal{P}$, let $\mathcal{P}^{+}_{i}$ be the set of products whose relevance with the query $q_{i}$ are explicitly labeled. Then the vanilla sampling process aims to randomly select $K$ products from the difference of sets $\mathcal{P}$ and $\mathcal{P}^{+}_{i}$, i.e., $\mathcal{P}-\mathcal{P}^{+}_{i}$, as negative sample set $\mathcal{P}^{-}_{i}$, satisfying

where $\tilde{r}_{i,k}=0$ denotes the irrelevance between query $q_{i}$ and any randomly sampled product $p_{k}$ in negative sample set $\mathcal{P}^{-}_{i}$.

Although the naive random sampling technique can generate negative pairs, it fails to utilize the most informative negative samples, resulting in insufficient learning. To avoid this issue, several studies (Kalantidis et al., 2020; Robinson et al., 2020) have tried to use “hard” negative samples during the sampling procedure due to their informativeness. At Instacart, we implement the selection of hard negative samples using our search embedding technique introduced in Section 2.1.1 and determining by the similarity scores between products and queries calculated by Eq. (2). The hard negative sampling process with a negative size equal to $K$ can be represented as follows:

However, indiscriminately assuming irrelevance between a given query $q_{i}$ and in batch product $p_{k}\in\mathcal{P}_{i}^{-}$ will inevitably introduce false negatives. Our case study based on the Instacart search log shows that for the query “Honey”, the product “Honey 100%”, which is definitely relevant to the query but is not the product in this pair, will be retrieved as the top-ranked hard negative sample by the hard negative sampler mentioned in Eq. (4), clearly proving the existence of pooling bias. The pooling bias diminishes the quality of augmented negative pairs, impeding the model from conducting precise relevance assessment and necessitating a concrete solution.

MiniLM-L3-v2 ⁵⁵5https://huggingface.co/sentence-transformers/paraphrase-MiniLM-L3-v2 and MiniLM-L-2-v2 ⁶⁶6https://huggingface.co/cross-encoder/ms-marco-MiniLM-L-2-v2 serve as bi-encoder and cross encoder in our experiments. During the training, the bi-encoder is frozen. We set the temperature hyperparameter $\tau$ to 2.

As the STS Benchmark evaluation dataset does not allow list-wise comparison, we decide to adopt the evaluation metrics utilized in script provided in the document of cross encoder⁷⁷7https://github.com/UKPLab/sentence-transformers/blob/master/examples/training/cross-encoder/training_stsbenchmark.py, which are Pearson correlation coefficient and Spearman correlation coefficient. Besides, we leverage the Area under the ROC Curve (AUROC) as an additional evaluation metric for a more fair comparison. These three metrics perform pair-wise evaluation simultaneously.

We compare our proposed Bias-mitigating Hard Negative Sampling (BHNS) with Vanilla Negative Sampling (VNS) and Hard Negative Sampling (HNS), they serve as natural baselines widely used in diverse scenarios. Furthermore, we extend our experiment by adding the following advanced baselines specially designed for handling false negatives:

• •

  STAR (Stable Training Algorithm for dense Retrieval) (Zhan et al., 2021) is a training pattern developed to combine random negative sampling with hard negative sampling to optimize the training process of the bi-encoder. In this study, we transfer their strategy of performing combination to the case of cross encoder.

• •

  FALSE (False negAtive sampLes aware contraStive lEarning model) (Zhang et al., 2022) is a strategy of detecting false negative samples in an unsupervised contrastive learning scenario. We transfer this method to the supervised training of the cross-encoder by adopting their idea of capturing false negatives based on inter-sample similarity.

• •

  CET (Coupled Estimation Technique) (Cai et al., 2022) leverages a subordinate cross encoder, i.e., observer, to monitor and correct false negative bias during the training of the primary cross encoder, i.e. reranker.

To perform a fair comparison, we also choose MiniLM-L-2-v2 as the cross-encoder for baselines. For methods involving hard negative sampling, i.e., HNS, FALSE and STAR, we adopt the bi-encoder same to BHNS (MiniLM-L3-v2) to guide the selection of hard negatives.

We test different sampling sizes to validate the superiority as well as the robustness of our proposed BHNS. The results are listed in Table  2, where $K$ is the negative sampling size. Comparison across different sampling strategies implies that hard negative sampling may not contribute to decent performance due to the introduced false negative bias, while BHNS equipped with our proposed BHNS is enabled to handle the pooling bias, thus performing better via the bias mitigation.

While efficiency can be regarded as a potential trouble as we introduce additional computations for bias exclusion, we observe 32% - 38% training time increase, compared with VNS method. Considering that inference speed will not be influenced, we confirm that our method contributes to superior performance with an acceptable compromise on efficiency.

It is also worth noting that BHNS outperforms CET, which involves an additional cross-encoder to observe and eliminate false negatives. It indicates that leveraging a frozen bi-encoder to simply estimate the likelihood of false negatives is superior to supervised sophisticated approaches demanding extra training, showcasing the soundness of the hypothesis we propose in Section 3.1, as well as validating the rationale behind our design.

We also conduct ablation study on each component in BHNS. The results are presented in Table 3. While both modules manifest competitive effectiveness compared with baselines, the pseudo label generation module performs better. Nevertheless, the combination of both modules, i.e., BHNS, is capable of achieving a more stable and superior performance. This study further validates that our design is non-redundant.

Datasets used for training and evaluating the cross-encoder model come from Instacart’s ongoing data collection effort to benchmark search relevance, where a random sample of search queries and the products ranked in top display positions are sent for human annotation every month to judge the relevance between queries and products. The annotators categorize each (query, product) pair into five categories:

• •

  Strongly Relevant: The product is exactly the type of product the search query is asking for.

• •

  Relevant: The product fits the search query, but there are likely better alternatives.

• •

  Somewhat Relevant: The product is not exactly what the search query asks for, but it is understandable why it shows up.

• •

  Not Relevant: The product is not what the search query asks for, and it is unclear why it shows up.

• •

  Offensive: The product is unacceptable and creates a bad user experience.

To help the model grasp the finer granularity in relevance, we convert the above categories into soft labels in the training dataset by taking “Strongly Relevant” as 1, “Relevant” as 0.5, “Somewhat Relevant” as 0.2, and “Not Relevant/Offensive” as 0.1. Scores from multiple annotators for the same (query, product) pair are averaged.

Annotated classes in the evaluation dataset are converted with a coarser stroke, with “Strongly Relevant” and “Relevant” as 1, and “Somewhat Relevant”/“Not Relevant”/“Offensive” as 0.1, in accordance with our previous work (Xie et al., 2022).

This golden dataset with human annotation is of extremely high quality, but it can introduce bias in model training in that all samples come from products occupying top display positions in search. As a result, they are much more relevant to the search query than other products not selected in the sample. To ensure that the model is capable of correctly scoring less relevant products, we further expand both the training and the evaluation datasets by including an additional 20% (query, product) pairs where the query and the product are randomly and independently sampled from all eligible queries and products from the full dataset. These random negative pairs get a score of 0, based on the assumption that, on average, they are even less relevant than the “Not Relevant”/ “Offensive” products that appeared in top search positions.

The present search embedding model serving as the bi-encoder for experiments is originally initialized with a MiniLM-L3-H384-uncased model⁸⁸8https://huggingface.co/nreimers/MiniLM-L6-H384-uncased pretrained on paraphrase dataset⁹⁹9https://www.sbert.net/examples/training/paraphrases/README.html. It is further finetuned using Instacart’s in-house dataset extracted from the search log as described in the reference (Xie et al., 2022). The search embedding model is frozen during the training of the cross-encoder model and is only leveraged for guiding negative sampling.

The cross-encoder is initialized with a pretrained MiniLM-L-2-V2 model¹⁰¹⁰10https://huggingface.co/cross-encoder/ms-marco-MiniLM-L-2-v2. The chunk size of negative sampling is set to 4. The temperature hyperparameter $\tau$ is set to 2 for our method.

We focus on the two evaluation metrics: Normalized Discounted Cumulative Gain at $M$ (NDCG@$M$), which measures the effectiveness of the model in a list-wise setting, and Area under the ROC Curve (AUROC), which measures the model’s pair-wise performance. For NDCG, we aggregate the data by search queries and evaluate the model’s ability to rank the products for each query by their relevance at $M$ = 5, 10, or 20. For AUROC, we convert the classes “Strongly Relevant” and “Relevant” to positive, and the classes “Somewhat Relevant”/“Not Relevant”/“Offensive” as well as the additional random (query, product) pairs to negative, to make the labels binary.

We compare our proposed Bias-mitigating Hard Negative Sampling (BHNS) method with Vanilla Negative Sampling (VNS, Eq. (3)) and Hard Negative Sampling (HNS, Eq. (4)) strategies as baselines. We set the chunk size of sampled negatives to 4 for all sampling strategies implemented for fair comparisons.

We conduct our experiment as described above and obtain results listed in Table 4. The comparison showcases the advantage of adopting a false negative sampling strategy, as the cross-encoder equipped with any negative sampling methods outperforms the one without them on nearly all the metrics. Compared to VNS, HNS does not contribute to performance increase in terms of AUROC, and the performance enhancement on NCDG is minimal, implying that the pooling bias introduced during the sampling process impedes the thorough exploitation of those informative true hard negatives. As an extension, our proposed Bias-mitigating Hard Negative Sampler addresses this shortcoming of HNS by introducing False Negative Estimation, and thus assists cross encoder to manifest more competitive performance.

To further demonstrate how our Bias-mitigating Hard Negative Sampling alleviates the pooling bias in the sampling process, we perform an intuitive case study to compare BHNS with conventional negative sampling approaches, i.e., VNS and HNS. Given the query “honey”, we retrieve corresponding negative products using VNS, HNS, and BHNS, the results are listed in Table 5.

It is obvious that the vanilla negative sampling method (VNS) is incapable of capturing informative hard negative samples, thus failing to boost the cross-encoder via learning from valuable sources. The hard negative sampling (HNS) strategy, although effectively collecting informative products, assumes the irrelevance between query and sampled products, thus unexpectedly generating false negatives and introducing the pooling bias. Our method, Bias-mitigating Hard Negative Sampling, successfully avoids probable false negatives by assigning lower scores to them. Also, even though potential false negative products are collected, the pseudo label generation mechanism specially designed to tackle this scenario is able to hint the model about the potential bias, thus achieving the most promising performance.

The search system at Instacart consists of a retrieval stage and a ranking stage. The retrieval stage casts a broad net and retrieves a large number of products related to the search query from multiple sources using both sparse and dense signals. It is followed by a multi-step ranking stage, where ranking models with increasing complexities are applied to filter out less relevant products at each step. This cascade pattern ensures that only high-quality search results are presented to the end user. Given the complexity and the latency implication of the cross-encoder model, it is tested as part of the final ranking stage, where the top 50 items get reranked to maximize search conversion.

The final ranker is a deep & cross network (DCN) model (Wang et al., 2017) with a variety of features, such as how relevant the product is to the search query, the product’s popularity and availability, the user’s personal preference, etc. In the baseline ranker model serving in production, the query-product cosine similarity from the existing search embedding model (Xie et al., 2022) is one of the dominant relevance features. In this experiment, we use the similarity score derived from the cross-encoder model, both with and without BHNS, and retrain the DCN ranker model using the same ranking objective. The retrained ranking is used as a replacement for the existing ranker in this test, while other components of the search pipeline remain unchanged.

In addition to evaluation metrics involved in Table 4, we extend to cover more evaluation metric, i.e., Mean Reciprocal Rank (MRR), for more comprehensive evaluation.

We benchmark the performance of the new ranker model against the baseline model on a sample of recent searches from Instacart’s business, and compare the two models’ performance in ranking, where converted products are taken as positive labels. The results are listed in Table  6. While the other features going into the two ranking models remain identical, the cross encoder boosted by Bias-mitigating Hard Negative Sampling strategy proves to be effective in significantly improving both list-wise and pair-wise ranking metrics, outperforming the one without BHNS. The comparison further demonstrates the superiority of Bias-mitigating Hard Negative Sampling in handling pooling bias in real-world e-commerce scenario.