AE-smnsMLC: Multi-Label Classification with Semantic Matching and Negative Label Sampling for Product Attribute Value Extraction

As both product text and labels (i.e., attribute values) contain textual information, our proposed model AE-smnsMLC makes use of both product text and label text information. Specifically, we design two encoders for encoding the product text and label text information respectively, which are text encoder and label encoder. Then a semantic matching approach with negative label sampling is designed to help both encoders learn better representations for product text and labels. The goal of semantic matching is to push the label feature of a given product towards its text feature which contains richer and complete information about the product. The overall architecture of our proposed method is shown in Fig. 2. The major components of our model includes text encoder, label encoder, semantic matching module, negative label sampling module, label selector and loss weight estimator. Other components such as predictor with attention mechanism and label prior matching are kept the same as HTCInfoMax [3].

We use pre-trained BERT-base model as part of the text encoder which takes the product title and description as input. The output of BERT is fed into a convolutional layer which does a convolution operation along the sequence length and outputs the final text representation for the product denoted as $T_{final}\in\mathbb{R}^{c\times d_{h}}$, where $c$ is the length of sequence after convolution and $d_{h}$ is the dimension of hidden states of BERT.

Each label has specific text information describing its meaning. It is intuitive to make use of such text information to help learn better representations for labels. Thus, we introduce a label encoder which takes advantage of the label text information to learn the semantic representations $H_{L}\in\mathbb{R}^{N\times d_{l}}$ for all labels, $d_{l}$ is the dimension of label embedding. The process and structure of the label encoder is shown in Fig. 3.

Since the product text contains more and complete information about the product, it represents the semantic meaning of the current product well. While the text for each label is relatively short which consists of only several short pieces of text that are attribute name and specific attribute value. However, for each product, the representation learned from its all attribute values (i.e., labels) should also uniquely identify this product, thus we design a semantic matching method to push the labels’ representation of a product towards its text feature learned from the complete product text. In this way, the semantic matching can help both text encoder and label encoder learn better text and labels’ representation respectively during the training process.

The details of semantic matching method is shown in Fig. 4. The left half shows the overall process of semantic matching, while the right half shows the details of max pooling used in it. For the side of text feature, the semantic matching module applies mean pooling on $T_{final}\in\mathbb{R}^{c\times d_{h}}$ to obtain pooled text feature $T_{p}\in\mathbb{R}^{1\times d_{h}}$. For the side of label feature, it first uses a label selector to get the ground truth labels’ representation $L_{gt}\in\mathbb{R}^{m\times d_{l}}$ for the current product from $H_{L}$, $m$ is the number of ground truth labels of the current product. Then max pooling is used to obtain the combined label representation $L_{cr}\in\mathbb{R}^{1\times d_{l}}$ for the current product as shown in the right half of Fig. 4. After this, the semantic matching module calculates the similarity between the combined label embedding and the text feature of the current product, the goal is to push the label embedding as close as possible to the text feature in the feature space which can help the label encoder learn informative representation for all labels. Thus it can help the model enhance the ability of identifying multiple attribute values for each product correctly. We use cosine similarity here for simplicity. The semantic matching loss is $L_{sm}=-\frac{1}{P}\sum_{i=1}^{P}S_{sim}^{i}.$ $P$ is the total number of products (i.e., instances) in the training set.

The labels (i.e., attribute values) belong to the same attribute may be difficult for models to distinguish because they are much closer to each other than other labels belong to a different attribute. Fig. 5 shows such an example. The label ”1 Liter” is much closer to the label “2 Liter” in the same attribute “Capacity” than to the label “Black” which belongs to another attribute “Color”. In other words, the difference between labels in the same attribute (i.e., intra-group difference) is smaller than the difference between labels belonging to two different attributes (i.e., inter-group difference).

To help the model enhance the ability of distinguishing similar labels (i.e., attribute values) belonging to the same attribute, we propose a negative label sampling method for training the model. The whole process of this method is shown in Fig. 6 and the specific sampling approach for negative labels is shown in Fig. 7. To be specific, we first construct a mapping dictionary $A2L$ which stores all the labels (i.e., attribute values) belong to the same attribute. Then the same label selector as in Fig. 4 is used to get the ground truth attribute names $AN$ and corresponding labels for each of the attributes $L_{gtIDs}^{AN}$ for the current product. They are taken as input for the negative label sampling algorithm together with the mapping dictionary $A2L$. The output of this sampling algorithm is a list of sampled negative labels for the current product and it is utilized to obtain the negative labels’ feature matrix $L_{neg}$ from all labels’ feature matrix $H_{L}$. A mean pooling is applied on the negative labels’ feature $L_{neg}$ to get the final compact negative labels’ representation $L_{cr-neg}$ for the current product. Similar to the semantic matching module, it calculates the similarity between this compact negative label representation $L_{cr-neg}$ and the pooled text feature $T_{p}$ of the current product. The goal is to push away the negative labels as far as possible to the current product which can help the model learn more discriminative feature representation for similar labels belong to the same attribute. Thus during the training process, the negative label sampling method aims at minimizing the similarity score $S_{sim-neg}$ between the negative labels and text feature of each product. The loss of negative label sampling is $L_{ns}=\frac{1}{P}\sum_{i=1}^{P}S_{sim-neg}^{i}$.

We use a real-world e-commerce dataset (it is not public due to commercial reasons) possessed by a commercial company to conduct experiments for all models. All the labels (i.e., attribute values) are annotated by the merchants when they create their products on the company’s website. Thus no additional human annotation is needed for training our model. The text and labels are in Japanese. Due to its large size (47 million products), we sample products from three different domain as three datasets whose statistics are shown in Table I. Evaluation metrics of multi-label classification including Precision (P), Recall (R), Micro-F1 (MiF1) and Macro-F1 (MaF1) are used to evaluate the performance of models.

Sequence labeling models need labor-intensive and time-consuming human annotation for every token in every product’s text (i.e., title and description) which is not available in the stated dataset. Thus no sequence labeling models can be trained on the dataset. Therefore, we only consider classification models as the baselines. Specifically, there are two groups of baselines used to compare with our model AE-smnsMLC which are described as follows. Same as in our model, the parameters of the pre-trained BERT-base model used in all these baselines are fixed due to out of GPU memory issue of fine-tuning. The pre-trained BERT is trained with 10 million product descriptions in Japanese.

All baselines and our model adopt the same setting for all hyper-parameters. To be specific, the maximum length of token sequence of product text is set to 256, the dimensions of text and label embedding are 768, the kernel size of CNN layer is 4. HTCInfoMax-based baselines are re-implemented by us using PyTorch based on the released code [3]. We implement our model and BERT-based baselines using PyTorch. All experiments are conducted on a single NVIDIA Quadro M6000 GPU sever with 12G GPU memory.

The experimental results of our model and baselines on the three datasets are shown in Table II. We can see that our proposed model AE-smnsMLC generally outperforms all baselines on Recall, Micro-F1 and Macro-F1 including the strong baseline in the group of HTCInfoMax-based baselines which utilizes pretrained CLS label embedding to initialize its structure encoder (i.e., the third from last row in Table II). Although this baseline has higher precision score on Gardening and Plants, our model obtains much better Recall, Micro-F1 and Macro-F1 scores by improvements of 7.40%, 1.31%, 10.37% on Gardening and by improvements of 8.47%, 3.16%, 4.34% on Plants respectively. This demonstrates that the semantic matching and negative label sampling methods help our model perform better on all labels, because Micro-F1 and Macro-F1 scores are more reliable metrics that evaluate models in a much more comprehensive way. Furthermore, our model outperforms the previously mentioned strongest baseline on Precision, Recall and Micro-F1 on Gradening Tools dataset by improvements of 0.34%, 2.30% and 1.39% respectively, which also verifies the superiority of our model. In addition, the consistent performance of our model across three different datasets demonstrates that our proposed model can learn better text representation and better representations for all attribute values (i.e, labels) by matching the semantic representation of all ground truth attribute values of a given product towards its text representation. In other words, the performance consistency validates the effectiveness of our proposed semantic matching method.

Besides, from the comparison between models in the group of HTCInfoMax-based baselines, we can see that pretrained label embedding generated by using the text (e.g., “stainless steel”) of attribute values can help improve the performance. This indicates that label text information is helpful for product attribute value extraction in terms of reformulating it as a multi-label classification task. In addition, the baselines with CNN layer performing much better than that without CNN indicates the importance of CNN layer in the text encoder.

We conduct ablation studies to demonstrate the effect of different components in our model such as negative label sampling and pooling method. We also design a third variant of our model called “AE-smnsMLC w/o LabelPrior” which removes the label prior matching module for more ablation study. The results of ablation studies are shown in Table III. For each of the three models in this table, two pooling methods for ground truth labels in semantic matching module are experimented. One is max pooling as shown in Fig. 4, the other is mean pooling which replaces the max pooling.

From Table III, one can see that max pooling is generally performing better than meaning pooling. And our models with (the last two rows) or without (the first two rows) label prior matching achieve comparably similar results, which verifies the effectiveness of our proposed model in capturing the semantic connections between attribute values of each product and its text. It also indicates that the label prior matching helps. Furthermore, the comparison between AE-smnsMLC and AE-smMLC in Table III (the last four rows) and Table II (the last two rows) shows that AE-smnsMLC generally performs better than AE-smMLC without negative label sampling. This demonstrates that negative label sampling can help the model distinguish similar labels by learning more discriminative feature for them and thus improves the performance.

To further verify the usefulness of negative label sampling, we conduct a case study on Gardening dataset. Specifically, we select the prediction results of AE-smnsMLC and its variant AE-smMLC on all labels belong to some attribute (e.g., “Events/Holiday”) from Gardening dataset and calculate Micro-F1 scores for these labels. The comparison between the performance results of our model and its variant without negative label sampling on all labels belong to the attribute “Events/Holiday” is shown in Fig. 8. One can see that AE-smnsMLC performs better on almost all labels belong to this attribute which demonstrates that negative label sampling can help the model learn much more discriminative feature for labels belong to the same attribute and thus improve the model’s ability of distinguishing similar labels.

Earlier models for product attribute value extraction are rule-based extraction methods [6, 15]. Later on, [12] treats this task as a sequence labeling task and many neural network based sequence labeling models with different techniques such as attention mechanism, using hierarchical taxonomy of product, adaptive CRF decoder and so on are designed [21, 18, 11, 9, 19]. Most of multi-label text classification models can be categorized into two groups. One group is local models which build a classifier for each label or labels in the same level of the label taxonomy [16, 1, 7]. The other group is global models building one classifier for all labels [8, 10, 17, 13, 14, 22, 3]. The latter four models make use of label structure information. In addition, attention-based models are popular for text classification in recent years [20, 2, 4].