An Application to Generate Style Guided Compatible Outfit

Outfit recommendation is a relatively well studied area in which researchers aim to recommend outfits based on the notion of compatibility between lifestyle items, see (Vasileva et al., 2018; Song et al., 2019; Li et al., 2020; Wang et al., 2021) for more details. A substantial volume of work has also been done on the specific area of personalised recommendations (Zhan et al., 2021; Lu et al., 2021). However, none of them specifically take style into account while learning compatibility within outfits. We realise that style is an essential component in modelling outfit compatibility as an outfit may look compatible under one style construct but not in another.

An example of style guided outfit creation is provided in Figure 1. The same top item (highlighted by a dotted rectangle) is used to create two outfits under two different styles, namely Formal and Casual. This is useful in the situation where we consider that a user likes the top item but is doubtful about making the final purchase. A style-guided algorithm will have two advantages: (a) it will reject an outfit which may be otherwise compatible but not in accordance with the desired style, and (b) it will pick compatible outfits from different styles, hence expanding the choice to the user. A style-independent algorithm, on the other hand, gets biased towards the dominant style (assuming formal style is dominant).

Style guided recommender system requires special attention from the research community as most of the work are unsupervised in nature. Some example research in this area are listed below: (a). Kuhn et al. (Kuhn et al., 2019) refer an outfit as a style fit and do not explicitly use style information for modeling compatibility; (b). Jeon et al. (Jeon et al., 2021) extract fashion attributes from full-body outfit images for classifying outfit style; (c). Li et al. (Li et al., 2019) models outfit level style from item descriptions; (d). Singhal et al. (Singhal et al., 2020) models context and type jointly using Graph Neural Network (GNN), and style between item pairs are modeled using autoencoder without any explicit style information. Each of these works lack in one way or the other the ability to generate style guided outfits.

Lai et al. proposed the Theme Matters paper (Lai et al., 2020) (archived work) which comes closest to our work. It projects a supervised approach that applies theme-aware attention to item pairs with fine-grained category tags (e.g., long-skirt, mini-skirt, t-shirt, etc.). There are two specific drawbacks, the first of which is that such fine-grained category information is not always available and can be ambiguous if done manually. Secondly, the size of the model increases exponentially with the number of fine-grained categories.

We propose a Style-Attention-based Compatible Outfit Generation (SATCOGen) framework that uses high-level categories (e.g., topwear, bottomwear, footwear, accessory, etc.) and outfit-level style information. It consists of a Style-Compatibility-Attention Network (SCA Net) (Lin et al., 2020) and a novel style encoder network called Variational Style Encoder Network (VSEN) which encodes the style of an outfit into a latent space. This encoding is used to provide style-specific subspace attention, along with category information during the computation of embedding. Multiple loss functions ensure style encoding, general as well as style-specific compatibility. For the generation task, given an anchor item, beam search is used to generate style-specific outfit. We have provided a demonstration api to showcase the kind of outfits that are generated for an anchor item given various styles.

The fundamental philosophy guiding the work in this paper is that in practical circumstances compatibility between lifestyle items present within an outfit is contingent on the style to which the outfit belongs. In a nutshell, we make use of Style-Compatibility-Attention Network (SCA Net), a compatibility learning framework that makes use of features extracted from the image of an item based on category information as previously done by (Lin et al., 2020) and then add style component to it. The methodology of SATCOGen is explained in greater detail below.

A smooth latent vector representation for outfit style is learnt using Variational inference in a novel style encoder named Variational Style Encoder Network (VSEN). There are two main trends in denoting an outfit, as an ordered sequence of items (Han et al., 2017; Nakamura and Goto, 2018) or as set (Bettaney et al., 2019; Chen et al., 2019). We choose the latter representation, which brings in two important properties, namely permutation invariance and allowance for varying length. This assumption enables us to select the set transformer approach proposed in (Lee et al., 2019) for our style encoder job. Keeping in mind that our work is not restricted only to compatibility learning and also involves outfit generation, we ensure that every outfit style is represented by the first two moments of a Gaussian distribution which is proximal to the unit Gaussian $\mathcal{N}(0,\mathbbm{1})$, a mechanism we borrowed from Variational inference (Blei et al., 2017). This further ensures smooth representation of the latent style space. The advantage of this step will be clear during the outfit generation stage.

A vector, sampled from the Gaussian distribution representing the style of the outfit is used to classify the style of the outfit. This ensures that VSEN is able to capture specific information about the style of the outfit as well. Thus, given the styles and their corresponding style vectors, this module solves a multi-class classification problem using an MLP with $n$ layers.

We modify the subspace attention network proposed in (Lin et al., 2020) to learn compatibility between items in an outfit. In the previous network, the image of an item within an outfit is passed through a ResNet18, and the embedding vector output is multiplied by learnt masks that help to learn the subspaces. The item and target categories are then consumed to estimate the subspace attention weights which subsequently leads to a weighted average of the masked embeddings to be denoted as the final embedding of the item in the tuple <item, item category, target category>. We tweak this and estimate the subspace attention weights by providing the outfit specific sample style vector from VSEN as an additional input. This helps to learn compatibility conditional on the style of the outfit.

There are four loss functions used in our method for learning style specific compatibility. We have the KL divergence loss from the VSEN network and the classification loss from the downstream job. We also have the compatibility loss from the SCA Net which is based on the popular triplet loss. And finally, we introduce one more loss function to account for penalisation when the wrong style is specified for an outfit. The overall loss for our method is given as the weighted sum of these four individual losses.

We created a female outfit dataset with style annotations.

Even though an outfit can have multiple style tags, for this work we ensure that each outfit has a single style. In the situation that an outfit has different style tags from the two annotations, we choose the randomly select a style. After the annotation task, it was found that some of the styles were heavily under-represented. To mitigate this issue, we merged certain styles that are semantically similar and discarded some as well. For example, party and wedding were merged into party while all outfits from classic and trendy were discarded. At the end we had ~28K outfits.

We constructed separate test sets for FITB and compatibility tasks similar to (Vasileva et al., 2018) by creating soft negative (SN) and hard negative (HN) samples corresponding to each positive outfit sample. In case of SN, we sample random items from the matching higher level categories (e.g., topwear, bottomwear, footwear, etc.), whereas for HN, we do the sampling from matching fine-grained categories²²2We make use of fine-grained category information only during evaluation or testing phase. (e.g., t-shirts, heels, shoes, etc.). It is to be noted that HN samples are relatively harder to differentiate from positives than SN samples. This gradation helps in evaluating the performance of SATCOGen at various difficulty levels. We consider five FITB and Compatibility test datasets and report the mean performance.

The two modules of SATCOGen, namely VSEN and SCA Net make use of RestNet18 as the backbone CNN architecture. We freeze all the layers of ResNet18 except the last convolutional block, which is connected to a new fully connected layer that outputs a 64 dimensional embedding vector similar to the state-of-the-art compatibility learning methods (Lin et al., 2020; Vasileva et al., 2018).

VSEN aggregates the CNN features of all the items in an outfit using the SAB Set Transformer (Lee et al., 2019) with a hidden dimension of 32 and two heads for mean and variance. The output of style vector of VSEN has a dimension of 64 which is followed by two fully connected MLP layers for the style classification task. Here we consider batch size as 128 and employ the Adam optimizer (Kingma and Ba, 2015) with an initial learning rate of $5\times 10^{-6}$. When combining the KL-Divergence loss with the cross entropy loss of classification, we consider 0.05 as the weight coefficient corresponding to KL-Divergence loss. After training the VSEN module, we freeze all of its parameters and use it in evaluation mode to train the rest of the SATCOGen framework. For learning the parameters of the SCA Net (five subspaces), we use the Adam optimizer with batch size of 32 triplets, and initial learning rate of $1\times 10^{-6}$. The Attention Network of SCA Net transforms the concatenated one-hot-encoded category vectors and the extracted style vectors from VSEN into 32 dimensions, respectively (using a single fully connected layer), which are then forwarded to two fully connected layers (after concatenation). Finally, the output is the five subspace attention weights.

For outfit generation, we have optimized the inference code using native spark implementation to get item embedding given style and category information and to run beam search at scale on a large scale of volume of anchor and candidate items, resulting in 20X run time reduction compared to single box implementation. We compared the run time on  60K anchor items with  30K average child candidates per category, 5 average categories in beam search template and beam width of 3.

We show the efficacy of SATCOGen in generating superior outfits for online shopping portals by comparing its performance against Theme Matters (TM) (Lai et al., 2020) based on FITB Acc. and Compat. AUC scores. Table 2 presents the FITB Acc. and Compat AUC results on SN and HN samples of Zal dataset for TM and SATCOGen (our proposed model). SATCOGen outperforms the TM results, mainly because of the requirement of fine-grained category information to have improved performance. Fig. 4 shows an example of the quality of parent-child category combinations with style pre-conditioning.

We have created a demonstration api to showcase the outfits generated by using SATCOGen framework given an anchor item, a style and candidate items from other categories. The outfit templates have been identified after discussion with fashion experts. An example of such template would be, (’dress’, ’heels’, ’bag’, ’jewellery’). In the api, we provide users to select a category and subsequently an anchor item from the category for which the user wants to view outfits under different legitimate styles. The top-5 outfits per style are displayed for the selected anchor item. In figure-5, we are showcasing the top-3 outfits for an anchor t-shirt given style sporty and template ’t-shirt’, ’leggings’, ’trainer-shoes’, ’bra’. The demo-api along with details and screenshots can be found here ³³3https://github.com/Lucky-Dhakad/SATCOGen-Demo-api.

In this paper, we presented SATCOGen - a novel outfit generation framework based on styles and evaluated its performance on Zal - a newly introduced outfit dataset with style tags associated with each outfit. In general, the outfit generation process using beam search algorithm is time consuming and not scalable for datasets having thousands of items for each category. We ensured scalability, by optimizing the code and making it suitable for execution on Hadoop Clusters, which reduced the execution time drastically. Finally, we presented a web-interface that demonstrates outfit generation starting from a chosen anchor item, a pre-defined style, and a template. In the future, we are planning to extend the outfit dataset with ethnic outfits (specific to Indian context) and productionize the SATCOGen framework to provide an enhanced shopping experience to the customers.