Exploiting Latent Codes: Interactive Fashion Product Generation, Similar Image Retrieval, and Cross-Category Recommendation using Variational Autoencoders

Variational Autoencoders (VAEs) are similar to the structure of autoencoders but with a different approach for latent representation learning. An autoencoder (AE) is a type of Artificial Neural Network (ANN) where the expected output is exactly the input by finding an efficient latent representation of the data or encodings using a hidden or latent layer called ”bottleneck” layer (often called latent codes in this paper). Both AE and VAE have similar structures made of an encoder that maps the input $x$ into a latent space $z$, the bottleneck layer, and a decoder that takes $z$ to reconstruct the input data into $x^{\prime}$, in which the target is to minimize the reconstruction loss or the difference between $x$ and $x^{\prime}$ Torres Fernandez (2018). The main challenge for autoencoders in general is to accurately learn a representation that embeds the most important features of the data and encode them to generalise beyond the training set by being able to gain control over the latent space to produce images with similar characteristics.

Instead of using the output of the encoder (i.e. the bottleneck), VAEs allow generative modelling by causing the latent variables to be normally distributed by forming a new layer in the architecture consisting of parameters mean $\mu$ and standard deviation $\sigma$ of a normal distribution $\mathcal{N}$($\mu,\sigma^{2}$) Spinner et al. (2018). During optimization and training, the target reference is is a standard normal distribution $\mathcal{N}$(0,1) where we force our normal distribution to diverge as close as possible to our target reference using Kullback-Leibler (KL divergence) Odaibo (2019). VAE now calculates loss function consisting two terms - KL-loss, which is dependent on the parameters of the distribution and can be computed easily, and reconstruction loss which is based on random and discrete outcome of $z$ that means its gradients are not computable preventing backpropagation over parameters $\mu$ and $\sigma$.

To solve this, Kingma and Welling (2013) proposed a method called reparameterization trick which tries to transform and approximate a normal distribution $\mathcal{N}$($\mu,\sigma^{2}$) into $\mathcal{N}$(0,1), allowing calculation of the gradients over $\mu$ and $\sigma$ by introducing a random component $\epsilon$ sampled from $\mathcal{N}(0,1)$ as seen in Equation 1 which eliminates randomness since the last remaining random entity in the equation is $\epsilon$. This enables the training of VAE to achieve both the generative and feature extraction capability applicable on this pipeline. Techniques and methods in this paper focus on how to take advantage of this inherent structure in VAE. The overall structure of VAE is seen in Figure 1.

The fashion industry has been quick to adapt in designing new products using deep learning. Two popular deep learning models for generative modelling are Generative Adversarial Networks (GANs) and Variational Autoencoders (VAEs). Deverall et al. (2017) experimented with a traditional GAN and DiscoGAN trained on athletic shoes using UT-Zap50K dataset and tested the generated images using a classifier based on GoogleNet and achieving 88% validation accuracy. Rostamzadeh et al. (2018) created a baseline results on high-resolution image generation conditioned on the given text description using their own created dataset of 293,008 high definition fashion images paired with item decriptions. Variational Autoencoders are also popular in generating new product designs. Zhu et al. (2018) used VAE on popular fashion image styles trained on DeepFashion dataset and was able to generate new products based on 30 discovered styles using Nonnegative matrix factorization (NMF). Current generative modelling puts emphasis on synthesizing new product designs that are realistic but enabling the end-user to be the one interacting with the model to create desired designs are still missing.

Deep learning has been used in content-based image retrieval and matching for quite a long time such as for medical images Qayyum et al. (2017) which uses deep Convolutional Neural Network (CNN) achieving 99.97% classification accuracy for retrieval tasks and for clothing products that uses CNN to match a real-world example of garment into same item in an online shop.

Traditional autoencoders and variational autoencoders are often used for image reconstruction or generation, but they can be used for image retrieval because of its capability to extract features. Krizhevsky and Hinton (2011) used a very deep autoencoder and semantic hashing for image retrieval by mapping images to short binary codes for extreme fast retrieval, while Torres Fernandez (2018) implemented a simple VAE with only 2 layers depth in both encoder and decoder and yielded good results using approximation of the latent space to $z$ = $\mu_{z}$ evaluated by mAP.

Content-based Recommendation has been dominated by deep learning where there is emphasis on finding similarity on two images using deep neural networks. Chen et al. (2017) used two convolutional neural network where the first neural network is used for classification and the second neural network to model similarity score between pair of images using Jaccard similarity. This method was able to achieve 0.5 accuracy on both classification and similarity score. Another work by Tuinhof et al. (2018) utilized a combination of CNN and a modified k-NN algorithm used for ranking in feature space. GANs can also be used to recommend products. A work by Kumar and Gupta (2019) implemented $c^{+}$GAN which tweaks traditional c-GAN where it recommends a generated pair of pants that might go well with the given shirt.

A work by Lin et al. (2019) proposed a neural co-supervision learning framework which they called FAshion Recommendation Machine (FARM) which is a neural network architecture similar to variational autoencoders that improves visual understanding and enhances visual matching by incorporating a layer-to-layer matching mechanism for combining aesthethic information. This was able to outperform state-of-the-art models for outfit recommendation. Works on cross-domain recommendation typically revolve around recommending one category.

Researches in fashion product generation, content-based similar image retrieval and recommendation have utilized deep learning to an extent but each study only focus on solving one problem using one deep learning architectures. In this work, the author experimented on using VAE by exploiting its ability to produce a latent representation of the image for generative modelling and feature extraction and introduced new ways to utilize these latent codes as features to solve all the problems aforementioned that are related in nature and can be integrated in a pipeline for fashion industry.

The dataset is obtained from the combination of Fashion Product Images Dataset from Kaggle and scraping images from Zappos website. The dataset is intentionally limited to footwear (8919 entries), eyewear (3378 entries) and bags (9462 entries) which totals to 21,759 data samples. Topwears and bottomwears are not considered given that majority of topwear and bottomwear includes human faces and poses that could add entropy and more features for the variational autoencoder to represent. The size of the images are 180 x 240 pixels.

The model is trained in Google Colab that uses K80 Tesla as GPU and includes 25GB of RAM for 200 epochs which took approximately four hours. The model architecture is composed of three Convolution layers and Dense layers for the encoder and three Deconvolution layers and Dense layers for the decoder and is built in Keras. The bottleneck layer is size 16.

Here, the user is able to directly control the z-vector that is used to synthesize new images by using range sliders. By controlling the range slider, the user is able to manipulate certain attributes such as length, color, thickness, etc. of the fashion products according to their preference. As the range slider is moved, the latent code is changed and is given to the decoder in the back-end of the application for reconstructing the image. For consistency, the user-generated vector from using the application sliders is denoted as $\bar{z}$-vector.

After the user interacts with the slider and synthesizes the image, its $\bar{z}$-vector is used to find similar images from the database. For this purpose, the author proposes two methods that tries to solve the problem in different manners. The first one takes advantage of the objective function being optimized by VAEs while the second one exploits the sampling process from reparameterization trick used to derive $z_{x}$ from $\mu_{x}$ and $\sigma_{x}$.

In order to use these techniques for image retrieval, encodings of each images are generated using the trained encoder and stored in a database. The encoding for an image is composed of n-dimensional $\mu_{x}$-vector and $\sigma_{x}$-vector and an n-dimensional vector computed from a fixed epsilon sampling that will be discussed in the next sections.

Image retrieval is computationally expensive, most especially if the algorithm has to compare the $\bar{z}$-vector to every entry in the database. In order to reduce the number of comparisons, images could be clustered using their generated latent codes. Creating clusters also allows us to find a method of recommending products from other clusters and categories that also uses the latent encodings.

To cluster each products, K-Means Clustering can be used as it allows choosing the number of $k$ clusters to group the data and provides the value of the centers for each clusters. These centers will be used for cross-product recommendation. In K-Means clustering, given a set of n data points in a high-dimensional space and integer $k$, the objective is to determine $k$ points called centers that minimizes the mean squared distance from each point to its nearest designated centerKhan and Ahmad (2004). Since the dataset comes with a subcategory tag, $k$ would be equal to three.

The author experimented with Euclidean distance as metric for K-means and evaluated using $\mu_{x}$-vector in comparison to using the fixed epsilon sampled $z_{x}$-vectors. Also, before the image retrieval process, the image constructed from the user-generated $\bar{z}$-vector is assigned to a cluster by calculating the distance in comparison to the stored cluster centers.

Aside from providing the most similar images to the user, latent codes can also be employed in recommending products from other clusters or categories. This also takes full advantage of the nature of the latent space VAEs generate which enables smooth interpolation and smooth mix of features a decoder can understand.

Simple vector arithmetic in the latent space can be utilized to navigate through the smooth interpolation of the latent space. For example, to generate specific features such as generating glasses on faces, we find two samples, one with glasses and one without and obtain the difference of their encoded vectors from the encoded. We add the difference as ”glasses” vector to the encodings of any other image that has no glasses.

Here, the clusters or categories are defined by the cluster centers. Take the difference of each center vectors (e.g. $center_{i}$ - $center_{j}$) as $diff_{ij}$ from one another where $i$ is the cluster assignment of $\bar{z}$-vector and $j$ is a cluster not equal to $i$. In order to recommend products from other clusters, add each difference vector $diff_{ij}$ to $\bar{z}$-vector as seen in Equation 3.

This does not recommend a reconstructed image using the decoder with $recommendation_{ij}$. It does, however, recommend a retrieved image from the database with the nearest to the calculated vector from the $j$th cluster using the presented image retrieval techniques.

Figure 3 shows comparison of the original image and the reconstruction of the image using the trained model. VAEs are known to produce blurry images due to the small bottleneck layer and latent space, but the reconstruction is still very recognizable as the original. The reconstruction of the first image in Figure 3 caught the right colors and structure but missed out some intricate details while the reconstruction of the glasses had some washed out colors.

Since the bottleneck layer is size sixteen, the user is provided sixteen range sliders for synthesizing images by manipulating the $\bar{z}$-vector. This is decoded by the decoder in the backend. The application provides the user an encoding from the database selected randomly as starting point. The user is given sixteen sliders to interactively control the $\bar{z}$ latent vector (Fig. 4).

Moving one slider changes some attribute of the the product image. In Figure 5, it illustrates how the product reconstructed image changes as the sixth slider is moved from left to right changing mostly its color and texture.

In traditional Variational Autoencoders, it is difficult to determine which specific attributes of the image will transform upon changing one latent code in the $z$-vector.

Each image from the training dataset is encoded using the encoder of the trained model and stored in the database. Both the $\mu_{x}$ and $\sigma_{x}$ vectors and fixed $z_{x}$ vectors are part of the encodings upon finishing training the model. These encodings will be used for retrieving similar images, clustering and recommendations.

K-means clustering with $k$ = 3 is used to cluster the entries in the database using both $\mu_{x}$ vector and fixed $z_{x}$ vector. Each entry has a subcategory tag (e.g. footwear, eyewear, or bag) and will be used for evaluating both clustering and similar image retrieval. All fashion products in the database are included in this clustering procedure.

As seen in Table 1 and 2, the F1 score of the two cases are almost similar. The F1 scores are decent considering that only a feature consisting of 16 dimensions is used to cluster images which resembles image classification although the scores are relatively lower on eyewear as K-means often tries to produce clusters of relatively uniform sizes and is very sensitive to skewed or imbalanced dataset Kumar et al. (2014). This means that using K-Means on this imbalanced dataset caused very low precision for eyewears and low recall for bags. This leads to having wrongly categorized items and cross-product recommendations that is not a member of the category. Even though each images have correct tags, the cluster assignments predicted by K-Means and the cluster centers shall be used.

The most similar images are retrieved using log-likelihood maximization and Euclidean distance on fixed-epsilon sampling encoding as discussed earlier. But before retrieving similar images to the one generated by the user, its cluster is predicted first using Euclidean distance between $\bar{z}$-vector and the cluster centers. The application retrieves similar images from the cluster it was assigned.

One evaluation metric for image retrieval is Mean Average Precision (mAP) Torres Fernandez (2018). Here, it is calculated on a randomly sampled 500 images for each category in retrieving top-10, top-25, top-50, and top-500 similar images. Figure 3 shows that using Euclidean Distance on Fixed-Epsilon Sampling Encodings yield higher mAP score most especially if the image retrieval is done on those which belong to the same cluster. Not only it is quicker but it has less false positives. This is highly encouraging since Torres Fernandez (2018) had mAP score of 0.95 on their best model using VAE for MNIST images which are a less complex dataset.

Figure 6 and 7 shows comparison of using these two methods for similar image retrieval. By observation, both techniques present the same image as the best match from the database but deviates on the second and third best match. From these two figures (Fig. 6 and 7), the first technique seems to prioritize similarities in color and texture while the second method tries to give emphasis on overall structure.

In recommending products from other categories, after getting $recommendation_{ij}$ (as discussed in 3.4), the closest image match from the database in that cluster is provided to the user using the two image matching techniques discussed in the paper so far.

Figures 8 and 9 illustrate sample recommendations for other clusters. Similar to the observation in Section 4.5, both similar image retrieval techniques for the cluster assignment of the generated product obtain the same output for the best match (b) but are different in the second best match (c). Recommendations using both methods are quite different from each other as observed both figures.

Figure 9 shows the fixed-epsilon sampling technique seems to provide a better suggestion of products to pair with the eyeglasses emphasizing color and patterns recommending a blue bag and blue footwear to go with an eyewear with bluish tint. This is not the case recommendations for eyewear in Figure 8 which tries to suggest a bag and a shoe of mostly white in color.

Both suggestions provide almost a similar items - a brown, leather bag and a thick-framed eyeglases. Fixed-epsilon sampling technique (Fig. 9 tries to recommend an eyeglass grayish tint similar to the generated image while the other method brings out the brown hue similar to the suggested bag and second best matching image.