Set2Box: Similarity Preserving Representation Learning for Sets

Concepts: A box is a $d$-dimensional hyper-rectangle whose representation consists of its center and offset [19]. The center describes the location of the box in the latent space and the offset is the length of each edge of the box. Formally, given a box $\mathrm{B}=(\mathrm{c},\mathrm{f})$ whose center $\mathrm{c}\in\mathbb{R}^{d}$ and offset $\mathrm{f}\in\mathbb{R}_{+}^{d}$ are in the same latent space, the box is defined as a bounded region:

$$\mathrm{B}\equiv\{\mathrm{p}\in\mathbb{R}^{d}:\mathrm{c}-\mathrm{f}\preceq\mathrm{p}\preceq\mathrm{c}+\mathrm{f}\},$$

where $\mathrm{p}$ is any point within the box. We let $\mathrm{m}\in\mathbb{R}^{d}$ and $\mathrm{M}\in\mathbb{R}^{d}$ be the vectors representing the minimum and the maximum at each dimension, respectively, i.e., $\mathrm{m}=\mathrm{c}-\mathrm{f}$ and $\mathrm{M}=\mathrm{c}+\mathrm{f}$. Given two boxes $\mathrm{B}_{X}=(\mathrm{c}_{X},\mathrm{f}_{X})$ and $\mathrm{B}_{Y}=(\mathrm{c}_{Y},\mathrm{f}_{Y})$, the intersection is also a box, represented as:

$\displaystyle\mathrm{B}_{X}\cap\mathrm{B}_{Y}\!\!\equiv$

$\displaystyle\{\mathrm{p}\in\mathbb{R}^{d}:\mathbf{max}(\mathrm{m}_{X},\mathrm{m}_{Y})\preceq\mathrm{p}\preceq\mathbf{min}(\mathrm{M}_{X},\mathrm{M}_{Y})\}.$

The volume $\mathbb{V}(\mathrm{B})$ of the box $\mathrm{B}$ is computed by the product of the length of an edge in each dimension, i.e., $\mathbb{V}(\mathrm{B})=\prod_{i=1}^{d}(\mathrm{M}[i]-\mathrm{m}[i])$. The volume of the union of the two boxes is simply computed by $\mathbb{V}(\mathrm{B}_{X})+\mathbb{V}(\mathrm{B}_{Y})-\mathbb{V}(\mathrm{B}_{X}\cap\mathrm{B}_{Y})$.

Representation: The core idea of Set2Box is to model each set $s$ as a box $\mathrm{B}_{s}=(\mathrm{c}_{s},\mathrm{f}_{s})$ so that the relations with other sets are properly preserved in the latent space. To this end, Set2Box approximates the volumes of the boxes to the relative sizes of the sets, i.e., $\mathbb{V}(\mathrm{B}_{s})\propto|s|$. In addition to the single-set level, Set2Box aims to preserve the relations between different sets by approximating the volumes of the intersection of the boxes to the intersection sizes of the sets, i.e., $\mathbb{V}(\mathrm{B}_{s_{i}}\cap\mathrm{B}_{s_{j}})\propto|s_{i}\cap s_{j}|$. Notably, Set2Box not only addresses limitations of random hashing and vector-based embeddings, but it also has various advantages benefited from unique properties of boxes, as we discuss in Section VI.

Objective: Now we turn our attention to how to capture such overlaps between sets using boxes. Recall that our goal is to derive accurate and versatile representations of sets, and towards the first goal, we take relations beyond pairwise into consideration. Specifically, we consider three different levels of set relations (i.e., single, pair, and triple-wise relations) to capture the underlying high-order structure of sets. In another aspect, we aim to derive versatile set representations that can be used to estimate various similarity measures (e.g., Jaccard Index and Dice Index). With these goals in mind, we design an objective function that aims to preserve elemental relations among triple of sets. Specifically, given a triple $\{s_{i},s_{j},s_{k}\}$ of sets, we consider seven cardinalities from three different levels of subsets: (1) $|s_{i}|$, $|s_{j}|$, $|s_{k}|$, (2) $|s_{i}\cap s_{j}|$, $|s_{j}\cap s_{k}|$, $|s_{k}\cap s_{i}|$, and (3) $|s_{i}\cap s_{j}\cap s_{k}|$ which contain single, pair, and triple-wise information, respectively, and we denote them from $c_{1}(s_{i},s_{j},s_{k})$ to $c_{7}(s_{i},s_{j},s_{k})$. These seven elements fully-describe the relations among the three sets, and we argue that any similarity measures are computable using them. In this regard, we aim to preserve the ratios of the seven cardinalities by the volumes of the boxes $\mathrm{B}_{s_{i}}$, $\mathrm{B}_{s_{j}}$, and $\mathrm{B}_{s_{k}}$ by minimizing the following objective:

$$\mathcal{J}(s_{i},s_{j},s_{k},\mathrm{B}_{s_{i}},\mathrm{B}_{s_{j}},\mathrm{B}_{s_{k}})=\\ \sum_{\ell=1}^{7}\left(p_{\ell}(s_{i},s_{j},s_{k})-\hat{p}_{\ell}(\mathrm{B}_{s_{i}},\mathrm{B}_{s_{j}},\mathrm{B}_{s_{k}})\right)^{2},$$

where $p_{\ell}$ is the ratio of the $\ell$^th cardinality among the three sets (i.e., $p_{\ell}=c_{\ell}/\sum_{\ell^{\prime}}c_{\ell^{\prime}}$) and $\hat{p}_{\ell}$ is the corresponding ratio estimated by the boxes. Since there exist $|\mathcal{S}|\choose 3$ possible triples of sets, taking all such combinations into account is practically intractable, and thus we resort to sampling some of them. We sample a set $\mathcal{T}$ of triples that consists of a set $\mathcal{T}^{+}$ of positive triples and a set $\mathcal{T}^{-}$ of negative triples, i.e., $\mathcal{T}=\mathcal{T}^{+}\cup\mathcal{T}^{-}$. Specifically, the positive set $\mathcal{T}^{+}$ and the negative set $\mathcal{T}^{-}$ are obtained by sampling three connected (i.e., overlapping) sets and three uniform random sets, respectively. Then, the final objective function we aim to minimize is:

$$\mathcal{L}=\sum\nolimits_{\{s_{i},s_{j},s_{k}\}\in\mathcal{T}}\mathcal{J}(s_{i},s_{j},s_{k},\mathrm{B}_{s_{i}},\mathrm{B}_{s_{j}},\mathrm{B}_{s_{k}}).$$

(1)

Notably, the proposed objective function aims to capture not only the pairwise interactions between sets, but also the triplewise relations to capture high-order overlapping patterns of the sets. In addition, it does not rely on any predefined similarity measure, but is a general objective for learning key structural patterns of sets and their neighbors. This prevents the model from overfitting to a specific measure and enables the model to yield accurate estimates to diverse metrics, as shown empirically in Section V.

Box Embedding: Then, given a set $s$, how can we derive the box $\mathrm{B}_{s}=(\mathrm{c}_{s},\mathrm{f}_{s})$, that is, its center $\mathrm{c}_{s}$ and offset $\mathrm{f}_{s}$? To make the method generalizable to unseen sets, Set2Box introduces a pair of learnable embedding matrices $\mathbf{Q}^{\mathrm{c}}\in\mathbb{R}^{|\mathcal{E}|\times d}$ and $\mathbf{Q}^{\mathrm{f}}\in\mathbb{R}_{+}^{|\mathcal{E}|\times d}$ of entities, where $\mathbf{Q}_{i}^{\mathrm{c}}\in\mathbb{R}^{d}$ and $\mathbf{Q}_{i}^{\mathrm{f}}\in\mathbb{R}^{d}$ represent the center and offset of an entity $e_{i}$, respectively. Then, the embeddings of the entities in the set $s$ are aggregated to obtain the center $\mathrm{c}_{s}$ and the offset $\mathrm{f}_{s}$:

$$\mathrm{c}_{s}=\textbf{pooling}(s,\mathbf{Q}^{\mathrm{c}})\;\;\;\text{and}\;\;\;\mathrm{f}_{s}=\textbf{pooling}(s,\mathbf{Q}^{\mathrm{f}})$$

where pooling is a permutation invariant function. Instead of using simple functions such as mean or max, we use attentions to highlight the entities that are important to obtain either the center or the offset of the box. To this end, we define a pooling function that takes the context of each set into account, termed set-context pooling (SCP). Specifically, given a set $s$ and an item embedding matrix $\mathbf{Q}$ (which can be either $\mathbf{Q}^{\mathrm{c}}$ or $\mathbf{Q}^{\mathrm{f}}$), it first obtains the set-specific context vector $\mathrm{b}_{s}$:

$$\mathrm{b}_{s}=\sum_{e_{i}\in s}\alpha_{i}\mathbf{Q}_{i}\;\;\;\text{where}\;\;\;\alpha_{i}=\frac{\exp(\mathrm{a}^{\intercal}\mathbf{Q}_{i})}{\sum_{e_{j}\in s}\exp(\mathrm{a}^{\intercal}\mathbf{Q}_{j})}$$

where $\mathrm{a}$ is a global context vector shared by all sets. Then using the context vector $\mathrm{b}_{s}$, which specifically contains the information on set $s$, it obtains the output embedding from:

$$\textbf{SCP}(s,\mathbf{Q})=\sum_{e_{i}\in s}\omega_{i}\mathbf{Q}_{i}\;\;\text{where}\;\;\omega_{i}=\frac{\exp(\mathrm{b}_{s}^{\intercal}\mathbf{Q}_{i})}{\sum_{e_{j}\in s}\exp(\mathrm{b}_{s}^{\intercal}\mathbf{Q}_{j})}.$$

To be precise, $\mathrm{c}_{s}=\textbf{SCP}(s,\mathbf{Q}^{\mathrm{c}})$ and $\mathrm{f}_{s}=|s|^{\frac{1}{d}}\textbf{SCP}(s,\mathbf{Q}^{\mathrm{f}})$. Note that, for the offset $\mathrm{f}_{s}$, we further take unique geometric properties of boxes into consideration. For any entity $e_{i}\in s$, the subset relation $\{e_{i}\}\subseteq s$ holds, and thus the same condition $\mathrm{B}_{\{e_{i}\}}\subseteq\mathrm{B}_{s}$ for boxes is desired, which should satisfy $\mathrm{f}_{\{e_{i}\}}\preceq\mathrm{f}_{s}$ and thus $\max_{e_{i}\in s}\mathrm{f}_{\{e_{i}\}}=\max_{e_{i}\in s}\mathbf{Q}_{i}^{\mathrm{f}}\preceq\mathrm{f}_{s}$. However, since SCP is the weighted mean of entities’ embeddings, the output of SCP is bounded by the input embeddings in all dimensions, i.e., $\min_{e_{i}\in s}\mathbf{Q}_{i}^{\mathrm{f}}\preceq\mathrm{f}_{s}\preceq\max_{e_{i}\in s}\mathbf{Q}_{i}^{\mathrm{f}}$, which inevitably contradicts the aforementioned condition. In these regards, for the offset $\mathrm{f}_{s}$, we multiply an additional regularizer $|s|^{\frac{1}{d}}$ that helps boxes to properly preserve the set similarity, i.e., $\mathrm{f}_{s}=|s|^{\frac{1}{d}}\textbf{SCP}(s,\mathbf{Q}^{\mathrm{f}})$.

Smoothing Boxes: By definition, a box $\mathrm{B}=(\mathrm{c},\mathrm{f})$ is a bounded region with hard edges whose volume is

$$\mathbb{V}(\mathrm{B})=\prod\nolimits_{i=1}^{d}(\mathrm{M}[i]-\mathrm{m}[i])=\prod\nolimits_{i=1}^{d}\text{ReLU}(\mathrm{M}[i]-\mathrm{m}[i])$$

where $\mathrm{m}=\mathrm{c}-\mathrm{f}$ and $\mathrm{M}=\mathrm{c}+\mathrm{f}$. This, however, disables gradient-based optimization when boxes are disjoint [30], and thus we smooth the boxes by using an approximation of ReLU:

$$\mathbb{V}(\mathrm{B})=\prod\nolimits_{i=1}^{d}\text{Softplus}(\mathrm{M}[i]-\mathrm{m}[i])$$

where $\text{Softplus}(\mathrm{x})=\frac{1}{\beta}\log\left(1+\exp(\beta\mathrm{x})\right)$ is an approximation to $\text{ReLU}(\mathrm{x})$, and it becomes closer to ReLU as $\beta$ increases (specifically, Softplus $\rightarrow$ ReLU as $\beta\rightarrow\infty$). In this way, any pairs of boxes overlap each other, and thus non-zero gradients are computed for optimization.

Encoding Cost: Each box consists of two vectors, a center and an offset, and it requires $2\cdot 32d=64d$ bits to encode them, assuming that we are using float-32 to represent each real number. Thus, $64|\mathcal{S}|d$ bits are required to store the box embeddings of $|\mathcal{S}|$ sets.

Box Quantization: We propose box quantization, a novel scheme for compressing boxes by using substantially smaller number of bits. Note that conventional product quantization methods [40], which are for vector compression, are straightforwardly applicable, by independently reducing the center and the offset of the box. However, it hardly makes use of geometric properties of boxes, and thus it does not properly reflect the complex relations between them. The proposed box quantization scheme effectively addresses this issue through two steps: (1) box discretization and (2) box reconstruction.

$\circ$ Box Discretization. Given a box $\mathrm{B}_{s}=(\mathrm{c}_{s},\mathrm{f}_{s})$ of set $s$, we discretize the box as a $K$-way $D$-dimensional discrete code $\mathrm{C}_{s}\in\{1,\cdots,K\}^{D}$ which is more compact and requires much less number of bits to encode than real numbers. To this end, we divide the $d$-dimensional latent space into $D$ subspaces ($\mathbb{R}^{d/D}$) and, for each subspace, learn $K$ key boxes. Specifically, in the $i$^th subspace, the $j$^th key box is denoted by $\mathrm{K}_{j}^{(i)}=(\mathrm{c}_{j}^{(i)},\mathrm{f}_{j}^{(i)})$ where $\mathrm{c}_{j}^{(i)}\in\mathbb{R}^{d/D}$ and $\mathrm{f}_{j}^{(i)}\in\mathbb{R}_{+}^{d/D}$ are the center and offset of the key box, respectively. The original box $\mathrm{B}_{s}$ is also partitioned into $D$ sub-boxes $\mathrm{B}_{s}^{(1)},\cdots,\mathrm{B}_{s}^{(D)}$ and the $i$^th code of $\mathrm{C}_{s}$ is decided by:

$$\mathrm{C}_{s}[i]=\operatorname*{arg\,min}_{j}\textbf{dist}\left(\mathrm{B}_{s}^{(i)},\;\mathrm{K}_{j}^{(i)}\right)$$

where dist($\cdot,\cdot$) measures the distance (i.e., dissimilarity) between two boxes, and we can flexibly select the criterion. In this paper, we specify the dist function, using softmax, as:

$$\mathrm{C}_{s}[i]=\operatorname*{arg\,max}_{j}\frac{\exp\left(\textbf{BOR}\left(\mathrm{B}_{s}^{(i)},\mathrm{K}_{j}^{(i)}\right)\right)}{\sum_{j^{\prime}}\exp\left(\textbf{BOR}\left(\mathrm{B}_{s}^{(i)},\mathrm{K}_{j^{\prime}}^{(i)}\right)\right)}$$

(2)

where BOR (Box Overlap Ratio) is defined to measure how much a box $\mathrm{B}_{X}$ and a box $\mathrm{B}_{Y}$ overlap:

$$\textbf{BOR}(\mathrm{B}_{X},\mathrm{B}_{Y})=\frac{1}{2}\left(\frac{\mathbb{V}(\mathrm{B}_{X}\cap\mathrm{B}_{Y})}{\mathbb{V}(\mathrm{B}_{X})}+\frac{\mathbb{V}(\mathrm{B}_{X}\cap\mathrm{B}_{Y})}{\mathbb{V}(\mathrm{B}_{Y})}\right).$$

$\circ$ Box Reconstruction. Once the discrete code $\mathrm{C}_{s}$ of set $s$ is generated, in this step, we reconstruct the original box based on it. To be specific, we obtain the reconstructed box $\widehat{\mathrm{B}}_{s}=(\widehat{c}_{s},\widehat{f}_{s})$ by concatenating $D$ key boxes from each subspace encoded in $\mathrm{C}_{s}$:

$$\widehat{\mathrm{B}}_{s}=\Big{\|}_{i=1}^{D}\mathrm{K}_{\mathrm{C}_{s}[i]}^{(i)}.$$

More precisely, $\widehat{\mathrm{B}}_{s}$ is reconstructed by concatenating the centers and the offsets of the $D$ key boxes respectively. Since $\mathrm{C}_{s}$ encodes key boxes that largely overlap with the box $\mathrm{B}_{s}$ (i.e., high BOR), if properly encoded, we can expect the reconstructed box $\widehat{\mathrm{B}}_{s}$ to be geometrically similar to the original box $\mathrm{B}_{s}$. In Section V-C, we demonstrate the effectiveness of the proposed box quantization scheme by comparing it with the product quantization method.

Differentiable Optimization: Recall that Set2Box⁺ is an end-to-end learnable algorithm, which requires all processes to be differentiable. However, the $\operatorname*{arg\,max}$ operation in Eq. (2) is non-differentiable, and to this end, we utilize the softmax with the temperature $\tau$:

$$\widetilde{\mathrm{C}}_{s}[i]=\frac{\exp\left(\textbf{BOR}\left(\mathrm{B}_{s}^{(i)},\mathrm{K}_{j}^{(i)}\right)/\tau\right)}{\sum_{j^{\prime}}\exp\left(\textbf{BOR}\left(\mathrm{B}_{s}^{(i)},\mathrm{K}_{j^{\prime}}^{(i)}\right)/\tau\right)}.$$

(3)

Note that $\widetilde{\mathrm{C}}_{s}[i]$ is a $K$-dimensional probabilistic vector whose $j$^th element indicates the probability for $\mathrm{K}_{j}^{(i)}$ being assigned as the closest key box, i.e., the probability of $\mathrm{C}_{s}[i]=j$. Then, the key box $\widetilde{\mathrm{K}}_{s}^{(i)}=(\widetilde{\mathrm{c}}_{s}^{(i)},\widetilde{\mathrm{f}}_{s}^{(i)})$ in the $i$^th subspace is the weighted sum of the $K$ key boxes:

$$\widetilde{\mathrm{K}}_{s}^{(i)}=\sum\nolimits_{j=1}^{K}\widetilde{C}_{s}[i][j]\cdot\mathrm{K}_{j}^{(i)}.$$

If $\tau=0$, Eq. (3) is equivalent to the $\operatorname*{arg\,max}$ function, i.e., a one-hot vector where $\mathrm{C}_{s}[i]$^th dimension is 1 and others are 0. In this case, $\widetilde{\mathrm{K}}_{s}^{(i)}$ becomes equivalent to $\mathrm{K}_{\mathrm{C}_{s}[i]}^{(i)}$, which is the exact reconstruction derivable from the discrete code $\mathrm{C}_{s}$. However, since this hard selection is non-differentiable and thus prevents an end-to-end optimization, we resort to the approximation by using the softmax with $\tau\neq 0$ which is fully differentiable. Specifically, we use different $\tau$s’ in forward ($\tau=0$) and backward ($\tau=1$) passes, which effectively enables differentiable optimization.

Joint Training: For further improvement, we introduce a joint learning scheme in the box quantization scheme. Given a triple $\{s_{i},s_{j},s_{k}\}$ of sets from the training data $\mathcal{T}$, we obtain their boxes $\mathrm{B}_{s_{i}}$, $\mathrm{B}_{s_{j}}$, and $\mathrm{B}_{s_{k}}$ and their reconstructed ones $\widehat{\mathrm{B}}_{s_{i}}$, $\widehat{\mathrm{B}}_{s_{j}}$, and $\widehat{\mathrm{B}}_{s_{k}}$ using the box quantization. While the basic version of Set2Box⁺ optimizes the following objective:

Encoding Cost: To encode the reconstructed boxes for each set, Set2Box⁺ requires (1) key boxes and (2) discrete codes to encode each set. There exist $K$ key boxes in each of the $D$ subspaces whose dimensionality is $d/D$, which requires $64Kd$ bits to encode. Each set is encoded as a $K$-way $D$-dimensional vector, which requires $D\log_{2}K$ bits. To sum up, to encode $|\mathcal{S}|$ sets, Set2Box⁺ requires $64Kd+|\mathcal{S}|D\log_{2}K$ bits. Notably, if $K\ll|\mathcal{S}|$, then $64Kd$ bits are negligible, and typically, $D\log_{2}K\ll 64d$ holds. Thus, the encoding cost of Set2Box⁺ is considerably smaller than that of Set2Box.

Similarity Computation: Once we obtain set representations, it is desirable to rapidly compute the estimated similarities in the latent space. Boxes, which Set2Box and Set2Box⁺ derive, require constant time to compute a pairwise similarity between two sets, as formalized in Lemma 1.

Proof. Assume that the true similarity $\text{sim}(s,s^{\prime})$ is computable using $|s|$, $|s^{\prime}|$, and $|s\cap s^{\prime}|$. They are estimated by $\mathbb{V}(\mathrm{B}_{s})$, $\mathbb{V}(\mathrm{B}_{s^{\prime}})$, and $\mathbb{V}(\mathrm{B}_{s}\cap\mathrm{B}_{s^{\prime}})$, respectively, and each of them is computed by the product of $d$ values, which takes $O(d)$ time. Hence, the total time complexity is $O(d)$. ∎

Machines & Implementations: All experiments were conducted on a Linux server with RTX 3090Ti GPUs. We implemented all methods including Set2Box and Set2Box⁺ using the Pytorch library.

Hyperparameter Tuning Table III describes the hyperparameter search space of each method. The number of training samples, $|\mathcal{T}^{+}|$ and $|\mathcal{T}^{-}|$, are both set to $10$ for Set2Box, Set2Box⁺, and their variants. For the vector-based methods, Set2Vec and Set2Vec⁺, since three pairwise relations are extractable from each triple, $\left\lceil\frac{7}{3}|\mathcal{T}^{+}|\right\rceil$ positive triples and $\left\lceil\frac{7}{3}|\mathcal{T}^{-}|\right\rceil$ negative samples are used for training. We fix the batch size to $512$ and use the Adam optimizer. In Set2Box⁺, we fix the softmax temperature $\tau$ to $1$.

Datasets: We use eight publicly available datasets in Table II. The details of each dataset is as follows:

• •

  Yelp (YP) consists of user ratings on locations (e.g., hotels and restaurants), and each set is a group of locations that a user rated. Ratings higher than 3 are considered.

• •

  Amazon (AM) contains reviews of products (specifically, those categorized as Movies & TV) by users. In the dataset, each user has at least 5 reviews. A group of products reviewed by the same user is abstracted as a set.

• •

  Netflix (NF) is a collections of movie ratings from users. Each set is a set of movies rated by each user, and each entity is a movie. We consider ratings higher than 3.

• •

  Gplus (GP) is a directed social network that consists of ‘circles’ from Google+. Each set is the group of neighboring nodes of each node.

• •

  Twitter (TW) is also a directed social networks consisting of ‘circles’ in Twitter. Each set is a group of neighbors of each node in the graph.

• •

  MovieLens (ML1, ML10, and ML20) are collections of movie ratings from anonymous users. Each set is a group of movies that a user rated. Sets of movies with ratings higher than 3 are considered.

Baselines: We compare Set2Box and Set2Box⁺ with the following baselines including the variants of the methods discussed in Section III:

• •

  Set2Bin encodes each set $s$ as a binary vector $\mathrm{z}_{s}\in\{0,1\}^{d}$ using a random hash function. See Section III for details.

• •

  Set2Vec embeds each set $s$ as a vector $\mathrm{z}_{s}\in\mathbb{R}^{d}$ which is obtained by pooling learnable entity embeddings using SCP. Precisely, given two sets $s$ and $s^{\prime}$ and their vector representations $\mathrm{z}_{s}$ and $\mathrm{z}_{s^{\prime}}$, it aims to approximate the predefined set similarity $\text{sim}(s,s^{\prime})$ by the inner product of $\mathrm{z}_{s}$ and $\mathrm{z}_{s^{\prime}}$, i.e., $\langle\mathrm{z}_{s},\mathrm{z}_{s^{\prime}}\rangle\approx\text{sim}(s,s^{\prime})$.

• •

  Set2Vec⁺ incorporates entity features $\mathbf{X}\in\mathbb{R}^{|\mathcal{E}|\times d}$ into the set representation. Features are projected using a fully-connected layer and then pooled to a set embedding.

In order to obtain the entity features for Set2Vec⁺, which incorporates features into the set representations, we generate a projected graph (a.k.a., clique expansion) where nodes are entities, and any two nodes are adjacent if and only if their corresponding entities co-appear in at least one set. Specifically, we generate a weighted graph by assigning a weight (specifically, the number of sets where the two corresponding entities co-appear) to each edge, and we apply node2vec [46], a popular random walk-based network embedding method, to the graph to obtain the feature of each entity. Recall that vector-based methods, Set2Vec and Set2Vec⁺, are not versatile, and thus they need to be trained specifically to each metric, while the proposed methods Set2Box and Set2Box⁺ do not separate training. It should be noticed that search methods (e.g., LSH) are not direct competitors of the considered embedding methods, whose common goal is similarity preservation. We summarize the encoding cost of each method, including the variants of Set2Box and Set2Box⁺ used in Section V-C, in Table IV.

Evaluation: For the Netflix dataset, whose number of sets is relatively large, we used $5\%$ of the sets for training. For the other datasets, we used $20\%$ of the sets for training. The remaining sets are split into half and used for validation and test. We measured the Mean Squared Error (MSE) to evaluate the accuracy of the set similarity approximation. Since the number of possible pairs of sets is $O(|\mathcal{S}|^{2})$, which may be considerably large, we sample 100,000 pairs uniformly at random for evaluation. We consider four representative set-similarity measures for evaluation: Overlap Coefficient (OC), Cosine Similarity (CS), Jaccard Index (JI), and Dice Index (DI), which are defined as:

$$\frac{|s\cap s^{\prime}|}{\min(|s|,|s^{\prime}|)},\;\;\;\frac{|s\cap s^{\prime}|}{\sqrt{|s|\cdot|s^{\prime}|}},\;\;\;\frac{|s\cap s^{\prime}|}{|s\cup s^{\prime}|},\;\;\;\text{and}\;\;\;\frac{|s\cap s^{\prime}|}{\frac{1}{2}\left(|s|+|s^{\prime}|\right)},$$

respectively, between a pair of sets $s$ and $s^{\prime}$.

Accuracy: As seen in Figure 3, Set2Box⁺ yields the most accurate set representations while using a smaller number of bits to encode them. For example, in the Twitter dataset, Set2Box⁺ gives $40.8\times$ smaller MSE for the Jaccard Index compared to Set2Bin. In the Amazon dataset, Set2Box⁺ gives $11.0\times$ smaller MSE for the Overlap Coefficient than Set2Vec⁺. In both cases, Set2Box⁺ uses about $31\%$ of the encoding costs used by the competitors.

Conciseness: To verify the conciseness of Set2Box⁺, we measure the accuracy of competitors across various encoding costs. As seen in Figure 5, Set2Box⁺ yields compact representations of sets while keeping them informative. Vector-based methods are prone to the curse of dimensionality and hardly benefit from high dimensionality. While the MSE of Set2Bin decreases with respect to its dimension, it still requires larger space to achieve the MSE of Set2Box⁺. For example, Set2Bin requires $8.0\times$ and $34.9\times$ more bits to achieve the same accuracy of Set2Box⁺ in MovieLens 1M and Yelp, respectively. This is more noticeable in larger datasets, where Set2Bin requires up to $96.8\times$ of the encoding cost of Set2Box⁺, as shown in Figure V(a). These results demonstrate the conciseness of Set2Box⁺. In addition, Figure 6 shows how the accuracy of estimations by Set2Box and Set2Box⁺ depend on their encoding costs.

Speed: For the considered methods, Figure 7 shows the loss (relative to the loss after the first epoch) over time in two large datasets, MovieLens 20M and Netflix. The loss in Set2Box⁺ drops over time and eventually converges within an hour.

Further Analysis of Accuracy: We analyze how much sets estimated to be similar by the considered methods are actually similar. To this end, for each set $s$, we compare its $k$ most similar sets $G_{s,k}$ to those $\widehat{G}_{s,k}$ estimated to be the most similar using each method. Then, as in [4], we measure the quality $q(\widehat{G}_{s,k})$ of $\widehat{G}_{s,k}$ which measures how sets in $\widehat{G}_{s,k}$ are similar enough to $s$ compared to those in $G_{s,k}$:

$$q(\widehat{G}_{s,k})=\frac{\sum_{s^{\prime}\in\widehat{G}_{s,k}}\text{sim}(s,s^{\prime})}{\sum_{s^{\prime}\in G_{s,k}}\text{sim}(s,s^{\prime})}$$

where $\text{sim}(\cdot,\cdot)$ is the similarity (spec., Jaccard Index) between sets. The quality ranges from $0$ to $1$, and it is ideally close to 1, which indicates that the sets estimated to be similar by the method are similar enough compared to the ideal sets. Based on the above criterion, we measure the quality of each method with different $k$’s. We use the above configuration for Set2Box⁺, while the dimensions of the other methods are adjusted to require a similar encoding cost to Set2Box⁺. As shown in Figure 8, the average quality $q(\widehat{G}_{s,k})$ is the highest in Set2Box⁺ in all considered $k$’s in MovieLens 1M and Yelp, implying that the sets estimated to be similar by the proposed methods are indeed similar to each other.

Effects of Box Quantization: We examine the effectiveness of the proposed box quantization scheme in Section IV-B by comparing Set2Box-BQ with Set2Box-PQ. To this end, we measure the relative MSE defined as:

$$\frac{\text{MSE of {Set2Box-BQ}}}{\text{MSE of {Set2Box-PQ}}}$$

(5)

of each dataset. Figure 4(a) demonstrates that Set2Box-BQ generally derives more accurate set representations compared to Set2Box-PQ, implying the effectiveness of the proposed box quantization scheme. As shown in Table V, on average, Set2Box-BQ yields up to $26\%$ smaller MSE than Set2Box-PQ while using approximately the same number of bits. For example, in MovieLens 10M, Set2Box-BQ gives $1.89\times$ more accurate estimation than Set2Box-PQ in approximating the Dice Index. While Set2Box-PQ discretizes the center and offset of the boxes independently, without the consideration of their geometric properties, the proposed box quantization scheme effectively takes the geometric relations between boxes into account and thus yields high-quality compression.

Effects of Joint Training: We analyze the effects of the joint training scheme of Set2Box⁺ by comparing Set2Box-BQ ($\lambda=0$) and Set2Box⁺ ($\lambda\geq 0$) and to this end, we measure the relative MSE defined as:

$$\frac{\text{MSE of {Set2Box}\textsuperscript{+}}}{\text{MSE of {Set2Box-BQ}}}$$

(6)

of each dataset. Figure 4(b) shows that Set2Box⁺ is superior compared to Set2Box-BQ in most datasets indicating that jointly training the reconstruction boxes with the original ones leads to accurate boxes. As summarized in Table V, joint training reduces the average MSEs on the considered datasets, by up to $44\%$, together with the box quantization scheme. For example, together with the box quantization scheme, joint training reduces estimation error by $64\%$ and $38\%$ for the Jaccard Index on Gplus and for the Overlap Coefficient on Netflix, respectively. These results imply that learning quantized boxes simultaneously with the original boxes improves the quality of the quantization and thus its effectiveness. To further analyze these results, we investigate how the loss decreases with respect to $\lambda$. In Figure 9, we observe that training the reconstructed boxes alone ($\lambda=0$) is unstable, and learning the original boxes together ($\lambda>0$) helps not only stabilize but also facilitate the optimization.

Effects of Boxes: To confirm the effectiveness of using boxes for representing sets, we consider Set2Box-order, which is also a region-based geometric embedding method:

• •

  Set2Box-order: A set $s$ is represented as a $d$-dimensional vector $\mathrm{z}_{s}\in\mathbb{R}_{+}^{d}$ whose volume is computed as $\mathbb{V}(\mathrm{z}_{s})=\exp(-\sum_{i}\mathrm{z}_{s}[i])$. The volume of the intersection and the union of two representations $\mathrm{z}_{s}$ and $\mathrm{z}_{s^{\prime}}$ of sets $s$ and $s^{\prime}$, respectively, are:

  	$\displaystyle\mathbb{V}(\mathrm{z}_{s}\wedge\mathrm{z}_{s^{\prime}})=\exp\left(-\textstyle\sum\nolimits_{i}\max(\mathrm{z}_{s}[i],\mathrm{z}_{s^{\prime}}[i])\right),$
  	$\displaystyle\mathbb{V}(\mathrm{z}_{s}\vee\mathrm{z}_{s^{\prime}})=\exp\left(-\textstyle\sum\nolimits_{i}\min(\mathrm{z}_{s}[i],\mathrm{z}_{s^{\prime}}[i])\right),$

  respectively. The encoding cost is $32d|\mathcal{S}|$ bits.

We set the dimensions for Set2Box-order and Set2Box to $8$ and $4$, respectively, so that their encoding costs are the same. In Table VI, we compare Set2Box with Set2Box-order in terms of the average MSE on the considered datasets for each measure. Set2Box yields more accurate representations than Set2Box-order, implying the effectiveness of boxes to represent sets for similarity preservation. For example, Set2Box achieved $62\%$ lower average MSE than Set2Box-order in preserving the Overlap Coefficient.