Hard-Negative Sampling for Contrastive Learning: Optimal Representation Geometry and Neural- vs Dimensional-Collapse

Our main theoretical contributions are Theorems 1, 2, and 3, which, under a widely-studied latent data model, hold for any convex, argument-wise non-decreasing contrastive loss function, any non-negative and argument-wise non-decreasing hardening function to generate hard-negative samples, and norm-bounded representations of dimension at least $C-1$.

Theorem 1 establishes that the HSCL loss dominates the SCL loss and similarly the HUCL loss dominates the UCL loss. In this context we note that Theorem 3.1 in [30] is a somewhat similar result for the UCL setting, but for a special loss function and it does not address directly hard-negatives.

Theorem 2 is a novel result which states that the globally optimal representation geometry for both SCL and HSCL corresponds to Neural-Collapse (NC) (see Definition 6) with the same optimal loss value. In contrast to existing works (see related works section 2), we show that achieving NC in SCL and HSCL does not require class-conditional independence of the positive samples.

Similarly, Theorem 3 establishes the optimality of NC-geometry for UCL if the representation dimension is sufficiently large compared to the number of latent classes which, in turn, is implicitly determined by the joint distribution of the positive examples that corresponds to the augmentation mechanism.

A comprehensive set of experimental results on one synthetic and three real datasets are detailed in Sec. 5 and Appendix B. These include experiments that study effects of two initialization methods, three different feature normalization methods, three different batch sizes, two very different families of hardening functions, and two different CL sampling strategies. Empirical results show that when using the Adam optimizer with random initialization, the matrix of class means for SCL is badly conditioned and effectively low-rank, i.e., it exhibits Dimensional-Collapse (DC). In contrast, the use of hard-negatives at appropriate hardness levels mitigates DC and enables convergence to the global optima. A similar phenomenon is observed in the unsupervised settings. We also show that feature normalization is critical for mitigating DC in these settings. Results are qualitatively similar across different datasets, a range of batch sizes, hardening functions, and CL sampling strategies.

The theoretical results for our SCL setting, where in contrast to [7] and [13] we make use of label information in positive as well as negative sampling, are novel. The debiased SCL loss in [5] corresponds to our SCL loss, but no analysis pertaining to optimal representation geometry was considered in [5]. A recent ArXiv preprint by [6], which appeared after our own ArXiv preprint, [11], considers using label information for negative sampling in SCL with InfoNCE loss, calling it the SINCERE loss. Our theoretical results prove that NC is the optimal geometry for the SINCERE loss. Furthermore, our results and analysis also apply to hard-negative sampling as well, a scenario not considered thus far for SCL.

We would like to note that our theoretical set-up for UCL under the sampling mechanism of Figure 1, can be seen to be aligned with the SCL analysis that makes use of label information only for positive samples as done in [13] and [7]. Therefore, our theoretical results provide an alternative proof of optimality of NC based on simple, compact, and transparent probabilistic arguments complementing the proof of a similar result in [7]. We note that similarly to [7], our arguments also hold for the case when one approximates the loss using batches.

We want to point out that in all key papers that conduct a theoretical analysis of contrastive learning e.g., [1, 7, 25], the positive samples are assumed to be conditionally i.i.d., conditioned on the label. However, this conditional independence may not hold in practice when using augmentation mechanisms typically considered in CL settings e.g., [19, 20, 31, 4].

Unlike the recent work of [33] which shows that the optimization landscape of supervised learning with least-squares loss is benign, i.e. all critical points other than the global optima are strict saddle points, in Sec. 5 we demonstrate that the optimization landscape of SCL is more complicated. Specifically, not only may the global optima not be reached by SGD-like methods with random initialization, but also the local optima exhibit the Dimensional-Collapse (DC) phenomenon. However, our experiments demonstrate that these issues are remedied via HSCL whose global optimization landscape may be better. Here we note that [32] show that with unit-sphere normalization, Riemannian gradient descent methods can achieve the global optima for SCL, underscoring the importance of optimization methods and constraints for training in CL.

[27] argue that asymptotically (in number of negative samples) the InfoNCE loss for UCL optimizes for a trade-off between the alignment of positive pairs while ensuring uniformity of features on the hypersphere. However, a non-asymptotic and global analysis of the optimal solution is still lacking. In contrast, for UCL in Theorem 3, we show that as long as the embedding dimension is larger than the number of latent classes, which in turn is determined by the distribution of the similar samples, the optimal representations in UCL exhibit NC-geometry.

Our results also complement several recent papers, e.g., [22], [29], [28], that study the role of augmentations in UCL. Similar to theoretical works analyzing UCL, e.g. [1] and [16], our results also assume conditional independence of positive pairs given the label. This assumption may or may not be satisfied in practice.

We demonstrate that a recent result, viz., Theorem 4 in [12], that attempts to explain DC in UCL is limited in that under a suitable initialization, the UCL loss trained with Adam does not exhibit DC (see Sec. 5). Furthermore, we demonstrate empirically, for the first time, that HUCL mitigates DC in UCL at moderate hardness levels. For CL (without hard-negative sampling), [34] characterize local solutions that correspond to DC but leave open the analysis of training dynamics leading to collapsed solutions.

A geometrical analysis of HUCL is carried out in [24], but the optimal solutions are only characterized asymptotically (in the number of negative samples) and for the case when hardness also goes to infinity, the analysis seems to require knowledge of supports of class conditional distributions. In contrast, we show that the geometry of the optimal solution for HUCL depends on the hardness level and is, in general, different compared to UCL due to the possibility of class collision.

Let $f:{\mathcal{X}}\rightarrow{\mathcal{Z}}$ denote a (deterministic) representation mapping from data space ${\mathcal{X}}$ to representation space ${\mathcal{Z}}\subseteq{\mathbb{R}}^{d_{\mathcal{Z}}}$. Let ${\mathcal{F}}$ denote a family of such representation mappings. Contrastive Learning (CL) selects a representation from the family by minimizing an expected loss function that penalizes “misalignment” between the representation of an anchor sample $z=f(x)$ and the representation of a positive sample $z^{+}=f(x^{+})$ and simultaneously penalizes “alignment” between $z$ and the representations of $k$ negative samples $z^{-}_{i}:=f(x^{-}_{i}),i=1:k$.

We consider a CL loss function $\ell_{k}$ of the following general form.

This subsumes and generalizes popular CL loss functions such as InfoNCE and triplet-loss with sphere-normalized representations. InfoNCE corresponds to $\psi_{k}(t_{1:k})=\log(\alpha+\sum_{i=1}^{k}e^{t_{i}})$ with $\alpha>0$¹¹1This is the log-sum-exponential function which is convex over ${\mathbb{R}}^{k}$ for all $\alpha\geq 0$ and strictly convex over ${\mathbb{R}}^{k}$ if $\alpha>0$. and $\psi_{k}(t)=\max\{t+\alpha,0\}$, $\alpha>0$, is the triplet-loss with sphere-normalized representations. However, some CL losses such as the spectral contrastive loss of [8] are not of this form.

The CL loss is the expected value of the CL loss function:

where $p_{CL}(x,x^{+},x^{-}_{1:k})$ is the joint probability distribution of the anchor, positive, and $k$ negative samples and is designed differently within the supervised and unsupervised settings as described below.

This joint distribution may be interpreted from a sample generation perspective as follows: first, a common class label $y\in{\mathcal{Y}}$ for the anchor and positive sample is sampled from a class marginal probability distribution $\lambda$. Then, the anchor and positive samples are generated by sampling from the conditional distribution $q(x,x^{+}|y)$. Then, given $x,x^{+}$ and their common class label $y$, the $k$ negative samples and their labels are generated in a conditionally IID manner. The sampling of $y^{-}_{i},x^{-}_{i}$, for each $i$, can be interpreted as first sampling a class label $y^{-}_{i}$ different from $y$ in a manner consistent with the class marginal probability distribution $\lambda$ (sampling from distribution $r(y^{-}_{i}|y)$) and then sampling $x^{-}_{i}$ from the conditional probability distribution $s(\cdot|\cdot)$ of negative samples given class $y^{-}_{i}$. Thus in SCL, the $k$ negative samples are conditionally IID and independent of the anchor and positive sample given the anchor’s label.

In the typical supervised setting, the anchor, positive, and negative samples all share the same common conditional probability distribution $s(\cdot|\cdot)$ within each class given their respective labels.

We denote the CL loss in the supervised setting by $L^{(k)}_{SCL}(f)$.

Latent labels in UCL can be interpreted as indexing latent clusters. Suppose that there are $C$ latent clusters from which the anchor, positive, and the $k$ negative samples can be drawn from. Then the joint distribution of all samples and their latent labels can be described by the following equation:

where $\lambda$ is the marginal distribution of the anchor’s latent label, $r(\cdot)$ the marginal distribution of the latent labels of negative samples, and $s(x^{-}|y^{-})$ is the conditional probability distribution of any negative sample $x^{-}$ given its latent label $y^{-}$. We have used the same notation as in SCL (and slightly abused it for $r(\cdot)$) in order to make the similarities and differences between the SCL and UCL distribution structures transparent.

In the typical UCL setting, $r=\lambda$ and the conditional distribution of $x$ given its label $y$ and the conditional distribution of $x^{+}$ given its label $y$ are both $s(\cdot|y)$ which is the conditional distribution of any negative sample given its label.

We denote the CL loss in the unsupervised setting by $L^{(k)}_{UCL}(f)$.

For UCL, we assume the same mechanism for sampling the positive samples as that for SCL, but the latent label $y$ is unobserved. Further, the negative samples are generated independently of the positive pairs using the same mechanism, e.g., IID augmentations of an $x^{ref-}$ chosen independently of $x^{ref}$ (see Fig. 1). Thus, the anchor, positive, and negative samples will all have the same marginal distribution given by $\sum_{i=1}^{C}\lambda_{i}s(\cdot|i)$.

Hard-negative sampling aims to generate negative samples whose representations are “more aligned” with that of the anchor (making them harder to distinguish from the anchor) compared to a given reference negative sampling distribution (whether unsupervised or supervised). We consider a very general class of “hardening” mechanisms that include several classical approaches as special cases. To this end, we define a hardening function as follows.

As an example, $\eta(t_{1:k}):=\prod_{i=1}^{k}e^{\beta t_{i}}$, $\beta>0$, is an exponential tilting hardening function employed in [25] and [10].

Hard-negative SCL (HSCL): From (2) it follows that in SCL, $p(x^{-}|x,x^{+},y)=p(x^{-}|y)=\sum_{y^{-}\in{\mathcal{Y}}}r(y^{-}|y)\,s(x^{-}|y^{-})=:p^{-}_{SCL}(x^{-}|y)$ is the reference negative sampling distribution for one negative sample and $p(x^{-}_{1:k}|x,x^{+},y)=p(x^{-}_{1:k}|y)=\prod_{i=1}^{k}p^{-}_{SCL}(x^{-}_{i}|y)$ is the reference negative sampling distribution for $k$ negative samples. Let $\eta$ be a hardening function such that for all $x\in{\mathcal{X}}$ and all $y\in{\mathcal{Y}}$,

Then we define the $\eta$-harder negative sampling distribution for SCL as follows.

Observe that negative samples which are more aligned with the anchor in the representation space, i.e., $f(x)^{\top}f(x^{-}_{i})$ is large, are sampled relatively more often in $p^{-}_{HSCL}$ than in the reference $p^{-}_{SCL}$ because $\eta$ is argument-wise non-decreasing throughout ${\mathbb{R}}^{k}$.

In HSCL, $x^{-}_{1:k}$ are conditionally independent of $x^{+}$ given $x$ and $y$ but they are not conditionally independent of $x$ given $y$ (unlike in SCL). Moreover, $x^{-}_{1:k}$ may not be conditionally IID given $(x,y)$ if the hardening function is not (multiplicatively) separable. We also note that unlike in SCL, $p^{-}_{HSCL}$ depends on the representation function $f$.

We denote the joint probability distribution of all samples and their labels in the hard-negative SCL setting by $p_{HSCL}$ and the corresponding CL loss by $L^{(k)}_{HSCL}(f)$.

Hard-negative UCL (HUCL): From (4) it follows that in UCL, $p(x^{-}|x,x^{+},y)=p(x^{-})=\sum_{y^{-}\in{\mathcal{Y}}}r(y^{-})\,s(x^{-}|y^{-})=:p^{-}_{UCL}(x^{-})$ is the reference negative sampling distribution for one negative sample and $p(x^{-}_{1:k}|x,x^{+},y)=p(x^{-}_{1:k})=\prod_{i=1}^{k}p^{-}_{UCL}(x^{-}_{i})$ is the reference negative sampling distribution for $k$ negative samples. Let $\eta$ be a hardening function such that for all $x\in{\mathcal{X}}$,

Then we define the $\eta$-harder negative sampling probability distribution for UCL as follows.

Again, observe that negative samples which are more aligned with the anchor in representation space, i.e., $f(x)^{\top}f(x^{-}_{i})$ is large, are sampled relatively more often in $p^{-}_{HUCL}$ than in the reference $p^{-}_{UCL}$ because $\eta$ is argument-wise non-decreasing throughout ${\mathbb{R}}^{k}$.

In HUCL, $x^{-}_{1:k}$ are conditionally independent of $x^{+}$ given $x$, but they are not independent of $x$ (unlike in UCL). Moreover, $x^{-}_{1:k}$ may not be conditionally IID given $x$ if the hardening function is not (multiplicatively) separable. We also note that unlike in UCL, $p^{-}_{HUCL}$ depends on the representation function $f$.

We denote the joint probability distribution of all samples and their (latent labels) in the hard-negative UCL setting by $p_{HUCL}$ and the corresponding CL loss by $L^{(k)}_{HUCL}(f)$.

We note that convexity of $\psi_{k}$ is not needed in Theorem 1. The proof of Theorem 1 is based on the generalized (multivariate) association inequality due to Harris, Theorem 2.15 [3] and its corollary which are stated below.

For clarity, we provide a detailed proof of the inequality $L^{(k)}_{HSCL}(f)\geq L^{(k)}_{SCL}(f)$. The detailed proof of the inequality $L^{(k)}_{HUCL}(f)\geq L^{(k)}_{UCL}(f)$ parallels that for the (more intricate) supervised setting and is omitted.

where (8) follows from (5) which defines $p^{-}_{HSCL}$, (9) follows from the application of the Harris-inequality (see Lemma 1) to the inner expectation where $x$ and $x^{+}$ are held fixed, and (10) follows from the definition of $\gamma(x,y,f)$ in (6). ∎

Consider the SCL model with anchor, positive, and $k$ negative samples generated as described in Sec. 3.1. Within this setting, we have the following lower bound for the SCL loss and conditions for equality.

$(i)$ and $(ii)$ Proof that additional condition (13) implies zero-sum and unit-norm class means: Inequality (21) together with condition (13) implies that

Thus $\|\sum_{j\in{\mathcal{Y}}}\mu_{y}\|^{2}=0$ and for all $j\in{\mathcal{Y}}$, $\|\mu_{j}\|^{2}=1$.

$(iii)$ Let $M:=[\mu_{1},\ldots,\mu_{C}]\in{\mathbb{R}}^{d_{\mathcal{Z}}\times C}$. Then from (13) and $(ii)$, the gram matrix $M^{\top}M=\tfrac{C}{C-1}I_{C}-\tfrac{1}{C-1}\mathbbm{1}_{C}\mathbbm{1}_{C}^{\top}$ where $I_{C}$ is the $C\times C$ identity matrix and $\mathbbm{1}_{C}$ is the $C\times 1$ column vector of all ones. From this it follows that $M^{\top}M$ has one eigenvalue of zero corresponding to eigenvector $\mathbbm{1}_{C}$ and $C-1$ eigenvalues all equal to $\tfrac{C}{C-1}$ corresponding to $(C-1)$ orthogonal eigenvectors spanning the orthogonal complement of $\mathbbm{1}_{C}$. Thus, $M$ has $C-1$ nonzero singular values all equal to $\sqrt{\frac{C}{C-1}}$ and a rank equal to $C-1\leq d_{\mathcal{Z}}$.

$(iv)$ Proof that additional condition (13) implies zero within-class variance: We just proved that additional condition (13) together with the unit-ball representation constraint implies unit-norm class means. This, together with (21) implies that for all $y\in{\mathcal{Y}}$,

This implies that we have equality in Jensen’s inequality, which can occur iff with probability one given $y$, we have $f(x)=\mu_{y}$ (since $\|\cdot\|^{2}$ is strictly convex). Since the anchor, positive and negative samples all have a common conditional probability distribution $s(\cdot|\cdot)$ within each class given their respective labels, it follows that for all $j\in{\mathcal{Y}}$ and all $i=1:k$, $\mathrm{Pr}(f(x)=\mu_{j}|y=j)=\mathrm{Pr}(f(x^{+})=\mu_{j}|y=j)=\mathrm{Pr}(f(x^{-}_{i})=\mu_{j}|y^{-}_{i}=j)=1$. Moreover, since the anchor and positive samples have the same label, for all $j\in{\mathcal{Y}}$, with probability one given $y=j$, we have $f(x)=f(x^{+})=\mu_{j}$.

Proof that additional condition (13) is sufficient for equality to hold in (11): From the proofs of $(i),(ii)$, and $(iv)$ above, if additional condition (13) holds, then we showed that with probability one given $y=j$ we have $f(x)=f(x^{+})=\mu_{j}$ (see the para below (22)). This equality of $f(x)$ and $f(x^{+})$ is a conditional equality given the class. Since this is true for all classes, it implies equality (with probability one) of $f(x)$ and $f(x^{+})$ without conditioning on the class:

From (23) we get

since $f(x)=f(x^{+})=\mu_{j}$ with probability one given $y=j$ and $\|\mu_{j}\|^{2}=1$. Equality in (14) then follows from (23) and (24). Moreover, due to zero within-class variance we will have

and then we will have equality in (15) and (19). Therefore additional condition (13) is a sufficient condition for equality to hold in (11).

$(v)$ Proof that support sets of $s(\cdot|y),y\in{\mathcal{Y}}$ are disjoint: From part $(iv)$, all samples belonging to the support set of $s(\cdot|y),y\in{\mathcal{Y}}$, are mapped to $\mu_{y}$ by $f$. From part $(ii)$ and condition (13), distinct labels have distinct representation means: for all $y,y^{\prime}\in{\mathcal{Y}}$, if $y^{\prime}\neq y$, then $\mu_{y}\neq\mu_{y^{\prime}}$. Therefore the support sets of $s(\cdot|y)$ for all $y\in{\mathcal{Y}}$ must be disjoint. Since the anchor, positive, and negative samples all share a common conditional probability distribution $s(\cdot|\cdot)$ and the same marginal label distribution $\lambda$, it follows that they share a common conditional distribution of label given sample (labeling function). Since the support sets of $s(\cdot|y)$ for all $y\in{\mathcal{Y}}$ are disjoint, the labeling function is deterministic and is defined by the support set to which a sample belongs.

$(vi)$ Proof of equality of HSCL and SCL losses under additional condition (13): under the equal inner-product class means condition, with probability one $f(x)^{\top}f(x^{-}_{i})=\tfrac{-1}{C-1}$ simultaneously for all $i=1:k$ and $\eta(f(x)^{\top}f(x^{-}_{1}),\ldots,f(x)^{\top}f(x^{-}_{k}))=\eta(\tfrac{-1}{C-1},\ldots,\tfrac{-1}{C-1})$, a constant. Consequently, for all $x,y$ and the given $f$, we must have $\gamma(x,y,f)=\eta(\tfrac{-1}{C-1},\ldots,\tfrac{-1}{C-1})$ which would imply that (see Equation 5) $p^{-}_{HSCL}(x^{-}_{1:k}|x,x^{+},y,f)=\prod_{i=1}^{k}p^{-}_{SCL}(x^{-}_{i}|y)$ and therefore $L^{(k)}_{HSCL}(f)=L^{(k)}_{SCL}(f)=\psi_{k}\big{(}\tfrac{-C}{(C-1)},\ldots,\tfrac{-C}{(C-1)}\big{)}$ where the last equality is because additional condition (13) is sufficient for equality to hold in (11).