Utilizing Expert Features for Contrastive Learning
of Time-Series Representations

A neural network encoder (or simply encoder) maps samples from the input domain $x\in{\mathbb{R}}^{c\times T}$, where $c$ denotes the number of input channels and $T$ the number of time steps of each sample¹¹1For notational simplicity we consider here time-series of fixed length, while our formalism is straightforward to extend to varying-length time-series., to an embedding or representation $E_{\theta}(x)\in{\mathbb{R}}^{e}$. The encoder’s parameters (weights) are $\theta\in{\mathbb{R}}^{P}$, which are updated in supervised learning by minimizing a loss-function ${\mathcal{L}}(E_{\theta}(X),Y)$ on the training set $(X,Y)=\{(x_{1},y_{1}),\dots,(x_{N},y_{N})\}$, where the labels $y_{i}$ are often discrete classes $y\in\{1,\dots,C\}$.

In contrastive representation learning the loss function is chosen in such a way that representations of samples with the same class label are pulled closer to each other w.r.t. the Euclidean norm, while samples with different class labels are pushed away from each other. This can be achieved by minimizing a contrastive loss function, e.g. the “triplet-Loss” (Chechik et al., 2010) or the “NCE-Loss” (Gutmann & Hyvärinen, 2010). Another prominent contrastive loss is the so called pair-loss function (Hadsell et al., 2006; Le-Khac et al., 2020):

where $s_{ij}$ is the discrete similarity measure defined by $s_{ij}:=\delta_{y_{i}=y_{j}}$, $\Delta\in\mathbb{R_{+}}$ is a hyperparameter, and $D_{ij}:=\|E(x_{i})-E(x_{j})\|_{2}$ denotes the Euclidean distance. While the first term in the sum is responsible for pulling closer together the representations of similar points, i.e. pairs with the same labels, the second term aims to push representations of dissimilar pairs to a distance of at least $\Delta$. So far, this is the supervised setting of CL, where all labels are provided. Next, we describe the semi-supervised setting, where labels are provided only for a fraction of the dataset, and the unsupervised setting as the extreme case with no labels available.

In the unsupervised setting, most CL algorithms make use of transformations $\{m_{1},\dots,m_{V}\}$ with $m_{v}:{\mathbb{R}}^{c\times T}\to{\mathbb{R}}^{c\times T}$, which leave the class label invariant. These transformations can then be used to create so-called “views” $x_{i}^{v}:=m_{v}(x^{0}_{i})$ of a data sample $x^{0}_{i}:=x_{i}$. By assumption, all $x_{i}^{v}$ have the same $y$-label as the original $x^{0}_{i}$ (even if this label is unknown) and therefore discrete similarity measure $s_{i,i}^{0,v}=1$. A number of other randomly selected data samples $x^{0}_{j}\ (j\neq i)$, e.g. the other samples in the batch, are then considered negative samples with similarity $s_{i,j}^{0,0}=0$, such that one can write down a CL loss like (Eq. 1) using these transformations. The expert knowledge used for unsupervised learning is thus the transformations $m_{v}$ leaving class labels invariant. The training pushes the encoder towards being invariant w.r.t. the transformations $m_{v}$. In the vision domain, a great number of sensible transformations $m_{v}$ are known (cropping, rotation, translation, etc.).

In the time-series domain, however, finding invariant transformations is less intuitive, and one can easily be led astray (Iwana & Uchida, 2021). But since many time-series datasets come from fields like industry or medicine in particular, expert features are often readily available from domain experts. These expert features may be discrete or (more usually) continuous, and could e.g. be calculated from the input time-series by an expert mapping $f:{\mathbb{R}}^{c\times T}\to{\mathbb{R}}^{d}$, i.e. $f_{i}:=f(x_{i})$, and/or be available from additional measurement sensors on the training data. In our work we assume to be given a set of expert features²²2Note, we do not assume the expert mapping $f$ to be given. It is unavailable e.g. in the HAR dataset (Sec. 4). $F=\{f_{1},\dots,f_{N}\}$ for the training inputs $X$. The full dataset to train our embedding is thus $(X,F,Y)$, where $Y$ may contain label information for any percentage of input data points, ranging between the supervised and the fully unsupervised setting.

We aim to employ the given expert features $F$ in a way to learn a good representation of the input time-series. To see how to best utilize $F$, we first discuss what properties a useful representation should have w.r.t. the expert features. We follow the general ideal behind CL, which is to push points with the same (resp. different) labels together (resp. apart). For continuous-valued expert features $f_{i}\in\mathbb{R}^{d}$, this motivates an encoder $E$ with the following properties:

• (P1)

  If expert features of two points are similar, i.e. $\|f_{i}-f_{j}\|_{2}$ is small, then $\|E(x_{i})-E(x_{j})\|_{2}$ should also be small, i.e. the corresponding representations be similar.

• (P2)

  If $\|f_{i}-f_{j}\|_{2}$ is large, then $\|E(x_{i})-E(x_{j})\|_{2}$ should also be large.

Here, $x_{i},x_{j}$ are any two samples from $X$, and $f_{i},f_{j}$ their given expert features. While both properties can be important for predictive models, (P1) is especially important for outlier detection, while (P2) is important for identifying similar samples and (safe) active learning.

The question may arise why one cannot directly use the expert features $f$ as the representation $E(x)$, which would satisfy both properties trivially. To start, note that we do not assume the full expert mapping $f$ to be available, but merely the features $f_{i}$ for the given inputs $X$; i.e. one couldn’t evaluate $E(x_{test})$ at test inputs with such a prescription. And even if $f$ were available, downstream tasks often benefit from fine-tuning the encoding function $E=E_{\theta}$ further (Sec. 4.5), which is generally not possible or successful with the expert feature mapping $f$. Second, the mapping $f$ would not allow to freely choose the dimension $e$ for the representation space, but bind it to the feature dimension $d$. Finally, we observe in experiments that a learned $E$ allows to exceed the performance over the original features $f$ in downstream tasks even for the unsupervised setting (Tab. 2).

We formalize the properties (P1) and (P2) by defining bilipschitz representations w.r.t. the given set of expert features:

Note that we require (and are able to evaluate) the condition in Def. 1 only on the training set $X$ and not on all potential input points $x\in{\mathbb{R}}^{c\times T}$. But from this, one can derive statistical bounds on the pair-Lipschitz constant $\|f-f^{\prime}\|_{2}/\|E(x)-E(x^{\prime})\|_{2}$ for test points $x,x^{\prime}$ (App. D), even though it is generally impossible to satisfy Def. 1 on an infinite set of inputs when the feature dimension $d>e$ exceeds the representation dimension.

The larger $l_{-}$ and the smaller $l_{+}$ is in Def. 1, the better are the guarantees one can provide for (P1) and (P2), respectively. In the ideal case we have $l_{-}=l_{+}$, i.e. the Euclidean distance in the representation space is proportional to the distance of the expert features.

Having established the properties a useful representation should have, we may ask how one can obtain such a representation from the expert features $F$. A first approach might be to put the expert features into bins to arrive again at discrete class labels, and then proceed with a standard contrastive loss function such as the pair-loss. Such an approach cannot generally lead to a bilipschitz representation; in particular, it cannot provide guarantees on $l_{-}$ and $l_{+}$ due to arbitrariness in choosing the bins and the absence of a relative distance measure between different bins. A similar problem occurs with other methods that generate pseudo-labels such as SleepPriorCL (Zhang et al., 2021).

As an alternative approach, one might add a linear layer $M=M_{\phi}$ on top of the encoder $E_{\theta}$ and jointly train $\theta$ and $\phi$ to predict the given expert features, e.g. by minimizing the MSE-loss. However, such a procedure does not necessarily lead to good guarantees for the two properties (P1), (P2) we are trying to fulfill, as the following shows:

Prop. 1 shows that learning to predict the expert features does not provide any guarantees for the two properties (P1) or (P2). The proof can be found in App. C.1.

The preceding section shows that two straightforward approaches do not guarantee an encoder $E$ to have the desired properties. We thus aim to find a new objective for CL which ensures the properties (P1) and (P2) to be fulfilled, and to a good degree at that. For this, we first propose to generalize the discrete similarity measure used within Eq. 1 to continuous labels or features. We do this by defining

where the maximum can either be taken over the complete dataset $X$ or only over a subsample (batch). Other ways of defining the similarity measure are equally possible, see also the possibilities discussed in Sec. 3.6. We point out that, for discrete class labels which we assume here to be one-hot-encoded, our generalized similarity measure reduces to the discrete similarity measure $s_{ij}=\delta_{f_{i}=f_{j}}$ from Sec. 3.1. While this is not a necessary condition, it allows us to treat discrete and continuous features $f$ in a uniform manner with our generalized similarity measure. When plugging the new similarity (Eq. 2) into the original pair-loss (Eq. 1), the resulting loss function encourages the desired properties (P1) and (P2) to be fulfilled, as shown below. However, the resulting loss function has the issue of discontinuities in its derivative, similar to versions of the pair-loss (Eq. 1) for continuous features (see also App. E), which might cause instabilities during optimization. We remedy this by designing a novel version of the continuous pair-loss, which we name the quadratic contrastive loss:

where again $D_{ij}:=\|E(x_{i})-E(x_{j})\|_{2}$. ${\mathcal{L}}_{quad}$ has the same minimum as the pair-loss (Eq. 1), but possesses continuous derivatives w.r.t. $D_{ij}$ and thus $E$. Furthermore, in contrast to the procedure from Prop. 1, minimizing ${\mathcal{L}}_{quad}$ does lead to (optimal) guarantees for $l_{-}$ and $l_{+}$:

While Prop. 2 gives guarantees for the desired properties only on the training set $X$ (App. C.2), this can be boosted to statistical bounds for (P1) and (P2) on unseen test data (App. D). Such guarantees are relevant for downstream tasks such as outlier detection, searching, or (safe) active learning.

In the previous section we introduced the quadratic contrastive loss ${\mathcal{L}}_{quad}$ (Eq. 3), which puts equal weight on all pairs of datapoints. While this works well, the performance in practice often improves when higher weight is put on high-loss datapoint pairs. Such a strategy is known as “hard-negative mining” in CL (Wang & Liu, 2021) and control-theory (Busseti et al., 2016), and also improves our method (Sec. 4.6). Rather than explicitly selecting the high-loss pairs, we perform implicit hard-negative mining through a version of the softmax-function. This changes the loss to:

where $L_{ij}:=\left((1-s_{ij})\Delta-D_{ij}\right)^{2}$ and $\tau\in{\mathbb{R}}^{+}$ is a temperature hyperparameter. The following proposition demonstrates that by changing $\tau$, one can control the strength of the implicit hard-negative mining:

Thus, the amount of hard-negative mining one wants to apply can be controlled by $\tau$ (proof in App. C.3). Similar limits exist for the NCE-Loss (Wang & Liu, 2021). A gradient-level analysis of the hard-negative mining loss can be found in App. F. We refer to the method of minimizing the loss ${\mathcal{L}}_{ExpCLR}$ (Eq. 4), i.e. our quadratic contrastive loss with implicit hard-negative mining, as ExpCLR.

Here we discuss two practical variations of ${\mathcal{L}}_{ExpCLR}$ we use in our experiments. First, while we introduced a simple similarity measure in Eq. 2, in practice we use its square,

due to its consistent superior performance in experiments. Other similarity measures, such as $s_{ij}=\exp\left(-\|f_{i}-f_{j}\|^{2}_{2}/\sigma\right)$ (Kim et al., 2019) with a hyperparameter $\sigma$, would be equally possible. For an empirical comparison of the three similarity measures, see Sec. 4.6.

Further, instead of using the unnormalized Euclidean metric and following (Kim et al., 2021), we normalize the Euclidean distance with $\mu_{i}=(1/N)\sum_{j}\|E(x_{i})-E(x_{j})\|_{2}$, i.e. we use $D_{ij}:=\|E(x_{i})-E(x_{j})\|_{2}/\mu_{i}$. Alternative normalizations would be possible as well.

In the following we compare ExpCLR to several state-of-the-art methods on three real-world time-series datasets. We start by introducing the three datasets; see Tab.  1 for detailed specifications.

Human Activity Recognition (HAR): The HAR dataset (Cruciani et al., 2019) contains multi-channel sensor signals of 30 subjects, each performing one out of six possible activities. A Samsung Galaxy S2 device embeds accelerometers and gyroscopes which collected the data at a constant rate of 50Hz. In addition, the dataset already contains a 561-dimensional expert feature vector for each sample.

Sleep Stage Classification (SleepEDF): In this classification task the goal is to classify five sleep stages from single-channel EEG signals, each sampled at 100Hz. The dataset originates from (Goldberger et al., 2000; Kemp et al., 2000) and subjects are selected and preprocessed following previous studies (Eldele et al., 2021). We equip each signal with expert features computed from the time and frequency domain, as suggested and identified in (Huang et al., 2020).

MIT-BIH Atrial Fibrillation (Waveform): This dataset (Goldberger et al., 2000) contains 23 long-term ECG recordings of humans suffering from atrial fibrillation. Two ECG signals are sampled at a constant rate of 250Hz and distinguish four different classes. We utilize the expert features designed by (Goodfellow et al., 2017) specifically for the artial fibrillation classification task.

To evaluate the expert features we report their resulting linear and KNN ($k=1$) classification accuracies in Tab. 2.

Next, we briefly present our model architecture and how we train our models in each setting; more details can be found in App. B.3. To capture relevant temporal properties and to improve training stability (Bai et al., 2018), we choose as a base encoder temporal convolutional network (TCN) (Lea et al., 2017) layers in a ResNet (He et al., 2016) architecture with eight such temporal blocks. Note that ExpCLR is not restricted to this architecture. To reach a pre-defined embedding dimensionality we add a two-layer fully connected neural network on top of the ResNet base encoder to arrive at our backbone encoder network.

In our work we consider three different modes of training:

1. 1.

   The unsupervised (U) training mode, where the encoder is optimized with the respective contrastive loss function on the input data time-series $X$ and the expert features $F$ only.

2. 2.

   The supervised (S) training mode, where the encoder is trained with either a supervised contrastive loss, which uses labels $y_{i}$ instead of expert features $f_{i}$, or with the cross-entropy loss function on the input time-series $X$ and the whole or part of the labels $Y$.

3. 3.

   The semi-supervised (SS) training mode, where the encoder is first trained with the unsupervised training mode on the whole training set $(X,F)$ and then this pretrained encoder is fine-tuned with a supervised training step on some percentage of the labels $Y$.

While for hyperparameter optimization we split the training set $X$ into $80\%$ training and $20\%$ validation data, for our comparisons experiments we make use of the full training set and evaluate the representations on the test set. The number of epochs for each dataset is selected such that all algorithms are able converge. For the optimization step we used the Adam optimizer with parameters $\beta_{1}=0.9$, $\beta_{2}=0.999$ and exponential decay $\gamma=0.99$. To enable a fair comparison between ExpCLR and the competing methods, we optimize the learning rate for each method and dataset individually via a grid search and identify $\tau=1$, $\Delta=1$ (Eq. 4), embedding dimension $e=100$ and batch size of 64 as a good compromise over all datasets and algorithms. For more information on model- and loss-specific parameters, see Sec. 4.6, App. A.2 and App. B.3.

To verify the goodness of our representations, we evaluate two kinds of classifiers on the representation: Linear classifiers perform well for representations where all classes are linearly separable. Second, KNN classifiers can even perform well when classes are not linearly separable, but tend to perform worse for clusters that are not separated by a large margin; this problem is most apparent for small $k$. We thus use the performance difference of the linear and KNN ($k=1$) classifier to investigate how well different classes are mapped into individual well-separated clusters.

For the unsupervised comparison of ExpCLR to other methods, we distinguish two different groups of CL algorithms. The first group consist of algorithms, which use transformations of the input data to create positive samples. Here we compare to SimCLR (Chen et al., 2020) and further to TS-TCC (Eldele et al., 2021), that combines classical CL with time-series forecasting to learn representations. For SimCLR we tested a range of different augmentations and found scaling and dropout to work the best, while for TS-TCC we employ weak and strong augmentations. Further details can be found in App. B.4. The second group includes methods which also use expert features. SleepPriorCL (Zhang et al., 2021) and the contrastive loss used by TREBA (Sun et al., 2021) both create expert feature based pseudo-labels via some form of discretization, which are then used with a version of the supervised contrastive loss introduced by (Khosla et al., 2020). Another method in this group is introduced by (Kim et al., 2021), a state-of-the-art metric learning method. It aims to achieve the same goal as we do and try to pull similar points closer together (Kim et al., 2021), while pushing dissimilar ones further apart. We simply replace their teacher model output with our expert features to be comparable. Lastly, we also compare to the embedding learned by Expert Feature Decoding: Here, the embedding is given by the output of the penultimate layer of a network that is trained by learning to predict the expert features from the input time-series (Sun et al., 2021). During training we minimized the MSE-loss and add a projection layer to the architecture used by the other methods. Expert Feature Decoding is used as part of the TREBA-objective (Sun et al., 2021) and is discussed theoretically in Sec. 3.3.

For the supervised fine-tuning step we use the approach described in the original works or utilize the natural extension for each algorithm. For SimCLR we use supervised CL (Khosla et al., 2020) and for (Kim et al., 2021) we use the pair-Loss (Hadsell et al., 2006; Le-Khac et al., 2020), because this is the loss that it naturally reduces to for class labels. Further, as ExpCLR allows to simply replace expert features with labels in order to perform a supervised fine-tuning step or a fully supervised training, we do this.

We selected the competing methods to cover a broad range of algorithms. All methods can be considered state-of-the-art in their respective domains. We implemented all methods except for TS-TCC inside our repository.

In this section we compare the performance of ExpCLR against several state-of-the-art unsupervised representation learning methods, using a linear and a KNN ($k=1$) classifier on top of the learned embedding. We compare ExpCLR on all three datasets against SimCLR, TS-TCC, SleepPriorCL, TREBA Contrastive Loss, Expert Feature Decoding and (Kim et al., 2021). In addition, we use the randomly initialized encoder network performance (Random Init), an encoder trained with a supervised cross-entropy loss, using the full labeled dataset, and the performance on the expert features themselves as baseline comparisons. The results of the comparison are shown in Tab. 2.

The superior performance of ExpCLR can be clearly seen, as we only perform on par with SimCLR on HAR and with SleepPriorCL on Waveform w.r.t. KNN ($k=1$) accuracy. ExpCLR also shows much higher consistency across all datasets, while most other algorithms significantly underperform on at least one of the datasets. Further, we are even able to exceed the expert feature performance on at least one performance metric on all three datasets. This could indicate that ExpCLR is able to learn new additional features from the raw time-series data on top of the provided expert features. In addition, we can even surpass the supervised performance on the SleepEDF dataset. This underlines how powerful the approach of ExpCLR.

Here, we compare ExpCLR to the competing methods in the semi-supervised setting. In this setting we fine-tune a representation, learned in the unsupervised setting, using some fraction of the full labeled data. The results of the comparison for the HAR, SleepEDF and Waveform datasets for labeled data ratios of $5\%,10\%,20\%,50\%,70\%$ and $100\%$ are shown in Fig. 1 (see App. A.1 for the KNN accuracy).

ExpCLR consistently surpasses the accuracy of all competing methods across all datasets over different label percentages. Further, using only $20\%$ labeled data ExpCLR (SS) is able to outperform any competing method with any amount of labeled data, even for $100\%$, on the HAR and SleepEDF datasets. In addition, ExpCLR’s unsupervised pretraining increases the label efficiency significantly, since ExpCLR is able to sustain its performance attained on the fully labeled dataset up to a minor decrease in accuracy of less than $3.5\%$ for the lowest labeled data percentage. In contrast, the representations learned by supervised cross-entropy drop by more than $17.5\%$.

The consistent superior performance of ExpCLR across all datasets is noteworthy as it does not use any data transformations. Further, the comparison clearly shows the superior performance of our loss function compared to Kim et al. (2021). Another interesting observation is that almost all methods that make use of an unsupervised pretraining outperform supervised methods for low percentages of labeled data. This underlines the importance and effectiveness of finding good approaches that make use of unlabeled data, as ours. All numerical values of Fig. 1 incl. variances can be found in App. A.5 (see also App. A.6).

To guide the design of our ExpCLR method we conducted several ablation studies. The first one investigates the choice of similarity measure in ExpCLR and is shown in the left panel of Fig. 2. We compare both similarities (Eq. 2) and (Eq. 5) from our methods section and the Gaussian similarity (Sec. 3.6) introduced by Kim et al. (2021) in the unsupervised setting for all three real-world datasets. The results show that the quadratic similarity measure (Eq. 5) outperforms the other similarity measures, therefore justifying our choice.

The second ablation study investigates the effectiveness of the hard-negative mining strategies introduced in Sec. 3.5, which ExpCLR employs. The right panel of Fig. 2 shows the unsupervised accuracy of ExpCLR for several values of the temperature $\tau$ and also compares to ExpCLR without hard-negative mining (NHNM). Using the insights gained from Prop. 3 (a), Fig. 2 shows that stronger hard-negative mining (decreasing $\tau$) can improve the performance. Further, as shown theoretically in Prop. 3 (b), the results nicely demonstrate that the performance convergences to ExpCLR (NHNM) for large values of $\tau$. More ablation studies and a sensitivity analysis for the hyperparameter $\Delta$, the batch-size and the dimension of the embedding can be found in the appendix (see App. A.2).