UNSUPERVISED LEARNING OF DEEP FEATURES FOR MUSIC SEGMENTATION

Music segmentation refers to the task of labeling distinct segments of music in a way that is similar to a human annotation. For example the chorus, verse, intro, outro and bridge in popular music.. The boundary between such segments may be due to a number of factors, for example, a change in melody or chord progression, a change in rhythm, changes in instrumentation, dynamics, key or tempo. This task is generally evaluated with two classes of metrics. The first class, boundary detection, refers to the ability of the algorithm to locate the locations of such boundaries in time. The second class, segment labelling, refers to the labelling of segments where two segments that are disconnected in time are labelled as same or different based on their perceptual similarity [1, 2, 3].

The approach of here is to create an embedding for audio features by transforming an audio representation via a CNN into a domain that is representative of musical structure. These embeddings may be trained by employing a loss function, such as contrastive loss [26], or triplet loss [27], that observes positive (similar) or negative (dissimilar) pairs or triplets of examples, where a triplet represents an anchor and both a positive example and a negative example. The triplet loss function is often shown to result in superior performance to contrastive loss, which has been argued to be due to the its relative nature [28]. That is, the triplet loss function is designed to provide a positive gradient with respect to increasing distance between positive pairs or the anchor and positive example of a triplet, relative to the distance between negative pairs, up to a given margin. As such, it is employed in this research.

If a transformation from an input feature, $x[q,k]$ (or equivalently $x$ for notational simplicity) to an embedding space, e.g., via a CNN, is described as $f:\mathbb{R}^{K\times Q}\rightarrow\mathbb{R}^{D}$, then a Euclidean triplet loss with a given margin, $\alpha$, and a mini-batch of training data consisting of the set $\mathcal{T}=\left\{x^{c}_{a},x^{c}_{p},x^{c}_{n}\right\}_{c=0}^{c=C-1}$ may be described as,

$$\mathcal{L}(\mathcal{T})=\sum_{c=0}^{C-1}\left[\left\lVert f\left(x^{c}_{a}\right)-f\left(x^{c}_{p}\right)\right\rVert^{2}_{2}-\left\lVert f\left(x^{c}_{a}\right)-f\left(x^{c}_{n}\right)\right\rVert^{2}_{2}+\alpha\right]_{+},$$

(1)

where subscript $a$, $p$ and $n$ represent an anchor, positive and negative input example respectively.

Ideally, in the context of music segmentation positive examples will be composed of audio feature representations from the same music segment as the anchor, while negative examples will represent audio features from distinct music segments. Thus, once trained, input features could be transformed via that CNN into a space where distinct clusters with respect to a Euclidean distance metric would represent distinct music segments. In an unsupervised context with no prior information about the music segmentation, exact selection of such examples is not possible, however strategies are described in Section 3 that may improve this selection.

The properties that are significant in forming clusters representing music segments in the embedded space are learned from the data at the CNN input. These embedded features may be representative of any of the musical qualities mentioned in Section 1, provided that these qualities are observable from the input features. Here, a constant-Q transform (CQT) [29] is used due to its ability to accurately represent transient and harmonic audio qualities, its translation invariant representation of harmonic structures, and the promising results observed for this feature specifically in [30].

A time window of CQT data providing $K$ frequency bins across $Q$ time windows forms a time frequency representation that may compose any of $x^{c}_{a}$, $x^{c}_{p}$ or $x^{c}_{n}$. In this work it is found significantly advantageous to synchronize the CQT analysis windows with the beat of the music. That is, if a beat at index $i$ occurs at time $b_{i}$, then $R$ CQT windows are analyzed centered at times $b_{i}+r(b_{i+1}-b_{i})/R$ for integers, $r\in\left\{0..R-1\right\}$. These beat synchronized CQT representations are then aggregated across $B$ beats providing $Q=BR$ time indices in the representation $x[q,k]$ at the input to the CNN.

Motivated by the unreasonable effectiveness of data in deep learning [31], methods are proposed here that create noisy positive / negative examples exploiting the facts that a) musical segments form contiguous regions in a song’s timeline, and b) each distinct musical segment label typically occurs for the minority of a song’s timeline. Specifically, it is proposed to use the time proximity information implicit in a song’s features - sampling features that occur close together or at a minimum distance apart for positive and negative examples respectively. That is, an anchor beat index, $i_{a}$, is selected via a uniform sampling of the beat indices in a given song, $\left\{0..L-1\right\}$. Thereafter, a positive beat index, $i_{p}$, is chosen uniformly sampled from beat indices $\left\{\max(i_{a}-\delta_{p},0)..\min(i_{a}+\delta_{p},L-1)\right\}$, and a negative beat index $i_{n}$ is chosen as uniformly sampled across two regions, $\left\{\max(i_{a}-\delta_{n,max},0)..\max(i_{a}-\delta_{n,min},0)\right\}$, and $\left\{\min(i_{a}+\delta_{n,min},L-1)..\min(i_{a}+\delta_{n,max},L-1)\right\}$. An example of the positive and negative sampling distributions is shown in Fig. 1. Intuitively, this results in an embedding in which clusters represent features that frequently occur close together. This is useful for the problem of the structural segmentation of music as segment boundaries are typically infrequent enough to be described as rare events, at least in comparison to lengths of the segments themselves.

It is interesting to consider the rates of false positives and negatives that result from the aforementioned sampling paradigm. Upon selection of $i_{a}$ it may be denoted to fall somewhere in the $n$th musical segment of class $s$ and length $l_{s,n}$. If $\delta_{p}<l_{sep}$, where $l_{sep}$ is the minimum number of indices between $s$ and any identically labeled segment, then the probability of the positive example being selected from a distinct segment, i.e., a false positive, is,

$$P(\mathrm{FP}|l_{s,n};\delta_{p})=\begin{cases}\frac{2\delta_{p}-l_{s,n}}{l_{s,n}}&l_{s,n}\leq\delta_{p}\\ \frac{\delta_{p}^{2}}{2l_{s,n}^{2}}-\frac{3\delta_{p}}{4l_{s,n}}+\frac{1}{2}&\delta_{p}<l_{s,n}<2\delta_{p}\\ \frac{\delta_{p}}{4l_{s,n}}&2\delta_{p}\leq l_{s,n}\end{cases}.$$

Similarly, if $\delta_{n,min}<l_{sep}$ and $\delta_{n,max}\geq L$, then the probabillity of a false negative is simply $\frac{\sum_{m}l_{s,m}}{L}-(1-P(\mathrm{FP}|l_{s,n};\delta_{n,min}))$.

In practice the aforementioned assumptions are realistic in many scenarios, but do not hold under all conditions. An empirical analysis of the rate of false positives is shown in Fig. 2. It is clear that with an increasing $\delta_{p}$, an increasing rate of false positives is observed, although it is important to note that smaller $\delta_{p}$ restricts the maximum observed time separation between the anchor and positive example. With musical phrases often lasting 16 beats, it is reasonable to set $\delta_{p}\geq 16$ to discourage distinct clustering of features within a single phrase. An empirical analysis false negatives is shown in Fig. 3. There it can be seen that for the datasets shown $\delta_{n,min}>28$ and $\delta_{n,max}>116$ result in relatively low false negative rates.

While structural annotations are not available under the scope of unsupervised learning, it is interesting to ask whether analysis of signal features may be employed to decrease the false-negative or false-positive rate. It was shown in [7] that 2D Fourier magnitude coefficients of HPCPs can be a useful feature in segment labeling. It is proposed here to use a similar feature to inform the sampling of positive and negative examples. That is, for every selected $i_{a}$, a comparison of the log-amplitude of the 2D Fourier transform of the log-amplitude of 8 beat long CQT segments is considered. The Euclidean distances between these CQT segments centered at two times before $i_{a}$, i.e., $i_{a}-4$ and $i_{a}-16$, and two times after $i_{a}$, i.e., $i_{a}+4$ and $i_{a}+16$ are considered. The side (“before” where $i<i_{a}$ or “after” where $i>i_{a}$) with minimum Euclidean distance is then assumed to be more likely in the same musical segment as $i_{a}$ and is chosen from which to sample $i_{p}$ while the opposite side may be chosen from which to sample $i_{n}$, in addition to the constraints, $\delta_{p}$, $\delta_{n,min}$ and $\delta_{n,max}$ above. The changes in false negative and false positive rates employing this sampling paradigm can be seen in Figures 2 and 3, respectively, where some improvement is observed.

To evaluate the effectiveness of the music embeddings described in this work, the problem of music segment boundary detection is considered. Perhaps the simplest and most well known method for music boundary detection is that of Foote [4]. While this has been surpassed in performance by several methods, e.g., [6, 9], its simplicity makes it effective in demonstrating the utility of the audio embeddings proposed here, as will be seen in the results in Section 5.

The SSM of the proposed features is computed as,

$$S[i,j]=\left\lVert f\left(x_{i}[q,k]\right)-f\left(x_{j}[q,k]\right)\right\rVert^{2}_{2}$$

(2)

where $x_{i}[q,k]$ and $x_{j}[q,k]$ correspond to beat synchronous CQT segments centered at beats $i$ and $j$, respectively.

Note that this work endeavors to create features that are close with respect to Euclidean distance at any point within a music segment. If successful, the SSM of embedded features typically contains block structures as opposed to the path structures typically representative of repetition. These structures are evident in Fig. 4. It was found beneficial in practice to perform median filtering on $S[i,j]$ to produce $\bar{S}[i,j]$ which reduces noise in the distances between embedded features while maintaining the aforementioned structures.

To detect segment boundaries in $\bar{S}[i,j]$, a checkerboard kernel,

$$g[i,j]=\mathrm{sgn}(i)\mathrm{sgn}(j)e^{\frac{-(i-j)^{2}}{\sigma}}$$

(3)

for $-\kappa\leq i,j\leq\kappa$ is convolved along the diagonal of the SSM,

$$\eta[\nu]=\sum_{i=-\kappa}^{\kappa}\sum_{j=-\kappa}^{\kappa}\bar{S}[\nu+i,\nu+j]g[\nu+i,\nu+j]$$

(4)

producing the novelty function $\eta[\nu]$. Note that it was found advantageous to set $g[i,j]$ to $0$ where $|i|\leq 1$ or $|j|\leq 1$. Finally, boundaries are detected as peaks in the novelty function. Specifically, peaks at which the peak-to-mean ratio exceeds a given threshold, $\tau$, are selected as segment boundaries, i.e., where,

$$\frac{(2T+1)\eta[\nu]}{\sum_{t=-T}^{T}\eta[\nu+t]}>\tau.$$

(5)

For evaluation, two datasets are considered: the BeatlesTUT dataset consisting of 174 hand annotated tracks from The Beatles catalogue [20], and the internet archive portion of the SALAMI dataset (SALAMI-IA) consisting of 375 hand annotated recordings [16]. The former dataset is perhaps the most widely evaluated in the music segmentation literature. The latter is employed here for two reasons, firstly the complete SALAMI dataset audio is not available to the author due to copyright restrictions, and secondly, the SALAMI-IA dataset is particularly interesting as it consists primarily of live recordings with many imperfections. It provides a dataset that is indicative of segmentation performance when there are mistakes either by musicians or recording engineers, resulting in imperfect repetitions and distorted or noisy audio in many cases.

For comparison, two baseline algorithms are included in the results as specified and measured in [30], note that these algorithms too included beat-synchronized features. Firstly, the method of [4] is included as it most closely mirrors the algorithm of Section 4. Secondly, the algorithm of [6] is widely evaluated as having the best performance with respect to unsupervised boundary detection in music segmentation. CQT features have been evaluated to provide superior performance in [30], and are used at the input to all algorithms, proposed and benchmark, in the results presented here. For the proposed algorithms, three methods are investigated, ”Unsynchronized”, employing a constant hop size of 3.6 ms between successive CQT windows; ”Beat-Synchronized”, employing 128 CQT windows centered at times linearly interpolated between successive beat markers (estimated during training and inference by an algorithm similar to [32]); and ”Biased”, employing the same features as the ”Beat-Synchronized” approach, but using the 2D Fourier Transform comparison sampling described in Section 3. The CQT features use a minimum frequency of 40 Hz, 12 bins per octave and 6 octaves.

The structure of CNNs employed in this paper is chosen to be simple enough so that it may be trained on most modern high performance GPUs. The architecture described in Fig. 5 was found to be effective and is employed in the experiments here. During training, mini-batches of size $C=96$ consisting of 16 triplets from each of 6 randomly selected tracks were formed in real-time by randomly selecting from 28,345 songs, excluding all songs in the SALAMI-IA and BeatlesTUT dataset. 256 min-batches form 1 epoch, and training took place over 240 epochs taking approximately 8 hours. The triplet margin was set to $\alpha=0.1$. Despite the observed error rates in Figures 2 and 3, [27, 28] argue that it is the difficulty of separation between examples that is important, and so not all false positives and negatives are equal. In practice it was found that $\delta_{p}=16$, $\delta_{n,min}=1$ and $\delta_{n,max}=96$ provide optimal results.

For boundary detection, embedded features were observed at every beat for beat-synchronized approaches, or once every 0.2484 seconds for ”Unsynchronized” (this is approximately twice per beat at a typical 120 BPM to ensure this method is not disadvantaged by any shortcoming in time resolution). SSM representations were median filtered using an 8x8 window. The checkerboard kernel was configured with $\sigma=18.5$ and $\kappa=40$ and for peak picking, the crest factor window size $T=10$ and threshold $\tau=1.35$ was employed.

For evaluation, the trimmed F-measure, precision and recall of the boundary detection hit rate at the 3 second tolerance level are employed [1]. The evaluation of the proposed and reference algorithms for the BeatlesTUT and SALAMI-IA dataset are shown in Table 1 and Table 2, respectively. It should be noted here that for the proposed algorithms, any selection of parameters was performed by observing results on the BeatlesTUT dataset only, and so the SALAMI-IA dataset displays the boundary detection algorithm’s ability to generalize to unseen data.

It is interesting to see that simply by employing the proposed deep features in an algorithm similar to that of Foote [4], such a method becomes competitive with the state of the art in unsupervised music segmentation. Furthermore, on the SALAMI-IA dataset, a significant performance improvement over the state of the art is observed without any additional parameter adjustment. This result might be postulated to be due to the poor quality of audio / music data in this portion of the SALAMI dataset. Because the algorithm of [6] is designed to detect changes in repetition patterns, when these patterns become imperfect, or corrupted by noise, a performance drop might be expected. In the proposed embedding, clustering of features is performed simply based on the time proximity of features observed from the training data, which contains many of the aforementioned imperfections providing some robustness.

In this work, methods for the unsupervised training of music embeddings were investigated with respect to their utility in the task of music segmentation. In particular, it was shown that by employing such embeddings in a traditional music segmentation algorithm, the performance of this algorithm can obtain state of the art performance. It was found that a common musical feature, rhythm, may be exploited in beat-synchronized sampling (and in the 2D Fourier Transform comparitive sampling of Section 3) to further improve performance.