Signal-domain representation of symbolic music for learning embedding spaces

Recent advances in machine learning and deep neural networks have produced extremely promising results. Notably, the fields of computer vision and Natural Language Processing (NLP) have been largely reshaped by these novel approaches Mikolov, Chen\BCBL \BOthers. (\APACyear2013); Bengio (\APACyear2009). Based on these ideas, several models have also been proposed to enhance musical data analysis Park \BBA Lee (\APACyear2015). One of the major reasons behind these successes was the development of efficient embedding spaces for symbolic data Mikolov, Sutskever\BCBL \BOthers. (\APACyear2013); Pennington \BOthers. (\APACyear2014). As these spaces encode semantic relationships between input data as metric properties, they allow to decompose a complex task into two separate and simpler sub-problems. First, a system learns an embedding that disentangles underlying semantic properties of a given type of data, which provides a continuous representation space. Then, another model can learn more advanced tasks based on this adequately organized space. Multiple attempts to mimic NLP embeddings for music have been proposed, based on the premises that they both target symbolic data Huang \BOthers. (\APACyear2016); Madjiheurem \BOthers. (\APACyear2016); Bretan \BOthers. (\APACyear2017). However, these attempts only achieved small enhancements for musical generation and analysis tasks. This lack of adequate embeddings for symbolic music stems from the large discrepancies between the properties of musical and textual data. Indeed, the music vocabulary is smaller, highly repeated, and occurs in a wider variety of contexts in which the semantic properties of each element will be very different. Hence, it is not relevant to encode musical symbols as co-activation matrices Pennington \BOthers. (\APACyear2014) or one-hot vectors Mikolov, Sutskever\BCBL \BOthers. (\APACyear2013) as in the NLP field.

Recently, the success of Variational Auto-Encoder (VAEs) in symbolic music have been demonstrated in the MusicVAE model Roberts \BOthers. (\APACyear2018) which has provided interesting generation results. In addition to a new recurrent architecture, the novelty of this approach comes from the use of an efficient representation of input melodies. Nevertheless, as most recent proposals, this representation only takes into account monophonic melodies, which drastically reduces the scope of its application. Indeed, the musical composition process usually features intricate polyphony, rather than simply stacking monophonic voices together. Motivated by this challenge, we propose in this paper a novel method, which allows to represent any symbolic score as a minimal audio signal. Different MIDI pitches are mapped to prime frequencies and summed across time resulting in a simplified waveform. The process being perfectly invertible, we can retrieve the original score without losing any information. We show that this novel representation (named signal-like) is able to outperform previous propositions on polyphonic data, by comparing it to previous symbolic music representations, namely, the piano-roll, the MIDI-like Oore \BOthers. (\APACyear2018) and the NoteTuple representation Hawthorne \BOthers. (\APACyear2018). To do so, we compute the correlation between the learned embedding and music theory properties, allowing to quantify how different spaces are organized along musical concepts.

In machine learning applied to music, the audio signal information offers several desirable properties. Indeed, it naturally contains polyphonic information as a decomposable sum. Moreover, there has been some large successes in using raw signal for learning music generation tasks Oord \BOthers. (\APACyear2016). In this paper, we show that relying on a compact signal-like representation for polyphonic symbolic music can lead to large enhancements in tasks related to learning embedding. Hence, we aim to transform any given piano-roll as a spectrogram generating the most compact signal representation possible. To do so, we first map each MIDI pitch to prime numbers, starting from 43 (for MIDI pitch 0) to 2063 (for MIDI pitch 127) and removing consecutive numbers with a gap smaller than 3 (in order to not obtain pseudo frequencies that are too close together). In this representation, harmonic relationships may create phase effects detrimental to the inversion. Indeed, as the frequencies are arbitrary chosen and each represent a single MIDI pitch, the emergence of new harmonics during the inversion process would lead to the addition of notes which are not present in the original score. The use of prime numbers allow to mitigate this undesirable effect. Note that this also restrains the maximum frequency to a rather small value, allowing for a compact resulting signal. Then, we add an artificial phase, by setting imaginary parts to $1$ that create a complex matrix. Doing so, we can apply the inverse Short-Time Fourier Transform (STFT) to the resulting matrix and obtain a compact signal-like representation of the score. In our experiments, we use a window size of 2048 with an hop size equal to $1$. Note that this whole process is invertible, as depicted in Figure  2.

In order to precisely evaluate how embedding spaces are able to capture musical theory principles, we design a principled generative approach. This allows to obtain large sets of evaluation data with controlled properties, while following known music theory rules. For our experiments, we produce a dataset composed by bars of synthetic chorales (see the support page for details).¹¹1https://github.com/MagiCzOOz/signallike-embedding We obtain a set of 21966 realisations from 370 different skeletons where the links between each realisation and its corresponding skeleton have been kept. In this paper, this corpus is used only for evaluation purpose. Hence, none of these data are used during the training.

Based on this corpus, we can analyze different aspects of the organization of learned embedding spaces. First, we can analyze the behavior of different spaces based on tonality. An adequate clustering of the different tonalities across the space would prove its capacity to encode this high level musical information. Second, we can compute the distance between a realisation and its corresponding skeleton based on the number of non-harmonic tones. In this case, if the space is well-organized, we expect these distances to evolve linearly with the number of additional tones. Third, we can compute the distance between a realisation and its corresponding skeleton based on the type of non-harmonic tones. A link between these distances and the note type would show that the embedding space can handle advanced musical concepts.

In order to compare the efficiency of piano-roll, MIDI-like, NoteTuple and Signal-like representations, we evaluate them with the same common learning approach. Based on the success of the MusicVAE model Roberts \BOthers. (\APACyear2018), we rely on this architecture for our experiments. We use a recurrent encoder with a two-layers bidirectional LSTM of 1024 units. For the decoder, we rely on a hierarchical recurrent architecture with a two-layer unidirectional LSTM with a hidden size of 1024 for both the conductor and decoder. We use a latent size of 256 which largely reduces the input dimensionality while keeping enough information for the reconstruction.

As discussed previously, we train the models on the JSB chorales dataset. We use data augmentation based on transposition as proposed in Hadjeres \BOthers. (\APACyear2017), ensuring that transposed chorales are still correct from a music theory standpoint. This leads to an extended set of 2418 training and 549 testing chorales. As the model takes musical bars as input, we split the MIDI files into separate bars and compute different representations on each bar. We filter bars to have a maximum of 64 events, retaining more than $95\%$ of all the data. After these operations, we obtain 36801 training and 8850 testing bars. Regarding the MIDI-like representation, we introduce a dummy event as padding to obtain a constant-size representation. We use a similar approach for the NoteTuple representation by taking 16 tuples by bar and padding with empty tuples.

All the models are trained with the ADAM optimizer Kingma \BBA Ba (\APACyear2014) with an initial learning rate of $10^{-4}$ and a batch size of $16$. In order to train the piano-roll and signal-like representation, we rely on a Mean Squared Error (MSE) loss. Since the MIDI-like and NoteTuple representation are categorical, we rather rely on a cross-entropy loss.

In this paper, we have proposed a new representation for symbolic music and evaluated it through the learning of musical embedding spaces. We have shown that our signal-like representation have improved the quality of the learned spaces for both the reconstruction quality and the organization of the underlying spaces. In future work, we are interested in optimizing the model for our representation to understand various musical qualities. Furthermore, in addition to the pure random latent generation, several creative applications could be explored by relying on these symbolic embedding spaces.