Learning interpretable representation for controllable polyphonic music generation

Most existing learning-based methods regard controlled music generation a conditional estimation problem. That is, to model $p(\text{music}|\text{control})$, in which both music and control are usually time-series features. Another approach that is closely related to conditional estimation is to first learn the joint distribution $p(\text{music},\text{control})$ and later on force the value of control during the generation process.

The above two methods have been used in various tasks, including generating chords based on the melody [9], creating the melody based on the chords [10, 11], completing the counterparts or accompaniment based on the melody or chord [12, 13, 14, 15, 16, 3], and producing the audio waveform based on timbre features [17, 18].

However, many abstract music factors, such as texture and melody contour, could hardly be explicitly coded by labels. Even if such labels are provided, the control still does not allow continuous manipulation, such as sampling and interpolation. Consequently, it remains a challenging task to control music by more abstract factors without complex heuristics [19].

Learning disentangled representations is an ideal solution to the problem above, since: 1) representation learning embeds discrete music and control sequences into a continuous latent space, and 2) disentanglement techniques can further decompose the latent space into interpretable subparts that correspond to abstract music factors. Recent studies show that VAEs [20, 21] are in general an effective framework to learn the representations of discrete music sequences, and the key to a successful disentanglement is to incorporate proper inductive biases into the representation learning models [22].

Under a VAE framework, an inductive bias can be realized in various forms, including constraining the encoder [23, 24, 25], constraining the decoder [26], imposing multitask loss functions [27, 28], and enforcing transformation invariant results during the learning process [29, 30]. This study is based on our previous work Deep Music Analogy [27] in which we disentangle pitch and rhythm factors for monophonic segments. We extend this idea to polyphonic composition while the model design is more similar to [24].

The chord encoder first applies rule-based methods [31, 32] to extract the chord progression under one-beat resolution. Each extracted chord progression is a 36 by 8 matrix, where each column denotes a chord of one beat. Each chord is a 36-D vector consisting of three parts: a 12-D one-hot vector for the pitch class of the root, a 12-D one-hot vector for the bass, and a 12-D multi-hot chroma vector.

The chord progression is then fed into a bi-directional GRU encoder [21], and the last hidden states on both ends of the GRU are concatenated and used to approximate the posterior distribution of $z_{\text{chd}}$. Following the assumption of a standard VAE, $z_{\text{chd}}$ has a standard Gaussian prior and follows an isotropic Gaussian posterior.

Note that although the chord progression here is extracted using algorithms, it can also be provided by external labels, in which case the whole model becomes a conditional VAE [33].

The chord decoder reconstructs the chord progression from $z_{\text{chd}}$ using another bi-directional GRU. The reconstruction loss of a chord progression is computed as a summation of 8 beat-wise chord loss using cross entropy functions [34]. For each beat, the chord loss is defined as the product of three parts: 1) the root loss, 2) the bass loss, and 3) the chroma loss. The root and bass are both considered 12-way categorical distributions and the chroma is regarded as 12 independent Bernoulli distributions.

The input of the texture encoder is an 8-beat segment of polyphonic piece represented by an image-like data format slightly modified from the piano-roll [14]. Each 8-beat segment is represented by a 128 by 32 matrix, where each row corresponds to a MIDI pitch and each column corresponds to $\frac{1}{4}$ beat. The data entry at $(p,t)$ records the duration of the note if there is a note onset, and zero otherwise.

The texture encoder aims to learn a chord-invariant representation of texture by leveraging both the translation invariance property of convolution and the blurry effect of max-pooling layers [35]. We use a convolutional layer with kernel size $12\times 4$ and stride $1\times 4$, which is followed by a ReLU activation [36] and max-pooling with kernel size $4\times 1$ and stride $4\times 1$. The convolutional layer has one input channel and 10 output channels. The convolutional layer design aims at extracting a blurry “concept sketch” of the polyphonic texture which contains minimum information of the underlying chord. Ideally, when such blurry sketch is combined with specific chord representation, the decoder can identify its concrete pitches in a musical way.

The output of the convolutional layer is then fed into a bi-directional GRU encoder to extract the texture representation $z_{\text{txt}}$, similar to how we encode $z_{\text{chd}}$ introduced in Section 3.1.

The PianoTree decoder takes the concatenation of $z_{\text{chd}}$ and $z_{\text{txt}}$ as input and decodes the music segment using the same decoder structure invented in PianoTree VAE [6], a hierarchical model structure for polyphonic representation learning. The decoder works as follows. First, it generates 32 frame-wise hidden states (one for each $\frac{1}{4}$ beat) using a GRU layer. Then, each frame-wise hidden state is further decoded into the embeddings of individual notes using another GRU layer. Finally, the pitch and duration for each note are reconstructed from the note embedding using a fully-connected layer and a GRU layer, respectively. For more detailed derivation and model design, we refer the readers to [6].

Let $x$ denote the input music piece and $c=f(x)$ denote the chord progression extracted by algorithm $f(\cdot)$. We assume standard Gaussian priors of $p(z_{\text{chd}})$ and $p(z_{\text{txt}})$, and denote the output posteriors of chord encoder and texture encoder by $q_{\phi}(z_{\text{chd}}|c)$, $q_{\psi}(z_{\text{txt}}|x)$, the output distributions of chord decoder and PianoTree decoder by $p_{\rho}(c|z_{\text{chd}})$ and $p_{\theta}(x|z_{\text{chd}},z_{\text{txt}})$. The objective of the model is:

By regarding chord progression content and texture style, we can achieve compositional style transfer by swapping the texture representations of different pieces. Figure 2 shows the transferred results ((c) & (d)) based on two 16-bar samples ((a) & (b)) in the test set by swapping $z_{\text{txt}}$ every 2 bars (without overlap).​​²²2The presented excerpts are converted from MIDI by the authors. The chord labels are inferred from the original/generated samples.

We see that such long-term style transfer is successful: The generated segment (c) follows the chord progression of (b) while mimicking the texture of (a), while (d) follows the chord progression of (a) while mimicking the texture of (b). As shown in the marked scores, the style transfer is effective. E.g., the cut-offs, melody contours, and the shape of the left-hand accompaniment are all preserved.

We can make variations of texture by sampling from $z_{\text{txt}}$ while keeping $z_{\text{chd}}$. Here, we investigate two sampling strategies: sampling from the posterior $q_{\psi}(z_{\text{txt}}|x)$, and sampling from the prior $p(z_{\text{txt}})$.

Sampling from the posterior distribution $q_{\psi}(z_{\text{txt}}|x)$ yields reasonable variations as shown in Figure 3(a). The variations of the right-hand melody can be seen as an improvisation following the chord progression and the melody. On the contrary, there is only small variation in the left-hand part, showing that the model regards the left-hand accompaniment as the dominant feature of texture.

Sampling from the prior distribution $p(z_{\text{txt}})$ changes the texture completely. Figure 3(b) shows a series of examples of prior sampling under the same chord progression C-Am-F-G. The resulting generations follow exactly the chord progression but with new textures.

We use a downstream predictive model to achieve accompaniment arrangement. For this task, we provide extra vocal melody tracks paired with the piano samples, and the model learns to generate 16-bar piano accompaniment conditioned on melody in a supervised fashion.

We encode the music every 2 bars (without overlap) into latent representations. For the accompaniment, we use the proposed model to compute the latent chord and texture representation, denoted by $\mathbf{z}_{\text{chd}}=[z_{\text{chd}}^{(1)},...,z_{\text{chd}}^{(4)}]$ and $\mathbf{z}_{\text{txt}}=[z_{\text{txt}}^{(1)},...,z_{\text{txt}}^{(4)}]$. For the melody, we use the EC²-VAE [27] to compute the latent pitch and rhythm representations, denoted by $\mathbf{z}_{\text{p}}=[z_{\text{p}}^{(1)},...,z_{\text{p}}^{(4)}]$ and $\mathbf{z}_{\text{r}}=[z_{\text{r}}^{(1)},...,z_{\text{r}}^{(4)}]$. Then, we adopt a vanilla Transformer [37] to model $p(\mathbf{z}_{\text{txt}},\mathbf{z}_{\text{chd}}|\mathbf{z}_{\text{p}},\mathbf{z}_{\text{r}})$, in which the encoder takes in the condition and the decoder’s input is a shifted right version $[\mathbf{z}_{\text{chd}},\mathbf{z}_{\text{txt}}]$. Both encoder and decoder inputs are incorporated with a positional encoding indicating the time positions and a learned factor embedding indicating the representation type (i.e., pitch, rhythm, chord or texture).

Figure 4 shows an example of accompaniment arrangement, where the first staff shows the melody and the second staff shows the piano accompaniment. In this case, the whole melody, together with the complete chord progression and the first 2 bars of accompaniment are given. The chord conditioning is done by forcing the decoded chord representation to match the given input during inference time. (A similar trick is used in [15].) From Figure 4, we see that the model predicts a similar texture to the given accompaniment. Moreover, it fills in a secondary melody line as a transition when the lead melody is rest.

Note that the arrangement can be generated in a flexible way by conditioning on different sets of latent factors. Much longer examples and more conditioning settings are available on our github page.

We train our model on the POP909 dataset[38], which contains about 1K MIDI files of pop songs (including paired vocal melody and piano accompaniment). We further extract the chord annotations using [31, 32]. We only keep the pieces with $\frac{2}{4}$ and $\frac{4}{4}$ meters and cut them into 8-beat music segments (so that each data sample in our experiment contains 32 time steps under 16${}^{\text{th}}$ note resolution). In all, we have 66K samples. We randomly split the dataset (at song-level) into training set (90%) and test set (10%). All training samples are further augmented by transposing to all 12 keys.

In our experiment, the VAE model uses 256, 512, and 512 hidden dimensions for the GRUs in chord encoder, chord decoder and texture encoder respectively. The latent dimension of $z_{\text{chd}}$ and $z_{\text{txt}}$ are both 256. The model size of the PianoTree decoder is the same as the implementation in the original paper [6]. The transformer model has the following size: hidden dimension $=256$, number of layers $=4$ and number of heads $=8$.

For both models, we use Adam optimizer [39] with a scheduled learning rate from 1e-3 to 1e-5. Moreover, for the VAE model, we use KL-annealing [40], i.e. setting a weight parameter for the KL-divergence loss starting from 0 to 0.1. We set batch size to be 128 and the training converges within 6 epochs. For the downstream transformer model, we use 12K warm-up steps for learning rate update [41]. We use the same batch size and the model converges within 40 epochs.

When $z_{\text{chd}}$ and $z_{\text{txt}}$ are well disentangled, small variations over the note pitches of the original music should lead to a larger change on $z_{\text{chd}}$, while variations of rhythm will influence more on $z_{\text{txt}}$. Following this assumption, we adopt a disentanglement evaluation via data augmentation method used in [42] and further developed in [27].

We define $F_{i}$ as the operation of transposing all the notes by $i$ semitones, and use the $L_{1}$-norm to measure the change of latent $z$ after augmentation. Figure 5(a) shows a comparison between $\Sigma|\Delta z_{\text{chd}}|$ and $\Sigma|\Delta z_{\text{txt}}|$ when we apply $F_{i}$ to all the music pieces in the test set (where $i\in[1,12]$).

It is conspicuous that when augmenting pitch in a small range, the change of $z_{\text{chd}}$ is much larger than the change of $z_{\text{txt}}$. At the same time, the change of $z_{\text{txt}}$ gets higher as the augmentation scale increases. Similar to the result in [27], the change of $z_{\text{chd}}$ reflects human pitch perception as $z_{\text{chd}}$ is very sensitive to a tritone transposition, and least sensitive for a perfect octave.

We further define $P_{i}$ as the function to randomly transpose all the notes in one beat either up or down one semitone under a certain probability $i$, and $R_{i}$ as the function to randomly reduce the note duration by half. Figure 5(b) shows a comparison between $\Sigma|\Delta z_{\text{chd}}|$ and $\Sigma|\Delta z_{\text{txt}}|$ when we apply $P_{i}$ and $R_{i}$ to all the music pieces in our test set (where $i\in[0.1,1.0]$).

For each value of $i$ in the figure 5(b), the first and second bars demonstrate $\Sigma|\Delta z_{\text{chd}}|$ and $\Sigma|\Delta z_{\text{txt}}|$ caused by $P_{i}$ function, while the third bar indicates $\Sigma|\Delta z_{\text{txt}}|$ caused by $R_{i}$ function. (We did not show $\Sigma|\Delta z_{\text{chd}}|$ caused by $R_{i}$ since they are all zero.) It again proves that the chord representation is more sensitive than texture representation under pitch variations, and conversely, texture representation is more sensitive than chord representation under rhythm variations.

Besides objective measurement, we conduct a survey to evaluate the musical quality of compositional style transfer (see Section 4.1). Each subject listens to ten 2-bar pieces with different chord progressions, each paired with 5 style-transfer versions generated by swapping the texture representation with a random sample from the test set. In other words, each subject evaluates 10 groups of samples, each of which contains 6 versions of textures (1 from the original piece and 5 from other pieces) under the same chord progression. Both the order of groups and the sample order within each group are randomized. After listening to each sample, the subjects rate them based on a 5-point scale from 1 (very low) to 5 (very high) according to three criteria: creativity, naturalness and musicality.

A total of 36 subjects (26 females and 10 males) participated in the survey. Figure 6 shows the comparison result among the original pieces (indicated by the orange bars) and the transferred pieces in terms of their mean and order statistics. The heights of bars represent averaged ratings across the subjects and the error bars represent the confidence intervals computed via paired t-test [43]. The result shows if we randomly transfer a piece’s texture 5 times, the best result is significantly better than the original version (with p-value $<0.005$), and there are only marginal differences between the second-largest statistics and the original (with p-value $>0.05$) in terms of creativity and musicality. We also see that on average the transferred results are still rated lower than the original ones. How to automatically decide the quality of a transferred result is considered a future work.