BERT-like Pre-training for Symbolic Piano Music Classification Tasks

We collect four existing public-domain piano MIDI datasets, including Pop1K7, ASAP, POP909, EMOPIA and compile ourselves a new dataset, named Pianist8. To simplify the token representation, we consider only MIDI files that specify 4/4 metre.²²2We note that the metre can be wrong due to errors in automatic music transcription, leading to noise in the data. Future work can be done to improve this. Moreover, future work can be done to use a more complicated token representation such as that proposed by [123] to include other time signatures. We list some important statistics of these five datasets in Tab. 1 and provide their details below.

• •

  The Pop1K7 dataset developed by [98]³³3https://github.com/YatingMusic/compound-word-transformer is composed of machine transcriptions of 1,747 audio recordings of piano covers (i.e., a new recording by someone other than the original artist or composer of a commercially released song) of Japanese anime, Korean and Western pop music, amounting to over 100 hours worth of data. The transcription was done with the “onsets-and-frames” RNN-based piano transcription model [95] (which is reported to attain 95.32 onset F1 score for note-level piano transcription [97]), and the RNN-based downbeat and beat tracking model from the Madmom library [73]. This dataset is the largest among the five, constituting half of our training data. We only use it for pre-training.

• •

  ASAP, the aligned scores & performances dataset compiled by \citeyearasap,⁴⁴4https://github.com/fosfrancesco/asap-dataset contains 1,068 MIDI performances of 222 Western classical music compositions from 15 composers, along with the MIDI performances of the 222 pieces compiled from the MAESTRO dataset [96]. We consider it as an additional dataset for pre-training, using only the MIDI that specifies 4/4 metre with no metre change at all throughout the piece. This leaves us with 65 pieces of MIDI files, which last for 3.5 hours in total. Tab. 1 shows that, being the only classical dataset among the five, ASAP features shorter average note duration and larger number of notes per bar.

• •

  POP909 comprises piano covers of 909 pop songs compiled by [136].⁵⁵5https://github.com/music-x-lab/POP909-Dataset It is the only dataset among the five that provides melody, non-melody labels for each note. Specifically, each note is labelled with one of the following three classes: vocal melody (piano notes corresponding to the lead vocal melody line in the original pop song, usually during the verse and chorus parts); instrumental melody (piano notes corresponding to the secondary melody line played by the instruments in the original pop song, usually during the intro, bridge, outro parts); and accompaniment (including arpeggios, broken chords and many other textures).⁶⁶6POP909 originally refers to vocal melody as melody and instrumental melody as bridge in their work [136]. We opt for our new naming to highlight the fact that the latter is also a type of melody. As it is a MIDI performance dataset, it also comes with velocity information. Therefore, we use it for the melody classification and velocity prediction tasks. We discard pieces that do not specify 4/4 metre, ending up with 865 pieces for this dataset.

• •

  Pianist8 consists of eight artists’ performances of piano music that we download from YouTube for training and evaluating symbolic-domain style classification.⁷⁷7https://zenodo.org/record/5089279 The artists are Richard Clayderman (pop), Yiruma (pop), Herbie Hancock (jazz), Ludovico Einaudi (contemporary), Hisaishi Joe (contemporary), Ryuichi Sakamoto (contemporary), Bethel Music (religious) and Hillsong Worship (religious). The artists are also the composers of the pieces, except for Richard Clayderman, Bethel Music and Hillsong Worship. The dataset contains a total of 411 pieces, with a fairly balanced number of pieces per artist. Each audio file is paired with its MIDI performance, which is machine-transcribed by the piano transcription model proposed by [110].

• •

  EMOPIA is a dataset of pop piano music collected recently by [103] from YouTube for research on emotion-related tasks.⁸⁸8https://annahung31.github.io/EMOPIA/ It has 1,087 clips (each around 30 seconds) segmented from 387 songs, covering Japanese anime, Korean & Western pop song covers, movie soundtracks and personal compositions. The emotion of each clip has been labelled using the following 4-class taxonomy: HAHV (high arousal high valence); LAHV (low arousal high valence); HALV (high arousal low valence); and LALV (low arousal low valence). This taxonomy is derived from the Russell’s valence-arousal model of emotion [127], where valence indicates whether the emotion is positive or negative and arousal denotes whether the emotion is high (e.g., angry) or low (e.g., sad) [142]. The MIDI performances of these clips are similarly machine-transcribed from the audio recordings by the model of [110]. We use this dataset for the emotion classification task. As Tab. 1 shows, the average length of the pieces in the EMOPIA dataset is the shortest among the five, since they are actually clips manually selected by dedicated annotators [103] to ensure that each performance expresses a single emotion.

All five datasets consist of MIDI performances. As mentioned in the introduction, we intend to build two PTMs, one for MIDI scores and the other for MIDI performances. We obtain the MIDI-score version of each performance by dropping velocity and tempo information, temporally quantising the onset time and duration of each the notes to the semiquaver resolution.

Similar to text, a piece of music in MIDI can be considered as a sequence of musical events or “tokens”. However, what makes music different is that musical notes are associated with a temporal length (i.e., note duration) and multiple notes can be played at the same time. Therefore, to represent music, we need note-related tokens describing, for example, the pitch and duration of the notes, as well as metric-related tokens placing the notes over a time grid.

In the literature, a variety of token representations for MIDI have been proposed, differing in many aspects such as the MIDI data being considered (e.g., melody [135], lead sheet [140], piano [100] and multi-track music [124, 85]), the temporal resolution of the time grid and the way the advancement in time is notated [101]. Auxiliary tokens describing, for example, the chord progression [101] or grooving pattern [76] underlying a piece can also be added.

In this work, we adopt the beat-based REMI token representation proposed by [101] to place musical notes over a discrete time grid comprising 16 equidistant sample points per bar. In addition to REMI, we experiment with the “token grouping” idea of the compound word (CP) representation [98], to reduce the length of the token sequences. We depict the two adopted token representations in Fig. 1 and provide some details below.

For symbolic-domain melody extraction, initial methodologies predominantly adopted rule-based approaches. These rule-based methods encompassed techniques such as utilising pitch contour characteristics [128], as well as the implementation of the “skyline" algorithm [75]. In recent years, deep learning-based approaches utilising convolutional neural networks (CNN) have been adopted [130, 99], We will review such CNN-based methods in Section 5, highlighting their specific details and implementation. Similar to [130], we regard melody extraction as a task that identifies the melody notes in a single-track ¹⁰¹⁰10It is common for MIDI files to consist of multiple tracks. We refer to “single-track” as MIDI files containing only one track, which is in contrast to multi-track MIDI files that have multiple tracks. homophonic or polyphonic music. Utilising the POP909 dataset [136], we can develop a model that classifies each Pitch event into vocal melody, instrumental melody or accompaniment, with classification accuracy (ACC) serving as the evaluation metric.¹¹¹¹11We note that there is a task closely related to melody extraction, called melody track identification. The goal of this task is to distinguish the melody track from other non-melody tracks present in a multi-track MIDI file [119, 106]. While melody extraction is a note-level classification task, melody track identification is a track-level task. The latter is also an important symbolic music classification task, but we do not consider it here for we exclusively focus on piano-only data.

Specifically, we consider two formulations of the task. Firstly, we adhere to the original configuration of POP909 and perform three-class melody classification, classifying each Pitch into three categories: vocal melody, instrumental melody or accompaniment. Secondly, we merge vocal melody and instrumental melody into a general "melody" category (while accompaniment is designated as "non-melody") and perform two-class classification. Doing so allows for a direct comparison with established baselines, such as the skyline algorithm and the baseline introduced in Section 5. For detailed results and a thorough examination, please refer to Section 7.1.

Dynamics is an important element in music, as they are often used by musicians to add excitement and emotion to songs. Given that the tokens we choose do not contain performance information, it is interesting to see how a machine model would “perform” a piece by deciding these volume changes, a task that is essential in performance generation [137, 104, 105] or expressive performance modelling [88, 89]. In the realm of MIDI, velocity is a parameter that scales the intensity or volume at which a sound sample is played back, with the value ranging from 0 to 127. Default MIDI velocity values are associated with dynamic indications. Apple’s Logic Pro 9 user manual correlates traditional volume indicators (pp, p, mp, mf, f, ff and fff) with specific MIDI velocity values (16, 32, 48, 64, 80, 96, 112 and 127), respectively.¹²¹²12https://help.apple.com/logicpro/mac/9.1.6/en/logicpro/usermanual/ (page 468 in the user manual; accessed 2023-06-22) In our work, we define and classify this information into six categories: pp (0–31), p (32–47), mp (48–63), mf (64–79), f (80–95) and ff (96–127). Our definition aligns with the Logic Pro 9 specifications, except that we treat fff as equivalent to ff. Our objective can be treated as a note-level classification task, aiming to classify Pitch events into six classes using the POP909 dataset [136].

Genre classification [83] can be considered as a type of style classification. While genre classification categorises music based on shared musical attributes and conventions, style classification seeks to capture the nuanced stylistic variations within either a specific genre, composer or performer, accounting for the diverse artistic choices and performance practices that shape musical expressions. We could relatively more easily find out which type of music we are listening to based on the similar patterns in that genre, while needing more musical insights to recognise the composer’s or performer’s style. Deep learning-based composer classification in MIDI has been attempted by [112] and [109], both treating MIDI pieces as 2D-representation matrices (via the piano-roll representation) and using CNN classifiers. Our work differs from theirs in that: 1) we encode MIDI pieces as token sequences, 2) we employ PTM, 3) we consider non-classical music pieces and 4) our task is about style classification because not all the pianists in Pianist8 composed the pieces they performed.

Emotion classification in MIDI has been approached by a few researchers, mostly using hand-crafted features and non-deep learning classifiers [90, 113, 121, 122]. Some researchers work on MIDI alone, while others use both audio and MIDI in multi-modal emotion classification [121]. The only deep learning-based approach we are aware of is presented by [103], using an RNN-based classifier called “Bi-LSTM-Attn” [114] but without employing PTMs, which is also used as a baseline in our experiment; see Section 5.

For PTMs, an unsupervised or self-supervised, pre-training task is needed to set the objective function for learning. We employ the masked language modelling (MLM) pre-training strategy of BERT, randomly masking 15% tokens of an input sequence and the Transformer will reconstruct these masked tokens from the context of the visible tokens by minimising the cross-entropy loss. As a self-supervised method, MLM needs no labelled data relating to the downstream tasks for pre-training. Following BERT, among all the masked tokens, we replace 80% by MASK tokens, 10% by a randomly chosen token and leave the last 10% unchanged. Doing so has the effect of helping mitigate the mismatch between pre-training and fine-tuning as MASK tokens do not appear at all during fine-tuning. For REMI, we mask the individual tokens at random. For CP, we mask the super tokens—when we mask a super token, we have to reconstruct all the four tokens composing it by different output heads [98], as shown in Fig. 2(a).

There are three steps for processing the input token. First, each input token $X_{i}$ is converted into a token embedding $E_{i}$ through an embedding layer. Second, it is augmented by addition with a relative positional encoding [102] that is related to its time step. Third, it is then fed $E_{i}$ to the stack of 12 self-attention layers to get a “contextualised” representation known as a hidden vector at the output of the self-attention stack. Because of the bi-directional self-attention layers, the hidden vector is contextualized in the sense that it has attended to information from all the other tokens from the same sequence. Finally, the hidden vector of a masked token is fed into a dense layer to predict what the missing super token is. As our network structure is rather standard, we refer readers to the literature [134, 84, 141] for details and the mathematical underpinnings due to space limits. Because the vocabulary sizes for the four token types are different, we weight the training loss associated with tokens of different types in proportion to the corresponding vocabulary size of REMI and CP, to facilitate model training.

We note that the original BERT article also used another self-supervised task called “next sentence prediction” (NSP) [84] together with MLM for pre-training. We do not use NSP for our model since it was later shown to be not that useful [107, 143, 82]; moreover, NSP requires a clear definition of “sentences”, which is not well-defined for our MIDI sequences. As a result, we do not use tokens such as CLS, SEP and EOS used by BERT for making the boundary of the sentences.¹⁶¹⁶16 CLS marks the beginning of a sentence, SEP the boundary between two consecutive sentences (useful for the so-called “next sentence prediction task” [84]) and EOS the end of a sentence.

It has been widely shown in NLP and related fields [81, 117, 131, 74] that, by storing knowledge in huge numbers of parameters and carrying out task-specific fine-tuning, the knowledge implicitly encoded in the parameters of a PTM can be transferred to benefit a variety of downstream tasks [93]. For fine-tuning, we extend the architecture shown in Fig. 2(a) by modifying the last few layers in two different ways, one for each of the two types of downstream tasks.

Fig. 2(b) shows the fine-tuning architecture for note-level classification. While the Transformer uses the hidden vectors to recover the masked tokens during pre-training, it has to predict the label of an input token during fine-tuning, by learning from the labels provided in the training data of the downstream task in a supervised way. To achieve this, we feed the hidden vectors to a stack of dense layers, a ReLU activation layer and finally another dense layer for the output classification, with 10% dropout probability. We note that this classifier design is fairly simple, as we expect much knowledge regarding the downstream task can already be extracted from the preceding self-attention layers.

Fig. 2(c) shows the fine-tuning architecture for sequence-level classification. Being inspired by the Bi-LSTM-Attn model [114], we employ an attention-based weighting average mechanism to convert the sequence of 512 hidden vectors for an input sequence to one single vector before feeding it to the classifier layer, which comprises two dense layers. We note that, unlike the baseline models introduced in Section 5, we do not use RNN layers in our models. An alternative approach is to add the CLS token to our sequences and simply use its hidden vector as the input to the classifier layer. We do not explore this alternative since we do not have CLS tokens.

Our implementation is based on the open-source library HuggingFace [138]. As mentioned in Section 3.1, we use Pop1K7 and ASAP for pre-training and the other three datasets (i.e., POP909, Pianist8 and EMOPIA) for the downstream tasks. From the combination of Pop1K7 and ASAP, we use 85% of them for pre-training as described in Section 6.1 and the rest as the validation set. We train with a batch size of 12 sequences for at most 500 epochs (i.e., around 500K iterations for REMI and 1M iterations for CP), using the AdamW optimizer with learning rate 2e$-$5 and weight decay rate 0.01. If the validation cross-entropy loss does not improve for 30 consecutive epochs, we stop the training process early. For pre-training, we can improve the validation “cloze” accuracy from 70.4% for REMI to 78.73% for CP. We observe that pre-training using the CP representation converges in 2.5 days on four GeForce GTX 1080-Ti GPUs, which is about 2.5 times faster than the case of REMI. Moreover, a smaller batch size degrades overall performance, including downstream classification accuracy. Because each sequence has 512 super tokens, we have 6,144 super tokens per batch.

For fine-tuning, we create training, validation and test splits for each of the three datasets of the downstream tasks with the 8:1:1 ratio at the piece level (i.e., all the 512-token sequences from the same piece are in the same split). With the same batch size of 12, we fine-tune the pre-trained our model for each task independently for at most 10 epochs, early stopping when there is no improvement for three consecutive epochs. Compared to pre-training, fine-tuning is less computationally expensive. All the results reported in our work can be reproduced with four GeForce GTX 1080-Ti GPUs within 30 minutes.

In our experiments, we will use the same pre-trained model parameters to initialise the models for different downstream tasks. During fine-tuning, we fine-tune the parameters of all the layers, including the self-attention and token embedding layers.

Fig. 3 presents the normalised confusion tables for three-class melody classification, illustrating distinct performance characteristics among the models. We note that the baseline exhibits a tendency to conflate vocal melody (M1) and instrumental melody (M2), whereas our model outperforms the RNN-based model, enhancing the overall accuracy by almost 8% (from 88.66% to 96.15%). A closer examination reveals our model’s superior ability to differentiate between vocal and instrumental melodies compared to the RNN baseline with minimal finetuning. This task is particularly challenging given the nature of the POP909 dataset, which exclusively features pop songs sung by humans. Consequently, the separation of vocal and instrumental melodies relies on the criterion of human vocalisation (which is absent in MIDI data), potentially leading to instances where notes between phrases are designated as instrumental melody despite sharing pitch and melodic characteristics with vocal melody.

Interestingly, an intriguing observation emerges as “our model (score) + CP” demonstrates a more effective distinction between vocal and instrumental melodies than “our model (performance) + CP”. This phenomenon suggests that even without velocity information, our model can discern segments designated for singing versus those serving as preludes, interludes or fills.

Tab. 3 compares our model with the “skyline” algorithm [75] and the CNN-based baselines [130] for the two-class “melody versus non-melody” melody classification task. As the dataset is highly unbalanced (i.e., the melody notes are much fewer than the accompaniment notes), we also report the precision, recall and F1 scores. It turns out that our model greatly outperforms other baselines across all the metrics, reaching 99.04% classification accuracy. A qualitative example demonstrating the superiority of the proposed model over the the skyline algorithm can be found in Fig. 4, using a randomly chosen testing piece from POP909${}_{\text{4/4}}$.

Moreover, we have extended the application of our melody extraction model to compositions from the Pianist8 dataset. Given the absence of manual labels for the melody notes within this dataset, we encourage readers to evaluate the results by listening to the prediction outputs.¹⁹¹⁹19https://github.com/wazenmai/MIDI-BERT/tree/CP/melody_extraction/audio We provide three versions of the melody MIDI file for each original song, generated respectively by the skyline algorithm, Simonetta et al.’s CNN and “our model (performance) + CP”. Taking “Clayderman_Yesterday_Once_More.mid” as an example, the melody generated by the skyline algorithm exhibits stiffness and lacks intricate details, retaining only the treble. The CNN version demonstrates considerable improvement over the skyline algorithm. However, a noticeable intermittent quality persists throughout the entire song, with some cohesive melodies omitted. Our model achieves commendable performance, successfully extracting the majority of the main melody and presenting a discernible melodic progression. It is worth highlighting the efficiency of our model, as it requires less than one hour for finetuning under the same hardware conditions that necessitate a full day of training for the CNN baseline on the POP909 training set.

Tab. 2 shows that the accuracy on our 6-class velocity classification task is not high, reaching 52.11% at best. This may be due to the fact that velocity is rather subjective, meaning that musicians can perform the same music piece fairly differently. Moreover, we note that the data is highly imbalanced, with the latter three classes (mf, f, ff) taking up nearly 90% of all labelled data. The confusion tables presented in Fig. 5 show that Bi-LSTM tends to classify most of the notes into f, the most popular class among the six. This is less the case for our model, but the prediction of p and pp, i.e., the two with the lowest dynamics, remains challenging. For future work, data augmentation is a potential solution to mitigate the impact of data imbalance.

Tab. 2 shows that our model greatly outperforms Bi-LSTM-Attn [114] and the CNN baseline [112] by 10–20% regardless of the token representation taken, reaching 81.75% testing accuracy at best for this 8-class classification problem. In addition, we see a large performance gap between REMI and CP in this task, the largest among the four tasks. Fig. 6 further shows that, both the baseline and “our model (score)$+$CP” confuse artists in similar genres and that our model performs fairly well in recognising Herbie Hancock and Ryuichi Sakamoto. In contrast, by considering velocity and tempo information, “our model (performance)$+$CP” gains lots of precision on classifying songs in pop and contemporary genres, boosting the classification accuracy from 67.46 (score) to 81.75 (performance).

Tab. 2 shows that our model outscores Bi-LSTM-Attn by around 14% and the CNN baseline [112] by around 7% in both REMI and CP for this 4-class classification problem, reaching 70.64% testing accuracy at best. There is little performance difference between REMI and CP in this task. Fig. 7 further shows that the evaluated models can fairly easily distinguish between high arousal and low arousal pieces (i.e., “HAHV, HALV” versus “LALV, LAHV”), but they have a much harder time along the valence axis (e.g., “HAHV” versus “HALV” and “LALV” versus “LAHV”). We see less confusion from the result of ‘our model (score)$+$CP’. By considering velocity and tempo, “our model (performance)$+$CP” can further classify variance difference in low-arousal songs, though there is still room for improvement.