Automatic Analysis and Influence of Hierarchical Structure on Melody, Rhythm and Harmony in Popular Music

Form and structure are among the most important elements in music and have been widely studied in music theory. Music structure has a hierarchical organization ranging from low-level motives to higher-level phrases and sections. These different levels influence the organization of other elements such as harmony, melody and rhythm, but these influences are not well formalized. MIR research has developed techniques for detecting music segmentation and repetition structures, but hierarchy is often ignored Dannenberg \BBA Goto (\APACyear2009); Paulus \BOthers. (\APACyear2010). A fundamental question is whether higher levels of hierarchy are essentially just larger groupings or whether different levels play different roles. If the latter is true, then a better representation and understanding of hierarchy should be useful for prediction, generation, analysis and other tasks. Long-term structure in music is also a recent topic in music generation with deep learning, and attention models such as the Transformer Vaswani \BOthers. (\APACyear2017); Huang \BOthers. (\APACyear2018) seem to improve results. While this suggests some data-driven discovery of structure, results are hard to interpret and, for example, it is not clear whether hierarchy plays a role. .

We began our study by developing a method to identify low-level structure in popular songs. Our approach identifies phrases with repetition of harmony, melody and rhythm. Next, we discovered a simple way to infer higher-level structure from this phrase-level structure. Beyond viewing structure as mere repetition, we show that chord progressions, melodic structures and rhythmic patterns are all related to music structure, and there are significantly different interactions at different levels of hierarchy. Our main contributions are: 1) a novel algorithm to extract repetition structure at both phrase and section levels from a MIDI data set of popular music, 2) formal evidence that melody, harmony and rhythm are organized to reflect different levels of hierarchy, 3) data-driven models offering new music features and insights for traditional music theory. We believe that this work is important in highlighting roles that structure can play in music analysis, music similarity, music generation, and other MIR tasks.

Section 2 discusses related work, and Section 3 presents our phrase-structure analysis method. Section 4 describes general properties of structures we found, while Section 5 explores the relationships between structures and harmony, melody and rhythm, respectively. Finally, Sections 6 and 7 present discussion and conclusions.

Computational analysis of musical form has long been an important task in Music Information Retrieval (MIR). Large-scale structure in music, from classical sonata form to the repeated structure in pop songs, is essential to music analysis as well as composition. Marsden \APACyear2010 implemented Schenkerian analysis and applied it to Mozart variations. Hamanaka, Hirata and Tojo \APACyear2014 describe a tool for Generative Theory of Tonal Music (GTTM) analysis that matches closely the analyses of musicologists. Allegraud et al. \APACyear2019 use unsupervised learning to segment Mozart string quartets. Go, Ryo, Eita and Kazuyoshi \APACyear2019 perform structural analysis using homogeneity, repetitiveness, novelty, and regularity. Our work builds on the idea of extracting structure by discovering repetition.

Identifying hierarchical structure is likely to play a role in music listening. Granroth-Wilding \APACyear2013 employs ideas from Natural Language Processing (NLP) to obtain a hierarchical structure of chord sequences. Marsden, Hirata and Tojo \APACyear2013 state that advances in the theory of tree structures in music will depend on clarity about data structures and explicit algorithms. Jiang and Müller \APACyear2013 propose a two-step segmentation algorithm for analyzing music recordings in predefined sonata form: a thumb-nailing approach for detecting coarse structure and a rule-based approach for analyzing the finer substructure. We present a detailed algorithm for segmenting music into phrases and deriving a higher-level sectional structure starting with a symbolic representation.

Segmentation of music audio is a common MIR task with a substantial literature. Dannenberg and Goto \APACyear2009 survey audio segmentation techniques based on repetition, textural similarity, and contrast. Barrington, Chan and Lanckriet \APACyear2009 perform audio music segmentation based on timbre and rhythmical properties. However, MIDI has the advantage of greater and more reliable rhythmic information along with the possibility of cleanly separating melody. Many chord recognition algorithms exist, e.g. Masada and Bunescu \APACyear2018 use a semi-Markov Conditional Random Field model. References to melody extraction from MIDI can be found in Zheng and Dannenberg \APACyear2019 who use maximum likelihood and Dynamic Programming. Rolland \APACyear1999 presents an efficient algorithm for spotting matching melodic phrases, which relates to our algorithm for segmentation based on matching sub-segments of music. Lukashevich \APACyear2008 proposes a music segmentation evaluation measure considering over- and under-segmentation. Collins, Arzt, Flossmann and Widmer \APACyear2013 develop a geometric approach to discover inexact intra-opus patterns in point-set representations of piano sonatas. Our work introduces new methods for the analysis of multi-level hierarchy in MIDI and investigates the interplay of structure with harmony, melody and rhythm.

We introduce a novel algorithm based on repetition and similarity to extract structure from annotated MIDI files. Given input consisting of a chord and melody sequence for each song together with its time signature (obtained from MIDI pre-processing), the algorithm outputs a repetition structure. In this section, we will introduce the design motivation, structure representation, details of the algorithm and some evaluation results.

In this section, we characterize the lower-level phrase structure and the higher-level section structure we found in our data set.

We could have used any number of ways to form higher-level structure (sections), but our choice is supported by the finding of interactions between sections, melody, harmony and rhythm that do not occur so strongly at the phrase level, suggesting that the section structure is not just an arbitrary construction.

In Figure 6, we show probabilities of different harmonies at different locations in phrases and sections in major mode. We are much more likely to see a I at the beginning of a phrase and at the end of a section. I and V chords are more popular at the ends of phrases (about equally). We expected to see a predominance of I chords at the ends of phrases, but as the last two categories reveal, the V is a more common ending within a section, while the I chord is more common at the end of a section. Here, we see significant interactions not only between structure and harmony but between different structural levels. We evaluate the significance of these differences by assuming a null hypothesis of equal probability everywhere (the background category in Figure 6) and using one-tailed unpaired t-tests. All the test results are significant ($P<.0001$).

Chord transitions at the ends of phrases or sections proved to be significantly different from general transition probabilities at other positions in the phrase. For example in major mode, 58% of progressions at the end of the section are V–I (Authentic Cadence in music theory). Transition probabilities from V–I at the end of phrase, end of phrase in the middle of a section, and end of section are 0.89, 0.84 and 0.94, while the average transition probability in all other positions is only 0.47.

Phrase and section structures also influence the distribution of melody pitches. Figure 7 shows probabilities of different melody pitches at different locations in phrases and sections, counting only pitches in the context of a I chord in the major mode. Scale degree $\hat{1}$ in melody tends to occur at the end of sections, but not at the start or middle of phrases. While scale degree $\hat{3}$ is common in the start of phrases, but not at the end of sections.

We are also interested in the duration and rhythm of the melody. The distribution of note length at the beginning and middle of phrases is about the same as the overall distribution, consisting mostly of short notes. In contrast, the phrase endings mostly consist of longer notes. We also observed a difference in phrase endings depending on position. For example, only 6.4% of whole-or-longer notes occur at the ends of phrases in the middle of a section, while 72% occur at the ends of phrases at the end of a section.

We also discovered two measures of phrase harmonic structure that correlate with year of composition. (Our data spans 7 decades of music.) Although space limits a complete discussion, we found cross-phrase similarity decreases with date (contrast between song sections increases) ($P<.01)$, and phrase complexity (in an information theoretical sense) increases with date, indicating generally longer phrases and more variety of chord types ($P<.002$).

The data-driven analysis results in this paper show that music elements such as harmony, melody and rhythm behave differently at different positions relative to the hierarchical music structure. These music-structure-related features support many aspects of traditional music theory. For example, in our analysis, half cadences are more often seen at the ends of phrases, but only in the middle of sections, consistent with the music theoretic concept that a half cadence calls for continuation. It is worth noting that the phrase structure extraction algorithm is fully based on repetition and similarity without using any knowledge of these music concepts. Thus, our approach forms a good test for music theory and existing domain knowledge.

The algorithms we proposed for extracting hierarchical repetition structures from MIDI files have a high accuracy of 93% compared to human labeling, and can be used to analyze other MIDI data sets. Our findings can guide music imitation or generation and can also be used to evaluate whether songs follow structural conventions. Notice that in the phrase-level structure analysis algorithm, parameters are manually tuned, but perhaps they could be adjusted automatically according to different styles of music.

Future work might strive to learn more about variations between similar phrases and how contrasting phrases are constructed. It would also be interesting to compare other data sets, including non-pop music. We have only begun to look for interactions between structure, melody, harmony and rhythm, and these initial results show this to be a promising research direction. The idea that structural tendencies change over time is also promising.

Our results with Chinese pop music are consistent with basic concepts of Western music theory, so we suspect that similar results would be obtained with Western pop music. Still, it would be interesting to conduct a comparative study with Western pop songs. Future work might also investigate more robust indicators of sections. It seems that the non-melodic phrases we use to detect sections are not present in all styles. Consider a repeated form such as AABA|AABA. There might be ways to identify these higher-level sections which are not separated by non-melodic phrases.

We believe this is the first study to analyze connections between different levels of music structure and the elements of harmony, melody and rhythm using a data-driven approach. We introduced a new hierarchical structure analysis algorithm. With it, we analyzed harmony, melody and rhythm in the context of multi-level structure. This work suggests there is still much to be learned about the role of structure in music, and that we can use hierarchical structure to inform future work on music style, analysis, evaluation and generation.

Our data set, annotations, and experimental results are released at: https://github.com/Dsqvival/hierarchical-structure-analysis.