Moving Beyond LDA: A Comparison of Unsupervised Topic Modelling Techniques for Qualitative Data Analysis of Online Communities

For qualitative researchers, social media serves as a rich and influential source of information (Andreotta et al., 2019). Reddit, in particular, has emerged as a crucial platform for gathering diverse and localized content; offering opinions, advice, and recommendations from trusted sources (Ryan et al., 2017; Ahn and Shin, 2013; Allen et al., 2014). This geographically specific and up-to-date information is invaluable for aiding research, synthesis, and presentation of data (Dugan et al., 2008; Skeels and Grudin, 2009; Steinfield et al., 2009; Morris et al., 2010). Researchers leverage social media to uncover nuanced insights in fields such as mental health, politics, consumer behavior, health and fitness, technology, gaming, education, and social movements (Andreotta et al., 2019; Ahn and Shin, 2013; Kapoor et al., 2018). For instance, public health researchers use Reddit to study health-related topics and understand community perspectives and communication norms, which aids in developing more effective interventions and policies (Eysenbach, 2005; Heldman et al., 2013; Gauthier et al., 2022; Rotolo et al., 2022; Gauthier et al., 2023).

Despite the availability of social media data, the sheer volume, diversity, noise, dynamic nature, and contextual nuances of this data present significant challenges (Andreotta et al., 2019). Computational techniques like topic modelling are thus necessary to effectively analyze this extensive content (Rana et al., 2016). However, traditional topic modelling methods like Latent Dirichlet Allocation (LDA), Biterm Topic Model (BTM), and Non-Negative Matrix Factorization (NMF) often fail to capture nuanced meanings and contextual usage and require extensive data cleaning and pre-processing, making them labor-intensive (Yan et al., 2013; Mazarura and De Waal, 2016; Zou and Song, 2016). In contrast, Large Language Model (LLM) based techniques like BERTopic, Top2Vec and GPT-3 offer substantial improvements. They provide advanced understanding of text with minimal pre-processing, adapting efficiently to various contexts (Brown et al., 2020; Devlin et al., 2019; Vaswani et al., 2017).

Further, qualitative researchers face significant challenges when integrating computational techniques like topic modelling into their qualitative workflows due to a lack of programming expertise (Lochmiller, 2021; Baden et al., 2022). They also have concerns about losing control over the nuanced aspects of their research and doubts about the accuracy of computer-generated insights (Abram et al., 2020; Jiang et al., 2021; Feuston and Brubaker, 2021). Ethical and privacy concerns further contribute to this reluctance (Siiman et al., 2023).

Nevertheless, AI tools can significantly aid in analyzing large volumes of social media data. These tools efficiently process and analyze data, identifying trends, sentiments, and patterns that would be challenging to uncover manually (Batrinca and Treleaven, 2015). Researchers could utilize AI’s capability for deep data analysis, providing a comprehensive understanding of social media interactions (Fan and Gordon, 2014). In particular, topic modelling is increasingly being integrated into qualitative research to manage and interpret large datasets, offering deeper and more objective insights (Gillies et al., 2022).

Our research is motivated by the desire to assist qualitative researchers who often face challenges with manual, time-consuming, and potentially biased data analysis methods due to limited programming skills. Recognizing these challenges, we aim to explore the potential of Large Language Models (LLMs) for unsupervised topic modelling, offering a more efficient and objective approach to data analysis. By understanding how qualitative researchers currently use topic modelling and identifying the specific difficulties they encounter, we can develop better tools to support their work.

We integrated BERTopic into the Computational Thematic Analysis (CTA) toolkit (Gauthier and Wallace, 2022). Integrating BERTopic into the CTA Toolkit required architectural changes to accommodate the model’s advanced requirements which uses word embeddings as compared to the original bag of words methodology, and leverages transformer-based processing. This included incorporating GPU utilization to meet BERTopic’s high computational demands and ensuring efficient processing of large datasets. Additionally, changes to the data filtering methods were necessary to ensure improved data processing and data integrity.

We then conducted interviews with qualitative researchers to discuss the challenges they face with existing topic modelling tools. Following these discussions, researchers engaged with the CTA toolkit, applying it to their own datasets and models to identify their preferred options. This hands-on exploration allowed them to evaluate the tools in the context of their specific needs. The choices and reasons for their preferences were then collaboratively analyzed, ensuring that the selected topic modelling techniques aligned well with the researchers’ needs and preferences.

Participants prioritized topic relevance, logical organization, and the capacity to reveal unexpected yet significant relationships within the data when evaluating topic modelling techniques. They desired detailed, coherent clusters that clearly separated and grouped significant topics, which facilitated deeper understanding and actionable insights, which were provided by BERTopic. Despite some visualization shortcomings, they valued BERTopic’s ability to uncover hidden connections, emphasizing the need for meaningful, comprehensive analysis tools that support their research objectives and enhance data interpretation. BERTopic was ranked first by 8 out of 12 participants (67%), LDA by 3 out of 12 participants (25%), and NMF by 1 out of 12 participants (8%).

In summary, we make three contributions:

1. (1)

   We integrated BERTopic into the Computational Thematic Analysis Toolkit.

2. (2)

   We conducted interviews with qualitative researchers to discuss the challenges they face when using topic modelling tools to analyse social media data.

3. (3)

   We discuss the findings on researchers’ priorities in topic modelling techniques.

The HCI community is increasingly exploring the use of social media to gain insights into human behavior (Gauthier, 2020; Gauthier et al., 2023; Andalibi et al., 2016; Ammari et al., 2018). Topic modelling in particular has been used to support social media analysis by identifying underlying themes and patterns within large datasets, allowing researchers to categorize and summarize vast amounts of unstructured text data efficiently (Jacobi et al., 2018; Curiskis et al., 2020). This facilitates the detection of trends, sentiment analysis, and the extraction of meaningful insights from complex and diverse social media content. Topic modelling has been applied across numerous fields – studies such as those by Jacobi et al. (2018) in journalism, Gauthier (2020); Gauthier et al. (2023); Rotolo et al. (2022); Eysenbach (2005); Han et al. (2021) in public health, Haghighi et al. (2018) in urban planning, Bail et al. (2018) in political science, and Pousti et al. (2021) in information systems illustrate the broad applicability and value of topic modelling in leveraging social media data platforms for research.

One such platform is Reddit, which facilitates extensive online interactions through user-generated communities called subreddits, covering diverse topics (Kumar et al., 2022; Rocha-Silva et al., 2023). Its structure, allowing long comments and user anonymity, encourages open and honest discussions (Alsinet et al., 2021; Ammari et al., 2018). Researchers utilize Reddit’s public data for qualitative and quantitative studies, benefiting from its organized subreddit format that simplifies data collection. This makes Reddit a valuable tool for examining online community interactions and behavior, providing rich datasets and insights into various aspects of online communication (Proferes et al., 2021).

Many studies demonstrate social media’s significant potential for mental health support and community engagement (De Choudhury and De, 2014; Rubya and Yarosh, 2017; Pretorius et al., 2020; Haimson et al., 2015; Wallace et al., 2017). They explore topics like mental health discussions on Reddit, video-mediated peer support, digital interventions, gender transition support, and support for older adults. Gauthier (2020) investigates online sobriety communities, while Arif et al. (2018) and Tsou et al. (2015) highlight social media analytics in crisis management and information operations. These studies collectively underscore social media’s diverse impact, emphasizing the importance of advanced topic modelling for deeper insights.

However, the vast and varied nature of social media data presents significant challenges for qualitative researchers (Stieglitz et al., 2020). Many qualitative researchers lack programming or data science expertise, making the adoption of advanced topic modelling tools particularly challenging. While they find machine learning techniques both empowering and exciting, appreciating the novel insights these tools can provide, they also harbor reservations about the trustworthiness of machine-generated results and the potential for missing nuanced content. Studies such as those by Lochmiller (2021) and Baden et al. (2022) highlight the dual perspectives of excitement and skepticism researchers feel towards integrating computational methods into their workflows. Additionally, research by Burgess and Bruns (2012), Lewis et al. (2013), and DiMaggio et al. (2013) underscores the critical need for intuitive, reliable, and ethically sound computational tools. These tools must address the complexities of social media, bridge the gap for researchers without technical expertise, and respect the methodological foundations of qualitative research.

Traditional topic modelling methods such as LDA (Blei et al., 2003), Biterm (Yan et al., 2013), NMF (Lee and Seung, 1999) often fall short in capturing the nuanced meanings and contextual usage of words, necessitating extensive data cleaning and pre-processing, which makes them labor-intensive (Xu et al., 2015; Hong and Davison, 2010; Yan et al., 2013; Mazarura and De Waal, 2016; Zou and Song, 2016). These methods rely on predefined topic numbers and treat words independently, leading to the loss of important contextual information and challenges with short texts like tweets or Reddit comments. Additionally, their effectiveness can be highly sensitive to hyperparameter settings, and they often require substantial computational resources. Conversely, Large Language Model (LLM) based techniques, such as BERTopic (Grootendorst, 2022), Top2Vec (Angelov, 2020) and GPT-3 (Brown et al., 2020), offer significant improvements. These models provide a more advanced understanding of text with minimal pre-processing and adapt efficiently to various contexts (Devlin et al., 2018; Brown et al., 2020). LLMs are leading in topic modelling due to their user-friendliness and advanced capabilities (Mu et al., 2024). Recent research highlights their effectiveness in zero-shot text summarization, achieving near-human performance, suggesting their potential for generating insightful topics (Zhang et al., 2024).

LDA and NMF are widely used for topic modelling in social media data analysis, but they have several limitations. LDA assumes a known number of topics and that words are generated independently given the topic, which can lead to loss of context and struggles with short texts like tweets or Reddit comments (Blei et al., 2003; O’Callaghan et al., 2015). Additionally, LDA’s performance is sensitive to hyperparameters and requires substantial computational resources (Wallach et al., 2009). NMF, while useful for producing sparse, interpretable results, is sensitive to matrix initialization and requires careful tuning, and it does not inherently capture probabilistic topic distributions (Lee and Seung, 1999, 2001). Studies show mixed results for these methods on social media data, highlighting the need for better algorithms to gain more accurate and insightful analyses (O’Callaghan et al., 2015).

Comparative studies, such as those by Egger and Yu (2022) demonstrated that BERTopic outperforms LDA, NMF, and Top2Vec in capturing semantic nuances and contextual information particularly on Twitter. Consequently, we incorporated BERTopic into our study to validate its effectiveness in enhancing our ability to extract nuanced insights and understand the complexities of social media data, specifically on Reddit. This approach is particularly beneficial for qualitative researchers from a Human-Computer Interaction (HCI) perspective, as it addresses the need for more intuitive and contextually rich analysis tools.

The Computational Thematic Analysis (CTA) Toolkit, developed by Gauthier and Wallace (2022), integrates qualitative thematic analysis and computational methods. We chose the CTA Toolkit as our experimental platform because its functionality aligned well with our research requirements; it provides a visual interface for qualitative analysis of social media data using computational techniques like topic modelling. We extended the toolkit by adding BERTopic as a topic modelling option, which allowed us to directly compare it to LDA and NMF under a single interface. We now provide a brief overview of the CTA toolkit’s key features for data collection, data cleaning and filtering, and modelling before describing how we integrated BERTopic.

The data collection module automates the retrieval and organization of data from online platforms like Twitter and Reddit. Researchers import data by defining parameters such as time periods and specific communities. The toolkit groups submissions and their corresponding comments into cohesive discussions, ensuring that the data is organized and ready for subsequent analysis. To address the ethical complexities inherent in online data collection, the toolkit prompts researchers to consider the ethical implications of their work before data is imported for analysis. These prompts guide researchers through considerations such as community consent and the handling of deleted posts.

The data cleaning and filtering module then employs natural language processing (NLP) tools, including NLTK and spaCy, to tokenize, stem, and lemmatize the text as it is imported. It also tags parts of speech and identifies stop words, facilitating a detailed and precise cleaning process. The toolkit enhances transparency by displaying a summary of the tokens and filtering rules applied, allowing researchers to see the impact of these steps on their data. This feature is essential for maintaining the integrity and reproducibility of the research.

The modeling and sampling module (Figure 1) allows researchers to interactively select and review data models, ensuring that the most relevant and representative data is used in the thematic analysis. This interactive approach supports the iterative nature of qualitative research, allowing researchers to refine their models based on emerging insights. To identify latent patterns within large datasets the CTA toolkit uses unsupervised topic modeling techniques — LDA (Blei et al., 2003), biterm (Yan et al., 2013), NMF (Lee and Seung, 1999). The toolkit’s visual interface includes a chord diagram visualization of generated topic models. The outer ring represents the different topics identified in the dataset. Each segment is labeled with a topic number (e.g., Topic 1, Topic 2, etc.). Alongside each topic label, there are word clouds showing the most relevant terms for each topic. The inner lines connecting different segments represent the number of overlapping documents between topics. Thicker lines imply more shared documents between the connected topics.

We integrated BERTopic within the CTA Toolkit’s pipeline. This work posed several challenges, primarily due to the inherent differences between BERTopic’s LLM-based processing requirements and the existing bag-of-words architecture. Additionally, BERTopic’s reliance on transformer models demands higher computational resources, including increased memory and processing power. For BERTopic, we leveraged the device’s built-in GPU to enhance processing efficiency. Moreover, the toolkit included filters and preprocessing steps tailored for bag-of-words models, which were not suitable for BERTopic’s needs. The preprocessing step involved default filters which eliminate terms with TF-IDF values less than 75%. This approach aimed to retain only the most significant terms; however, it led to the loss of almost 80% of the data, which proved to be detrimental to the analysis. Therefore, we decided to eliminate this filtering step for BERTopic to preserve the integrity and comprehensiveness of the data.

To understand the specific needs and challenges faced by qualitative researchers, we conducted a series of interviews with experts. We conducted interviews to identify researchers’ requirements and gather insights into their experiences with traditional topic modelling techniques, including manual coding methods, or using proprietary software, and their impressions of modern unsupervised topic modelling techniques like BERTopic.

We interviewed 12 participants, each an expert in qualitative research with distinct interests and varying levels of experience in their respective fields. We asked each participant to identify a subreddit of interest, downloaded the archived data, and loaded it into the Computational Thematic Analysis Toolkit. The datasets spanned a wide range of domains, including gaming, technology, COVID-19, health, environment, and politics.

We then asked participants to reflect on their data set through three unsupervised topic models: LDA, NMF, and BERTopic. The three topic modelling techniques were selected to reflect different conceptual approaches. LDA is a probabilistic approach which is widely used in the literature (e.g. (Grimmer, 2010; Hoffman et al., 2010; Teh et al., 2006)). NMF offers computational efficiency and a matrix-based perspective that is fast and effective for large datasets (Wu et al., 2008). BERTopic leverages cutting-edge capabilities of language models to provide deep, contextually rich topic modelling (Grootendorst, 2022). We asked participants to examine their data to help explain how the topic models might be useful.

This study has undergone a thorough review process and received ethics clearance from our Research Ethics Board (REB #46062) from University. The review process involved a detailed examination of the study’s methodology, participant recruitment procedures, data collection methods, and potential risks to ensure compliance with ethical standards.

We used a standard procedure to create models for each subreddit, following the same protocol for LDA, NMF, and BERTopic. This process involved performing tokenization, which were then converted into formats suitable for each model: a bag-of-words corpus for LDA, a TF-IDF matrix for NMF, and dense vector embeddings for BERTopic. Each model was configured according to its specific requirements, with hyperparameters and settings tailored to optimize performance, and kept to default settings, as detailed in the following sections.

We began by merging submissions and comments from JSON files (RS_<filename>.json and RC_<filename>.json) in a specified subreddit directory. Submissions and comments are merged into threads using their IDs, creating a cohesive narrative for each discussion. Threads are saved as JSON for analysis. We then tokenized each thread with gensim’s simple_preprocess, and stored tokens in a dictionary. Stopwords and irrelevant terms are removed using NLTK, while bigrams are identified with gensim.

LDA Implementation: The LDA model was implemented using the gensim library. For tokenization and preprocessing, gensim handled tokenization and bigrams, NLTK removed stopwords, and spaCy performed lemmatization. A dictionary mapped unique words to IDs, and token sets were converted to a bag-of-words corpus. Hyperparameters $\alpha$ (symmetric) and $\eta$ (auto) were chosen to balance topic distribution.

NMF Implementation: The NMF model was implemented using the scikit-learn library. Token sets were converted into space-separated strings for the TF-IDF vectorizer. The vectorizer transformed text into a TF-IDF matrix, which was saved and fitted to the NMF model, with a set random state for reproducibility.

BERTopic Implementation: BERTopic used the sentence-transformers library for embeddings, umap-learn for dimensionality reduction, and hdbscan for clustering. The bertopic library integrated these components, facilitating effective topic modelling analysis.

The number of topics (num_topics) for LDA and NMF were chosen based on the maximum topic coherence value observed from generating topic models ranging from 5 to 50 topics, in increments of 5. To determine an optimal value BERTopic, we generated 11 topic models and chose the median value. In doing so, we observed that the mean and median values for nr_topics were almost in conjunction, differing by average standard deviation between the median and average values being $2.09\pm 1.37$ across 12 datasets.

BERTopic consistently outperformed LDA and NMF in terms of quantitative metrics, including topic coherence, topic diversity, and KL divergence. On the other hand, and as expected, NMF was the least computationally-demanding model and required the lowest amount of time to compute. LDA tended to lay in-between the other two models across measures. Participant preferences closely align with these findings. BERTopic was ranked first by 8 out of 12 participants (67%), LDA by 3 out of 12 participants (25%), and NMF by 1 out of 12 participants (8%). A Friedman test revealed that these differences were statistically significant, $\chi^{2}_{(2)}=108$, $p<0.001$. Post-hoc analysis using the Nemenyi test showed significant differences between BERTopic and LDA ($p<0.05$), BERTopic and NMF ($p<0.01$), but not between LDA and NMF ($p>0.05$).

We first present these quantitative results in detail. Then, in presenting our analysis of interviews with researchers, we highlight rationale for these preferences alongside practical concerns for each topic modelling technique.

For LDA, the execution times range from a minimum of 200 seconds to a maximum of 3178 seconds, with a mean time of 1013 seconds. NMF exhibits a much lower range of execution times from 51 seconds to 836 seconds, with a mean time of 149 seconds. BERTopic shows a range of execution times between 153 seconds and 2891 seconds, with a mean time of 771 seconds. Table 7 compares the execution time of these three topic modelling methods.

A one-way ANOVA revealed a significant effect of the topic modelling method on execution time, $F_{2,33}=5.41$, $p=.009$, $\eta^{2}=.247$. To explore these differences further, a Tukey HSD post hoc test was conducted. The Tukey HSD test indicated that the mean execution time for LDA ($M=1013$) was significantly higher than NMF ($M=149$), with a mean difference of 863.88 (95% CI, 199.46 to 1528.29), $p=.007$. Additionally, NMF was significantly lower than BERTopic ($M=771$), with a mean difference of 621.200 (95% CI, -43.214 to 1285.61), $p=.013$. However, there was no significant difference between LDA and BERTopic, with a mean difference of 242.68 (95% CI, -421.73 to 907.095), $p=.905$.

We now discuss the results of our comparative analysis, evaluating the strengths and weaknesses of BERTopic, NMF, and LDA, and exploring their implications for topic modeling. We will examine why BERTopic outperformed other models, the trade-offs involved in using each technique, and the considerations for choosing the most suitable model and evaluation metric for topic modeling.

Our study also uncovered several important limitations that highlight areas for improvement. Researchers highlighted the need for simplified interfaces and better visualizations, such as interactive chord diagrams and hierarchical representations, to manage large datasets. They also suggested future features, including a more in-depth search function, to further improve the design and guide the next phase of our project.

Optimizing LDA and NMF Models: Future research should focus on optimizing parameters and evaluation metrics, especially the criteria for determining the number of topics. While topic coherence proved to be a valuable metric, relying solely on it for model evaluation posed challenges. This was evident when we used coherence to determine the number of topics for LDA and NMF, potentially limiting their performance and applicability. The presence of irrelevant symbols and mathematical characters in the topics highlighted a need for more precise and automated data cleaning processes. Both LDA and NMF required extensive manual effort to clean the data accurately.

BERTopic’s Number of Topics: Participants often found the large number of topics generated by BERTopic to be overwhelming, which led some to prefer other models despite acknowledging BERTopic’s superior detail and depth. Exploring hierarchical topic models may provide more organized and interpretable results. However, those models would necessitate improved visualization techniques. Developing more intuitive and effective visual interfaces will help users better understand and interact with the topic models.

Generalizability: Ensuring the generalizability and practical utility of our findings requires a larger, more diverse participant base and longitudinal studies. Our study included a diverse group of researchers, but the sample size was relatively small, with only 12 participants. This limitation restricts the generalizability of our findings. A larger and more varied participant base would help validate and extend our conclusions, ensuring that the results are applicable across a broader spectrum of qualitative research contexts. Further, conducting longitudinal studies where researchers use the software over extended periods of time can provide deeper insights into its practical utility and areas for improvement.

Integrating BERTopic into the Computational Thematic Analysis (CTA) Toolkit significantly enhanced its capabilities. Researchers praised BERTopic for generating detailed, coherent topics, their ease of interpretation, and their utility in uncovering hidden relationships within data. Evaluation metrics showed BERTopic’s superiority in topic coherence and topic diversity. This enhancement improved qualitative data analysis efficiency and depth, combining advanced computational methods with an intuitive interface for comprehensive and logical data representation.

In reflecting on our interviews, we found that researchers prefer models that produce detailed, organized, and coherent topic clusters, making it easier to recognize relevant terms and understand related concepts. They value models that reveal unexpected yet significant relationships within the data, providing comprehensive and actionable insights. The ability to capture a wide range of nuances and generate a larger number of topics is particularly important for deep understanding of complex subjects. Our results demonstrate the potential LLM-based methods like BERTopic for supporting qualitative analysis of large data sets.