Mental Illness Classification on Social Media Texts using Deep Learning and Transfer Learning

Mental illness could be a sort of health condition that changes a person’s intellect, feelings, or behavior (or all three) and has been appeared to affect an individual’s physical health [28, 23]. Mental health issues including depression, schizophrenia, attention-deficit hyperactivity disorder (ADHD), autism spectrum disorder (ASD), etc., are highly prevalent today, and it is estimated that around 450 million people worldwide suffer from such problems[28].

To way better get the mental health conditions and provide care, early detection of mental health problems is a basic step. Different from the diagnosis of other chronic conditions that depend on research facility tests and measurements, mental illnesses are regularly diagnosed based on an individual’s self-report to particular surveys planned for the detection of specific patterns of feelings or social interactions [18].

Amid these uncertain times when COVID-19 torments the world, many people have indicated clinical anxiety or depression. This could be due to lockdown, limited social activities, a higher unemployment rate, economic depression, and fatigue related to work. American Foundation for Suicide Anticipation reported that individuals encounter anxiety (53%) and sadness (51%) more regularly now as compared to the time before covid-19 widespread.

Within the past decade, social media has changed social interaction. Along with sharing data and news, individuals effectively communicate their day-to-day activities, experiences, hopes, emotions, etc., and generate tons of data online. This textual data gives information that can be utilized to design systems to predict people’s mental health. Moreover, the current limited social interaction state has forced people to express their thoughts on social media. It gives people an open stage to share their opinions with others numerous times attempt to find assistance [25].

A previous study explored the application of Machine Learning (ML) techniques in mental health [31]. They reviewed literature by grouping them into four main application domains, such as detection and diagnosis (ii) prognosis, treatment and support, (iii) public health applications, and (iv) research and clinical administration. Another study explored the emerging area of application of DL techniques in psychiatry. They focused on DL by embedding semantically interpretable computational models of brain dynamics or behavior into a statistical machine learning context [15].

This study uses reddit.com¹¹1https://www.reddit.com/ Last visited: 23-01-2022 user data proposed by Murarka and Radhakrishnan [25] to determine mental illnesses, see sample of dataset instances in Table 1. We applied traditional machine learning, deep learning, and transfer learning approaches to automatically detect mental disorders in social media texts. Our extensive experiments demonstrate that machine learning, deep learning, and transfer learning techniques have the potential to complement clinical procedures in the prediction of mental health between two classes of individuals: those seeking help online and those who are unaware of their condition.

The rest of the paper is organized as follows: section 2 describes the studies on mental illness in literature. Section 3 explain the problem and gives dataset insights. Section 4 give details of methodology applied to detect mental disorders. Section 5 presents results and their analysis. 6 concludes the paper with possible future work.

In recent years, people have been using social media to communicate and seek advice on mental health issues. This has motivated researchers to take the information and apply a variety of NLP and ML approaches to help individuals who may want assistance. Initially, many researchers have focused on Twitter text [27, 10, 12], later on the focus has shifted on Reddit platform [21, 17, 10, 34].

A wide range of approaches has been applied to mental health text analysis, from traditional ML to advanced DP. According to [12], they employed character-level language models to see how probable a user with mental health concerns would create a series of characters. [10] determined different types of mental health disorders by applying neural MTL, regression, and multi-layer perceptron single-task learning (STL) models. [1] trained the SVMs to distinguish 200 text messages into two classes: ”ADHD or not.” The most crucial step was the elimination of the acronym ADHD from the messages before learning, and further information concerning attention disorders was removed from the texts. The goal was to see how well the SVMS learns when keywords and even semantically relevant material are unavailable.

Deep feedforward neural network has outperformed typical ML models in a variety of data mining tasks [9, 7], and it has been used in the study of clinical and genetic data to predict mental health disorders. To diagnose depression, [27] used word embeddings in combination with a range of neural network models such as CNNs and RNNs. To conduct binary classification on mental health textual posts, [17] used Feed Forward Neural Networks, CNNs, traditional machine learning such as SVMs, and Linear classifiers. [30] detected depression, ADHD, anxiety, and other types of mental illnesses by training a binary classifier for each disease with Hierarchical Attention Networks. The most recent work on this was a CNN-based classification model [21]. In which, the team trained a separate binary classifier for each type of mental disorder to conduct the detection. Reference [19] found the potential factors to influence a person’s mental health during the Covid-19 pandemic by applying machine learning classifiers such as Naive Bayes, Logistic Regression, Support Vector Machine, Decision Tree, Random Forest, and Gradient Boosting. They have also presented an analysis of the feature selection technique LIME.

In today’s research world, transfer learning is extremely important. By using several types of transformers, researchers attempt to acquire greater accuracy and performance in each research study. The authors of [25] examined three approaches for identifying and diagnosing mental illness on Reddit, including LSTM, BERT, and RoBERTa. Among these three methods, RoBERTa outperformed. Reference [14] employed RoBERTa to categorize COVID-19-related informative tweets, and their method yielded the best results.

Murarka et al. [25] developed a benchmark multi-class dataset from Reddit social media platform for mental illnesses detection.

This section describes our machine learning, deep learning, and transfer learning models applied for multi-class mental illness problems.

Our machine learning, deep learning, and transfer learning results are documented in Table 3. In this Table, “ML Algorithms” indicates traditional machine learning algorithms. The “LinearSVC” indicates to Linear Support Vector Classifier, “LR” indicates to Logistic Regression, “NB” indicates to Naive Bayes, “RF” indicates to Random Forest classifier. The “DL Algorithms” indicates deep learning algorithms used in this study such as GNN, Bi-GNN, CNN, LSTM, and Bi-LSTM. The “TL Algorithms” refers to pre-trained transfer learning algorithms applied to evaluate Reddit corpus, i.e., BERT, XLNet, and RoBERTa.

Our pre-trained transfer learning RoBERTa model outperformed other traditional machine learning and deep learning algorithms with an accuracy score of 0.80, which is quite good on this challenging multi-class mental illness detection problem. The performance of the XLNet model is close to the RoBERTa difference of 0.01.

The overall best results using the deep learning algorithm were on Bi-LSTM. This shows that Bi-LSTM is the most suitable algorithm to detect mental illness among deep learning algorithms. Interestingly, Bi-LSTM results are similar to the ones obtained with BERT. This indicates that the BERT model performs equally well on multi-class mental illness detection problems as advanced pre-trained transfer learning models. The accuracy scores of GRU, Bi-GRU, and CNN are not very high, highlighting the fact that multi-class mental illness detection on Reddit post’s text is a challenging task.

Using traditional machine learning algorithms, overall, best results are obtained using combination of word n-grams when length of N was minimum = 1, maximum = 3 (Accuracy = 0.78, F1 = 0.67). This shows that combinations of word grams (length of N = 1-3) were the most suitable features when we trained the model on the Reddit social media platform.

The RoBERTa model’s detailed class-wise results are shown in Table 4. The first surprising outcome is the model’s ability to recognize non-illness-related postings with high accuracy. The model can categorize the none class with an F1 score of more than 0.98. That gives us the indication that when it comes to detecting mental illness on social media, this model will suffer from very few false positives.

The two highest performing classes among mental disorder are adhd and ptsd, while on the contrary the two poorest performing classes are depression and anxiety.

There are number of factors contribute to the result of depression and anxiety classes. depression and anxiety have the fewest average number of tokens (words) per post among all classes.

For example, when comparing depression and ptsd classes, depression class have around 53% fewer textual content. Furthermore, researchers demonstrate that depression is frequently associated with another mental disorder, and our findings support this. In 12% percent of ptsd posts, 12% percent of anxiety posts, and 31% of bipolar posts, the term depression appears. Anxiety is present in 12% of adhd posts, 14% of bipolar posts, and 20% of ptsd posts, respectively. This means that, unlike with the rest of the diseases, the model is unable to place a high value on the mention of these class names, making the categorization of these two labels challenging.

Figure 2 shows the confusion matrix for RoBERTa model. The terms depression and anxiety are mentioned in more instances in the adhd and ptsd classes than the name these class themselves. One could expect poor outcomes as a result of this, but these classes outperform all others. This exhibits the actual potential of our approach since it does not depend solely on the mention of class names in the post but also has a deep awareness of the post’s context.

The present Covid-19 outbreak and globally forced isolation are our primary motivations for multi-class mental illness detection efforts. We feel that social media platforms have become the most widely used communication medium for individuals, allowing them to express themselves without fear of being judged. We applied state-of-the-art traditional machine learning, deep learning, and transfer learning-based methods for multi-class mental illness detection problem. The best results (see Table 3) obtained using pre-trained RoBERTa transfer learning model (accuracy = 0.83, F1-score = 0.83).

In the future, we plan to develop a multi-label dataset for mental illness problems, which would be more reflective of the situation than a multi-class dataset, as a post can have more than one mental disease instead of one per post, i.e., depression, anxiety. We can also use the data augmentation technique on top of this existing mental health data [8]. We plan to apply other transfer learning-based models such as DistilBERT, etc., in the future. An ensemble modeling would be considered to improve classification performance.

The work was done with support from the Mexican Government through the grant A1-S-47854 of the CONACYT, Mexico and grants 20211784, 20211884, 20211178 of the Secretaría de Investigación y Posgrado of the Instituto Politécnico Nacional, Mexico, and grants of PAPIIT-UNAM project TA101722. The authors utilize the computing resources brought to them by the CONACYT through the Plataforma de Aprendizaje Profundo para Tecnologías del Lenguaje of the Laboratorio de Supercómputo of the INAOE, Mexico.