OSN-MDAD: Machine Translation Dataset for Arabic Multi-Dialectal Conversations on Online Social Media.

In today’s digital and global age, $\approx$ 52% of internet users access and post information in their own language and dialect. Seventy-five percent of world’s population do not speak English, and French, Arabic and Spanish are among the most spoken languages after English on social media. In order to accurately analyze social behaviors of social users who speak different languages, we need to understand the expressions of the languages they use to share and communicate their opinions and thoughts [1]. One current approach to addressing the multilingual aspect on OSNs is to use a translation approach between languages [2, 3, 4]. The challenge with the translation approach is that it heavily depends on the quality of the parallel translation datasets [5, 6, 7]. Low-quality datasets lead to low-quality models and vice versa [5, 3, 6, 8]. Hence, a careful content translation methodology need to be considered when crafting translation datasets.

Th advent of social media has revolutionized the language that people use for communication; the informal nature of conversations and communication has become the new norm on online social media platforms like Twitter. This is plainly evident in the Arabic language, where urban dialects have become the dominant means of communication instead of Modern Standard Arabic (MSA), which is different from dialectal Arabic in terms of morphology, lexicons, and expressions. As a result, existing translation systems designed for MSA fail to work well with Arabic dialects [9, 10]. In light of this, it is necessary to adapt to the informal nature of communication on OSNs by developing translation systems that can effectively handle the various Arabic dialects in addition to MSA. Unlike MSA, for which considerable progress has been made in translation systems such as Google Translator, little effort has been exerted to include Arabic dialects in machine translation systems. The main limitation contributing to the immaturity of Arabic dialect translation systems is the insufficiency of data resources and datasets [10]. Only few efforts [11] have been made to build datasets to support the Arabic multidialect translation systems ; however, the construction design of those dataset is not OSN-cultural language friendly, which might limit the translation performance of Arabic multidialect systems on OSNs. The nature of conversations on OSNs involves short and unstructured messages. In addition to hashtags and short forms, slang expressions are commonly used on OSNs. With the informal language of day-to-day talks being transferred on OSNs, idiomatic expressions such as "it is a piece of cake" and "it is raining cats and dogs" are naturally used within messages. This has resulted in a new nature of contents with a mix of spoken and online language that the existing datasets have ignored. Therefore, this linguistic representational challenge must be be considered to utilize the online language in machine translation systems. It is well known that OSNs platforms (Twitter in our work case) are an open environment for conversations of different topics and domains; hence, domain-specific data resources (The Bible¹¹1 https://www.biblesociety.ma, ²²2 https://www.bible.com, [12]) introduce a major limitation to the capability of translation models [13]. That being said, the domain-specific approach in designing translation datasets narrows the spectrum of vocabularies and expressions to be solely specific to the particular domains. This limitation, in turn, has raised other challenges, such as out-of-vocabulary (OOV) and out-of-domain test data, which therefore would affect the generalizability or adaptability of such datasets to open contents existing on OSNs.

In addition to the insufficiency of Arabic multidialectal data resources [11], the protocol followed for dataset crafting and content translation in the existing datasets is not clear. The criteria and guidelines for choosing source sentences, translating content, qualifying translators, and evaluating the quality of target translations are ambiguous and are sometimes absent [14, 15, 16]. In this paper, we aim to overcome the abovementioned challenges and propose an OSN-based multidialect Arabic dataset (OSN-MDAD). OSN-MDAD is created by contextually translating English tweets into four main urban dialects: Gulf, Yemeni, Iraqi, and Levantine/Shami. We do not accept word-to-word translation; instead, we propose a contextual translation of tweets from English to multidialect Arabic. We adopt the content localization approach in our translation. The proposed dataset is designed and built to work in the wild under the uncontrolled conditions of content sharing on social media. To the best of our knowledge, we are the first to construct a multidialectal Arabic-English parallel dataset for machine translation that is optimized for the informal language used on social media. Moreover, we are the first to create a guideline framework that is applicable for translating not only a foreign language into different Arabic dialects , but also from any language to any other language and dialect.

We summarize the contributions of this work as follows:

• •

  Design a guideline framework for constructing online social media (OSN)-based contextual translation datasets.

• •

  Construct a benchmark translation dataset by translating tweets from English into multidialect Arabic (i.e., Gulf, Levantine/Shami, Iraqi, and Yemeni).

• •

  Develop four OSN-based English-to-Arabic deep learning-based translation models for four Arabic dialects: Gulf, Levantine/Shami, Iraqi, and Yemeni.

The rest of the paper is organized as follows. Section 2 presents in detail the related works in the literature and our proposed methodology is presented in Section 3. Section 4 explains the experimental design and evaluation protocol followed in this work followed by an illustration and discussion of the results and analysis. Finally, in Section 5 we summarize our proposed work and findings and discuss future directions.

The phenomenon of urban dialects (i.e. informal conversations) has become the dominant mean of communication on online social media platforms such as Twitter. This phenomenon is plainly evident in the Arabic language, where the informal dialects have become more common than Modern Standard Arabic (MSA), for communication among social users on OSNs. Modern Standard Arabic (MSA) is different from the dialectal Arabic in terms of morphology, lexicons, and expressions. As a result, existing translation systems designed for MSA fail to work well with Arabic dialects [17]. Considering this, it is necessary to adapt to the informal nature of communication on OSNs by developing translation systems that can effectively handle the multidialect aspect of Arabic language, besides the MSA. Unlike MSA that shows advanced progress in translation systems such as Google Translator, little efforts have been exerted to utilize Arabic dialects for machine translation systems [11]. The main limitation contributing to the immaturity of Arabic dialect translation systems is the insufficiency of data resources and datasets [17]. Only a few efforts have been made to create datasets for dialectal Arabic. The Bible was among the first data resources that was translated into Arabic Moroccan 1 and Arabic Tunisian dialects 2. In the context of online data, Zbib et al. [15] created Egyptian-to-English and Levantine-to-English dataset collected from dialectal Arabic weblogs and online user groups ³³3https://catalog.ldc.upenn.edu/LDC2012T09. The translation from Arabic to English was carried out through crowdsourcing technique on Mechanical Turk by Arabic users. Bouamor et al. [14] used a subset of 2K sentences from Zbib’s dataset and extended the translation to Palestinian, Syrian, and Tunisian dialects. The authors asked native dialectal Arabic speakers to translate Egyptian sentences into their own native dialects. Later, Bouamor et al. [16] created parallel-phrase and parallel-sentence datasets that cover various city-level Arabic dialects. The corpus was created by translating a subset of phrases and sentences taken from the Basic Traveling Expression Corpus (BTEC) [12]. Although those Arabic dialect datasets were created as an attempt to support the Arabic multidialect translation systems, they reveal a number of issues that might limit the translation performance of Arabic multidialect systems on online social networks (OSNs). Nine observed reasons support this claim: (1) the number of sentences translated into each dialect is small; some datasets comprise as few as 2K sentences per dialect [16]; (2) translation was done by non-professional translators, and it is unknown whether the translators were native speakers of an Arabic dialect and possessed sufficient/native English proficiency to ensure accurate translation [15]; (3) translators were not screened regarding whether they were familiar with informal and slang English and with OSN language (i.e., whether they were active on social media); (4) some datasets are domain dependent 1, 2, [12, 18] and might not be adapted to other domains; (5) idiomatic expressions were not taken into consideration; (6) most of the datasets did not consider identical sentence translation into different dialects; (7) ONS cultural expressions were not included in the datasets and were not translated; (8) code-borrowing terms were not taken into consideration; and (9) the translation criteria and guidelines for most of the datasets have some ambiguity or are even absent [14, 15, 16]. To overcome the above limitations and to complement the existing datasets of dialectal Arabic for machine translation on social media, we propose an OSN-based multidialect Arabic dataset (OSN-MDAD). OSN-MDAD is created by contextually translating English tweets into four main urban dialects: Gulf, Yemeni, Iraqi, and Levantine/Shami. To the best of our knowledge, we are the first to construct a multidialectal Arabic-English dataset for machine translation that is optimized for the informal language used on social media. Moreover, we are the first to create a guideline framework that is applicable for translating not only a foreign language into different Arabic dialects but also for translating any language to any other language and dialect.

Machine translation models trained on MSA data are not capable of performing well in translating from and into Arabic dialects [19]. Hence, research efforts have been made to enrich such models to explicitly handle the translation of Arabic dialects. The very first efforts toward building an Arabic dialectal machine translation were based on statistical learning approach [20, 21]. Some studies adopted the approach of appending a dialect to MSA as a preprocessing step [20, 20, 22]. Other studies adapted MSA-to-English systems to dialectal data [21]. Recently, deep learning methods have become more dominant than statistical learning approaches, and neural machine translation (NMT) has been shown to outperform STM with state-of-the-art results [23]. NMT for Arabic dialects has not been extensively explored; however, some work on translating Arabic dialects using NMT has been recently introduced [24, 25, 9, 11]. Baniata et al. [26] designed a recurrent neural network-based encoder-decoder neural machine translation model that uses multitask learning with individual encoders for both MSA and dialects and with a shared decoder. Transformer neural networks have become an alternative to sequence-to-sequence neural networks for neural machine translation. Shapiro and Duh [27] trained transformer-based models for MSA, Egyptian, and Levantine dialects. Their results showed that training a model on multidialectal data can benefit the translation of unknown dialects. Sajjad et. al. [19] found that transformer based NMT models produced better translations when trained on large-scale datasets compared to small-sized datasets in the settings of training from scratch. On the other hand, their results revealed that using transfer learning through fine-tuning pretrained NMT models (i.e. on MSA) improved the learning of Arabic dialectal translations. In our work, we use transformer networks and transfer learning to train our NMT models using our proposed OSN-MDAD dataset.

Without a doubt, the informal and slang nature of conversations has become the dominant mode for communication on social media. Moreover, a variety of languages other than English are used in social talks and conversations. Accordingly, monitoring systems for online social behavior have to extend their ability to understand and analyze interactions expressed in different languages and even dialects. In response, we propose a framework that targets low-resource language (i.e., Arabic in this work). Considering the translation approach used to solve the multilingual challenges, we propose a translation dataset (OSN-MDAD) from English to multidialectal Arabic in an effort to contribute to the low-resource status [37] of multidialectal Arabic resources. Four different Arabic dialects were considered in this work: Gulf, Yemeni, Iraqi, and Levantine/Shami. OSN-MDAD dataset was then used for the purpose of building translation models -one for each dialect- from English to multidialectal Arabic optimized for OSN data. Our experimental results have illustrated the superior performance of our English-dialectal Arabic translations models trained using Transformer architecture -with transfer learning- compared to the performance yielded by the sequence-to-sequence with attention mechanism -from scratch-, using our proposed translation dataset. We believe that this study contributes to the research community by providing a high-quality benchmark Arabic multidialectal translation dataset that can be used by researchers to learn from and benchmark against and also extend upon in future related tasks. Another contribution of this work is the proposed guideline framework for constructing contextual translation datasets. Some of the instructions were customized to fit the purpose of this study; however, they could be customized to fit other problems of interest. The proposed guidelines were carefully studied, developed, and considered after rounds of expert consultations.

For future directions, an extension of additional Arabic dialects will be considered to expand the ability of systems to understand and analyze broader online Arabic content. This extension will assist in enriching the under-resourced status of the Arabic language in order to benefit many downstream tasks. Additionally, we are interested in investigating long sentence translation in the context of multiple Arabic dialects, as this work has only experimented with short texts. The benefits of translation systems that are able to handle both short and long sentences are great and will significantly reduce the expensive cost of building individual models for each case. While the BLUE evaluation metric provides the average performance measure of how well a machine translation model translates content, it is limited in providing insights into where the model struggles to produce fluent translations. Therefore, we plan to explore the evaluation area in order to study aspects such as linguistics to enhance the performance of machine translation models.