CMLFormer: A Dual Decoder Transformer with Switching Point Learning for Code-Mixed Language Modeling

We introduce a synchronous coupling of two decoders with the same shared encoder. In contrast to the vanilla Transformer architecture, we add a third attention layer to introduce inter-decoder cross-attention in each decoder block to mutually share the latent characteristics of each constituent language. Denoting the output hidden states from layer normalization after the encoder-decoder attention sub-layer of the base and mixing language decoder as $a^{b}_{l}$ and $a^{m}_{l}$ respectively, the decoder-decoder cross-attention is computed as,

The corresponding output attention scores $o_{l}^{b}$ and $o_{l}^{m}$ are then passed as input to the feed-forward and layer normalization sub-layers as in the vanilla Transformer. This allows each decoder to attend to and "peek" at the other decoder’s hidden states while decoding the input CM sentence into their corresponding language, helping it capture inter-block interactions and driving cross-lingual learning and alignment. The two decoders are thus fully synchronized as each requires the hidden states of the other in each block to compute its own.

We incorporate the widely used Masked Language Modeling Devlin et al. (2019) task as the base pre-training objective. Masked Language Modelling (MLM) was first introduced in BERT to mitigate the problem of traditional bidirectional language modelling, allowing the model to "see" the tokens being predicted. Given a token sequence $S=\{s_{1},s_{2},...,s_{N_{S}}\}$, a subset of tokens are randomly sampled denoted by $L_{mask}$, typically 15% of which 80% are replaced by $[MASK]$ token, 10% are replaced by a random token, and 10% remain unchanged. The objective of MLM is to predict the masked tokens by using the surrounding unmasked tokens in the input sequence as context. We pre-train on the MLM objective using all three $C$, $B$ and $M$ language sequences and average over the three cross-entropy losses $\mathcal{L}_{MLM_{c}}$, $\mathcal{L}_{MLM_{b}}$ and $\mathcal{L}_{MLM_{m}}$.

The multilingual setting of CM languages enables concepts to be represented by words from the base or mixing languages. It is necessary to model cross-lingual relationships between these word variants by learning rich contextual representations of the CM, base and mixing languages. We propose a new Bilingual Translated Sentence Prediction (BTSP) objective to capture these cross-lingual relationships. Given a CM token sequence $C$, we randomly sample another token sequence $S$ with 50% positive examples (25% of the time $S$ is $B$, 25% of the time it is $M$) and 50% negative examples (25% of the time it is a randomly selected sequence $B^{\prime}$ and the remaining 25% of the time it is a randomly selected sequence $M^{\prime}$ from the dataset). The objective of BTSP, formulated using binary cross-entropy as $\mathcal{L}_{BTSP}$, is to predict whether sequence $C$ is equivalent to the sampled sequence $S$.

To capture the inherent multilingual characteristics of CM languages, it is crucial to not only learn rich contextual representations for the CM, base and mixing languages but also model a three-way alignment among them. Given a CM token sequence $C$, and its corresponding translations in the base and mixing language translations $B$ and $M$ respectively, we introduce a multi-target Bilingual Language Translation (BiLTM) objective. The base and mixed language decoders are tasked to simultaneously generate translations of the CM sequence into its equivalent base language and mixing language forms $L_{B}$ and $L_{M}$, respectively. The objective of BiLTM can be formulated as the average over the autoregressive modeling losses $\mathcal{L}_{BiLTM_{b}}$ and $\mathcal{L}_{BiLTM_{m}}$ for each decoder.

Code-mixing enables words to be shared from its constituent languages, also known as lexical borrowing. These loan words can thus occur within the context of a base or mixing language sentence, with each having a different syntactic and lexical structure within its local language’s context. CM representations must capture this phenomenon to effectively represent the mixing of two languages and thus, be able to identify the language in which a word occurs from a given local context window.

We propose a Token Language Classification (TLC) objective $\mathcal{L}_{TLC}$ using binary cross-entropy to predict the language in which each token $w_{i}$ occurs from its given local context. We construct a concatenated sequence $S$ comprising $C$, $B$ and $M$ in a randomly shuffled order. This concatenation allows loan words to concurrently occur in two out of the three segments of $S$. Consequently, the model must rely solely on local context while predicting, enabling it to learn the usage of loan words across different languages and linguistic structures.

Switching points mark the locations where a language transition occurs between consecutive tokens in a code-mixed sequence. Modeling these transitions explicitly allows the encoder to capture the structural dynamics of language alternation, which are often critical to understanding the syntactic and semantic properties of code-mixed text.

Given a code-mixed sequence $C=\{c_{1},c_{2},\dots,c_{N_{C}}\}$ and a corresponding switching point label sequence $T=\{t_{1},t_{2},\dots,t_{N_{C}}\}$, where $t_{i}=1$ if a switch occurs between $c_{i}$ and $c_{i-1}$ and $0$ otherwise, the task is to predict the probability $p_{i}$ of a switch occurring at each token position. We treat the SPP objective $\mathcal{L}_{\mathrm{SPP}}$ as a token-level binary classification task, where the loss is computed using binary cross-entropy between the predicted switching probabilities $p_{i}$ and the ground truth labels $t_{i}$.

The Code Mixing Index (CMI) is a commonly used metric that quantifies the degree of code-mixing in a sentence by measuring the relative proportion of tokens across its constituent languages Das and Gambäck (2014). Given a sequence of $N$ tokens, with $P$ switching points and $N_{d}$ tokens from the base language, we adapt the CMI formulation for our objective as,

where $w_{n}$ and $w_{p}$ are weighting factors. This extension balances the contribution of language distribution and the frequency of transitions in the code-mixed sentence.

To encourage the encoder to model these sentence-level code-mixing dynamics, we introduce CMI Prediction as a regression objective. The encoder is tasked with predicting a scalar $\hat{c}_{C}$ for each input sequence $C$, with the ground truth $c_{C}$ computed directly from the labels, which is then optimized using Mean Squared Error (MSE). This task complements local objectives such as Switching Point Prediction and Token Language Classification by providing a global signal for code-mixing complexity, capturing both global mixing and local switching behavior.

CMLFormer is jointly trained on all the above pre-training objectives to minimise the overall training loss $\mathcal{L}_{total}$. This can be formulated as,

where $\alpha$, $\beta$, $\gamma$, $\eta$, $\zeta$, and $\delta$ are scaling constants.

CMLFormer is pre-trained on the L3Cube-HingCorpus Nayak and Joshi (2022), a large-scale Hindi-English code-mixed corpus in Roman script. It comprises 52.93M¹¹1Due to computational limitations and time constraints, it was not possible to augment and pre-train on all 52.3M sentences. Instead, we sampled 10,000 sentences for pre-training. sentences and 1.04B tokens from Twitter and reflects real-world code-mixing patterns. To support our auxiliary pre-training objectives, we augment the dataset by generating parallel translations of code-mixed Hinglish sentences into pure Hindi and English in Roman script and annotate each code-mixed sentence with its corresponding switching points. More details about the data augmentation can be found in Appendix A.1.

CMLFormer is evaluated on downstream tasks by detaching the decoders and fine-tuning just the encoder on an established Hinglish CM language classification benchmark, the HASOC 2021 Mandl et al. (2021) dataset for hate speech detection. It comprises 7,000 code-mixed Hinglish tweets distributed across 5,740 train and 1,348 test examples. We compare its performance against HingBERT, a code-mixed $\text{BERT}_{\text{base}}$ model for Hinglish with MLM as its only pre-training objective.²²2We pre-trained $\text{BERT}_{\text{base}}$ on our sample of the dataset to produce a comparable HingBERT to benchmark against.

Pre-training Complexity. We observe that CMLFormer converges quickly on most tasks before reaching a long stagnating tail. The lack of improvement beyond the first epoch could be attributed to the large number of pre-training objectives, increasing the complexity of the overall pre-training stage. The small size of the pre-training dataset also contributes to a lack of robust generalization across training examples.

We observe a steady decrease in the decoder-based losses $\mathcal{L}_{{BiLTM}_{b}}$ and $\mathcal{L}_{{BiLTM}_{m}}$ before stagnating while the encoder-based losses $\mathcal{L}_{{MLM}_{c}}$, $\mathcal{L}_{{MLM}_{b}}$ and $\mathcal{L}_{{MLM}_{m}}$ do not show much convergence before stagnating. This could hint towards the higher overall complexity of the encoder-based objectives compared to decoder-based ones. Additionally, the encoder is responsible for not only generalizing well to its own tasks but also learning rich representations that aid the decoder in autoregressively generating sequences.

Encoder Optimization. Since the encoder is optimized for multiple training objectives, it bears a heavier representational burden and faces a greater optimization challenge than each decoder, which is trained for a single task. Consequently, the encoder’s loss exhibits slower convergence during training. Due to its involvement in multiple training objectives, the base variant’s encoder might not be robust enough and could benefit from a larger model with a higher number of parameters. We will explore this in our future experiments, where we restrict our encoder to match the parameters in $\text{BERT}_{\text{large}}$ while retaining the same size of the decoder.