ViNMT: Neural Machine Translation Tookit

Neural Machine Translation is a fully-automated translation based on neural network. (Cho et al., 2014) suggest a new architectural and called Sequence-to-Sequence (Seq2Seq) model. They apply memory unit such as Long Short-Term Memory (LSTM) or Gated Recurrent Units (GRU) to surmount the exploding or vanishing gradient problem in recurrent networks. The architecture includes an encoder and decoder. The one Encoder uses to present the sentence in the source language with n tokens X = ($x_{1}$; $x_{2}$; …; $x_{n}$) and a decoder to generate the predicted sentence in the target language with m tokens Y = ($y_{1}$; $y_{2}$; …; $y_{m}$). The model evaluates the conditional probability $P(y_{j}\mid Y_{<j},X)$ generating output sentence Y when the input sequence X is given:

In the Seq2seq model, the global attentions mechanism (Luong et al., 2015) is considered to compute alignment attention from source sentences to corresponding target sentences. However, due to the Seq2seq model computing the probability sequentially, they have limited parallelization in the training process.

To solve the parallelization problem, the transformer architecture for machine translation is mentioned for the first time by (Vaswani et al., 2017). They are noted to be highly parallelizable as well as better in translating long sentences. Instead of using GRU or LSTM units to encode source sentences sequentially, they use encoder layers that help the encoder look at other words in the input sentence as it encodes a specific word. This architecture allows to train much faster and gives a better quality compared to RNN architecture. The self-attention mechanism as the following:

Here, K (key), Q (query), V (value) is the outputs of the encoder or decoder to present tokens in the input sentence, and d is the size of the input. And then, the Transformer is trained to optimize parameters $\theta$ by minimizing the maximum likelihood of the sentences pairs:

where T is the number of sentence pairs in the multilingual corpus. We use transformer-based for all our experiments.

Multilingual neural machine translation (MNMT) system can translate many language pairs, which has been useful in improving translation quality as a result of translation knowledge transfer (transfer learning), even low-resource, zeros-shot issues. (MNMT) has some strategy to translate between many languages:

• •

  (1) Many to many (Ha et al., 2016): from many sources to many target languages;

• •

  (2) Many to one (Gu et al., 2019): from many sources to a target language;

• •

  (3) One to many (Wang et al., 2018): from a sources to many target languages;

Our motivation is to improve low-resource translation tasks and focus on translating from many languages to one language, so we systems are the same as the case (2). In a MNMT system from many to one, the objective function uses maximum likelihood estimation on the whole parallel pairs:

Where M is the number of languages and K is the total number of sentences in languages m. Of course, the vocabulary of the source side is mixed from all source language:

Code-mixed Language: The benefit of shared vocab in MNMT has been shown by Gu et al. (2019) that if the languages shared the same alphabet and had many similar words, MNMT system can get many advantages of common features of languages. In order to do that they may share the same subwords in all language pairs. This significantly reduces the number of rare words in the MT systems. In our experiments, we choose English, Chinese, Khmer, and Lao are source languages and hope that they can share many tokens with each other to translate into Vietnamese.

In order to support the technical system above, our code base is designed from an Object-Oriented point of view, with interchangeable base classes following PyTorch’s nn.Module class. This modular approach allows easy and simple modification when experimenting or upgrading specific parts of the system, since in most cases we can simply inherit the base class that needed to be changed and use our new class as an initiation argument.

Our system revolves around three main important classes:

• •

  the Layer class, which are layers that directly involve in neural network’s training and processing. As such, they are exclusively contained by the neural network modules in the system such as the Encoder and Decoder modules.

• •

  the Module class, which make up the bulk of our coding base. A Module represents a specific duty to be filled, for example, our Transformer base model currently contain a Loader object for converting text data to numeric form and vice versa; a pair of Encoder-Decoder objects for the main translation process, and an additional DecoderStrategy object to support the model during autoregressive inference.

• •

  the Model class, which contains all the respective modules required when training and inferring. This packaging allows (1) the highest level of user interference into translating process without the need of understanding supporting codes such as the \bin or \utils code; and (2) simple usage during TorchScript serving, as the converted object would be a stand-alone module that doesn’t have to worry about re-implementing specific non-neural modules.

Using an example to demonstrate the modularity of our system, assuming that we have a new Decoder object to put into our original model, all we have to do is to replace the corresponding decoder_cls argument within the constructor function of the Model class. This can be easily accomplished by writing a lambda function in the model name list specified within \models \__init__.py

In addition, our code base contains several additional code and resources: \bin contains command line scripts to run the system, \config contains basic configuration files which contain model hyperparameters that will be preserved during runs, \web contains serving scripts and resources, \utils contains extra support functions that don’t fit other folders.

We use the standard beam search algorithm to generate texts on the target side. The baseline configuration includes beam size $k=4$, length penalty $alpha=0.6$ and maximum number of decode steps is 128. There are some techniques we can use to speed up the inference process and reduce memory usage:

We want the deployment and training process are independent as much as possible, the models can be deployed on any environment besides Python. Therefore, we use TorchScript which is a way to create serializable and optimize models from PyTorch code. Any TorchScript program can be saved from a Python process and loaded in a process where there is no Python dependency. It transitions a model from a pure Python program to a TorchScript program that can be run independently from Python and then export the model via TorchScript to a production environment where Python programs may be disadvantageous for performance and multi-threading reasons.

We crawled data from the news domain for four language pairs English-Vietnamese, Chinese-Vietnamese, Laos-Vietnamese and Khmer-Vietnamese. The details of those datasets are described in Table 1.

All parallel texts were tokenized and truncated using sentencepiece scripts, and then they are applied to Sennrich’s BPE (Sennrich et al., 2016). We explore 32K operators are learned to generate BPE codes for all languages.

For Vietnamese, we only apply Moses’s scripts for tokenization and true-casing.

We implement our NMT systems from zeros-base to train all our experiments. The same settings are used for all experiments.

We trained our Transformer model using the number of encoder 12, decoder layers are 6, 8 head is used, $d_{model}$ is 512, dropout value is 0.1, batch size of 64, learning rate value is 0.4 with the aid of Adam optimizer. The learning rate has warmup updates by 8000 steps and label smoothing value is 0.1.

We evaluate the quality of two systems (1) bilingual system, (2) multilingual system.

The results of our experiments are shown in table 2 and table 3.