ChatGPT MT: Competitive for High- (but not Low-) Resource Languages

We evaluated ChatGPT’s (gpt-3.5-turbo) MT for our full language set. We compared with NLLB-MOE nllbteam2022language as our baseline, as it is the current state-of-the-art open-source MT model that covers such a wide variety of languages. NLLB is a discriminative transformer trained on supervised bi-text data (the traditional MT paradigm). We obtained scores for NLLB outputs of ENG$\rightarrow$X translation into 201 of the language varieties in our set (as reported by nllbteam2022language).

We used both zero- and five-shot prompts for ChatGPT MT. (See §2.3.) Previous studies hendy2023good; gao2023design; moslem2023adaptive; NEURIPS2020_1457c0d6; zhu2023multilingual suggest that few-shot prompts produce slightly (albeit not consistently) better translations. But zero-shot prompts are more convenient and affordable for users.

We also compare with results for subsets of our selected languages from two other MT engines. Google Translate API was an important baseline for our analysis because it is popular among end users. We also included it to represent commercial MT systems in our study. Because Google’s API does not support all 204 of the FLORES-200 languages, we obtained results only for the 115 non-English languages it supports.

Lastly, we obtained MT results from GPT-4, since it is a popular LLM and has been shown to outperform ChatGPT on MT jiao2023chatgpt; wang2023document. Because the cost of GPT-4 use exceeds that of ChatGPT by 1900%, our resources did not permit its evaluation on all 203 non-English languages. Instead we selected a 20-language subset by picking approximately every 10th language, with languages sorted by chrF++ differentials between ChatGPT and NLLB ($chrf_{GPT}-chrf_{NLLB}$). We chose this criterion in order to have 20 languages with a range of relative ChatGPT performance and a variety of resource levels. We used only five-shot prompts for GPT-4.

We conducted all LLM experiments with gpt-3.5-turbo (ChatGPT) and gpt-4-0613 (GPT-4). We used top_p $1$, temperature $0.3$, context_length $-1$, and max_tokens⁴⁴4Although some languages had higher token counts than others (see §3.4), we found that adjusting max_tokens had a minimal effect on MT performance. We thus decided to maintain the same value of max_tokens across all languages for experimental consistency. $500$.

To evaluate the outputs, we used:⁵⁵5We excluded learned MT metrics like COMET rei2020comet and BLEURT sellam2020bleurt, since they do not support many LRLs.

Previous works gao2023design; jiao2023chatgpt investigated LLM prompting to optimize MT performance. We adopt gao2023design’s recommended prompts for both zero- and few-shot MT (Table 1). We are interested in multiple $n$-shot prompt settings because, as mentioned in §2.1, they present different benefits to LLM users. We explored zero-shot (no in-context example), one-shot (1 example), and five-shot (5 examples). We employed both zero- and five-shot prompts in our main experiments over 203 languages, and we analyzed all three $n$-shot settings for a subset of languages on FLORES-200 dev sets.

The languages in FLORES-200 represent 22 language families. To experiment with multiple $n$-shot settings, we selected one language from each of the 12 families containing at least two members in the set. We chose four HRLs ($\geq$1M Wikipedia pages⁶⁶6Throughout the paper we use the ”Total pages” count from https://en.wikipedia.org/wiki/List_of_Wikipedias, accessed 7 August 2023, as a proxy for the resource level of a language.), four LRLs (25K-1M pages), and four extremely LRLs ($\leq$25K pages). These languages also employ a variety of scripts. See Table 2.

Table 3 shows the number of languages we evaluated for each MT system, as noted in §2.1, with average chrF and BLEU scores across those languages. The best performing model on average was (1) Google, then (2) NLLB, (3) GPT-4, and (4) ChatGPT. Unabridged results are in Table LABEL:tab:main_results in Appendix A. Supplementary materials can also be browsed on our repository.⁷⁷7%\def\github{https://anonymous.4open.science/r/GPT_MT_benchmark-7EB1}https://github.com/cmu-llab/gpt_mt_benchmark (Also see the interactive score visualizer on our Zeno browser.⁸⁸8https://hub.zenoml.com/project/cabreraalex/GPT%20MT%20Benchmark)

Table 4 shows a meaningful subset of scores: chrF for the 20 languages evaluated on both LLM systems. Of the 11 languages evaluated on all four systems, Google performed best for 10 of them. Notably, GPT-4 surpassed NLLB in five languages and Google in one (Moroccan Arabic, acm_Arab).

On the 20 languages for which we tested it, GPT-4 improved over ChatGPT by 6.5 chrF on average. The standard deviation of performance difference with NLLB ($chrF_{GPT}-chrF_{NLLB}$) was 8.6 for GPT-4, compared with ChatGPT’s 12.7 for the same languages, suggesting a more consistent advantage across language directions. GPT-4 offered larger improvements for LRLs, whereas HRL performance plateaued between the LLMs. Previous studies have found GPT-4 improving multilingual capabilities over ChatGPT on a range of tasks xu2023superclue; zhang2023m3exam; openai2023gpt4. This may account for its superior MT performance.

Google Translate outperformed all other systems in chrF on 100 of the 115 languages for which we evaluated it, with an average improvement of 2.0 chrF points over the next best system for each language. (See Appendix A for unabridged results.) Google’s was the best performing MT system overall, though NLLB has broader language coverage.

NLLB outperformed ChatGPT in chrF on 169 (84.1%) of the 201 languages for which we obtained scores for both, with NLLB scoring an average of 11.9 chrF points higher than the better $n$-shot ChatGPT setting for each language. This trend is corroborated by zhu2023multilingual. Table 5 has both BLEU and chrF scores from both systems for the five languages with the most negative chrF deltas ($chrF_{GPT}-chrF_{NLLB}$) on top, followed by the five languages with the highest positive deltas on bottom. For many of the subsequent sections of this paper we focus on comparing ChatGPT and NLLB, since we evaluted them on the most languages.

Using nllbteam2022language’s (nllbteam2022language) resource categorization, we find that ChatGPT performs worse on LRLs than HRLs, corroborating findings of previous works jiao2023chatgpt; zhu2023multilingual. There is a strong positive correlation between ChatGPT and NLLB chrF scores, but the correlation is higher for HRLs ($\rho$=0.85) than LRLs ($\rho$=0.78), indicating that ChatGPT struggles to keep up with NLLB for LRLs.

Figure 1 shows scatter plots where dots represent languages, with ChatGPT’s (positive or negative) relative improvement over NLLB chrF ($\frac{chrf_{GPT}-chrf_{NLLB}}{chrf_{NLLB}}$) on the y-axis. When languages are grouped by family or script, some trends are apparent (in part because we ordered groups by descending average scores). For example, ChatGPT fairs better with Uralic and Indo-European languages and clearly worse with Niger-Congo and Nilo-Saharan languages. However, the clearest natural correlation appears when languages are grouped by resource level, approximated by number of Wikipedia pages (Figure 1, bottom). Note the relative improvement (y-axis) is typically negative since ChatGPT rarely outperformed NLLB.

In the five-shot setting, ChatGPT outperformed NLLB on 47% of the HRLs designated by nllbteam2022language, but only on 6% of the LRLs. These findings contrast with what is commonly observed in multilingual MT models liu-etal-2020-multilingual-denoising; m2m-paper; siddhat-synergy; bapna2022building; nllbteam2022language, where LRLs benefit the most. This highlights the need to investigate how decoder-only models may catch up with encoder-decoder models in low-resource applications. It underscores the importance of smaller specialized models when large multitask models cannot overcome low-resource challenges.

Our main experiments suggested that $n$-shot setting had only a modest effect on MT performance. We conducted a more concentrated study of $n$-shot prompts using $dev$ sets for the 12 languages in Table 2. Results in Table 6 show five-shot prompts performing best. For some LRLs, this was simply a result of ChatGPT’s failure to model the language. In Santali’s case, for example, zero-shot ChatGPT was unable to produce the Ol Chiki script at all. In the five-shot setting, it was able to imitate the script characters from the context, but without any coherence or accuracy. Excepting Santali as an outlier, five-shot settings offered generally marginal improvements over zero-shot (the most cost-effective of the settings), with an average improvement of only 1.41 chrF across all 12 languages (0.31 if we exclude Santali). Zero-shot prompts actually produced the best chrF score for six of the 12 languages. The one-shot setting performed worst. We noted this trend of few-shot contexts offering only meager and inconsistent improvements throughout our experiments, with five-shot MT improving on zero-shot by only 0.88 average chrF across all 203 language directions. (See Appendix A.)

We were interested in which language features determined LLMs’ effectiveness compared to traditional MT. Analyzing this may reveal trends helpful to end users deciding which MT system to use, especially if their language is not represented here but shares some of the features we consider. In this section we focus on comparing ChatGPT and NLLB, since we evaluated the most languages with them. We focus on zero-shot ChatGPT, as it is the most common and convenient setting for end users.

We encoded each of the 203 languages in our set as a feature vector. In these language feature vectors we included four numerical features: number of Wikipedia pages in the language (wiki_ct), size of the language’s bi-text corpus in the Oscar MT database⁹⁹9https://oscar-project.org (oscar_ct) oscar2022, percentage of ASCII characters¹⁰¹⁰10Percentage of characters with an encoding between 0 and 128, inclusive, using the Python built-in ord function in the FLORES-200 dev set for the language (ascii_percentage), and average number of tokens per dev set sentence in FLORES-200 with ChatGPT’s tokenizer (token_ct). We also included two categorical features: language family (family) and script the language was written in (script); and one binary feature: the FLORES resource designation of the language–with 1 for high-resource and 0 for low-resource (hi/lo). Before analysis, we one-hot encoded the two categorical features into 48 binary features like family_Niger-Congo and script_Latn.

We selected token_ct as a feature because we observed languages in low-resource scripts having many tokens. For example, ChatGPT’s tokenizer encodes multiple tokens for every character in Ol Chiki script. This tendency for GPT models with low-resource scripts has been noted in previous studies ahia2023languages.

We fit a decision tree with these feature vectors to regress on ChatGPT’s relative improvement over NLLB in chrF ($\frac{chrf_{GPT}-chrf_{NLLB}}{chrf_{NLLB}}$), for each of the 201 languages with NLLB scores. When we used max_depth 3, the tree in Figure 2 was learned. Languages are delimited first by wiki_ct; then LRLs are separated into Niger-Congo languages and others, while HRLs are delimited by token_ct. The only group where ChatGPT beat NLLB is of languages with more than 58,344 Wikipedia pages, fewer than 86 tokens per average sentence, and less than 15.5% ASCII characters. This group contains some East Asian HRLs. The group where ChatGPT was least advantaged contains Niger-Congo languages with fewer than 3,707 Wikipedia pages.

We also fit a random forest regressor with the same features and labels to find feature importance values. Only ten features had importance $\geq 0.01$, shown in Table 7. The most important feature by far was wiki_ct. (This feature correlates strongly with ChatGPT’s relative improvement, $\rho=0.68$.) family_Niger-Congo was much more important than any other family feature. No script feature had an importance exceeding $0.01$. In general, features for resource level and tokenization were more important than family or script.

ChatGPT has a blind spot not only for Niger-Congo languages, but for African languages in general. Figure 1 shows ChatGPT is least advantaged for the two exclusively African families, Niger-Congo and Nilo-Saharan; and the two exclusively African scripts, Tifinagh (Tfng) and Ge’ez (Ethi).

Prior research suggests that ChatGPT output quality is sensitive to language script bang2023multitask. Our own analysis in §3.4 actually suggests that script is the least important language feature in predicting ChatGPT’s MT effectiveness. However, differences in performance are clear when comparing scripts used for the same language. Table 8 shows one script typically outperforming the other, by an average of 14.3 chrF points for zero-shot. Five-shot contexts narrowed the gap slightly to 12.0. Although transliteration is a deterministic process for many languages, these performance gaps suggest that ChatGPT has not implicitly learned it as part of a translation task. We hypothesize that ChatGPT’s observed sensitivity to script in earlier studies may be particular to the languages and tasks evaluated.

LLMs’ performing worse than NLLB may be due in large part to their translating into the wrong language. Using FLORES-200’s dev data, we trained a logistic regression language identifier for 100 epochs. Language identification accuracies for four of the models we evaluated are in Table 9. Zero-shot ChatGPT only translated on target 72% of the time. This expectedly improved with five-shot prompts, and GPT-4 performed even better, still just shy of NLLB. LLMs’ tendency to translate off target is corroborated by zhu2023multilingual.

Our results suggest that GPT-4 is a better translator than ChatGPT. However in considering the needs of MT end users, it would be remiss not to consider the respective costs of the systems evaluated. GPT-4’s high cost (roughly 2000% that of ChatGPT’s) prohibited us from evaluating it on all FLORES-200 languages. In general, using few-shot prompts for LLMs is more costly than zero-shot prompts, since users are charged for both input and output tokens. And for this same reason, some languages are more costly than others in LLM MT. Previous work has found that Google Translate has associated costs comparable to those of five-shot ChatGPT Neubig_Zeno_GPT_Machine_2023. NLLB is the least expensive system we evaluated.

We estimated cost values for each MT system and language: the expense, in USD, of translating the full FLORES-200 devtest English set into the language. We estimated costs of GPT models using the prompts employed in our experiments, the tiktoken tokenizer¹¹¹¹11https://github.com/openai/tiktoken used by both models, and inference prices from OpenAI’s website.¹²¹²12https://openai.com/pricing Conveniently, Google Translate costs nothing for the first 500K input characters. But since frequent MT users may have already expended this allowance, we calculated costs from their rates beyond the first 500K.¹³¹³13https://cloud.google.com/translate/pricing As the full NLLB-MOE model (54.5B parameters) is difficult to run on standard computing devices, nllbteam2022language also provided a version with only 3.3B parameters that achieves similar performance. Since users commonly opt for the smaller model, and since the performance difference does not impact our estimates significantly, we estimated the costs to run the 3.3B-parameter NLLB model using a single GPU on Google Colab. Details of our estimation method are in Appendix B.1. Table 10 contains the average cost for each system across the languages we evaluated with it.

Figure 3 displays chrF scores for the 11 languages on which we evaluated all four MT systems (top), and the same scores divided by the approximate cost of each model (bottom). Bars for GPT-4 drop significantly in the bottom chart because of its high cost. Note from the top chart that Google Translate scores the best, but the bottom chart shows that NLLB has the best scores for its price. Zero-shot ChatGPT also tops five-shot in the bottom chart, suggesting that while few-shot prompts provide modest score improvements, they may not be worth the extra cost. See Appendix B for fuller visualizations with all 203 languages.