Research map / Large language models / Instruction and fine-tuning
A Paradigm Shift in Machine Translation: Boosting Translation Performance of Large Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes ALMA, a two-stage fine-tuning recipe (monolingual data fine-tuning then small high-quality parallel data fine-tuning) that turns LLaMA-2 7B/13B into strong translators rivaling NLLB-54B and GPT-3.5.
Problem
Moderate-size (7B/13B) decoder-only LLMs lag far behind supervised encoder-decoder translation models, and prior LLM translation fine-tuning gave only modest gains.
Approach
Two-stage fine-tuning: first causal LM fine-tuning on non-English and English monolingual data to build multilingual proficiency, then fine-tuning on a small set (~58K) of human-written WMT/Flores parallel data, showing LLMs need little parallel data and too much dilutes their knowledge.
Key results
- ALMA-7B/13B improve over LLaMA-2 zero-shot by more than 12 BLEU and COMET averaged over 10 directions (ALMA-7B +16.12 BLEU, +17.61 COMET for en→xx).
- ALMA-13B-LoRA averages 31.87 BLEU / 87.00 COMET en→xx, exceeding NLLB-54B (30.92 / 85.04) and GPT-3.5-D (28.96 / 84.59) but trailing GPT-3.5-T (32.17 / 86.56) on BLEU.
- Fine-tuning on only 1B monolingual tokens followed by parallel fine-tuning already gives performance comparable to NLLB-54B and GPT-3.5-D (7B model), taking ~18 hours on 16 MI200 GPUs.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Apply two-stage fine-tuning: non-English monolingual data followed by high-quality human-written parallel data | LLaMA-2-7B · WMT’22/’21 en→xx, average | COMET: 68.88 -> 86.49 | Helped |
| Apply two-stage fine-tuning: non-English monolingual data followed by high-quality human-written parallel data | LLaMA-2-7B · WMT’22/’21 xx→en, average | COMET: 77.16 -> 84.08 | Helped |
| Add monolingual fine-tuning before filtered parallel data fine-tuning (ablation) | LLaMA-2-7B · WMT’22/’21 en→xx, average | COMET: 74.35 -> 83.98 | Helped |
| Add monolingual fine-tuning before filtered parallel data fine-tuning (ablation) | LLaMA-2-7B · WMT’22/’21 xx→en, average | COMET: 78.94 -> 83.03 | Helped |
| Use human-written rather than filtered parallel data after monolingual fine-tuning (ablation) | LLaMA-2-7B · WMT’22/’21 en→xx, average | COMET: 83.98 -> 86.49 | Helped |
| Use human-written rather than filtered parallel data after monolingual fine-tuning (ablation) | LLaMA-2-7B · WMT’22/’21 xx→en, average | COMET: 83.03 -> 84.08 | Helped |
| Use LoRA instead of full-weight fine-tuning for parallel-data stage (ablation) | ALMA-13B · WMT’22/’21 en→xx, average | COMET: 86.80 -> 87.00 | Helped |
| Use LoRA instead of full-weight fine-tuning for parallel-data stage (ablation) | ALMA-13B · WMT’22/’21 xx→en, average | COMET: 84.40 -> 84.59 | Helped |
| Use prefix LM or mixture-of-denoisers instead of causal language modeling (ablation) | MPT-7B translation fine-tuning · Russian→English with 100K or 1M parallel examples | translation performance | Hurt |
Surprising findings
- Fine-tuning LLaMA-2-7B on large parallel data (5M-20M) reduces COMET despite rising BLEU, suggesting parallel data dilutes pre-existing knowledge.
- Models trained from scratch on 20M parallel data reach similar performance to fine-tuned LLMs, indicating catastrophic forgetting dominates at large data scales.
Limitations
- Comparison with much larger systems (GPT-3.5, NLLB-54B) is not fully fair due to differences in training data and model size.
- Monolingual fine-tuning covers only six languages (de, cs, is, zh, ru, en), and results are limited to 10 English-centric directions.
- Performance continues to improve with more monolingual data, so the reported training budget may not be optimal.
Experimental setup
Datasets: OSCAR WMT'17-WMT'20 test sets Flores-200 dev and test WMT'21 test (dev) WMT'22 test
Benchmarks: WMT'21 WMT'22
Metrics: BLEU COMET-22
Baselines: NLLB-54B GPT-3.5-text-davinci-003 GPT-3.5-turbo-0301 GPT-4 BigTranslate BayLing TIM SWIE
Builds on: LLaMA-2 NLLB LoRA DeepSpeed
Closest papers by meaning
- When Scaling Meets LLM Finetuning: The Effect of Data, Model and Finetuning Method (ICLR 2024 · similarity 0.92)
- Contrastive Preference Optimization: Pushing the Boundaries of LLM Performance in Machine Translation (ICML 2024 · similarity 0.91)
- Teaching Large Language Models to Translate with Comparison (AAAI 2024 · similarity 0.91)
- Calibrating Translation Decoding with Quality Estimation on LLMs (NeurIPS 2025 · similarity 0.91)
- DelTA: An Online Document-Level Translation Agent Based on Multi-Level Memory (ICLR 2025 · similarity 0.90)
- Supervised Knowledge Makes Large Language Models Better In-context Learners (ICLR 2024 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.