Research map / Large language models / Language, safety and interpretability
Mufu: Multilingual Fused Learning for Low-Resource Translation with LLM
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Mufu finetunes LLMs on postediting prompts that include automatically generated multilingual translation candidates from a teacher model, improving En-XX low-resource translation, with distillation to cut inference cost.
Problem
LLMs translate poorly into low-resource languages due to scarce pretraining data, and few-shot prompting is less effective than finetuning, while specialized NMT systems like NLLB remain stronger.
Approach
Turn translation into a postediting task: the prompt includes multilingual auxiliary translations (from a teacher, PaLM2 S, selected by URIEL language proximity) plus a draft target translation, and the student is finetuned to output the corrected translation against references, combining in-context multilingual context with supervised finetuning.
Key results
- PaLM2 XXS-NTL with mufu20 reaches mean chrF 48.4 on FLORES-200 devtest across 201 pairs versus 43.3 for the PaLM2 S teacher and 46.0 for NLLB 1.3B distilled (n=198).
- Best Mufu model PaLM2 XXS-NTL outperforms NLLB 1.3B distilled in 64% of very-low and low-resource language pairs on FLORES-200.
- Distilled PaLM2 XXS-NTL retains gains, averaging about 3.1 chrF over the finetune-only baseline in low-resource translations, per the abstract.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add 20 selected auxiliary multilingual translations and a postediting target | PaLM2 XXS–NTL finetuning · FLORES-200 devtest (201 En-XX pairs) | chrF: 39.2 -> 48.4 | Helped |
| Add 20 selected auxiliary multilingual translations and a postediting target | PaLM2 XXS–NTL finetuning · NTREX test (112 En-XX pairs) | chrF: 36.3 -> 48.8 | Helped |
| Add 20 selected auxiliary multilingual translations and a postediting target | Gemma 7B finetuning · FLORES-200 devtest (201 En-XX pairs) | chrF: 39.9 -> 47.6 | Helped |
| Add a postediting target without multilingual auxiliary translations (ablation) | PaLM2 XXS–NTL · FLORES-200 devtest (201 En-XX pairs) | chrF: 39.2 -> 42.5 | Helped |
| Add five selected auxiliary multilingual translations to postediting (ablation) | PaLM2 XXS–NTL · FLORES-200 devtest (201 En-XX pairs) | chrF: 42.5 -> 47.1 | Helped |
| Replace selected related auxiliary languages with five high-resource languages (ablation) | PaLM2 XXS–NTL mufu5 · FLORES-200 devtest (201 En-XX pairs) | chrF: 47.1 -> 42.9 | Hurt |
| Remove the postediting target while retaining five auxiliary candidates (ablation) | PaLM2 XXS–NTL mufu5 · FLORES-200 devtest (201 En-XX pairs) | chrF: 47.1 -> 44.4 | Hurt |
| Add five high-resource auxiliary languages to mufu20 (ablation) | PaLM2 XXS–NTL · FLORES-200 devtest (201 En-XX pairs) | chrF: 48.4 -> 47.1 | Hurt |
| Use LoRA finetuning (r = 16) instead of full finetuning (ablation) | PaLM2 S mufu20 · FLORES-200 devtest | chrF: 47.0 -> 47.2 | Mixed |
| Distill the mufu20 model using teacher-predicted sequences (ablation) | PaLM2 XXS–NTL · FLORES-200 devtest (201 En-XX pairs) | chrF: 48.4 -> 45.1 | Hurt |
| Distill the mufu20 model using teacher-predicted sequences (ablation) | PaLM2 XXS–NTL · NTREX test (112 En-XX pairs) | chrF: 48.8 -> 49.0 | Helped |
Surprising findings
- Mufu models trained on only 787 sentences outperform the teacher model that generated the auxiliary candidates in low- and very-low-resource languages.
- Using only high-resource auxiliary languages (Dutch, Russian, French, Chinese, Spanish) yields much smaller gains than URIEL-selected related languages, and adding them to mufu20 hurts PaLM2 XXS-NTL.
Limitations
- Mufu underperforms baseline when auxiliary candidates are of very poor quality (three or more bad candidates).
- Substantially higher inference latency and cost due to generating auxiliary translations in a two-iteration pipeline.
- Evaluation is limited to En-XX translation and to training and testing on FLORES-200-derived data, so out-of-domain gains are smaller (NTREX).
Experimental setup
Datasets: FLORES-200 NTREX NLLB seed data WMT General Tasks test sets (2009-2018)
Benchmarks: FLORES-200 devtest NTREX
Metrics: chrF win rate
Baselines: PaLM2 S teacher NLLB 1.3B distilled NLLB 54B MoE postedit-only finetuning baseline finetuning
Builds on: chain-of-thought prompting sequence-level knowledge distillation LoRA
Closest papers by meaning
- A Paradigm Shift in Machine Translation: Boosting Translation Performance of Large Language Models (ICLR 2024 · similarity 0.88)
- Can LLMs Really Learn to Translate a Low-Resource Language from One Grammar Book? (ICLR 2025 · similarity 0.87)
- X-ALMA: Plug & Play Modules and Adaptive Rejection for Quality Translation at Scale (ICLR 2025 · similarity 0.87)
- A Benchmark for Learning to Translate a New Language from One Grammar Book (ICLR 2024 · similarity 0.86)
- Knowledge Fusion of Large Language Models (ICLR 2024 · similarity 0.86)
- Teaching Large Language Models to Translate with Comparison (AAAI 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.