Research map / Large language models / Language, safety and interpretability
Tradutor: Building a Variety Specific Translation Model
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Builds PTradutor, a ~1.7M-document European Portuguese-English parallel corpus via back-translation with Google Translate, and fine-tunes Gemma-2 2B, Phi-3-mini and LLaMA-3 8B (full and LoRA) into English-to-European-Portuguese translation models.
Problem
Machine translation and LMs neglect European Portuguese, which is overwhelmed by Brazilian Portuguese in available data, and no open-source translation model targets European Portuguese.
Approach
Back-translate monolingual European Portuguese text into English with a commercial MT system to create a parallel corpus, then fine-tune instruction-tuned LLMs to translate English into European Portuguese, checking variety with a Portuguese variety classifier (VID score).
Key results
- Fine-tuned LLaMA-3 8B with full fine-tuning reaches BLEU 41.12 on FRMT and 35.76 on NTrex, beating open-source Opus-MT (40.41 and 32.99) and ArgosTranslate.
- LoRA variants score highest on the VID variety metric (up to 1.203 for Phi-3 on NTrex) but have much lower translation quality, with repetition loops for Phi-3 and LLaMA-3.
- Fine-tuned LLaMA-3 VID scores (0.968 FRMT, 0.933 NTrex) are comparable to Google pt and DeepL and well above Google br (0.445, 0.361), though DeepL leads on BLEU, ROUGE-L and COMET.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Full fine-tune LLaMA-3 on PTradutor rather than use it zero-shot | LLaMA-3 · FRMT | BLEU: 31.47 -> 41.12 | Helped |
| Full fine-tune LLaMA-3 on PTradutor rather than use it zero-shot | LLaMA-3 · NTrex | BLEU: 21.09 -> 35.76 | Helped |
| Use full fine-tuning instead of LoRA (ablation) | LLaMA-3 · FRMT | BLEU: 25.42 -> 41.12 | Helped |
| Use full fine-tuning instead of LoRA (ablation) | LLaMA-3 · NTrex | BLEU: 17.74 -> 35.76 | Helped |
| Use full fine-tuning instead of LoRA (ablation) | Phi-3 · FRMT | BLEU: 24.70 -> 38.16 | Helped |
| Use full fine-tuning instead of LoRA (ablation) | Gemma-2 · FRMT | BLEU: 19.83 -> 33.76 | Helped |
| Increase model size among full fine-tuned models | PTradutor fine-tuned models · FRMT | BLEU: 33.76 -> 38.16 -> 41.12 | Helped |
| Use LoRA instead of zero-shot inference (ablation) | Phi-3 · FRMT | VID: 0.942 -> 1.178 | Helped |
| Use LoRA instead of zero-shot inference (ablation) | Gemma-2 · FRMT | BLEU: 25.37 -> 19.83 | Hurt |
| Use full fine-tuning instead of LoRA (ablation) | Phi-3 · FRMT | VID: 1.178 -> 1.055 | Hurt |
| Use full fine-tuning instead of LoRA (ablation) | LLaMA-3 · FRMT | VID: 1.092 -> 0.968 | Hurt |
| Use Google Translate to generate English from either Portuguese variety | PTradutor corpus construction · FRMT | BLEU: 96.8% | No clear effect |
Surprising findings
- Google Translate output for European and Brazilian Portuguese inputs was identical for about 87.2% of FRMT examples, so the back-translation step is largely insensitive to variety.
- LoRA-trained models score highest on variety fidelity yet produce much poorer translations than full fine-tuning.
Limitations
- Greedy decoding only; the authors did not explore beam search.
- Results rely on automatic metrics; no human evaluation of variety or quality yet.
- Documents over 900 tokens were removed, so the model is limited to shorter inputs.
Experimental setup
Datasets: PTradutor DSL-TL PtBrVid CETEM Publico FRMT NTrex
Benchmarks: FRMT NTrex
Metrics: BLEU ROUGE-L COMET VID score
Baselines: Google Translate DeepL ArgosTranslate Opus-MT Gemma-2 zero-shot Phi-3 zero-shot LLaMA-3 zero-shot
Builds on: LoRA Opus-MT COMET FRMT
Closest papers by meaning
- Enhancing Portuguese Variety Identification with Cross-Domain Approaches (AAAI 2025 · similarity 0.87)
- Trans-EnV: A Framework for Evaluating the Linguistic Robustness of LLMs Against English Varieties (NeurIPS 2025 · similarity 0.85)
- X-ALMA: Plug & Play Modules and Adaptive Rejection for Quality Translation at Scale (ICLR 2025 · similarity 0.84)
- RedPajama: an Open Dataset for Training Large Language Models (NeurIPS 2024 · similarity 0.83)
- Teaching Large Language Models to Translate with Comparison (AAAI 2024 · similarity 0.83)
- A Paradigm Shift in Machine Translation: Boosting Translation Performance of Large Language Models (ICLR 2024 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.