Research map / Large language models / Instruction and fine-tuning
Can LLMs Really Learn to Translate a Low-Resource Language from One Grammar Book?
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Disentangles grammar-book parallel examples from grammatical explanations for extremely low-resource translation (Kalamang, Nepali, Guarani), showing parallel data drives gains, fine-tuned NLLB matches Gemini with the book, and a typological feature prompt helps grammaticality judgment and gloss prediction.
Problem
Whether long-context LLMs really learn to translate an extremely low-resource language from a grammar book (as claimed by MTOB), and what kind of grammatical knowledge helps XLR tasks.
Approach
Split grammar books into parallel examples and grammatical explanations to ablate their contributions; show parallel examples account for nearly all translation gains, fine-tuning a small MT model matches long-context prompting, and introduce a typological feature prompt (from Grambank) that helps linguistic tasks.
Key results
- Gemini with the book's parallel subset (Book_para) nearly matches the full book for kgv translation, while the non-parallel explanations alone (Book_non-para) score far lower (e.g., 22.6 vs 30.8 ChrF++ eng-kgv).
- Fine-tuned NLLB-1.3B on Para_book reaches 34.2 ChrF++ eng-kgv, versus 26.6 for Gemini with the same parallel data in-context.
- The typological prompt with Book_para achieves the highest morpheme accuracy on kgv IGT prediction (46.1), above supervised baselines.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace grammar explanations with book parallel examples (ablation) | Gemini Kalamang translation · Kalamang eng→kgv | ChrF++: 22.6 -> 30.8 | Helped |
| Replace grammar explanations with book parallel examples (ablation) | Gemini Kalamang translation · Kalamang kgv→eng | ChrF++: 27.5 -> 34.7 | Helped |
| Add grammatical explanations to parallel data (ablation) | Gemini Kalamang translation · Kalamang eng→kgv | ChrF++: +0.7 ChrF++ | No clear effect |
| Add grammatical explanations to parallel data (ablation) | Gemini Kalamang translation · Kalamang kgv→eng | ChrF++: -0.3 points | Hurt |
| Remove parallel examples and retain explanatory text (ablation) | Gemini Kalamang translation · Kalamang translation | ChrF++: up to 8 points | Hurt |
| Add typological prompting to book parallel examples (ablation) | Gemini Kalamang translation · Kalamang eng→kgv | ChrF++: 30.8 -> 31.4 | Helped |
| Add typological prompting to book parallel examples (ablation) | Gemini Kalamang translation · Kalamang kgv→eng | ChrF++: 34.7 -> 35.2 | Mixed |
| Add parallel training examples to book parallel data (ablation) | NLLB Kalamang translation · Kalamang eng→kgv | ChrF++: 34.2 -> 38.7 | Helped |
| Add backtranslated training examples (ablation) | NLLB Kalamang translation · Kalamang eng→kgv | ChrF++: 38.7 -> 32.0 | Hurt |
| Add backtranslated training examples (ablation) | NLLB Kalamang translation · Kalamang kgv→eng | ChrF++: +3 ChrF++ | Helped |
| Add typological prompt to book parallel examples (ablation) | Gemini grammaticality judgment · Kalamang grammaticality judgment | accuracy: up to 3% over Book_p | Helped |
| Add typological prompt to book parallel examples (ablation) | Gemini IGT prediction · Kalamang IGT prediction | morpheme accuracy: 45.4 -> 46.1 | Helped |
Surprising findings
- Grammatical explanations add no significant advantage over the type coverage they provide; the full grammar book performs worse than its parallel subset despite ~5x more tokens.
- Fine-tuning Gemini on the parallel data underperforms NLLB and in-context Gemini by 6-12 ChrF++.
Limitations
- Kalamang test set is only 100 examples and no human evaluation is possible without Kalamang speakers.
- Main experiments use a single API-only long-context model (Gemini-1.5-Flash), with only short-context tests on Llama-3.1-8B.
- Many ChrF++ differences are small and may not be perceptible to humans.
Experimental setup
Datasets: Kalamang grammar book Kalamang parallel examples Kalamang wordlist FLORES Dictionaria IGT Grambank
Benchmarks: MTOB Kalamang test set (100 examples) FLORES devtest (npi, gug) Dictionaria IGT test set
Metrics: ChrF++ BLEU morpheme accuracy word accuracy stem F1 gram F1 grammaticality judgment accuracy
Baselines: MTOB Gemini-1.5-Flash 0-shot 5*-shot retrieval W4W Top-Class SMP-Base TücL-Morph ByT5-ft GlossLM-ft
Builds on: Machine Translation from One Book (MTOB) LoRA NLLB Grambank GlossLM
Closest papers by meaning
- A Benchmark for Learning to Translate a New Language from One Grammar Book (ICLR 2024 · similarity 0.92)
- Language Ranker: A Metric for Quantifying LLM Performance Across High and Low-Resource Languages (AAAI 2025 · similarity 0.89)
- LLM Augmented LLMs: Expanding Capabilities through Composition (ICLR 2024 · similarity 0.89)
- A Paradigm Shift in Machine Translation: Boosting Translation Performance of Large Language Models (ICLR 2024 · similarity 0.88)
- When Scaling Meets LLM Finetuning: The Effect of Data, Model and Finetuning Method (ICLR 2024 · similarity 0.88)
- MindMerger: Efficiently Boosting LLM Reasoning in non-English Languages (NeurIPS 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.