Research map / Large language models / Language, safety and interpretability
A Benchmark for Learning to Translate a New Language from One Grammar Book
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces MTOB, a benchmark where LLMs learn to translate between English and the endangered language Kalamang from a single grammar book, word list, and small parallel corpus, compared against a human learner.
Problem
Measuring whether LLMs can genuinely adapt to new tasks unseen in web-scale training data, and how well they can learn a language from reference materials alone for a language with virtually no web presence.
Approach
Frame low-resource translation as learning a new language from one human-readable grammar book (plus word list and ~400 parallel sentences), using in-context retrieval and long-context prompting, and calibrate against a data-matched human learner.
Key results
- Claude 2 with word list, sentence, and full grammar context reaches 44.7 chrF on kgv→eng and 45.8 chrF on eng→kgv, versus a human learner's 51.6 and 57.0.
- Without any context, all models score near zero BLEU and are essentially unable to translate Kalamang.
- Fine-tuning Llama 2 on the grammar text harmed results relative to in-context baselines.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Provide long grammar-book context with word-list and paired-sentence context | Claude 2 · Kalamang to English | chrF: 42.9 -> 44.7 | Helped |
| Provide long grammar-book context with word-list and paired-sentence context | Claude 2 · English to Kalamang | chrF: 40.8 -> 45.8 | Helped |
| Add paired-sentence context | Claude 2 with no context · Kalamang to English | chrF: 15.6 -> 35.1 | Helped |
| Add paired-sentence context | Claude 2 with no context · English to Kalamang | chrF: 13.0 -> 36.2 | Helped |
| Combine word-list and paired-sentence context | Claude 2 · Kalamang to English | chrF: 35.1 -> 43.6 | Helped |
| Combine word-list and paired-sentence context | Claude 2 · English to Kalamang | chrF: 36.2 -> 39.3 | Helped |
| Add retrieved grammar passages using LCS instead of embeddings | Claude 2 with grammar-book retrieval · Kalamang to English | chrF: 18.7 -> 26.1 | Helped |
| Add retrieved grammar passages using LCS instead of embeddings | Claude 2 with grammar-book retrieval · English to Kalamang | chrF: 12.5 -> 22.1 | Helped |
| Add retrieved grammar passages to word-list and paired-sentence context | Claude 2 · Kalamang to English | chrF: 43.6 -> 42.9 | No clear effect |
| Add retrieved grammar passages to word-list and paired-sentence context | Claude 2 · English to Kalamang | chrF: 39.3 -> 40.8 | Helped |
| Fine-tune on grammar-book text | Llama 2-7B and Llama 2-13B · Kalamang translation | chrF | Hurt |
| Increase base-model scale and training data | LLaMA and Llama 2 model families · Kalamang translation | chrF | Helped |
Surprising findings
- Finetuning Llama 2 directly on the grammar book text degraded translation quality compared with prompting.
- Embedding-based retrieval (all-mpnet-base-v2) underperformed longest-common-substring retrieval for grammar passages.
Limitations
- Test sentences are drawn from the same causally-structured corpus as the training materials, so the test domain is somewhat artificial.
- Only one language (Kalamang) and sentence-level translation are evaluated; the task is only instrumentally useful and not immediately socially useful.
- Human baseline is a single non-native author, and finetuning experiments were limited to models up to 13B.
Experimental setup
Datasets: A grammar of Kalamang Kalamang-English bilingual word list (Dictionaria) Kalamang-English parallel sentences (400 train / 100 test)
Benchmarks: MTOB
Metrics: chrF BLEU
Baselines: LLaMA-7B LLaMA-13B LLaMA-30B Llama 2-7B Llama 2-13B Llama 2-70B text-davinci-003 gpt-3.5-turbo gpt-4 Claude 2
Builds on: LoRA chrF
Closest papers by meaning
- Can LLMs Really Learn to Translate a Low-Resource Language from One Grammar Book? (ICLR 2025 · similarity 0.92)
- A Paradigm Shift in Machine Translation: Boosting Translation Performance of Large Language Models (ICLR 2024 · similarity 0.90)
- Teaching Large Language Models to Translate with Comparison (AAAI 2024 · similarity 0.88)
- MT-Ranker: Reference-free machine translation evaluation by inter-system ranking (ICLR 2024 · similarity 0.88)
- metabench - A Sparse Benchmark of Reasoning and Knowledge in Large Language Models (ICLR 2025 · similarity 0.87)
- Automating Dataset Updates Towards Reliable and Timely Evaluation of Large Language Models (NeurIPS 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.