Research map / Large language models / Language, safety and interpretability
Translate Meanings, Not Just Words: IdiomKB’s Role in Optimizing Idiomatic Translation with Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Builds IdiomKB, a machine-generated multilingual idiom knowledge base of figurative meanings distilled from LLMs, and uses retrieved meanings as KB-CoT prompts to improve idiomatic translation by smaller LMs, with a GPT-4-based reference-free evaluation metric.
Problem
Machine translation and general LMs translate idioms literally because idioms are non-compositional; existing idiom knowledge bases are small, lack multilingual meanings, and small models struggle to translate idiomatic text.
Approach
Distill multilingual figurative meanings of idioms from LLMs via in-context learning into an offline KB, then retrieve the meaning and insert it as a chain-of-thought transition ("KB-CoT") before translation, plus a GPT-4 1-3 point rubric for idiomatic translation quality.
Key results
- KB-CoT with IdiomKB improved GPT-4-judged translation for InstructGPT (6.7B) on Zh-En from 1.66 to 2.08 and BLOOMZ (7.1B) from 2.11 to 2.21 (Direct vs KB-CoT).
- GPT-4 judge scores correlated with humans (Pearson 0.69 Zh-En, 0.79 En-Zh, 0.67 Ja-En), outperforming sacreBLEU and COMET.
- Human evaluation of IdiomKB meanings averaged 2.92 out of 3.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add IdiomKB meanings through KB-CoT prompting (ablation) | InstructGPT (6.7B) · Zh→En | GPT-4: 1.66 -> 2.08 | Helped |
| Add IdiomKB meanings through KB-CoT prompting (ablation) | InstructGPT (6.7B) · En→Zh | GPT-4: 1.50 -> 1.99 | Helped |
| Add IdiomKB meanings through KB-CoT prompting (ablation) | InstructGPT (6.7B) · Ja→En | GPT-4: 1.34 -> 1.64 | Helped |
| Add IdiomKB meanings through KB-CoT prompting (ablation) | BLOOMZ (7.1B) · Zh→En | GPT-4: 2.11 -> 2.21 | Helped |
| Add IdiomKB meanings through KB-CoT prompting (ablation) | BLOOMZ (7.1B) · En→Zh | GPT-4: 2.08 -> 2.17 | Helped |
| Add IdiomKB meanings through KB-CoT prompting (ablation) | BLOOMZ (7.1B) · Ja→En | GPT-4: 1.22 -> 1.54 | Helped |
| Add IdiomKB meanings through KB-CoT prompting (ablation) | Alpaca (7B) · Zh→En | GPT-4: 1.54 -> 2.12 | Helped |
| Add IdiomKB meanings through KB-CoT prompting (ablation) | Alpaca (7B) · En→Zh | GPT-4: 1.11 -> 1.46 | Helped |
| Add IdiomKB meanings through KB-CoT prompting (ablation) | Alpaca (7B) · Ja→En | GPT-4: 1.23 -> 1.57 | Helped |
| Add IdiomKB meanings through KB-CoT prompting (ablation) | ChatGPT · Zh→En | GPT-4: 2.74 -> 2.82 | Helped |
| Add IdiomKB meanings through KB-CoT prompting (ablation) | GPT-4 · En→Zh | GPT-4: 2.73 -> 2.83 | Helped |
| Use ChatGPT-generated meanings instead of self-generated meanings (ablation) | InstructGPT (6.7B) · Zh→En | GPT-4: 1.69 -> 2.08 | Helped |
Surprising findings
- Self-generated meanings by small models (Self-CoT) gave little benefit, with InstructGPT (6.7B) rising only from 1.66 to 1.69 on Zh-En, versus 2.08 with ChatGPT-generated KB meanings.
- KB-CoT lowered sacreBLEU for InstructGPT (6.7B) on Zh-En (14.26 to 9.64) even as GPT-4 scores rose.
Limitations
- Evaluation sets are small (800 samples per source, 500 test instances, 20 annotated pairs per score point), and budget constraints limited scale.
- Idiom identification is bypassed by using gold idiom-sentence pairs from source datasets.
- GPT-4 judge may be biased toward GPT-generated translations.
Experimental setup
Datasets: MAGPIE IMIL EPIE PIE PETCI CCT ChID OpenMWE ID10M WMT22
Benchmarks: WMT22 Zh-En idiomatic subset PIE-derived En-Zh set OpenMWE-derived Ja-En set no-leakage news set
Metrics: sacreBLEU COMET CometKiwi GPT-4 1-3 idiom score Pearson Spearman Kendall human 1-3 score
Baselines: mBART NLLB InstructGPT (6.7B) direct prompting BLOOMZ (7.1B) direct prompting Alpaca (7B) direct prompting ChatGPT GPT-4 sacreBLEU CometKiwi
Builds on: chain-of-thought prompting symbolic knowledge distillation BLOOMZ InstructGPT
Closest papers by meaning
- Enhancing Machine Translation Experiences with Multilingual Knowledge Graphs (AAAI 2024 · similarity 0.85)
- MT-Ranker: Reference-free machine translation evaluation by inter-system ranking (ICLR 2024 · similarity 0.85)
- A Benchmark for Learning to Translate a New Language from One Grammar Book (ICLR 2024 · similarity 0.85)
- A Paradigm Shift in Machine Translation: Boosting Translation Performance of Large Language Models (ICLR 2024 · similarity 0.84)
- Are Large Reasoning Models Good Translation Evaluators? Analysis and Performance Boost (NeurIPS 2025 · similarity 0.84)
- Can LLMs Really Learn to Translate a Low-Resource Language from One Grammar Book? (ICLR 2025 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.