Research map / Large language models / Instruction and fine-tuning

Language Imbalance Driven Rewarding for Multilingual Self-improving

ICLR 2025 · Large language models › Instruction and fine-tuning · approach: Fine-tuning generalization and delta compression · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes Language Imbalance Driven Rewarding, using the inherent gap between dominant (English) and non-dominant language responses from an LLM's self-translation as preference pairs for iterative DPO+NLL multilingual self-improvement.

Problem

LLMs perform much better in dominant languages (English, Chinese) than in others, and existing multilingual fixes rely on scarce human-authored multilingual data or are bottlenecked by the dominant language's performance.

Approach

Treat the natural quality gap between English and non-dominant language responses as a free preference ranking: the model self-translates responses in both directions, so the English response is preferred over the translated non-English one and vice versa, then trains iteratively with DPO+NLL so the dominant language also improves and keeps providing signal.

Key results

• On X-AlpacaEval, Llama-3-8B-Instruct after two iterations gains an average of 7.46% win rate over GPT-4 Turbo across five languages, comparable to 70B-level models.
• On MGSM, M2 improves average accuracy on training languages by 11.6% and unseen languages by about 16% for M1 (average over unseen languages), versus 7.8% for multilingual alignment on training languages.
• English win rate improves by 9.19% on X-AlpacaEval while multilingual alignment drops English by 3.02%; MT-Bench average rises from 6.80 to 7.51.

Measured effects reported in the paper

Surprising findings

• Multilingual alignment via SFT degrades English performance, whereas the imbalance-driven DPO improves the dominant language as well.
• Reward strength is inversely related to language quality: weaker languages give stronger preference signals, but signals weaken over iterations.

Limitations

• The reward signal derived from language imbalance is coarse-grained.
• Self-translation may introduce translation artifacts that hurt response quality.
• Gains are limited for very low-resource languages where the base model's generation is weak.

Experimental setup

Datasets: AlpaGasus GSM8K X-AlpacaEval MT-Bench MGSM MMLU HellaSwag ARC Challenge TruthfulQA

Benchmarks: X-AlpacaEval MT-Bench (multilingual) MGSM MMLU HellaSwag ARC Challenge TruthfulQA

Metrics: win rate accuracy GPT-4 score off-target rate reward accuracy

Baselines: Multilingual Alignment (SFT on self-translated responses) MAPO GPT-4o GPT-4 Qwen2-7B-Instruct Llama-3-8B-Instruct

Builds on: DPO Self-Rewarding MAPO

Closest papers by meaning

• Beyond English-Centric Training: How Reinforcement Learning Improves Cross-Lingual Reasoning in LLMs (ICLR 2026 · similarity 0.89)
• Toward Self-Improvement of LLMs via Imagination, Searching, and Criticizing (NeurIPS 2024 · similarity 0.88)
• Importance Weighting Can Help Large Language Models Self-Improve (AAAI 2025 · similarity 0.88)
• Mind the Gap: Examining the Self-Improvement Capabilities of Large Language Models (ICLR 2025 · similarity 0.88)
• XIFBench: Evaluating Large Language Models on Multilingual Instruction Following (NeurIPS 2025 · similarity 0.88)
• RMB: Comprehensively benchmarking reward models in LLM alignment (ICLR 2025 · similarity 0.88)