Research map / Large language models / Alignment and preferences
Meta-Learning Preferences for Multilingual LLM Alignment
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes MAML-based meta-learning over languages (as tasks) for preference alignment, with MAML-RLHF (meta-reward model) and MAML-DPO (meta-policy), with convergence analysis and experiments on BLOOM-7.1B and Gemma3-270M.
Problem
Low-resource languages lack sufficient human preference data, so RLHF/DPO alignment performs poorly there; adapting to a target language with very few preference samples is hard.
Approach
Treat each language as a meta-learning task and meta-train a reward model (MAML-RLHF) or policy (MAML-DPO) across non-target languages so that a few gradient steps on target-language preferences yield an aligned model; provides first MAML convergence guarantees for Bradley-Terry/DPO losses.
Key results
- With 100 target-language samples, MAML-DPO on Gemma3-270M reaches 79.83% win rate on Romanian vs 51.56% for DPO (Table of Romance targets).
- MAML-RLHF on Gemma3-270M reaches 85.23% win rate on Romanian with 100 samples vs 60.28% for RLHF and 63.41% for Multitask-RLHF.
- BLOOM-7.1B MAML-DPO with 8K samples on French reaches 66.2% win rate vs 57.8% for DPO; with 40K, 71.4% vs 67.4%.
Surprising findings
- Group 2 (linguistically distant meta-training languages) showed the highest point estimate for French adaptation, with no significant effect of linguistic distance.
- Gains of MAML over baselines shrink as adaptation data grows, and MAML-RLHF is sometimes below multitask-RLHF on French with larger data.
Limitations
- Theory uses linear reward and log-linear policy classes, far from the LLM setting.
- Experiments are limited to small models (270M, 7.1B) and few target languages; the RLHF pipeline depends on GRPO with a small number of prompts.
- Win rates are judged by in-house reward models rather than human or strong LLM judges, and the paper notes multitask baselines can be competitive.
Experimental setup
Datasets: Okapi preference data (derived from Alpaca-64K) multilingual/orca_dpo_pairs
Benchmarks: ARC HellaSwag MMLU
Metrics: win rate vs base model (judge reward model) reward accuracy zero-shot accuracy and normalized accuracy
Baselines: DPO RLHF Multitask-DPO Multitask-RLHF Translate-test with gpt-4o
Builds on: MAML DPO RLHF GRPO
Closest papers by meaning
- Towards Efficient Exact Optimization of Language Model Alignment (ICML 2024 · similarity 0.92)
- SimPER: A Minimalist Approach to Preference Alignment without Hyperparameters (ICLR 2025 · similarity 0.91)
- MetaAligner: Towards Generalizable Multi-Objective Alignment of Language Models (NeurIPS 2024 · similarity 0.91)
- Align-Pro: A Principled Approach to Prompt Optimization for LLM Alignment (AAAI 2025 · similarity 0.91)
- Cal-DPO: Calibrated Direct Preference Optimization for Language Model Alignment (NeurIPS 2024 · similarity 0.91)
- Spread Preference Annotation: Direct Preference Judgment for Efficient LLM Alignment (ICLR 2025 · similarity 0.91)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.