Research map / Large language models / Alignment and preferences
Fluent Alignment with Disfluent Judges: Post-training for lower-resource languages
David Samuel, Lilja Øvrelid, Erik Velldal, Andrey Kutuzov
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use on-policy RL instead of SFT on machine-translated data | NorMistral 11B · Norwegian fluency (human) | pairwise win-rate: 67.5 | Improved |
| use on-policy RL instead of SFT on machine-translated data | NorMistral 11B · Norwegian fluency (automatic) | fluency score: 85.7% -> 92.2% | Improved |
| use a disfluent judge (Mistral Nemo 12B) rather than the most fluent judge Mistral Large for reward (ablation) | on-policy RL · policy fluency | fluency of trained policy: 92.2 vs 94.2; Pearson correlation 0.067 | No clear effect |
| train the SFT stage for 2 epochs instead of 1 on English data (ablation) | initial SFT before RL · RL fluency | fluency: 94.2 -> 93.2 | Hurt |
| train the SFT stage for 4 epochs instead of 1 on English data (ablation) | initial SFT before RL · RL fluency | fluency: 94.2 -> 92.8 | Hurt |
| use translated Norwegian dataset instead of English for initial SFT (ablation) | initial SFT before RL · RL fluency | fluency: 94.2 -> 91.0 | Hurt |
| translate with Tower-Plus 72.7B instead of OPUS Eng-Gem 0.1B (ablation) | translated SFT · Norwegian fluency | fluency: 68.2 -> 85.7 | Improved |
| translate with Tower-Plus instead of MADLAD-400 (ablation) | translated SFT · Norwegian fluency | fluency: 82.4 -> 85.7 | Improved |
| translate with Tower-Plus instead of Seed-X (ablation) | translated SFT · Norwegian fluency | fluency: 73.4 -> 85.7 | Improved |
| continue RL training beyond first 50 steps | on-policy RL with Mistral Large judge · fluency over training | fluency score: 87.5% -> around 93% | No clear effect |
Closest papers by meaning
- Meta-Learning Preferences for Multilingual LLM Alignment (NeurIPS 2026 · similarity 0.91)
- Advantage-Guided Distillation for Preference Alignment in Small Language Models (ICLR 2025 · similarity 0.90)
- SimPER: A Minimalist Approach to Preference Alignment without Hyperparameters (ICLR 2025 · similarity 0.89)
- Preference-Oriented Supervised Fine-Tuning: Favoring Target Model over Aligned Large Language Models (AAAI 2025 · similarity 0.89)
- Towards Efficient Exact Optimization of Language Model Alignment (ICML 2024 · similarity 0.89)
- Noise Contrastive Alignment of Language Models with Explicit Rewards (NeurIPS 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
