Research map / Large language models / Alignment and preferences
BLEUBERI: BLEU is a surprisingly effective reward for instruction following
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows BLEU with synthetic references rivals strong reward models in agreeing with human preferences, then trains Qwen2.5 and Llama-3.1 models with GRPO using BLEU as the reward on hard prompts, matching reward-model-guided RL.
Problem
Reward models are costly to train and need large human preference data; it is unclear whether simple reference-based string-matching metrics can replace them as RL rewards for general instruction following.
Approach
Use BLEU against multiple synthetic reference responses as a verifiable reward in GRPO, applied to a subset of hard instructions where the base model's BLEU is low, instead of a learned reward model.
Key results
- On Chatbot Arena, BLEU with five synthetic references reaches 74.2% human agreement versus 75.6% for RM-27B (note: paper reports RM-8B 76.7% and RM-27B 77.6% in text).
- BLEUBERI on Qwen2.5-14B reaches an average of 47.6 across four benchmarks, versus 50.5 for GRPO-RM and 44.2 for SFT.
- Human evaluation on 120 examples: soft preference for BLEUBERI outputs of 67.5% (annotator 1) and 52.5% (annotator 2) versus GRPO-RM outputs.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Optimize BLEU with GRPO instead of using RM-8B as the reward | Qwen2.5-14B · Four instruction-following benchmarks (Average) | average score: 50.5 -> 47.6 | Mixed |
| Optimize BLEU with GRPO instead of using RM-8B as the reward | Qwen2.5-7B · Four instruction-following benchmarks (Average) | average score: 45.9 -> 44.6 | Mixed |
| Select hard examples by low initial BLEU rather than random examples (ablation) | BLEUBERI on Qwen2.5-7B · 1K training examples (Average) | average score: 40.7 -> 45.3 | Helped |
| Select hard examples rather than easy examples (ablation) | BLEUBERI on Qwen2.5-7B · 1K training examples (Average) | average score: 33.4 -> 45.3 | Helped |
| Enforce chain-of-thought using <think>/<answer> tokens and a format reward (ablation) | BLEUBERI on Qwen2.5-7B · 1K hardest examples (Average) | average score: 45.3 -> 38.6 | Hurt |
| Use five references instead of Tulu3 references (ablation) | BLEUBERI on Qwen2.5-7B · 1K hardest examples (Average) | average score: 45.3 -> 44.2 | Hurt |
| Replace BLEU reward with a random reward (ablation) | GRPO on Qwen2.5-7B · Four instruction-following benchmarks (Average) | average score: 44.6 -> 35.3 | Hurt |
| Replace BLEU reward with a random reward (ablation) | GRPO on Qwen2.5-3B · Four instruction-following benchmarks (Average) | average score: 33.8 -> 25.8 | Hurt |
| Use one reference rather than five references for BLEU training (ablation) | BLEUBERI on Qwen2.5-7B · Four instruction-following benchmarks (Average) | average score | Mixed |
| Use n-gram precision and brevity penalty together rather than either component alone (ablation) | BLEU human-preference scoring · Chatbot Arena preference analysis | human agreement | Helped |
Surprising findings
- BLEU with multiple synthetic references matches strong reward models in agreement with human preferences on Chatbot Arena despite BLEU being long considered inadequate for open-ended generation.
- Random rewards did not improve instruction-following performance, unlike reported gains on math reasoning.
- Applying SFT before GRPO hurt Qwen models, though it helped Llama-3.1-8B.
Limitations
- Only two model scales, a 50K data pool, and a single string-overlap metric (BLEU) were studied.
- BLEU is sensitive to reference quality and performs poorly in math reasoning and multilingual settings due to surface-form variation.
- Limited hyperparameter tuning due to computational constraints.
Experimental setup
Datasets: LMSYS chatbot_arena_conversations Tulu 3 SFT mixture MT-Bench ArenaHard v1 ArenaHard v2 WildBench FreshQA LongFact AskHistorians
Benchmarks: MT-Bench ArenaHard v1 ArenaHard v2 WildBench FreshQA LongFact AskHistorians Chatbot Arena human preferences
Metrics: win rate human agreement accuracy F1@K factuality soft preference rate Cohen's kappa 95% bootstrap confidence intervals
Baselines: GRPO-RM SFT Qwen2.5-7B-Instruct ROUGE BERTScore Skywork-Reward-Gemma-2-27B Skywork-Reward-Llama-3.1-8B
Builds on: GRPO RLVR Tulu 3 SFT mixture Skywork reward models
Closest papers by meaning
- Reward-Augmented Data Enhances Direct Preference Alignment of LLMs (ICML 2025 · similarity 0.92)
- RMB: Comprehensively benchmarking reward models in LLM alignment (ICLR 2025 · similarity 0.91)
- Data with High and Consistent Preference Difference Are Better for Reward Model (AAAI 2025 · similarity 0.90)
- MM-RLHF: The Next Step Forward in Multimodal LLM Alignment (ICML 2025 · similarity 0.90)
- Generative RLHF-V: Learning Principles from Multi-modal Human Preference (NeurIPS 2025 · similarity 0.90)
- Sequence to Sequence Reward Modeling: Improving RLHF by Language Feedback (AAAI 2025 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.