Research map / Reinforcement learning / RL from feedback
ESRL: Efficient Sampling-Based Reinforcement Learning for Sequence Generation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes ESRL, a sampling-based RL method for sequence generation that uses two-stage sampling (sample without graphs, then one parallel forward pass for probabilities) and dynamic sampling that adapts sample size and temperature per input from estimated model capability.
Problem
Policy-gradient RL for sequence generation (MT, summarization, RLHF) needs large-scale sampling over huge action spaces and long sequences, which is computationally and memory expensive.
Approach
Two-stage sampling decouples sampling (no computational graph) from probability computation (single parallel forward pass), and dynamic sampling reduces the sample size and adjusts temperature for inputs the model already handles well, estimated from previous-epoch samples via BLEU/ROUGE or entropy; the loss uses MRT when k>1 and REINFORCE when k=1, with a FIFO-queue baseline reward.
Key results
- On IWSLT'14 De-En with sampling size 15, ESRL-BLEU reaches 35.63 BLEU versus 35.28 for REINFORCE, while cutting memory by about 47% and training time by about 39%.
- On the Vicuna benchmark with LLaMA-7B-LoRA, ESRL-Entropy scores 626.00 versus 596.00 for PPO with two-stage sampling, and trains in 10.19 hours versus 13.87.
- On CNN/DM summarization, ESRL-ROUGE with sampling size 5 gets 38.13 ROUGE-L versus 37.06 for MLE and runs within 6.59 GB memory where REINFORCE and MRT exceed 24 GB.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use ESRL-BLEU with sampling size 20 instead of MLE | Transformer base · WMT'14 En-De test | BLEU: 27.26 -> 28.30 | Helped |
| Use ESRL-ROUGE instead of MLE | Transformer base · CNN/DM | ROUGE-L: 37.06 -> 37.72 | Helped |
| Use ESRL-Entropy instead of PPO with two-stage sampling | LLaMA-7B-LoRA · Vicuna benchmark | Vicuna Score: +30.00 points | Helped |
| Add GOLD-s after ESRL training | ESRL · IWSLT test | SBLEU: 34.95 -> 35.12 | Helped |
| Add two-stage sampling (ablation) | ESRL · IWSLT development set | Memory (G): >24.00 -> 16.35 | Helped |
| Add dynamic sampling (ablation) | ESRL · IWSLT development set | SBLEU: 35.87 -> 36.34 | Helped |
| Add temperature adjustment (ablation) | ESRL · IWSLT development set | SBLEU: 36.02 -> 36.34 | Helped |
| Add FIFO-based baseline reward (ablation) | ESRL · IWSLT development set | SBLEU: 36.17 -> 36.34 | Helped |
| Use top-k sampling instead of beam search, diverse beam search, or top-p sampling (ablation) | ESRL · IWSLT development set | BLEU: 36.46; 36.78; 37.01 -> 37.15 | Helped |
| Use model-capability-based sampling adjustment instead of random adjustment (ablation) | ESRL · Machine translation, abstractive summarization, and RLHF | generation quality | Helped |
| Use elimination ratio 0.3 instead of 0 (ablation) | ESRL-BLEU · IWSLT | SBLEU | Helped |
| Vary the dynamic sampling temperature interval (ablation) | ESRL · IWSLT | BLEU | Mixed |
Surprising findings
- Elimination of samples (elimination ratio 0.3) slightly improved SBLEU over no elimination, which the authors attribute to better exploration-exploitation balance.
- Without dynamic sampling, ESRL failed to gain a well-performed translation model in the ablation, while two-stage sampling alone was insufficient to reduce memory for the sampling-20 setting.
Limitations
- Two-stage sampling adds an extra forward pass, incurring additional time cost.
- Capability estimation relies on samples from the previous epoch and uses greedy decoding for unseen inputs.
- RLHF evaluation uses GPT-4 as judge on the Vicuna benchmark with a single sampling size of 1.
Experimental setup
Datasets: IWSLT'14 De-En WMT'14 En-De CNN/DM Alpaca 52k GPT-4 Alpaca (English) GPT-4 Comparison (English)
Benchmarks: IWSLT'14 De-En WMT'14 En-De CNN/DM Vicuna benchmark
Metrics: sacreBLEU BLEU COMET-22 ROUGE-L BARTScore Vicuna total score training time peak memory
Baselines: MLE REINFORCE MRT ESRL-Random PPO SFT GOLD-s GOLD-p
Builds on: REINFORCE minimum risk training PPO GOLD LoRA
Closest papers by meaning
- TTRL: Test-Time Reinforcement Learning (NeurIPS 2025 · similarity 0.89)
- Sequence to Sequence Reward Modeling: Improving RLHF by Language Feedback (AAAI 2025 · similarity 0.88)
- Overcoming Slow Decision Frequencies in Continuous Control: Model-Based Sequence Reinforcement Learning for Model-Free Control (ICLR 2025 · similarity 0.88)
- Parallelizing Model-based Reinforcement Learning Over the Sequence Length (NeurIPS 2024 · similarity 0.88)
- On Predictability of Reinforcement Learning Dynamics for Large Language Models (ICLR 2026 · similarity 0.88)
- FASTER: Value-Guided Sampling for Fast RL (NeurIPS 2026 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.