Research map / Reinforcement learning / RL from feedback
Is Reinforcement Learning (Not) for Natural Language Processing: Benchmarks, Baselines, and Building Blocks for Natural Language Policy Optimization
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| fine-tune with RL (PPO) instead of supervised learning | GPT-2 · IMDB | sentiment score: 0.539 -> 0.602 | Helped |
| replace PPO with NLPO (top-p action masking) | GPT-2 · IMDB | sentiment score: 0.602 -> 0.611 | Helped |
| warm-start RL with supervised fine-tuning (ablation) | NLPO · IMDB | sentiment score: 0.611 -> 0.620 | Helped |
| remove KL penalty (ablation) | NLPO · IMDB | perplexity: 33.832 -> 41.429 | Hurt |
| lower top-p of NLPO masking policy from 0.9 to 0.1 (ablation) | NLPO · IMDB | sentiment score: 0.611 -> 0.579 | Hurt |
| reward classifier trained on 10% of data (ablation) | NLPO · IMDB | sentiment score: 0.611 -> 0.599 | Hurt |
| use bandit setting (gamma=1) instead of discounted token-level MDP (ablation) | NLPO · IMDB | perplexity: 33.832 -> 43.720 | Hurt |
| use bandit setting (gamma=1) instead of discounted token-level MDP (ablation) | PPO · IMDB | sentiment score: 0.602 -> 0.651 | Mixed |
| learned human preference reward instead of METEOR-only reward | Supervised+RL · CommonGen | pairwise human preference (count): 587 -> 682 | Helped |
| RL on 10% reward data vs supervised with 5x more data (ablation) | GPT-2 · IMDB | sentiment score | Helped |
Closest papers by meaning
- Leftover Lunch: Advantage-based Offline Reinforcement Learning for Language Models (ICLR 2024 · similarity 0.92)
- MA-RLHF: Reinforcement Learning from Human Feedback with Macro Actions (ICLR 2025 · similarity 0.92)
- Degeneration-free Policy Optimization: RL Fine-Tuning for Language Models without Degeneration (ICML 2024 · similarity 0.91)
- Sequence to Sequence Reward Modeling: Improving RLHF by Language Feedback (AAAI 2025 · similarity 0.91)
- Online Preference Alignment for Language Models via Count-based Exploration (ICLR 2025 · similarity 0.91)
- Towards Efficient Exact Optimization of Language Model Alignment (ICML 2024 · similarity 0.91)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.