Research map / Large language models / Alignment and preferences
MA-RLHF: Reinforcement Learning from Human Feedback with Macro Actions
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes MA-RLHF, which groups tokens into macro actions (fixed/randomized n-gram, parsing, or perplexity-based termination) inside PPO to speed credit assignment in RLHF, reporting 1.7-2x faster reward gains on summarization, dialogue, QA and code.
Problem
Token-level RLHF suffers from credit assignment over long sequences because rewards are delayed to the end of long token sequences, slowing learning and convergence.
Approach
Treat contiguous token chunks (macro actions) as options in an SMDP-style PPO, computing macro-level advantages and importance ratios over joint token probabilities, while keeping the original token vocabulary so no extra training or inference cost is added.
Key results
- On TL;DR, MA-PPO (2B) reaches test RM 1.41 vs 0.84 for vanilla PPO (+68%), and MA-PPO (7B) reaches 2.47 vs 1.90 (+30%).
- MA-PPO reaches parity with vanilla PPO roughly 1.7-2x faster in training steps; GPT-4 win rate vs PPO on TL;DR is 78% (2B) and 86% (7B).
- On APPS code generation, MA-PPO on 7B improves pass@1 by 35% and pass@5 by 30% over vanilla PPO.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use macro actions instead of token-level PPO | Gemma-2B · TL;DR | RM score: 0.84 -> 1.41 (+68%) | Helped |
| Use macro actions instead of token-level PPO | Gemma-2B · HH-RLHF | RM score: 1.31 -> 1.55 (+18%) | Helped |
| Use macro actions instead of token-level PPO | Gemma-2B · WebGPT | RM score: -0.62 -> -0.60 (+3%) | Helped |
| Use macro actions instead of token-level PPO | Gemma-7B · TL;DR | RM score: 1.90 -> 2.47 (+30%) | Helped |
| Use macro actions instead of token-level PPO | Gemma-7B · HH-RLHF | RM score: 1.05 -> 1.24 (+18%) | Helped |
| Use macro actions instead of token-level PPO | Gemma-7B · WebGPT | RM score: -0.61 -> -0.56 (+8%) | Helped |
| Use macro actions (n=5) instead of PPO | Llama-3.2-3B · TL;DR | RM score: 3.33 -> 3.96 | Helped |
| Use parsing-based rather than fixed 5-gram termination (ablation) | MA-PPO · HH-RLHF | RM score: 1.55 -> 1.64 | Helped |
| Use parsing-based rather than fixed 5-gram termination (ablation) | MA-PPO · TL;DR | RM score: 1.40 -> 1.37 | Hurt |
| Use parsing-based rather than fixed 5-gram termination (ablation) | MA-PPO · HH-RLHF | GPT-4 win rate vs PPO: 58% -> 62% | Helped |
| Use parsing-based rather than fixed 5-gram termination (ablation) | MA-PPO · TL;DR | GPT-4 win rate vs PPO: 78% -> 78% | No clear effect |
| Use perplexity-based rather than fixed 5-gram termination (ablation) | MA-PPO · TL;DR | RM score: 1.40 -> 1.27 | Hurt |
Surprising findings
- Setting n to infinity (treating the whole sequence as one macro action, REINFORCE-like) gave the highest RM score on TL;DR.
- PPO on WebGPT showed reward hacking via repetition tokens, requiring early stopping.
Limitations
- Macro action termination rules are hand-designed and rule-based rather than learned.
- Experiments limited to models up to 27B parameters and mostly Gemma family; no complex reasoning tasks evaluated.
- Relies on reward models that show limited agreement with GPT-4 and human judgments; WebGPT policy optimization was unstable and needed early stopping.
Experimental setup
Datasets: TL;DR (OpenAI Summarization) Anthropic HH-RLHF WebGPT Comparisons APPS
Benchmarks: TL;DR HH-RLHF WebGPT Comparisons APPS
Metrics: test reward model score GPT-4 win rate human win rate pass@1 pass@5 best-of-N RM score
Baselines: vanilla PPO DPO RLOO
Builds on: PPO options framework semi-Markov decision process InstructGPT RLHF pipeline DeepSpeed-Chat
Closest papers by meaning
- Sequence to Sequence Reward Modeling: Improving RLHF by Language Feedback (AAAI 2025 · similarity 0.94)
- RLAIF vs. RLHF: Scaling Reinforcement Learning from Human Feedback with AI Feedback (ICML 2024 · similarity 0.92)
- Provably Efficient Online RLHF with One-Pass Reward Modeling (NeurIPS 2025 · similarity 0.92)
- SELF-EVOLVED REWARD LEARNING FOR LLMS (ICLR 2025 · similarity 0.91)
- Dense Reward for Free in Reinforcement Learning from Human Feedback (ICML 2024 · similarity 0.91)
- Nash Learning from Human Feedback (ICML 2024 · similarity 0.91)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.