Research map / Large language models / Alignment and preferences
Towards Cost-Effective Reward Guided Text Generation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes FaRMA, a reward model that scores all next-token extensions of a prefix in one forward pass and is trained with a Bradley-Terry loss plus a max-consistency TD-style constraint, enabling faster reward-guided text generation.
Problem
Reward-guided text generation (RGTG) incurs high test-time overhead because the reward model must be called once per candidate token, and existing partial-sequence reward models can prefer suboptimal prefixes.
Approach
A single reward model head outputs values for all vocabulary tokens at once, trained with a BT loss on full sequences alternated with a Bellman-style constraint that the value of a prefix equals the max over its next-token extensions, so that the value of a prefix reflects its best completion.
Key results
- On TL;DR, FaRMA achieves average reward 2.05 in about 5 minutes versus 1.73 for CARDS in 17 minutes and 1.46 for ARGS in 32 minutes.
- FaRMA requires 106.54 total model calls per TL;DR response versus 656.69 for ARGS and 338.11 for CARDS.
- On HH dialogue, FaRMA-2.8b reaches reward 1.80 in 5 minutes, competitive with DPO (1.73) and PPO (1.92), and achieves the most diverse outputs (ROUGE-L 0.24).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace conventional candidate-by-candidate reward scoring with one-call vocabulary-wide token scoring | FaRMA versus RGTG baselines · TL;DR | total model calls per response: 656.69 -> 106.54 (ARGS vs FaRMA); 642.29 -> 106.54 (PARGS vs FaRMA); 662.86 -> 106.54 (CD vs FaRMA) | Helped |
| use FaRMA reward-guided generation instead of ARGS, PARGS, or CD | Llama3.2-1B-Instruct · TL;DR Summarization | average reward: 1.46$\pm$0.16 -> 2.05$\pm$0.15; 1.56$\pm$0.19 -> 2.05$\pm$0.15; 1.15$\pm$0.16 -> 2.05$\pm$0.15 | Helped |
| use FaRMA generation instead of DPO | Llama3.2-1B-Instruct · TL;DR Summarization | average reward: 2.08$\pm$0.18 vs 2.05$\pm$0.15 | No clear effect |
| use FaRMA generation instead of conventional RGTG baselines | Pythia-2.8B · HH Dialogue | average reward: 1.41$\pm$0.18 -> 1.80$\pm$0.18; 1.63$\pm$0.17 -> 1.80$\pm$0.18; 1.24$\pm$0.13 -> 1.80$\pm$0.18 | Helped |
| reduce the FaRMA reward model size from 2.8b to 1b | FaRMA · HH Dialogue | average reward: 1.80$\pm$0.18 -> 1.56$\pm$0.18 | Hurt |
| reduce the FaRMA reward model size from 2.8b to 400m | FaRMA · HH Dialogue | average reward: 1.80$\pm$0.18 -> 1.49$\pm$0.12 | Hurt |
| use FaRMA rather than PARGS | Zephyr-7B · Ultra Feedback | average reward: -1.01 $\pm$ 0.26 -> -1.20 $\pm$ 0.21 | Hurt |
| use FaRMA rather than ARGS | Zephyr-7B · Ultra Feedback | average reward: -1.35 $\pm$ 0.31 -> -1.20 $\pm$ 0.21 | Helped |
| set $\beta$ to 1.5 rather than 0.5 (ablation) | FaRMA · TL;DR Summarization | average reward: $1.33 \pm 0.18$ -> $2.11 \pm 0.16$ | Helped |
| set $\beta$ to 1.5 rather than 1.0 (ablation) | FaRMA · TL;DR Summarization | average reward: $1.77 \pm 0.17$ -> $2.11 \pm 0.16$ | Helped |
| set $\beta$ to 2.0 rather than 1.5 (ablation) | FaRMA · TL;DR Summarization | average reward: $2.11 \pm 0.16$ -> $2.10 \pm 0.14$ | No clear effect |
Surprising findings
- CARDS produces very low diversity (ROUGE-L 0.86 on HH), indicating near-identical generations.
- A smaller FaRMA reward model (400M) still improves over the frozen base model on HH dialogue.
Limitations
- Evaluation relies on a small number of samples (50-100) with overlapping standard errors.
- Theoretical guarantee holds only in the limit of infinite data and expressive value function.
- Rewards are measured by a full-sequence reward model that may be correlated with the training reward model.
Experimental setup
Datasets: Reddit TL;DR preference dataset Anthropic HH-RLHF UltraFeedback
Benchmarks: Reddit TL;DR Anthropic HH-RLHF UltraFeedback
Metrics: average reward from full-sequence reward model GPT-4 win rate ROUGE-L diversity number of LLM and RM calls generation time
Baselines: ARGS PARGS CD CARDS DPO PPO top-k sampling
Builds on: ARGS PARGS Controlled decoding Value Augmented Sampling DPO
Closest papers by meaning
- Sequence to Sequence Reward Modeling: Improving RLHF by Language Feedback (AAAI 2025 · similarity 0.90)
- GRAM: A Generative Foundation Reward Model for Reward Generalization (ICML 2025 · similarity 0.89)
- RM-Bench: Benchmarking Reward Models of Language Models with Subtlety and Style (ICLR 2025 · similarity 0.88)
- RM-R1: Reward Modeling as Reasoning (ICLR 2026 · similarity 0.88)
- Exploring Data Scaling Trends and Effects in Reinforcement Learning from Human Feedback (NeurIPS 2025 · similarity 0.88)
- Provably Efficient Online RLHF with One-Pass Reward Modeling (NeurIPS 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.