Research map / Large language models / Alignment and preferences
GenARM: Reward Guided Generation with Autoregressive Reward Model for Test-Time Alignment
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces GenARM, a test-time alignment method that trains an Autoregressive Reward Model predicting next-token rewards (reward as a log-probability) and combines it with a frozen LLM's logits for guided decoding, supporting weak-to-strong and multi-objective alignment.
Problem
Test-time alignment methods guide frozen LLMs with trajectory-level reward models, which are trained only on complete responses and so give inaccurate next-token rewards for partial responses, or they are costly because they generate full continuations per candidate token.
Approach
Parametrize the reward of a full response as a sum of token-level log-probabilities from a learnable autoregressive distribution, trained with the standard Bradley-Terry loss; at inference, sample each token from the base LLM probability times the reward model's next-token probability raised to 1/beta, requiring one forward pass per token.
Key results
- On HH-RLHF with GPT-4 judging, GenARM beats ARGS (65.33% win vs 8.22% tie, 26.44% lose) and Transfer-Q (66.22% win) and slightly beats DPO (48.00% win vs 45.11% lose).
- Inference for 128 tokens with 7B base and 7B RM: GenARM 7.28 s vs ARGS 7.74 s, CARDS 87.09 s, Transfer-Q 130.53 s.
- Weak-to-strong: a 7B Autoregressive RM guiding Tulu2-70B recovers over 70% of the raw and LC win-rate gap to Tulu2-DPO-70B on AlpacaEval 2.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use GenARM instead of ARGS | LLaMA-7B-SFT test-time alignment · HH-RLHF test | Win + ½ Tie (%): 26.89 ± 0.19 -> 69.44 ± 0.38 | Helped |
| Use GenARM instead of Transfer-Q | LLaMA-7B-SFT test-time alignment · HH-RLHF test | Win + ½ Tie (%): 33.72 ± 0.25 -> 69.17 ± 0.29 | Helped |
| Use GenARM instead of CARDS | LLaMA-7B-SFT test-time alignment · HH-RLHF test | Win + ½ Tie (%): 41.94 ± 0.25 -> 57.27 ± 0.19 | Helped |
| Use GenARM instead of DPO | LLaMA-7B-SFT alignment · HH-RLHF test | Win + ½ Tie (%): GenARM: 51.44 ± 0.35 | Helped |
| Use autoregressive next-token rewards instead of trajectory-level reward guidance | 7B base LLM and 7B RM · Generate 128 tokens | Inference time (s): 7.74 -> 7.28 | Helped |
| Use autoregressive next-token rewards instead of Transfer-Q full-response candidate evaluation | 7B base LLM and 7B RM · Generate 128 tokens | Inference time (s): 130.53 -> 7.28 | Helped |
| Use autoregressive next-token rewards instead of CARDS segment selection | 7B base LLM and 7B RM · Generate 128 tokens | Inference time (s): 87.09 -> 7.28 | Helped |
| Use a 7B Autoregressive RM to guide larger frozen base models | Tulu2-13B and Tulu2-70B · AlpacaEval 2 | Raw and length-controlled win rates | Helped |
| Use a 7B Autoregressive RM to guide Tulu2-70B rather than train the 70B model with DPO | Tulu2-70B · AlpacaEval 2 | Raw and length-controlled win rates | Mixed |
| Train an Autoregressive RM instead of a trajectory-level RM | LLaMA-7B-SFT reward model · HH-RLHF training | Training time (seconds): 11,722 seconds -> 12,162 seconds | Hurt |
| Guide the Alpaca-65B model using a 7B Autoregressive RM for multiple preference dimensions | Alpaca-65B · PKU-SafeRLHF-10K | Helpfulness and harmlessness | Helped |
Surprising findings
- Best-of-N with a 7B trajectory-level RM beats GenARM on length-controlled win rate against GPT-4 at the 70B scale, though GenARM wins on raw win rate.
- ARGS struggles to produce coherent long responses at larger guidance weights, producing gibberish.
Limitations
- Evaluation relies on GPT-4 judging and win-rate metrics with limited human validation.
- Experiments focus on human preference alignment; applicability to math and coding reasoning is left for future work.
- The 65B and 70B base models are quantized (GPTQ), and BoN beats GenARM on length-controlled win rate against GPT-4.
Experimental setup
Datasets: HH-RLHF UltraFeedback PKU-SafeRLHF-10K AlpacaEval 2
Benchmarks: HH-RLHF test prompts AlpacaEval 2 PKU-SafeRLHF-10K
Metrics: GPT-4 win/tie/lose rate win + 1/2 tie rate AlpacaEval 2 raw win rate length-controlled win rate inference time for 128 tokens
Baselines: ARGS Transfer-Q CARDS DPO Best-of-N Rewarded Soups MORL
Builds on: ARGS Transfer-Q DPO RLHF Rewarded Soups
Closest papers by meaning
- RMB: Comprehensively benchmarking reward models in LLM alignment (ICLR 2025 · similarity 0.91)
- Test-Time Preference Optimization: On-the-Fly Alignment via Iterative Textual Feedback (ICML 2025 · similarity 0.91)
- PARM: Multi-Objective Test-Time Alignment via Preference-Aware Autoregressive Reward Model (ICML 2025 · similarity 0.91)
- Reward-Augmented Data Enhances Direct Preference Alignment of LLMs (ICML 2025 · similarity 0.90)
- Larger or Smaller Reward Margins to Select Preferences for LLM Alignment? (ICML 2025 · similarity 0.89)
- ReMoDetect: Reward Models Recognize Aligned LLM's Generations (NeurIPS 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.