Research map / Large language models / Alignment and preferences
REAR: Test-time Preference Realignment through Reward Decomposition
Fuxiang Zhang, Pengcheng Wang, Chenran Li, Yi-Chen Li, Yuxin Chen, Lang Feng, Chenfeng Xu, Masayoshi Tomizuka, Bo An
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use REAR scoring in best-of-N instead of generative reward model scoring | BoN · PrefEval Explicit Preference | score: 69.0 -> 74.1 | Improved |
| replace best-of-N with DVTS tree search under REAR | REAR · PrefEval Explicit Preference | score: 74.1 -> 77.7 | Improved |
| DVTS instead of BoN with REAR on role-play | REAR · Ping-Pong Bench | score: 3.07 -> 3.03 | Hurt |
| apply Linear Alignment test-time method | Qwen2.5-7B-Instruct greedy · PrefEval Explicit Preference | score: 67.0 -> 64.2 | Hurt |
| apply Amulet test-time method | Qwen2.5-7B-Instruct greedy · Ping-Pong Bench | score: 2.97 -> 2.87 | Hurt |
| BoN with generative reward model | Qwen2.5-7B-Instruct · PrefEval (long context) | score: 67.0 -> 69.0 at 0 turns, then near greedy at long contexts (6.7 -> 6.7 at 20 turns) | Hurt |
| add REAR-guided DVTS under long conversation context | Qwen2.5-7B-Instruct · PrefEval Explicit Preference (5 turns) | score: 13.3 -> 33.4 | Improved |
| use REAR to select answers instead of majority voting | Qwen2.5-7B-Instruct and Qwen3-4B-Instruct-2507 · MATH500, AIME24, AIME25, AMC23 | score | Improved |
| best-of-N with REAR on a vision-language model with factual-grounding preference | Qwen3-VL-8B-Instruct · MMHal-Bench | hallucination rate: 28.12 -> 21.87 | Improved |
| use REAR instead of external reward model Skywork-Reward-Llama-8B in BoN | BoN · preference benchmarks | score | Improved |
| vary realignment coefficient lambda (ablation) | BoN w/ REAR · PrefEval explicit and implicit choice | score | Mixed |
| apply REAR with lambda 20 without retuning to Llama | Llama-3.1-8B-Instruct · PrefEval Explicit | score: 74.6 -> 86.4 | Improved |
Closest papers by meaning
- Test-Time Preference Optimization: On-the-Fly Alignment via Iterative Textual Feedback (ICML 2025 · similarity 0.94)
- Reward-Augmented Data Enhances Direct Preference Alignment of LLMs (ICML 2025 · similarity 0.92)
- Transforming and Combining Rewards for Aligning Large Language Models (ICML 2024 · similarity 0.92)
- Sequential Preference Optimization: Multi-Dimensional Preference Alignment with Implicit Reward Modeling (AAAI 2025 · similarity 0.92)
- Online Preference Alignment for Language Models via Count-based Exploration (ICLR 2025 · similarity 0.92)
- Proximalized Preference Optimization for Diverse Feedback Types: A Decomposed Perspective on DPO (NeurIPS 2025 · similarity 0.91)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
