AI Paper Atlas

Research map / Large language models / Alignment and preferences

REAR: Test-time Preference Realignment through Reward Decomposition

Fuxiang Zhang, Pengcheng Wang, Chenran Li, Yi-Chen Li, Yuxin Chen, Lang Feng, Chenfeng Xu, Masayoshi Tomizuka, Bo An

ICML 2026 · Large language models › Alignment and preferences · approach: Reward model design and preference models · 0 citations (OpenAlex)

Read the paperCheck your idea against 137,217 papers

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
use REAR scoring in best-of-N instead of generative reward model scoringBoN · PrefEval Explicit Preferencescore: 69.0 -> 74.1Improved
replace best-of-N with DVTS tree search under REARREAR · PrefEval Explicit Preferencescore: 74.1 -> 77.7Improved
DVTS instead of BoN with REAR on role-playREAR · Ping-Pong Benchscore: 3.07 -> 3.03Hurt
apply Linear Alignment test-time methodQwen2.5-7B-Instruct greedy · PrefEval Explicit Preferencescore: 67.0 -> 64.2Hurt
apply Amulet test-time methodQwen2.5-7B-Instruct greedy · Ping-Pong Benchscore: 2.97 -> 2.87Hurt
BoN with generative reward modelQwen2.5-7B-Instruct · PrefEval (long context)score: 67.0 -> 69.0 at 0 turns, then near greedy at long contexts (6.7 -> 6.7 at 20 turns)Hurt
add REAR-guided DVTS under long conversation contextQwen2.5-7B-Instruct · PrefEval Explicit Preference (5 turns)score: 13.3 -> 33.4Improved
use REAR to select answers instead of majority votingQwen2.5-7B-Instruct and Qwen3-4B-Instruct-2507 · MATH500, AIME24, AIME25, AMC23scoreImproved
best-of-N with REAR on a vision-language model with factual-grounding preferenceQwen3-VL-8B-Instruct · MMHal-Benchhallucination rate: 28.12 -> 21.87Improved
use REAR instead of external reward model Skywork-Reward-Llama-8B in BoNBoN · preference benchmarksscoreImproved
vary realignment coefficient lambda (ablation)BoN w/ REAR · PrefEval explicit and implicit choicescoreMixed
apply REAR with lambda 20 without retuning to LlamaLlama-3.1-8B-Instruct · PrefEval Explicitscore: 74.6 -> 86.4Improved

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy