Research map / Large language models / Alignment and preferences
Test-Time Preference Optimization: On-the-Fly Alignment via Iterative Textual Feedback
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Test-time Preference Optimization (TPO), which aligns frozen LLM outputs to a reward model at inference by turning reward scores into textual critiques (textual loss/gradient) and iteratively rewriting responses.
Problem
Aligning LLMs with human preferences usually requires retraining via gradient-based methods (RLHF, DPO), which is costly and slow to adapt to new preferences.
Approach
Treat the policy LLM as an optimizer in text: it compares best and worst reward-scored responses to write a textual loss, derives a textual gradient, and regenerates candidates, iterating at test time with model parameters fixed (TextGrad-style).
Key results
- TPO (D2-N5) with Llama-3.1-70B-SFT raises Arena-Hard WR to 70.5%, above Llama-3.1-70B-Instruct (59.0%) and Llama-3.1-70B-DPO (50.4%) in the table.
- Mistral-Small-Instruct-2409 with TPO reaches 53.4 LC on AlpacaEval 2 (from 45.7) and 72.2 WR on Arena-Hard (from 53.8).
- With the ultra setting (D5-N20), TPO on Llama-3.1-70B-SFT reaches 37.8 LC and 55.7 WR on AlpacaEval 2 and 77.5 on Arena-Hard.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Apply TPO (D2-N5) to Llama-3.1-70B-SFT using Llama-3.1-Tulu-3-8B-RM | Llama-3.1-70B-SFT · Arena-Hard | WR: 44.1 -> 70.5 | Helped |
| Apply TPO (D2-N5) to Llama-3.1-70B-SFT using FsfairX-LLaMA3-RM-v0.1 | Llama-3.1-70B-SFT · MATH-500 | pass@1 accuracy: 61.8 -> 71.2 | Helped |
| Apply TPO (D2-N5) | Llama-3.1-70B-Instruct · AlpacaEval 2 | LC(%): 36.9 -> 39.1 | Helped |
| Apply TPO (D2-N5) | Mistral-Small-Instruct-2409 · Arena-Hard | WR: 53.8 -> 72.2 | Helped |
| Increase TPO iterations and samples from D2-N5 to D5-N20 (ablation) | Llama-3.1-70B-SFT w/ TPO · AlpacaEval 2 | WR(%): 40.5 -> 55.7 | Helped |
| Use FsfairX-LLaMA3-RM-v0.1 instead of Llama-3.1-Tulu-3-8B-RM for TPO (ablation) | Llama-3.1-70B-SFT · BeaverTails-Evaluation | reward: -4.1 -> -4.8 | Hurt |
| Replace iterative TPO with recursive revision of the best cached response, without a rejected response (ablation) | Llama-3.1-70B-SFT · Reward-model score across TPO iterations | average reward | Mixed |
| Increase search width from 5 to 20 samples per iteration (ablation) | TPO · HH-RLHF | reward-model score | Helped |
| Add sequential revision depth rather than relying only on Best-of-N sampling (ablation) | TPO · AlpacaEval 2, Arena-Hard, and HH-RLHF | GPT-4 win rate: 65.2%; 57.5% | Helped |
| Use TPO iterations on Llama-3.1-8B-Instruct | Llama-3.1-8B-Instruct · HH-RLHF | reward-model score | Hurt |
Surprising findings
- Llama-3.1-8B-Instruct degrades under TPO, with reward scores falling over iterations, suggesting a minimum instruction-following ability is required.
- TPO with 15 total samples (D2-N5) beats Best-of-N with 30 and 60 samples in GPT-4 win-rate comparisons.
Limitations
- Relies on the policy model's instruction-following ability; small models such as Llama-3.1-8B-Instruct fail to improve.
- Gains depend on the reward model proxy, and evaluation uses reward models rather than human preferences directly.
- Test-time inference cost grows with samples and iterations, and models were not trained specifically for TPO prompts.
Experimental setup
Datasets: UltraFeedback HH-RLHF AlpacaEval 2 Arena-Hard BeaverTails XSTest MATH-500
Benchmarks: AlpacaEval 2 Arena-Hard HH-RLHF BeaverTails XSTest MATH-500
Metrics: length-controlled win rate win rate average reward model score accuracy pass@1 standard deviation of reward scores
Baselines: Llama-3.1-70B-DPO Llama-3.1-70B-Instruct Best-of-N sampling revision baseline Mistral-Small-Instruct-2409
Builds on: TextGrad DPO Best-of-N sampling
Closest papers by meaning
- Sequential Preference Optimization: Multi-Dimensional Preference Alignment with Implicit Reward Modeling (AAAI 2025 · similarity 0.92)
- Earlier Tokens Contribute More: Learning Direct Preference Optimization From Temporal Decay Perspective (ICLR 2025 · similarity 0.92)
- Proximalized Preference Optimization for Diverse Feedback Types: A Decomposed Perspective on DPO (NeurIPS 2025 · similarity 0.92)
- Reward-Augmented Data Enhances Direct Preference Alignment of LLMs (ICML 2025 · similarity 0.92)
- 3D-Properties: Identifying Challenges in DPO and Charting a Path Forward (ICLR 2025 · similarity 0.92)
- On-the-fly Preference Alignment via Principle-Guided Decoding (ICLR 2025 · similarity 0.92)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.