Research map / Large language models / Alignment and preferences
Amulet: ReAlignment During Test Time for Personalized Preference Adaptation of LLMs
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Amulet, a training-free test-time personalized preference adaptation method that treats each token's decoding as an online FTRL-proximal learning problem with a closed-form per-iteration update.
Problem
Aligning LLMs with personalized, changing, and diverse user preferences without collecting new data or retraining, at acceptable inference cost.
Approach
Formulates per-token decoding as an independent online learning problem solved with FTRL-proximal, using a utility that contrasts preference-prompted and base-prompted policies, and derives a closed-form update per iteration so the optimization is cheap.
Key results
- Across 64 settings (4 LLMs x 4 datasets x 4 preferences), Amulet achieves the best reward-model score in 75% of settings, versus 20% for LA, 3% for BS16, 2% for Pref, and 0% for Base.
- With Llama-3.1-8B-Instruct, Amulet gains up to 79% over Base on the creative preference (Personal dataset) and beats LA by 35% in that setting.
- On the Personal dataset with GPT-4o win rate, Amulet achieves at least 62.2% win rate against LA on Qwen2-7B-Instruct.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Apply Amulet instead of LA | Llama-3.1-8B-Instruct · Creative preference, average across four datasets | ArmoRM-8B instruction-following reward score: 0.43 -> 0.56 | Helped |
| Apply Amulet instead of LA | Llama-3.1-8B-Instruct · Verbose preference, average across four datasets | ArmoRM-8B instruction-following reward score: 0.35 -> 0.44 | Helped |
| Apply Amulet instead of LA | Llama-3.1-8B-Instruct · Concise preference, average across four datasets | ArmoRM-8B instruction-following reward score: 0.51 -> 0.57 | Helped |
| Apply Amulet instead of LA | Llama-3.1-8B-Instruct · Uplifting preference, average across four datasets | ArmoRM-8B instruction-following reward score: 0.45 -> 0.52 | Helped |
| Increase the number of optimization iterations from 0 to 20 (ablation) | Amulet · Personal dataset; four LLMs and preference dimensions | Reward model score and GPT-4o BT score | Helped |
| Use too many optimization iterations (ablation) | Amulet · Personal dataset; various LLMs and preference dimensions | Reward model score and GPT-4o BT score | Hurt |
| Vary the learning rate eta (ablation) | Amulet · Truthful QA, Llama-3.1-8B-Instruct, creative preference | Reward model score | Mixed |
| Vary alpha and lambda (ablation) | Amulet · Truthful QA, Llama-3.1-8B-Instruct, creative preference | Reward model score | Mixed |
| Use Amulet with different model sizes (ablation) | Amulet · Personal dataset, four preferences | ArmoRM-8B instruction-following reward score | Mixed |
Surprising findings
- Too many optimization iterations degrade performance, which the authors attribute to overfitting the approximate utility.
- Amulet did not win on every setting; for example, it lost to LA on Llama-2-7B-Chat HelpSteer under GPT-4o judging, and it was less effective on Mistral-7B and Qwen2-7B for formal and complex preferences.
Limitations
- Utility is only an approximation of the latent true preference, so more iterations can overfit.
- Relies on GPT-4o as judge and an ArmoRM reward model as the main metric, with no dedicated preference-matched reward model.
- Gains are uneven across models and preferences; some settings underperform LA or baselines.
Experimental setup
Datasets: HelpSteer UltraFeedback Truthful QA UltraChat Personal Preference Eval
Benchmarks: HelpSteer Truthful QA UltraChat Personal Preference Eval
Metrics: ArmoRM-8B instruction-following reward score GPT-4o win rate Bradley-Terry score
Baselines: Base Preference prompting Beam Search (BS16) Linear Alignment
Builds on: Linear Alignment Contrastive Decoding FTRL RAIN URIAL
Closest papers by meaning
- Spread Preference Annotation: Direct Preference Judgment for Efficient LLM Alignment (ICLR 2025 · similarity 0.89)
- Panacea: Pareto Alignment via Preference Adaptation for LLMs (NeurIPS 2024 · similarity 0.89)
- Leveraging robust optimization for llm alignment under distribution shifts (NeurIPS 2025 · similarity 0.88)
- Aligning to Thousands of Preferences via System Message Generalization (NeurIPS 2024 · similarity 0.88)
- Q-Adapter: Customizing Pre-trained LLMs to New Preferences with Forgetting Mitigation (ICLR 2025 · similarity 0.88)
- TODO: Enhancing LLM Alignment with Ternary Preferences (ICLR 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.