Research map / Large language models / Alignment and preferences

PAD: Personalized Alignment of LLMs at Decoding-time

ICLR 2025 · Large language models › Alignment and preferences · approach: Steering and correcting LLM behavior · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes PAD, a training-free decoding-time method that guides a frozen general-aligned LLM toward personalized preferences using token-level rewards from a single personalized reward model (PersRM) trained with a decoupled successor-feature formulation.

Problem

Aligning LLM outputs with diverse personalized preferences (cultural, educational, political, etc.) is costly because traditional alignment methods need per-preference data and policy training.

Approach

Decouple text-generation dynamics from personalized preferences by defining the reward as a linear function of state-action features and a preference weight vector; train one reward model whose backbone yields features and a value head yields preference weights, then at decoding time re-rank the base model's top-k tokens by the preference-weighted implicit log-ratio reward added to the base log-li

Key results

• On P-Soups, PAD (1-dim) reaches an average win rate of 84% and raises the average reward model score from 0.32 to 1.28, surpassing baselines on 7 of 10 metrics.
• On HelpSteer2, PAD (1-dim) achieves overall reward score 1.19 and GPT-4 win rate 80%, best on 7 of 10 metrics.
• PAD improves Gemma-2b-it, Mistral-7B-SFT, Llama-2-7b-chat and Llama-3-8B-it overall RM scores (e.g., 0.61 to 0.96 for Gemma-2b-it) using the same PersRM, and generation time increases 2-3x over greedy decoding.

Measured effects reported in the paper

Surprising findings

• Reward model scores sometimes disagree with GPT-4 judgments, which the authors note indicates the reward model can be unreliable.
• Decoding-time PAD adds 2-3x generation time and about 17 GB extra memory relative to greedy decoding.

Limitations

• Decoding-time alignment increases test-time compute and significantly slows LLM inference.
• Personalized preferences are simulated via explicit prompts; implicit preferences in dialogue are not studied.
• Reward model scores are sometimes inconsistent with GPT-4 judgments.

Experimental setup

Datasets: HelpSteer2 Rewards-in-Context SafeRLHF BeaverTails-30k hh-rlhf UltraFeedback P-Soups Koala

Benchmarks: P-Soups HelpSteer2 validation

Metrics: reward model score (harmless, helpful, humor) ArmoRM score GPT-4 win rate diversity score

Baselines: MORLHF MODPO Personalized soups Rewarded soups RiC MetaAligner Preference Prompting MOD Aligner

Builds on: RLHF DPO Rewards-in-Context ARGS successor features

Closest papers by meaning

• MATO: Multi-objective Personalized Alignment with Test-time Optimization for Large Language Models (NeurIPS 2026 · similarity 0.90)
• Inference-Time Personalized Alignment with a Few User Preference Queries (NeurIPS 2025 · similarity 0.90)
• Aligning to Thousands of Preferences via System Message Generalization (NeurIPS 2024 · similarity 0.89)
• ReMoDetect: Reward Models Recognize Aligned LLM's Generations (NeurIPS 2024 · similarity 0.88)
• TODO: Enhancing LLM Alignment with Ternary Preferences (ICLR 2025 · similarity 0.88)
• On Evaluating LLM Alignment by Evaluating LLMs as Judges (NeurIPS 2025 · similarity 0.88)