Research map / Large language models / Alignment and preferences

On-the-fly Preference Alignment via Principle-Guided Decoding

ICLR 2025 · Large language models › Alignment and preferences · approach: Steering and correcting LLM behavior · 1 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes OPAD, a tuning-free inference-time method that reweights next-token distributions by the log-ratio between principle-conditioned and unconditioned model predictions, derived from a KL-regularized reward maximization.

Problem

Aligning LLM outputs with human principles and personalized preferences without costly RLHF/DPO training or large preference datasets, at inference time.

Approach

Treats the divergence between the principle-constrained and unconstrained policy as a residual alignment reward (over the last two decoding steps) and derives a closed-form token-level reweighting p ∝ π(y|x,c)·exp(r/β), so decoding is adjusted on the fly with two forward passes per token.

Key results

• On general alignment, OPAD beats Direct Prompting on Summarization with Vicuna-7B (54.3% win vs 23.0% lose) and on HH-RLHF with Mistral-7B (42.0% win vs 22.3% lose), per GPT-4-Turbo judging.
• On DSP personalized alignment with Vicuna-7B, OPAD vs Direct Prompting wins 74.8% and loses 7.0%.
• Inference cost: per-token generation time roughly doubles (3.69 to 7.31 x10^-2 s/token for Vicuna) because two forward passes are required.

Measured effects reported in the paper

Surprising findings

• Principle prompting alone sometimes underperforms direct prompting, and the method shows largest gains on mid-scale models rather than the smallest or largest ones.
• On HH-RLHF, OPAD's advantage over Linear Alignment is small or absent for some models, while RLHF-aligned Llama-3 models still show gaps that OPAD closes.

Limitations

• KL-based reward may fail to capture alignment nuances when constrained and unconstrained policies have little overlap.
• Strict principle adherence can lead to overfitting, producing formulaic or rigid outputs.
• Requires two forward passes per token, doubling inference time.

Experimental setup

Datasets: HH-RLHF Summarization (OpenAI summarize-from-feedback) Domain-Specific Preference (DSP) P-soups

Benchmarks: HH-RLHF Summarization DSP P-soups

Metrics: win rate lose rate Distinct-1 Distinct-2 perplexity ROUGE

Baselines: Direct Prompting Principle Prompting In-context Learning Best-of-N Self-Contrastive Decoding Linear Alignment PPO DPO

Builds on: Linear Alignment Best-of-N DPO RLHF residual energy-based models

Closest papers by meaning

• Linear Alignment: A Closed-form Solution for Aligning Human Preferences without Tuning and Feedback (ICML 2024 · similarity 0.92)
• Latent Preference Coding: Aligning Large Language Models via Discrete Latent Codes (ICML 2025 · similarity 0.92)
• Test-Time Preference Optimization: On-the-Fly Alignment via Iterative Textual Feedback (ICML 2025 · similarity 0.92)
• Beyond Bradley-Terry Models: A General Preference Model for Language Model Alignment (ICML 2025 · similarity 0.91)
• Understanding the Learning Dynamics of Alignment with Human Feedback (ICML 2024 · similarity 0.91)
• Proximalized Preference Optimization for Diverse Feedback Types: A Decomposed Perspective on DPO (NeurIPS 2025 · similarity 0.91)