Research map / Large language models / Alignment and preferences
Linear Alignment: A Closed-form Solution for Aligning Human Preferences without Tuning and Feedback
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Linear Alignment, a tuning-free method that aligns LLM outputs to human preferences in one inference step by linearly shifting output logits along the difference between principle-prompted and plain next-token distributions, with a closed-form policy under divergence constraints.
Problem
Aligning LLMs with general and personalized human preferences normally requires preference annotation, reward modelling and parameter tuning (e.g., PPO, DPO); the paper asks whether aligned decoding can be obtained without training or feedback.
Approach
Derive a closed-form policy update under a divergence constraint via a first-order linear approximation of the Q function, and estimate the gradient direction without training by contrasting the model's logits with and without preference principle prompts (self-contrastive decoding), then amplify this shift by a tunable step size.
Key results
- On Vicuna-7B-v1.5, Linear Alignment achieves 40.9% win vs 19.0% lose against the SFT model in GPT-4 evaluation, versus PPO's 32.4% win and 16.8% lose.
- On the personal preference dataset, Mistral-7B with Linear Alignment reaches 77.2% total accuracy, above ChatGPT (76.6%) and below GPT-4 (91.1%).
- Linear Alignment reduces inference throughput modestly on Vicuna-7B (46 to 42.4 tokens/s) while increasing GPU memory from 14129 MB to 15347 MB.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| apply Linear Alignment | Vicuna-7B · Personal preference dataset (Total) | accuracy: 57.1 -> 67.0 | Helped |
| apply Linear Alignment | Llama2-7B-Chat · Personal preference dataset (Total) | accuracy: 53.8 -> 57.1 | Helped |
| apply Linear Alignment | Llama2-13B-Chat · Personal preference dataset (Total) | accuracy: 59.1 -> 63.7 | Helped |
| apply Linear Alignment | Llama2-13B-Chat · Technology | accuracy: 56.2 -> 54.2 | Hurt |
| apply Linear Alignment | Llama2-13B-Chat · Daily Life | accuracy: 67.6 -> 73.1 | Helped |
| apply Linear Alignment | Mistral-7B · Personal preference dataset (Total) | accuracy: 66.1 -> 77.2 | Helped |
| apply Linear Alignment | Vicuna-7B-v1.5 · Anthropic-RLHF-HH vs SFT | win rate: 32.4 -> 40.9 | Helped |
| apply Linear Alignment | Mistral-7B-Instruct · Anthropic-RLHF-HH vs SFT | win rate: 36.1 -> 35.2 | Mixed |
| apply helpful and harmless Linear Alignment | Vicuna · MMLU (ppl) | score: 50.96 -> 50.95 | No clear effect |
| apply helpful and harmless Linear Alignment | Mistral · GSM8K (gen) | score: 43.37 -> 42.76 | Hurt |
| apply Linear Alignment | Vicuna-7b-v1.5 · Inference | tokens/s: 46 -> 42.4 | Hurt |
Surprising findings
- Best-of-N achieves a higher win rate against the SFT model (48.6-48.7% on Vicuna) than Linear Alignment, though the paper attributes this to GPT-4 bias toward responses from the same distribution.
- Llama2-13B-Chat with Linear Alignment drops on the Technology domain, and RLHF-tuned Llama2-7B-Chat benefits less from linear alignment than its SFT-only counterpart, suggesting over-alignment.
Limitations
- Step size is a manually tuned hyperparameter rather than derived from the constraint, as the authors admit.
- Inference requires two forward passes (with and without principle), roughly doubling compute, and the paper reports reduced downstream PPL-based performance on some tasks.
Experimental setup
Datasets: Anthropic-RLHF-HH OpenCompass downstream tasks
Benchmarks: Anthropic-RLHF-HH Personal preference dataset MMLU CSQA HellaSwag PIQA SIQA DROP GSM8K SQuAD2.0 Natural Questions HumanEval
Metrics: win rate tie rate lose rate multiple-choice accuracy tokens per second GPU memory
Baselines: PPO DPO Best-of-16 Best-of-50 Llama2-7B-Chat Llama2-13B-Chat ChatGPT GPT-4
Builds on: PPO DPO Contrastive Decoding TRPO Constitutional AI
Closest papers by meaning
- Preference Ranking Optimization for Human Alignment (AAAI 2024 · similarity 0.93)
- Understanding the Learning Dynamics of Alignment with Human Feedback (ICML 2024 · similarity 0.93)
- Towards Efficient Exact Optimization of Language Model Alignment (ICML 2024 · similarity 0.92)
- RLTHF: Targeted Human Feedback for LLM Alignment (ICML 2025 · similarity 0.92)
- Distortion of AI Alignment: Does Preference Optimization Optimize for Preferences? (NeurIPS 2025 · similarity 0.92)
- On-the-fly Preference Alignment via Principle-Guided Decoding (ICLR 2025 · similarity 0.92)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.