Research map / Large language models / Alignment and preferences
Decoding-Time Language Model Alignment with Multiple Objectives
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes multi-objective decoding (MOD), a training-free decoding-time method that combines the token distributions of separately aligned single-objective LMs via a Legendre-transform-derived closed form, with optimality guarantees under f-divergence regularization.
Problem
Aligning language models to multiple, user-specific weightings of objectives without retraining for every preference weighting.
Approach
Under strong-barrier f-divergence regularization, each single-objective optimal policy has a closed-form bijection with its reward, so the optimal policy for any linear reward combination can be approximated token-wise by a weighted combination of the base models' transformed probabilities (for reverse KL, a product of powered probabilities, i.e., logit combination), requiring no retraining.
Key results
- On Helpful Assistant with 3 objectives, MOD achieves 12.8% overall relative reward improvement over rewarded soups (parameter merging).
- Combining CodeTülu-2-7B, Tülu-2-HH-13B and Tülu-2-Ultra-13B with w=(0.75,0.1,0.15) reduces Toxigen to 0.58 and improves Codex@1 to 45.12 vs 41.68 for CodeTülu-2-7B (about 8% relative gain).
- The paper claims parameter-merging (rewarded soups) can be sub-optimal under common settings, and MOD generally lies over RS Pareto frontiers on Reddit Summary and Safety Alignment.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use MOD with weights (0.75, 0.1, 0.15) to combine CodeTulu-2-7B, Tulu-2-HH-13B, and Tulu-2-Ultra-13B | Three-model combination · Open Instruction-Following | Toxigen: 5 -> 0.58 | Helped |
| Use MOD with weights (0.75, 0.1, 0.15) to combine CodeTulu-2-7B, Tulu-2-HH-13B, and Tulu-2-Ultra-13B | Three-model combination · Open Instruction-Following | Codex@1: 41.68 -> 45.12 | Helped |
| Use MOD with weights (0.75, 0.1, 0.15) to combine CodeTulu-2-7B, Tulu-2-HH-13B, and Tulu-2-Ultra-13B | Three-model combination · Open Instruction-Following | GSM COT: 33 -> 44 | Helped |
| Use MOD with weights (0.75, 0.1, 0.15) to combine CodeTulu-2-7B, Tulu-2-HH-13B, and Tulu-2-Ultra-13B | Three-model combination · Open Instruction-Following | BBH COT: 49.1 -> 52.96 | Helped |
| Increase the negative weight on harmlessness from 0 to -4 while increasing helpfulness weight | MOD with f-DPO models · Safety Alignment | Harmless: -8.26 -> -14.88 | Hurt |
| Increase the helpfulness weight from 1 to 5 while setting the harmlessness weight from 0 to -4 | MOD with f-DPO models · Safety Alignment | Helpful: 7.28 -> 7.49 | Mixed |
| Use MOD rather than parameter merging (RS) for three equally weighted objectives | Helpful Assistant · Helpful Assistant | overall reward: 12.8% | Helped |
Surprising findings
- Rewarded soups produce degenerate outputs (e.g., repeated 'Start') for negative weightings, while MOD still produces meaningful responses for weights like (5,-4), implying the model learned the negated reward.
Limitations
- Analysis is primarily in a tabular setting and does not account for function approximation error.
- Token-level greedy decoding of a response-level distribution may degrade performance.
- Requires loading multiple models simultaneously, incurring higher memory cost.
Experimental setup
Datasets: Summarize-from-Feedback Anthropic-HH BeaverTails-10K UltraFeedback HelpSteer
Benchmarks: Reddit Summary Helpful Assistant Safety Alignment (BeaverTails) Open Instruction-Following Toxigen Codex@1 GSM COT BBH COT
Metrics: Pareto frontier area reward model scores toxicity pass@1 accuracy
Baselines: Rewarded soups MORLHF MODPO parameter merging
Builds on: PPO DPO f-DPO MODPO proxy-tuning
Closest papers by meaning
- Decoding-time Realignment of Language Models (ICML 2024 · similarity 0.91)
- Aligner: Efficient Alignment by Learning to Correct (NeurIPS 2024 · similarity 0.91)
- MATO: Multi-objective Personalized Alignment with Test-time Optimization for Large Language Models (NeurIPS 2026 · similarity 0.91)
- Language Model Decoding as Direct Metrics Optimization (ICLR 2024 · similarity 0.90)
- Collab: Controlled Decoding using Mixture of Agents for LLM Alignment (ICLR 2025 · similarity 0.90)
- Aligning Large Language Models with Representation Editing: A Control Perspective (NeurIPS 2024 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.