Research map / Large language models / Alignment and preferences

Language Model Decoding as Direct Metrics Optimization

ICLR 2024 · Large language models › Alignment and preferences · approach: LLM-as-judge and feedback data · 1 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Frames LM decoding as reverse-KL projection subject to matching expected metric scores with human text, yielding an energy-reweighted LM distribution sampled via sampling-importance-resampling (DAEMON).

Problem

Existing decoding methods (sampling-based truncation vs. search-based contrastive methods) fail to align generated text with human text across multiple aspects like repetition, coherence, diversity and information content simultaneously.

Approach

Decode from the reverse-KL information projection of the LM onto the set of distributions that match human-text expectations of chosen metrics; the solution is an exponential-family reweighting p_theta * exp(-mu^T f), with mu estimated by WIS and sampled via SIR over temperature-modulated candidates. The authors prove this distribution has lower perplexity on human text than the LM.

Key results

• On Wikipedia with GPT-2 XL, DAEMON achieves the lowest sr-4 (0.42 vs. human 0.48) and highest MAUVE (88.1) among compared decoding methods.
• Perplexity on human text drops from 23.1 to 22.0 for GPT-2 XL on Wikipedia and from 13.9 to 13.1 on News.
• In human pairwise evaluation on Wikipedia, DAEMON wins on coherence against Nucleus (0.66 vs. 0.15 lose) and Typical (0.62 vs. 0.19).

Measured effects reported in the paper

Surprising findings

• Contrastive Search achieves higher MAUVE than Contrastive Decoding at length 256, contradicting prior results obtained at length 128.

Limitations

• Decoding requires sampling M candidates per step and resampling, increasing latency over greedy decoding.
• Only equality constraints are handled; inequality or structural constraints are left for future work.
• Evaluated mainly on Wikipedia and News with GPT-2 XL and OPT-6.7B.

Experimental setup

Datasets: WikiText-103 Wikinews

Benchmarks: WikiText-103 Wikinews

Metrics: seq-rep-n tok-rep-l coherence (SimCSE cosine) diversity entropy exponent MAUVE perplexity

Baselines: Greedy Top-k sampling Nucleus sampling Typical decoding Contrastive Decoding Contrastive Search

Builds on: top-k sampling nucleus sampling contrastive search contrastive decoding

Closest papers by meaning

• Decoding-Time Language Model Alignment with Multiple Objectives (NeurIPS 2024 · similarity 0.90)
• Approximately Aligned Decoding (NeurIPS 2025 · similarity 0.88)
• Weak-to-Strong Search: Align Large Language Models via Searching over Small Language Models (NeurIPS 2024 · similarity 0.88)
• Fast Best-of-N Decoding via Speculative Rejection (NeurIPS 2024 · similarity 0.87)
• Decoding-time Realignment of Language Models (ICML 2024 · similarity 0.87)
• Judge Decoding: Faster Speculative Sampling Requires Going Beyond Model Alignment (ICLR 2025 · similarity 0.87)