Research map / Large language models / Alignment and preferences

Latent Preference Coding: Aligning Large Language Models via Discrete Latent Codes

ICML 2025 · Large language models › Alignment and preferences · approach: Reward model design and preference models · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes Latent Preference Coding (LPC), which models hidden preference factors as discrete latent codes learned via variational inference, and plugs this into DPO, SimPO and IPO to improve alignment of Mistral-7B and Llama3-8B models.

Problem

Holistic human preferences mix multiple, sometimes conflicting factors that vary across prompts and tasks, which a single reward function or fixed sub-reward weights cannot capture.

Approach

Introduce a discrete codebook of latent preference codes with a prior network p(z|x) inferred from the prompt and a posterior network q(z|x,y_w≻y_l) from the preference pair; the policy is conditioned on the Gumbel-softmax-mixed code embedding and trained with an ELBO that plugs into offline objectives (DPO, SimPO, IPO).

Key results

• DPO w. LPC on Llama3-8B-Instruct improves average downstream score from 65.16 to 65.90 across four tasks.
• IPO w. LPC on Mistral-7B raises GSM8K from 30.48 to 42.61 and average from 55.60 to 58.61.
• Preference accuracy on UltraFeedback test rises for most settings, e.g., SimPO on Llama3-8B from 69.2 to 71.8.

Measured effects reported in the paper

Surprising findings

• On Llama3-8B-Instruct, LPC's gain on preference accuracy is muted, which the authors attribute to extensive instruction tuning.
• LPC does not significantly improve MMLU and some variants slightly decrease it, which the authors attribute to domain gap.

Limitations

• Population-level (personalized) preferences are not modeled; left for future work.
• Gains on ARC commonsense reasoning are modest, and gains on preference accuracy are small for some settings.
• Continuous-latent ablation shows only small differences from discrete codes.

Experimental setup

Datasets: UltraFeedback

Benchmarks: ARC-Challenge ARC-Easy GSM8K TruthfulQA MMLU AlpacaEval 2 UltraFeedback preference accuracy

Metrics: accuracy preference accuracy length-controlled win rate raw win rate standard deviation over 5 runs

Baselines: DPO IPO SimPO Base model

Builds on: DPO SimPO IPO Gumbel-softmax VAE

Closest papers by meaning

• Sequential Preference Optimization: Multi-Dimensional Preference Alignment with Implicit Reward Modeling (AAAI 2025 · similarity 0.93)
• On-the-fly Preference Alignment via Principle-Guided Decoding (ICLR 2025 · similarity 0.92)
• Reward-Augmented Data Enhances Direct Preference Alignment of LLMs (ICML 2025 · similarity 0.91)
• Panacea: Pareto Alignment via Preference Adaptation for LLMs (NeurIPS 2024 · similarity 0.91)
• Limited Preference Data? Learning Better Reward Model with Latent Space Synthesis (NeurIPS 2025 · similarity 0.91)
• Transforming and Combining Rewards for Aligning Large Language Models (ICML 2024 · similarity 0.91)