Research map / Large language models / Alignment and preferences

Compositional Preference Models for Aligning LMs

ICLR 2024 · Large language models › Alignment and preferences · approach: Robust and calibrated DPO · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes Compositional Preference Models, which score interpretable features with a prompted LM and combine them via logistic regression, yielding PMs that are more robust to overoptimization and better for best-of-n selection than standard PMs.

Problem

Standard preference models overfit the preference data, are prone to reward hacking/overoptimization, and are opaque, collapsing complex preferences into a single scalar.

Approach

Decompose a global preference judgment into 13-14 human-interpretable features, score each with a prompted LM (GPT-3.5 or Flan-T5-XL) without weight updates, and aggregate the standardized scores with a logistic regression trained on pairwise comparisons, so the only trained parameters are a small set of coefficients.

Key results

• On HH-RLHF, CPM-GPT-3.5 achieves a 0.810 win rate vs. vanilla Flan-T5 under BoN(n=16), versus 0.588 for standard PM and 0.742 for CPM-Flan-T5.
• Rank correlation between gold and proxy PM scores on HH-RLHF is 0.997 for CPM-GPT-3.5 and 0.665 for standard PM; on SHP it is 0.981 vs 0.057.
• CPMs show markedly smaller divergence between independently trained PMs on 10K subsets across BoN sizes up to 256, and win rate improves with extractor LM size.

Measured effects reported in the paper

Surprising findings

• A single holistic 'helpfulness' feature prompted through an LM (win rate 0.707) already outperforms the standard fine-tuned PM (0.588).
• On SHP, CPM-GPT-3.5 (0.672) is outperformed by the externally trained reference PM only in HH-RLHF; the reference PM scored 0.829 on HH-RLHF, above CPM-GPT-3.5's 0.810.

Limitations

• Evaluation relies on Claude 2 as a proxy for human judgment rather than genuine human evaluation.
• Features are hand-specified and may overlap; feature diversity and selection are not learned.
• GPT-3.5 BoN experiments were limited to n<=16 due to computational constraints.

Experimental setup

Datasets: HH-RLHF SHP

Benchmarks: HH-RLHF SHP

Metrics: win rate vs vanilla Flan-T5 BoN score gap between PMs (robustness) rank correlation between gold and proxy PM scores

Baselines: standard preference model (Flan-T5-XL with scalar head) reference PM (DeBERTa-v3-large, OpenAssistant) CPM-Flan-T5

Builds on: RLHF best-of-n sampling Goodhart's law overoptimization scaling

Closest papers by meaning

• Latent Preference Coding: Aligning Large Language Models via Discrete Latent Codes (ICML 2025 · similarity 0.90)
• Spread Preference Annotation: Direct Preference Judgment for Efficient LLM Alignment (ICLR 2025 · similarity 0.89)
• Panacea: Pareto Alignment via Preference Adaptation for LLMs (NeurIPS 2024 · similarity 0.89)
• Reward-Augmented Data Enhances Direct Preference Alignment of LLMs (ICML 2025 · similarity 0.89)
• Leveraging robust optimization for llm alignment under distribution shifts (NeurIPS 2025 · similarity 0.89)
• Larger or Smaller Reward Margins to Select Preferences for LLM Alignment? (ICML 2025 · similarity 0.89)