Research map / Large language models / Alignment and preferences
Limited Preference Data? Learning Better Reward Model with Latent Space Synthesis
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes LENS, which trains a contrastive VAE on frozen LLM response embeddings, perturbs latents to synthesize preference pairs in embedding space, and trains an MLP reward head on original plus synthetic pairs, with theory bounding synthesis quality and learnability.
Problem
Reward modeling is bottlenecked by costly human preference data, and existing textual synthesis (generate responses plus LLM annotation) is computationally expensive; the paper asks how to expand limited preference data efficiently.
Approach
Synthesize preference pairs directly in the LLM's embedding space by encoding positive and negative response embeddings with a VAE (with a Wasserstein divergence term separating their latent distributions), adding Gaussian noise to latent codes, decoding back to embeddings, and training a lightweight MLP reward head on the augmented set.
Key results
- On HH-RLHF with 8x augmentation, LENS reaches a gold reward of 2.20 versus 1.93 for the best textual baseline (fully fine-tuned).
- On TL;DR at 8x augmentation, LENS reaches 1.48 versus 1.23 for fully fine-tuned textual synthesis.
- Latent synthesis is reported as 18x faster in generation (0.2h vs 3.6h) with a 16,000x smaller generator (0.5M vs 8B parameters).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use latent-space synthesis instead of the strongest textual synthesis baseline at 4× augmentation | Reward modeling with Llama-3.1-8B-Instruct · HH-RLHF | reward score: 1.78 -> 1.94 | Helped |
| Use latent-space synthesis instead of the strongest textual synthesis baseline at 4× augmentation | Reward modeling with Llama-3.1-8B-Instruct · TL;DR | reward score: 0.97 -> 1.44 | Helped |
| Use latent-space synthesis rather than textual synthesis for reward-model generation | HH-RLHF synthesis pipeline · HH-RLHF, 8× | generation time: 3.6h -> 0.2h | Helped |
| Use latent-space synthesis rather than textual synthesis for reward-model generation | HH-RLHF synthesis pipeline · HH-RLHF, 8× | total runtime: 5.2h -> 0.4h | Helped |
| Use the latent-synthesis reward model to select rejection-sampling targets instead of the textual-synthesis reward model | SFT model trained on selected targets · HH-RLHF, 4× rejection sampling | GPT-4 win rate: 39% -> 61% | Helped |
| Set divergence-loss weight gamma to 0.1 rather than 0 (ablation) | LENS VAE · HH-RLHF | reward performance | Helped |
| Increase divergence-loss weight gamma excessively to 0.5 or 1.0 (ablation) | LENS VAE · HH-RLHF | reward performance | Hurt |
| Use moderate synthesis noise variance 0.01 instead of 0.001 (ablation) | LENS latent sampling · HH-RLHF, 4× | reward score: 1.63 -> 1.96 | Helped |
| Increase synthesis noise variance to 1.0 rather than 0.01 (ablation) | LENS latent sampling · HH-RLHF, 4× | reward score: 1.96 -> 1.51 | Hurt |
| Set KL divergence weight beta to 1.0 rather than 0.1 (ablation) | LENS VAE · HH-RLHF, 4× | reward score: 1.48 -> 1.96 | Helped |
| Set KL divergence weight beta to 5.0 rather than 1.0 (ablation) | LENS VAE · HH-RLHF, 4× | reward score: 1.96 -> 1.23 | Hurt |
| Share the VAE encoder/decoder while retaining separate positive/negative distribution parameters (ablation) | LENS VAE · HH-RLHF, 4× | reward score: 1.35 -> 1.96 | Helped |
Surprising findings
- Direct embedding perturbation and Gaussian sampling baselines degrade performance as augmentation grows, with Gaussian sampling falling to 0.94 on HH-RLHF at 8x.
- Textual synthesis with the IPO baseline underperforms the original unaugmented data at 4x and 8x augmentation on HH-RLHF.
Limitations
- Evaluation relies on a single gold reward model (Skywork) and best-of-16 selection, so gains may not transfer to other metrics.
- Theoretical bounds rely on assumptions such as Lipschitz decoder and Hölder-continuous reward, and reconstruction error decay that are only loosely verified.
- Reward model is an MLP head on frozen embeddings, not a full fine-tuned reward model, limiting comparability.
Experimental setup
Datasets: HH-RLHF TL;DR summarization
Benchmarks: HH-RLHF TL;DR summarization
Metrics: best-of-n gold reward (Skywork) GPT-4 win rate
Baselines: Fully fine-tune Low rank adaptation Embedding MLP Self-rewarding Self-evolved IPO Direct perturbation Gaussian sampling
Builds on: variational autoencoder Bradley-Terry model self-rewarding language models
Closest papers by meaning
- Beyond Bradley-Terry Models: A General Preference Model for Language Model Alignment (ICML 2025 · similarity 0.92)
- Reward-Augmented Data Enhances Direct Preference Alignment of LLMs (ICML 2025 · similarity 0.91)
- Latent Preference Coding: Aligning Large Language Models via Discrete Latent Codes (ICML 2025 · similarity 0.91)
- Transforming and Combining Rewards for Aligning Large Language Models (ICML 2024 · similarity 0.90)
- Sequential Preference Optimization: Multi-Dimensional Preference Alignment with Implicit Reward Modeling (AAAI 2025 · similarity 0.90)
- Data with High and Consistent Preference Difference Are Better for Reward Model (AAAI 2025 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.