Research map / Large language models / Alignment and preferences

Preference Alignment with Flow Matching

NeurIPS 2024 · Large language models › Alignment and preferences · approach: Reward model design and preference models · 2 citations (OpenAlex)

Read the paper Check your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes Preference Flow Matching (PFM), an add-on flow-matching module that learns to transport less-preferred samples to preferred ones on top of a frozen pre-trained model, without reward modeling or fine-tuning the base model.

Problem

Preference alignment of pre-trained models normally requires fine-tuning (RLHF, DPO) and reward models that overfit, which is impractical for black-box APIs and large models.

Approach

Learn a conditional flow matching vector field from less-preferred to more-preferred samples in the preference dataset, then apply it at inference to samples from the frozen reference policy, avoiding Bradley-Terry reward estimation; iterating the flow further sharpens alignment with theoretical convergence guarantees.

Key results

On D4RL MuJoCo, PFM averages 63.10 normalized return vs 62.49 for DPO fine-tuned and 59.97 for BC.
On IMDB, the iterative PFM with 5 iterations on the reference policy reaches GPT-4 win rate of 85% against the PPO-tuned policy, and PPO+PFM reaches 99%.
On MNIST, PFM achieves preference score 0.9841 vs 0.9171 for RLHF and 0.8936 for DPO; iterative PFM reaches 0.9996.

Measured effects reported in the paper

What was changed	Where	Measured	Effect
add PFM to the reference model	DCGAN · MNIST conditional image generation	preference score: 0.8389 -> 0.9841	Helped
apply the learned preference flow iteratively for two iterations (ablation)	PFM on DCGAN · MNIST conditional image generation	preference score: 0.9841 -> 0.9996	Helped
add PFM to the reference model	GPT-2 SFT · IMDB test instances	average preference score: -0.3607 -> 0.6399	Helped
apply the learned preference flow iteratively five times (ablation)	GPT-2 SFT + PFM · IMDB test instances	average preference score: 0.6399 -> 2.7469	Helped
add PFM to the PPO policy	PPO · IMDB test instances	average preference score: 2.0156 -> 2.178	Helped
apply the learned preference flow iteratively five times (ablation)	PPO + PFM · IMDB test instances	average preference score: 2.178 -> 2.7894	Helped
use the true less-preferred marginal p₀ instead of the reference policy as the flow source (ablation)	PFM on BC · walker2d-medium-v2	normalized return: 72.59 ± 15.8 -> 80.58 ± 1.80	Helped
use the true less-preferred marginal p₀ instead of the reference policy as the flow source (ablation)	PFM on BC · walker2d-random-v2	normalized return: 1.77 ± 0.13 -> 2.03 ± 1.01	Helped
use the true less-preferred marginal p₀ instead of the reference policy as the flow source (ablation)	PFM on BC · walker2d-expert-v2	normalized return: 108.38 ± 0.28 -> 109.19 ± 0.37	Helped
use the true less-preferred marginal p₀ instead of the reference policy as the flow source (ablation)	PFM on BC · walker2d average	normalized return: 60.91 -> 63.93	Helped

Surprising findings

On expert D4RL datasets PFM gives little gain and can underperform, which the authors attribute to a shifted source distribution when using the reference policy instead of the true marginal.
Applying the same learned flow iteratively (without re-collecting data) gave the best IMDB performance, outperforming retraining the module iteratively.

Limitations

Text generation relies on a T5 autoencoder to map variable-length texts to fixed-size embeddings, which limits scaling to long-form text.
Performance gains are small or absent on near-optimal expert datasets because the reference policy's samples are unlikely under the true marginal source.
Relies on non-deterministic preference support assumptions for inference from the reference policy rather than the true source distribution.

Experimental setup

Datasets: MNIST IMDB D4RL (ant, hopper, halfcheetah, walker2d) 8-Gaussians toy density

Benchmarks: MNIST conditional generation IMDB controlled sentiment generation D4RL MuJoCo (12 datasets)

Metrics: preference score normalized episode return GPT-4 win rate

Baselines: RLHF DPO PPO Marginal BC Planning CPL IPO FTB

Builds on: flow matching RLHF DPO conditional flow matching PPO

Closest papers by meaning

Reward-Augmented Data Enhances Direct Preference Alignment of LLMs (ICML 2025 · similarity 0.89)
Data with High and Consistent Preference Difference Are Better for Reward Model (AAAI 2025 · similarity 0.89)
Latent Preference Coding: Aligning Large Language Models via Discrete Latent Codes (ICML 2025 · similarity 0.89)
Direct Alignment with Heterogeneous Preferences (NeurIPS 2025 · similarity 0.89)
Sequential Preference Optimization: Multi-Dimensional Preference Alignment with Implicit Reward Modeling (AAAI 2025 · similarity 0.89)
On-the-fly Preference Alignment via Principle-Guided Decoding (ICLR 2025 · similarity 0.89)

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft