Research map / Reinforcement learning / RL from feedback

WTF?! Simulation-Free Reinforcement Learning with Wasserstein-Tilted Flow Maps

NeurIPS 2026 · Reinforcement learning › RL from feedback · approach: RL fine-tuning of generative models · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes Wasserstein-Tilted Flow Maps (WTF), a simulation-free reward fine-tuning method that regularizes a flow map with an optimal transport cost built from the pre-trained drift, equivalent to deterministic optimal control.

Problem

Reward fine-tuning of flow-based generative models is usually cast as sampling a KL reward-tilted distribution, which requires diffusion conversion and expensive rollouts; flow maps (few-step flows) lack an end-to-end fine-tuning recipe.

Approach

Replace the KL regularizer with an optimal transport (prior-action) cost built from the pre-trained drift, which transports samples toward higher reward and yields a deterministic optimal control problem; a flow map gives unbiased, critic-free value-gradient targets in O(1) evaluations and is fine-tuned end-to-end.

Key results

• On ImageNet-256, WTF at 1 NFE reaches HPSv2 0.319 vs 0.307 for VFM and 0.243 for Adjoint Matching.
• On text-to-image at 4 NFE, WTF reaches HPSv2 0.401 vs 0.357 for Adjoint Matching at 50 NFE.
• WTF reaches peak rewards of baselines with up to 280x less training compute than Flow-GRPO and 47x less than Adjoint Matching.

Surprising findings

• Euclidean reward gradient (omitting flow-map Jacobian) improves both reward and diversity over the exact pullback gradient in experiments.

Limitations

• Main result is a population-level fixed-point proposition; practical algorithm relies on approximations.
• Assumes access to a pre-trained flow map; starting from a velocity model requires distillation whose cost is excluded.
• Higher reward can reduce sample diversity, controlled by reward scale.

Experimental setup

Datasets: ImageNet-256

Benchmarks: ImageNet-256 HPSv2 PickScore ImageReward DreamSim diversity CLIP diversity

Metrics: HPSv2 PickScore ImageReward DreamSim CLIP diversity

Baselines: Adjoint Matching MFM VFM Flow-GRPO Base Model

Builds on: Adjoint Matching Flow Map Matching Flow-GRPO Benamou-Brenier formulation Schrödinger bridge

Closest papers by meaning

• Online Reward-Weighted Fine-Tuning of Flow Matching with Wasserstein Regularization (ICLR 2025 · similarity 0.91)
• Aligning Flow Map Policies with Optimal $Q$-Guidance (NeurIPS 2026 · similarity 0.88)
• Flow Density Control: Generative Optimization Beyond Entropy-Regularized Fine-Tuning (NeurIPS 2025 · similarity 0.88)
• Flow-Based Policy for Online Reinforcement Learning (NeurIPS 2025 · similarity 0.87)
• Beyond the Dirac Delta: Mitigating Diversity Collapse in Reinforcement Fine-Tuning for Image Generation (NeurIPS 2026 · similarity 0.87)
• Directly Fine-Tuning Diffusion Models on Differentiable Rewards (ICLR 2024 · similarity 0.86)