Research map / Reinforcement learning / RL from feedback

The Reward Was in Your Data All Along: Correcting Flow Matching with Discriminator-Guided RL.

NeurIPS 2026 · Reinforcement learning › RL from feedback · approach: Reward modeling versus direct optimization · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes Discriminator-Guided RL (DRL), which trains a linear discriminator on frozen DINOv2 features to separate data from model samples and uses its logit as a KL-regularized RL reward to correct flow-matching models without preference data.

Problem

Flow and score matching models often need preference-based RL to recover data properties (visual realism, coherent structure) that matching losses should learn from data; the paper asks why matching fails and how to correct it without preference rewards.

Approach

Estimate the log density ratio between data and base model in a frozen SSL representation space with a discriminator, and use its logit as the reward for KL-regularized adjoint-matching RL; the optimum aligns the pushforward of the model with the data in that feature space.

Key results

• DRL improves FD in 16 of 16 model-feature pairs without CFG at tuned lambda, often by more than half in DINOv2 and DINOv3.
• DRL improves all four held-out preference rewards (ImageReward, PickScore, Aesthetics v2.5, HPSv2) on every architecture without preference data.
• DRL+PRL shifts the reward-drift frontier upward versus Base+PRL across five low-level statistics.

Surprising findings

• R1 regularization helps at lambda=10 but degrades performance at lambda=1, unlike its role in standard GANs.
• A student distilled by flow matching from a DRL teacher, trained for over 900k steps on 50M samples, fails to replicate the teacher, suggesting RL objective matters beyond samples.

Limitations

• Classifier-free guidance cannot be cleanly incorporated during RL training, and DRL's gains shrink under best CFG.
• Requires tuning lambda; reverse-KL can become mode-seeking at large lambda.
• Only corrects distributional differences visible to the chosen frozen embedder.

Experimental setup

Datasets: ImageNet

Benchmarks: ImageNet

Metrics: Fréchet Distance Kernel Distance FID precision-coverage (PRDC)

Baselines: Base model (SiT, JiT, REPA, RAE) Preference-based RL from base model Score-matching distillation from RL teacher

Builds on: adjoint matching REPA GAIL AIRL classifier-free guidance

Closest papers by meaning

• Reward Score Matching: Unifying Reward-based Fine-tuning for Flow and Diffusion Models (NeurIPS 2026 · similarity 0.89)
• Flow-GRPO: Training Flow Matching Models via Online RL (NeurIPS 2025 · similarity 0.88)
• Preference Alignment with Flow Matching (NeurIPS 2024 · similarity 0.88)
• Online Reward-Weighted Fine-Tuning of Flow Matching with Wasserstein Regularization (ICLR 2025 · similarity 0.88)
• Designing Reinforcement Learning for Diffusion Models: A Unified Path-Space View (NeurIPS 2026 · similarity 0.87)
• Value Gradient Guidance for Flow Matching Alignment (NeurIPS 2025 · similarity 0.87)