Research map / Image generation / Fast diffusion sampling
Diff2Flow: Training Flow Matching Models via Diffusion Model Alignment
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Diff2Flow converts a pretrained diffusion model (Stable Diffusion) into a flow matching model by rescaling timesteps, aligning interpolants, and deriving velocity predictions from diffusion outputs, enabling efficient FM finetuning (including LoRA).
Problem
Foundation flow matching models are too large to finetune cheaply, while diffusion models like Stable Diffusion are efficient but misaligned with flow matching in interpolants, timestep scaling, and objectives, so naive transfer converges slowly and performs poorly, especially under PEFT.
Approach
Map the diffusion trajectory to a linear flow matching trajectory via invertible timestep and state reparameterizations (f_t, f_x), and compute the FM velocity from the diffusion v-prediction using the boundary-condition relations, so the pretrained diffusion network is used without learning a new parameterization.
Key results
- On COCO-2017 5k with 25 steps, Diff2Flow achieves FID 52.80 vs 56.36 for SD1.5 continued diffusion training and CLIP 26.54 vs 26.33.
- With LoRA at 2 sampling steps, Diff2Flow (62M params) reaches FID 32.31 vs 31.35 for 0.9B Rectified Flow, and at 4 steps FID 25.29 vs 22.97 for PeRFlow.
- On zero-shot NYUv2 depth, full-finetuned Diff2Flow reaches AbsRel 5.7 / delta1 96.7, competitive with Marigold+E2E FT (5.2 / 96.6) and outperforming DepthFM on DIODE (21.6 vs 22.4 AbsRel).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Align diffusion timesteps, interpolants, and prediction objective using Diff2Flow rather than continued diffusion training | Stable Diffusion 1.5 fine-tuned on LAION-Aesthetics · COCO-2017 5k | FID: 56.36 -> 52.80 | Helped |
| Use Diff2Flow rather than continued diffusion training | Stable Diffusion 1.5 fine-tuned on LAION-Aesthetics · COCO-2017 5k | CLIP: 26.33 -> 26.54 | Helped |
| Use Diff2Flow rather than continued diffusion training | Stable Diffusion 1.5 fine-tuned on LAION-Aesthetics · COCO-2017 5k | Aesthetics Score: 5.90 -> 5.99 | Helped |
| Add trajectory and objective alignment when adapting the diffusion prior to monocular depth estimation | Stable Diffusion depth model · NYUv2 | AbsRel: 6.0 -> 5.7 | Helped |
| Add trajectory and objective alignment when adapting the diffusion prior to monocular depth estimation | Stable Diffusion depth model · KITTI | AbsRel: 9.1 -> 8.7 | Helped |
| Add trajectory and objective alignment when adapting the diffusion prior to monocular depth estimation | Stable Diffusion depth model · ETH3D | AbsRel: 6.5 -> 5.5 | Helped |
| Add trajectory and objective alignment when adapting the diffusion prior to monocular depth estimation | Stable Diffusion depth model · DIODE | AbsRel: 22.4 -> 21.6 | Helped |
| Limit trainable parameters with LoRA rather than full fine-tuning (ablation) | Diff2Flow depth model · NYUv2 | AbsRel: 5.7 -> 5.9 | Hurt |
| Use the LoRA small configuration rather than the LoRA base configuration (ablation) | Diff2Flow depth model · NYUv2 | AbsRel: 5.9 -> 6.9 | Hurt |
| Train with Diff2Flow alignment rather than the standard diffusion objective | Stable Diffusion text-to-image model · COCO 2017 validation | FID | Helped |
| Train with Diff2Flow alignment rather than a naive flow-matching objective under LoRA | Stable Diffusion text-to-image model · COCO 2017 validation | FID | Helped |
| Apply Diff2Flow alignment rather than directly applying flow matching to the diffusion model | Stable Diffusion depth model with LoRA · NYUv2 | AbsRel | Helped |
Surprising findings
- Continuous (non-integer-shifted) DDIM timesteps, which were never trained, still yield high-quality images.
- Naive flow matching finetuning with LoRA fails to close the gap, while the objective-aligned Diff2Flow works well with LoRA.
Limitations
- Continuous timestep interpolation of the diffusion schedule is heuristic and not trained for.
- Demonstrated mainly on Stable Diffusion 1.5/2.1 scale models, not on large flow models like Flux or SD3.
- Depth results are trained only on synthetic data and the performance is mixed across benchmarks (e.g., KITTI behind some baselines).
Experimental setup
Datasets: LAION-Aesthetics COCO 2017 synthetic image-depth data (Marigold setup) NYUv2 KITTI ETH3D ScanNet DIODE
Benchmarks: COCO 2017 5k FID/CLIP NYUv2 KITTI ETH3D ScanNet DIODE
Metrics: FID CLIP score Aesthetics Score AbsRel delta1
Baselines: Stable Diffusion 1.5 continued training naive flow matching finetuning DepthFM Marigold E2E FT PeRFlow Rectified Flow DPM Solver Depth Anything Depth Anything v2 Metric3D GeoWizard
Builds on: Stable Diffusion flow matching Rectified Flow (reflow) LoRA DepthFM
Closest papers by meaning
- Shortcutting Pre-trained Flow Matching Diffusion Models is Almost Free Lunch (NeurIPS 2025 · similarity 0.93)
- Elucidating Flow Matching ODE Dynamics via Data Geometry and Denoisers (ICML 2025 · similarity 0.90)
- Rectified Diffusion: Straightness Is Not Your Need in Rectified Flow (ICLR 2025 · similarity 0.90)
- Neural Flow Diffusion Models: Learnable Forward Process for Improved Diffusion Modelling (NeurIPS 2024 · similarity 0.90)
- Structured Coupling for Flow Matching (NeurIPS 2026 · similarity 0.90)
- MissPath-FM: Flow Matching with Structured Priors for Partially Observed Time Series (NeurIPS 2026 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.