Research map / Image generation / Text-to-image diffusion
Text-to-Image Rectified Flow as Plug-and-Play Priors
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes RFDS, iRFDS and RFDS-Rev, which use pretrained text-to-image rectified flow models (SD3, InstaFlow) as plug-and-play priors for text-to-3D generation and image inversion/editing.
Problem
Whether large pretrained rectified flow models can serve as effective plug-and-play priors (loss functions) like diffusion models, for 3D generation and 2D image inversion/editing.
Approach
Derive SDS-like loss (RFDS) from the flow-matching objective by dropping the network Jacobian; reverse it to optimize noise (iRFDS) using time-symmetry; and combine them iteratively (RFDS-Rev) to recover the noise and then refine the input, improving detail and diversity.
Key results
- RFDS-Rev with SD3 reaches an average T3Bench quality score of 56.5, versus 20.5 for SDS and 46.4 for VSD (diffusion).
- RFDS-Rev with InstaFlow averages 48.9 on T3Bench, above the diffusion VSD baseline of 46.4.
- iRFDS with InstaFlow attains the best editing CLIP score of 0.306 (vs. 0.298 for null inversion) and 43.6% semantic user preference.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add iRFDS noise reversal before RFDS (ablation) | RFDS with InstaFlow · T³Bench Average | quality score: 36.5 -> 48.9 | Helped |
| Add iRFDS noise reversal before RFDS (ablation) | RFDS with SD3 · T³Bench Average | quality score: 43.9 -> 56.5 | Helped |
| Use iRFDS with InstaFlow rather than the listed editing baselines | 2D image editing · 2D editing | CLIP Score: 0.298 -> 0.306 | Helped |
| Insert learned noise into an intermediate flow generation step (ablation) | iRFDS · Image reconstruction | PSNR: 28.52 -> 28.96 | Helped |
| Keep the rectified-flow network Jacobian instead of ignoring it (ablation) | RFDS · 2D image generation | meaningful image generation | Hurt |
| Run iRFDS for one step in RFDS-Rev (ablation) | RFDS-Rev · 2D and 3D generation | efficiency-performance trade-off | Helped |
| Use classifier-free guidance greater than 10 (ablation) | RFDS and RFDS-Rev · Text-to-3D generation | shape quality | Helped |
| Increase CFG excessively (ablation) | RFDS · Text-to-3D generation | color saturation | Hurt |
| Use large CFG values (ablation) | RFDS-Rev · Text-to-3D generation | robustness | Helped |
| Replace RFDS-Rev with RFDS-VSD (ablation) | InstaFlow text-to-3D generation · Text-to-3D generation | generation quality | Hurt |
| Use one-step iRFDS inversion with a Reflow-finetuned model | iRFDS · Image inversion and editing | editing performance | Helped |
| Use RFDS-Rev instead of the RFDS baseline (ablation) | 2D optimization with InstaFlow · 2D text-guided generation | generation performance | Helped |
Surprising findings
- Keeping the rectified flow network Jacobian in the loss makes it hard to generate meaningful images, so dropping it is necessary.
- With diffusion models, the RFDS baseline reduces exactly to the vanilla SDS loss.
Limitations
- Multi-face (Janus) issue inherited from SDS/VSD because the 2D prior lacks camera pose conditioning.
- iRFDS does not account for the CFG mismatch between inversion and generation.
- Diffusion models require multiple noise steps, so the single-step iRFDS trick used for RFDS-Rev does not carry over to them well.
Experimental setup
Datasets: T3Bench prompts 15 internet real images for editing
Benchmarks: T3Bench
Metrics: T3Bench quality score (CLIP and ImageReward) CLIP score user preference PSNR
Baselines: SDS (DreamFusion) VSD (ProlificDreamer) DDS Null inversion RFDS
Builds on: Dreamfusion SDS ProlificDreamer VSD Stochastic Interpolants Reflow DDS
Closest papers by meaning
- Rectified Diffusion: Straightness Is Not Your Need in Rectified Flow (ICLR 2025 · similarity 0.92)
- Solving Inverse Problems with FLAIR (NeurIPS 2025 · similarity 0.91)
- Improving the Training of Rectified Flows (NeurIPS 2024 · similarity 0.90)
- Semantic Image Inversion and Editing using Rectified Stochastic Differential Equations (ICLR 2025 · similarity 0.90)
- Scaling Rectified Flow Transformers for High-Resolution Image Synthesis (ICML 2024 · similarity 0.90)
- InstaFlow: One Step is Enough for High-Quality Diffusion-Based Text-to-Image Generation (ICLR 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.