Research map / Image generation / Text-to-image diffusion
OmniFlow: Any-to-Any Generation with Multi-Modal Rectified Flows
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes OmniFlow, a modular multi-modal rectified-flow transformer that extends SD3's MMDiT with audio and text streams for any-to-any text/image/audio generation.
Problem
Existing any-to-any generators either combine separate modality encoders/decoders with weak cross-modal fusion or train unified models from scratch at high cost; there is no open unified model covering text, image, and audio.
Approach
Extends rectified flow to a multi-modal setting with per-modality timesteps, so any-to-any tasks are paths between noise levels; uses a modular MMDiT extension where only joint attention is shared, enabling separate pretraining and merging with SD3; adds multi-modal classifier-free guidance weighting cross-modal influence.
Key results
- On MSCOCO-30K text-to-image, OmniFlow reaches FID 13.40 and CLIP 31.54, beating UniDiffuser and CoDi on CLIP and SD3-Medium on FID.
- On GenEval, OmniFlow scores 0.62, versus 0.38 for CoDi, 0.43 for UniDiffuser, and 0.62 for SD3-Medium.
- On AudioCaps text-to-audio, OmniFlow gets FAD 1.75 and CLAP 0.183, improving on AudioLDM 2 FAD (1.86) and CoDi (1.80 FAD, 0.053 CLAP).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use OmniFlow for text-to-image generation | OmniFlow · MSCOCO-30K | CLIP: 30.69 -> 31.54 | Helped |
| Use OmniFlow for text-to-image generation | OmniFlow · GenEval | score: .38 -> .62 | Helped |
| Use OmniFlow for text-to-audio generation | OmniFlow · AudioCaps | FAD: 1.86 -> 1.75 | Helped |
| Use OmniFlow for text-to-audio generation | OmniFlow · AudioCaps | CLAP: .182 -> .183 | No clear effect |
| Use OmniFlow instead of CoDi for text-to-audio generation | OmniFlow · AudioCaps | CLAP: .053 -> .183 | Helped |
| Use rectified flow with log-normal timestep sampling (ablation) | audio-text diffusion model · AudioCaps | FAD: 2.08 -> 1.79 | Helped |
| Use rectified flow with log-normal timestep sampling (ablation) | audio-text diffusion model · AudioCaps | CLAP: .141 -> .254 | Helped |
| Use discrete SEDD instead of continuous-space diffusion (ablation) | audio-text diffusion model · AudioCaps | CLAP: .180 | No clear effect |
| Use discrete MDLM instead of continuous-space diffusion (ablation) | audio-text diffusion model · AudioCaps | CLAP: .163 | No clear effect |
Surprising findings
- Among audio/text formulations, rectified flow with a log-normal timestep distribution performed best, while discrete diffusion (SEDD, MDLM) gave no clear advantage over continuous text diffusion.
- A timestep shift that gives a concave schedule also improved audio and text sample quality, even though those modalities have no resolution variation.
Limitations
- Text-to-image FID is higher than SDv1.5 on MSCOCO, partly due to distribution mismatch of high-quality training data.
- Discrete text diffusion (SEDD, MDLM) did not show advantages over continuous alternatives in this setting.
Experimental setup
Datasets: COYO-700M LAION-Aesthetic-3M LAION-COCO CC12M AudioSet AudioCaps WavCaps VGGSound SoundNet MSCOCO-30K
Benchmarks: MSCOCO-30K GenEval AudioCaps
Metrics: FID CLIP score GenEval score FAD CLAP score
Baselines: CoDi UniDiffuser UIO-2XXL SDXL SD3-Medium AudioLDM 2 Chameleon Transfusion
Builds on: Stable Diffusion 3 rectified flow DiT classifier-free guidance
Closest papers by meaning
- Scaling Rectified Flow Transformers for High-Resolution Image Synthesis (ICML 2024 · similarity 0.88)
- UniFlow-Audio: Unified Flow Matching for Audio Generation from Omni-Modalities (NeurIPS 2026 · similarity 0.88)
- InstaFlow: One Step is Enough for High-Quality Diffusion-Based Text-to-Image Generation (ICLR 2024 · similarity 0.86)
- STARFlow2: Bridging Language Models and Normalizing Flows for Unified Multimodal Generation (NeurIPS 2026 · similarity 0.86)
- OmniTalker: One-shot Real-time Text-Driven Talking Audio-Video Generation With Multimodal Style Mimicking (NeurIPS 2025 · similarity 0.86)
- DFlow: A Generative Model Combining Denoising AutoEncoder and Normalizing Flow for High Fidelity Waveform Generation (ICML 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.