Research map / Image generation / Fast diffusion sampling
Flowing from Words to Pixels: A Noise-Free Framework for Cross-Modality Evolution
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes CrossFlow, which trains flow matching to map directly from one modality's latent (e.g., text) to another's (e.g., image) using a variational encoder and an indicator-based classifier-free guidance, without noise or cross-attention.
Problem
Standard diffusion and flow matching models start from Gaussian noise and inject conditions via cross-attention or concatenation, which makes cross-modal generation rely on conditioning mechanisms and task-specific designs; direct evolution between different modalities has been unexplored and CFG is
Approach
Replace the Gaussian noise source with a variational encoder that maps the input modality (e.g., text embeddings) to a regularized latent of the target shape, flow-match directly to the target latent with a vanilla transformer, and enable classifier-free guidance via learnable conditional/unconditional indicator tokens.
Key results
- Text-to-image: CrossFlow (0.95B) reaches zero-shot FID-30K 10.13 vs 10.79 for standard flow matching with cross-attention at same settings; with sin-cos matching, FID-30K 8.95 and GenEval 0.57.
- Image captioning on COCO Karpathy: CrossFlow achieves CIDEr 116.2 and BLEU-4 36.4, close to SCD-Net's 118.0 CIDEr among non-autoregressive methods trained without CIDEr optimization.
- Super-resolution on ImageNet 64to256: CrossFlow FID 3.0 vs 3.4 for flow matching and 5.2 for SR3.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Directly evolve text latents into image latents instead of starting from noise with text cross-attention | CrossFlow · COCO validation | FID-30K: 10.79 -> 10.13 | Helped |
| Use sin-cos matching instead of linear flow matching (ablation) | CrossFlow · COCO validation | FID-30K: 9.63 -> 8.95 | Helped |
| Directly map images to text latents for captioning | CrossFlow · COCO Karpathy split | CIDEr: 118.0 -> 116.2 | Hurt |
| Map images directly to depth maps | CrossFlow · KITTI Eigen split | AbsRel: 0.050 -> 0.053 | Hurt |
| Map images directly to depth maps | CrossFlow · NYUv2 official split | AbsRel: 0.085 -> 0.094 | Mixed |
| Directly evolve upsampled low-resolution images into high-resolution images without additional conditioning | CrossFlow · ImageNet validation, 64×64→256×256 | FID: 3.4 -> 3.0 | Helped |
| Add a regularized Variational Encoder instead of using a standard encoder (ablation) | CrossFlow · Text-to-image ablation | FID: 66.65 -> 40.78 | Helped |
| Add a regularized Variational Encoder instead of adding Gaussian noise to a standard encoder (ablation) | CrossFlow · Text-to-image ablation | FID: 59.91 -> 40.78 | Helped |
| Use text-text contrastive loss instead of text-text reconstruction loss (ablation) | Text Variational Encoder · Text-to-image ablation | FID: 40.78 -> 34.67 | Helped |
| Use image-text contrastive loss instead of text-text contrastive loss (ablation) | Text Variational Encoder · Text-to-image ablation | FID: 34.67 -> 33.41 | Helped |
| Use CFG with an indicator instead of Autoguidance (ablation) | CrossFlow · Text-to-image ablation | FID: 26.36 -> 24.33 | Helped |
| Use CFG with an indicator instead of no guidance (ablation) | CrossFlow · Text-to-image ablation | FID: 33.41 -> 24.33 | Helped |
Surprising findings
- A vanilla transformer without cross-attention slightly outperforms standard flow matching with cross-attention for text-to-image, despite starting from text latents rather than noise.
- At small model sizes CrossFlow underperforms the baseline, but it surpasses it as model size and training steps increase.
Limitations
- Main text-to-image training uses a proprietary 350M-pair dataset, limiting reproducibility and fair comparison.
- Underperforms the cross-attention baseline at small model scales.
- Captioning and depth results are comparable to, not clearly above, state-of-the-art methods on several metrics.
Experimental setup
Datasets: proprietary image-text pairs (~350M) COCO KITTI NYUv2 ImageNet
Benchmarks: COCO validation FID-30K GenEval COCO Karpathy captioning KITTI NYUv2 ImageNet 64to256 super-resolution
Metrics: FID CLIP score GenEval overall score BLEU-4 METEOR ROUGE CIDEr SPICE AbsRel delta1
Baselines: Standard flow matching with cross-attention DALL·E 2 LDMv1.5 PixArt-α LDMv3 SR3 DiffusionDepth SCD-Net Autoguidance CFM
Builds on: flow matching DiMR classifier-free guidance Autoguidance CFM InterFlow
Closest papers by meaning
- Cross Flow: One-Step Generation Across Latent and Pixel Spaces (NeurIPS 2026 · similarity 0.91)
- Dual Diffusion for Unified Image Generation and Understanding (CVPR 2025 · similarity 0.89)
- De-Diffusion Makes Text a Strong Cross-Modal Interface (CVPR 2024 · similarity 0.89)
- Towards General Modality Translation with Contrastive and Predictive Latent Diffusion Bridge (NeurIPS 2025 · similarity 0.88)
- Diffuse Everything: Multimodal Diffusion Models on Arbitrary State Spaces (ICML 2025 · similarity 0.88)
- Connect, Collapse, Corrupt: Learning Cross-Modal Tasks with Uni-Modal Data (ICLR 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.