Research map / Image generation / Editing and style transfer
Frequency-Controlled Diffusion Model for Versatile Text-Guided Image-to-Image Translation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes FCDiffusion, a ControlNet-style text-guided image-to-image translation framework that conditions Stable Diffusion on DCT-filtered latent features of the source image, with separate frequency branches (mini/low/mid/high) for different I2I correlations.
Problem
Text-guided image-to-image translation with large text-to-image diffusion models needs different structural/style correlations between source and output depending on the application, and existing methods are each suited to only some of these scenarios and are slow or tuning-unfriendly.
Approach
Filter source latent features in the DCT domain into distinct spectral bands (mini, low, mid, high) and use each filtered signal as a control input to a separately fine-tuned FreqControlNet branch, so switching bands at inference switches the I2I correlation (style, structure, layout, contour). An Equifrequency Shuffle further removes spatial structure for style-only control.
Key results
- Achieves 9.4s inference time per 512x512 image on RTX 3090 Ti, versus 11.2s for InstructPix2Pix and 69.7s to 474.6s for other baselines.
- Quantitative comparison on 200 translations per scenario shows competitive text fidelity and structure similarity trade-off across semantic manipulation, style translation, and style-guided creation, with SDEdit(0.85) and VQGAN-CLIP reaching higher text fidelity at very low structure similarity.
- Ablations show removing time embedding injection yields unstable results with noise and artifacts, and Equifrequency Shuffle decorrelates spatial structure in style-guided content creation.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use FCDiffusion instead of insPix2Pix | text-guided image-to-image translation · Single 512 × 512 image translation | inference time: 11.2s -> 9.4s | Helped |
| Add Equifrequency Shuffle to the mini-frequency branch (ablation) | FCDiffusion · Style-guided content creation | spatial structure disentanglement | Helped |
| Remove time embedding injection from FCNet (ablation) | FCDiffusion · Text-guided image-to-image translation | image quality | Hurt |
| Feed null text to FCNet during training (ablation) | FCDiffusion · Text-guided image-to-image translation | image quality | Hurt |
| Raise the low-pass filtering threshold (ablation) | FCDiffusion · Image semantic manipulation | resemblance to source image | Helped |
| Lower the low-pass filtering threshold (ablation) | FCDiffusion · Image semantic manipulation | resemblance to source image | Hurt |
Surprising findings
- SDEdit(0.85) and VQGAN-CLIP achieve higher text fidelity than FCDiffusion on style-guided content creation, but at very low structure similarity in other tasks.
Limitations
- Each new DCT filter requires allocating and training a new frequency control branch, reducing flexibility.
- Authors state the approach is less flexible than training-free spectral control would be.
- Evaluation relies on automatic CLIP and DINO metrics on 200 translations per scenario, with no human evaluation reported.
Experimental setup
Datasets: LAION-Aesthetics 6.5+
Benchmarks: LAION-Aesthetics 6.5+ test split
Metrics: CLIP cosine similarity (text fidelity) DINO-ViT self-similarity structure distance inference time
Baselines: VQGAN-CLIP DiffuseIT Text2LIVE InstructPix2Pix Prompt-to-Prompt Plug-and-Play SDEdit
Builds on: ControlNet Latent Diffusion Model DDIM SDEdit
Closest papers by meaning
- AltDiffusion: A Multilingual Text-to-Image Diffusion Model (AAAI 2024 · similarity 0.88)
- FDS: Frequency-Aware Denoising Score for Text-Guided Latent Diffusion Image Editing (CVPR 2025 · similarity 0.88)
- One Stone with Two Birds: A Null-Text-Null Frequency-Aware Diffusion Models for Text-Guided Image Inpainting (NeurIPS 2025 · similarity 0.86)
- R&B: Region and Boundary Aware Zero-shot Grounded Text-to-image Generation (ICLR 2024 · similarity 0.86)
- Prompt-Free Diffusion: Taking "Text" out of Text-to-Image Diffusion Models (CVPR 2024 · similarity 0.86)
- Disentangling Dual Image References in Frequency Aware Diffusion Models for Personalized Generation (NeurIPS 2026 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.