Research map / Image generation / Editing and style transfer
Image Translation as Diffusion Visual Programmers
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Diffusion Visual Programmer (DVP), a training-free neuro-symbolic image translation framework that pairs GPT-4 planning and off-the-shelf vision modules with a diffusion model using instance-normalization guidance instead of a hand-tuned classifier-free guidance scale.
Problem
Diffusion-based image translation is sensitive to the manually tuned classifier-free guidance scale, and existing methods lack precise region-of-interest (RoI) editing, context-free reasoning, and explainable control.
Approach
Replace the linear classifier-free guidance combination with instance normalization guidance (shifting unconditional noise statistics toward conditional ones via a 1x1 conv), and wrap the diffusion model in a GPT-4-planned visual program (segment, inpaint, position manipulation, prompt generation) that yields human-readable intermediate symbols.
Key results
- DVP reaches CLIP-Score 0.839 and DINO-Score 0.697, outperforming VISPROG (0.813 / 0.651) and Prompt2Prompt (0.825 / 0.657) in the comparison table.
- In the user study, DVP scores 4.28 on fidelity versus 4.04 for VISPROG and 3.92 for Prompt2Prompt.
- Instance normalization guidance (CLIP 0.839, DINO 0.697) beats the best fixed guidance scale tested (w=2.5 gives CLIP 0.833; w=7.5 gives DINO 0.712) without per-image tuning.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use DVP instead of DiffuseIT | DVP · Image translation benchmark (100 text-image pairs) | Quality: 3.88 -> 3.95 | Helped |
| use DVP instead of VISPROG | DVP · Image translation benchmark (100 text-image pairs) | Fidelity: 4.04 -> 4.28 | Helped |
| use DVP instead of Text2Live | DVP · Image translation benchmark (100 text-image pairs) | Diversity: 3.73 -> 3.56 | Hurt |
| use DVP instead of Prompt2Prompt | DVP · Image translation benchmark (100 text-image pairs) | CLIP-Score: 0.825 -> 0.839 | Helped |
| use DVP instead of VQGAN-CLIP | DVP · Image translation benchmark (100 text-image pairs) | DINO-Score: 0.667 -> 0.697 | Helped |
| apply instance normalization guidance instead of classifier-free guidance with w = 2.5 (ablation) | Condition-flexible diffusion model · Instance normalization study | CLIP-Score: 0.833 -> 0.839 | Helped |
| apply instance normalization guidance instead of classifier-free guidance with w = 5 (ablation) | Condition-flexible diffusion model · Instance normalization study | CLIP-Score: 0.795 -> 0.839 | Helped |
| apply instance normalization guidance instead of classifier-free guidance with w = 7.5 (ablation) | Condition-flexible diffusion model · Instance normalization study | CLIP-Score: 0.742 -> 0.839 | Helped |
| apply instance normalization guidance instead of classifier-free guidance with w = 10 (ablation) | Condition-flexible diffusion model · Instance normalization study | CLIP-Score: 0.686 -> 0.839 | Helped |
| apply instance normalization guidance instead of classifier-free guidance with w = 2.5 (ablation) | Condition-flexible diffusion model · Instance normalization study | DINO-Score: 0.664 -> 0.697 | Helped |
| apply instance normalization guidance instead of classifier-free guidance with w = 5 (ablation) | Condition-flexible diffusion model · Instance normalization study | DINO-Score: 0.681 -> 0.697 | Helped |
| apply instance normalization guidance instead of classifier-free guidance with w = 7.5 (ablation) | Condition-flexible diffusion model · Instance normalization study | DINO-Score: 0.712 -> 0.697 | Hurt |
Surprising findings
- Fixed guidance-scale variants of the diffusion model show large swings in output quality, with CLIP-Score falling from 0.833 at w=2.5 to 0.686 at w=10.
Limitations
- Fails on occluded objects, since segmentation only covers the visible part of an object.
- Struggles under poor photometric conditions such as bad lighting or shadows.
- Depends on the GPT-4 planner and off-the-shelf modules, so errors in segmentation or planning propagate through the pipeline.
Experimental setup
Datasets: ImageNet-R ImageNet MS COCO
Benchmarks: 100-pair text-image translation benchmark
Metrics: CLIP-Score DINO-Score user study (quality, fidelity, diversity on 1-5 Likert) runtime
Baselines: VQGAN-CLIP Text2LIVE SDEdit Prompt-to-Prompt DiffuseIT VISPROG
Builds on: VISPROG Prompt-to-Prompt null-text optimization classifier-free guidance
Closest papers by meaning
- Synthesizing Visual Concepts as Vision-Language Programs (CVPR 2026 · similarity 0.88)
- InstructCV: Instruction-Tuned Text-to-Image Diffusion Models as Vision Generalists (ICLR 2024 · similarity 0.87)
- Prompt-Free Diffusion: Taking "Text" out of Text-to-Image Diffusion Models (CVPR 2024 · similarity 0.87)
- MMTIT-Bench: A Multilingual and Multi-Scenario Benchmark with Cognition-Perception-Reasoning Guided Text-Image Machine Translation (CVPR 2026 · similarity 0.87)
- MMaDA: Multimodal Large Diffusion Language Models (NeurIPS 2025 · similarity 0.86)
- SynthRGB-T: Language-Vision Guided Image Translation for Diversity Synthesis (CVPR 2026 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.