Research map / Image generation / Text-to-image diffusion
PixWizard: Versatile Image-to-Image Visual Assistant with Open-Language Instructions
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Builds PixWizard, a DiT-based (Lumina-Next) instruction-tuned image-to-image visual assistant trained on a 30M-sample Omni Pixel-to-Pixel dataset spanning generation, editing, restoration, grounding, and dense prediction.
Problem
Existing visual generalists either rely on in-context visual prompts without language instructions or diffusion-based instruction models that cover few tasks and lag specialist models; a versatile, language-instructed image-to-image assistant is lacking.
Approach
Unify diverse vision tasks as image-text-to-image generation with a flow-matching DiT, adding structure-aware (VAE latent channel concatenation) and semantic-aware (zero-initialized gated cross-attention over CLIP features) guidance, a task-aware dynamic token sampler, any-resolution handling, and a two-stage data-balanced training recipe.
Key results
- On RefCOCO val, PixWizard reaches 46.44 cIoU, 4.8 above InstructDiffusion, and gets 0.287 RMSE on NYUv2 depth (10.0% better than Unified-IO).
- On Emu Edit test, PixWizard achieves L1 0.069 and CLIP_out 0.248, beating InstructPix2Pix, MagicBrush and UltraEdit on most metrics and matching Emu Edit overall.
- Zero-shot COCO-30K text-to-image FID of 9.56 with HPSv2 27.47, and outpainting FID 7.54 on Places, reported as state-of-the-art among compared baselines.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| combine structure-aware and semantic-aware guidance rather than using structure-aware guidance alone (ablation) | PixWizard · Rain100L | PSNR: 29.91 -> 30.24 | Helped |
| combine structure-aware and semantic-aware guidance rather than using semantic-aware guidance alone (ablation) | PixWizard · Rain100L | PSNR: 14.72 -> 30.24 | Helped |
| add dynamic semantic token sampling (DSTS) (ablation) | PixWizard · Rain100L | PSNR: 30.19 -> 30.24 | No clear effect |
| add dynamic semantic token sampling (DSTS) (ablation) | PixWizard · Emu Edit | CLIP_dir: 0.091 -> 0.088 | Hurt |
| add two-stage training and data balancing (ablation) | PixWizard · Rain100L | PSNR: 29.17 -> 30.24 | Helped |
| add two-stage training and data balancing (ablation) | PixWizard · RefCOCO val | cIoU: 40.18 -> 41.72 | Helped |
| add two-stage training and data balancing (ablation) | PixWizard · NYUv2 depth estimation | RMSE: 0.322 -> 0.324 | Hurt |
| use PixWizard instead of InstructDiffusion | visual grounding · RefCOCO val | cIoU: 41.64 -> 46.44 | Helped |
| use PixWizard instead of Unified-IO | depth estimation · NYUv2 test | RMSE | Helped |
| use PixWizard instead of the specialized best reported baseline | image restoration · SIDD | PSNR: 39.52 -> 38.67 | Hurt |
| use PixWizard instead of the strongest reported editing baseline | image editing · Emu Edit Test | L1: 0.071 -> 0.069 | Helped |
| use PixWizard instead of the best reported outpainting baseline | outpainting · Places | FID: 7.80 -> 7.54 | Hurt |
Surprising findings
- Semantic-aware guidance alone (M2) performs poorly on most restoration and dense tasks, with the deraining PSNR dropping to 14.72, while structure-aware guidance alone (M1) is stronger on structure-preserving tasks.
- Dynamic semantic token sampling has minimal effect on task performance (mostly within ±0.1), though the authors report it improves average performance and reduces attention cost.
Limitations
- Architecture does not support multi-image input conditions.
- Lags specialized models on segmentation and image grounding.
- Performance depends on the text encoder and foundation model capacity.
Experimental setup
Datasets: Omni Pixel-to-Pixel Instruction-Tuning Dataset gRefCOCO RefCOCO Visual Genome LAION Art UltraEdit MagicBrush GQA-Inpaint InstructPix2Pix SEED-X-Edit GIER HQ-Edit
Benchmarks: Rain100L SIDD RefCOCO RefCOCO+ gRefCOCO NYUv2 SUNRGB-D ADE20K Emu Edit Test MagicBrush Test MultiGen-20M Places
Metrics: RMSE mIoU mean angle error PSNR SSIM cIoU CLIP-dir CLIP-im CLIP-out L1
Baselines: Unified-IO Painter InstructCV InstructDiffusion Emu Edit UltraEdit MagicBrush InstructPix2Pix
Builds on: Lumina-Next DiT InstructPix2Pix DynamicViT ControlNet
Closest papers by meaning
- PixArt-$\alpha$: Fast Training of Diffusion Transformer for Photorealistic Text-to-Image Synthesis (ICLR 2024 · similarity 0.87)
- UNIC-Adapter: Unified Image-instruction Adapter with Multi-modal Transformer for Image Generation (CVPR 2025 · similarity 0.87)
- VersaGen: Unleashing Versatile Visual Control for Text-to-Image Synthesis (AAAI 2025 · similarity 0.86)
- Imagine Before You Draw: Visual Prompt Engineering for Image Generation (NeurIPS 2026 · similarity 0.86)
- InsightEdit: Towards Better Instruction Following for Image Editing (CVPR 2025 · similarity 0.85)
- Guiding Instruction-based Image Editing via Multimodal Large Language Models (ICLR 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.