Research map / Vision-language and multimodal / Human and embodied modeling
InstructDiffusion: A Generalist Modeling Interface for Vision Tasks
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Casts diverse vision tasks (segmentation, keypoint detection, low-level restoration, image editing) as instruction-driven image editing with a fine-tuned Stable Diffusion v1.5 that outputs pixels.
Problem
Vision tasks have heterogeneous output formats (masks, coordinates, images) and separate methodologies, so a single generalist model that follows human instructions across understanding and generative tasks is hard to build with quantized or task-specific output spaces.
Approach
Represent all outputs as RGB pixel images (masks and keypoints encoded as colored overlays/circles) and train a diffusion model conditioned on text instructions and source image, so tasks become instructional image editing; joint multi-task training improves generalization to unseen datasets.
Key results
- Keypoint detection on HumanArt reaches 51.4 AP vs 12.4 for Painter, and AP-10K 15.9 vs 15.3 for Painter, outperforming generalist baselines.
- Referring segmentation on RefClef reaches 49.58 cIoU vs 40.13 for Unified-IO, exceeding specialized LAVT and ReLA on that split.
- Joint multi-task training improves open-set semantic segmentation over single-task training (e.g., ADE20K-847 19.7 vs lower single-task score in Figure 7).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use highly detailed instructions instead of coarse task labels and fixed colors (ablation) | InstructDiffusion · HumanArt | average precision: 7.0 -> 51.4 | Helped |
| Use highly detailed instructions instead of coarse task labels and fixed colors (ablation) | InstructDiffusion · AP-10K | average precision: 5.2 -> 15.9 | Helped |
| Use highly detailed instructions instead of coarse task labels and fixed colors (ablation) | InstructDiffusion · COCO-Stuff | mcIoU: 41.28 -> 53.17 | Helped |
| Train jointly on multiple vision tasks instead of segmentation alone (ablation) | InstructDiffusion · unseen datasets | evaluation metrics | Helped |
| Train jointly with other tasks instead of image editing alone (ablation) | InstructDiffusion · image editing | editing accuracy | Helped |
Surprising findings
- Replacing detailed instructions with simple task indicators such as 'keypoint detection' drops COCO keypoint AP from 71.2 to 22.7.
- Specialized keypoint models (PCT, ViTPose) still outperform the generalist model on COCO val, and the generalist model underperforms on several editing metrics relative to specialized baselines.
Limitations
- Image enhancement quality is bounded by the VAE reconstruction loss (e.g., deblur PSNR 23.58 vs 29.54 upper bound).
- Keypoint localization accuracy lags specialized methods, and the model relies on a lightweight U-Net to convert outputs to coordinates.
- Editing evaluation metrics (CLIP-Sim, aesthetic score) may favor minimal edits and are limited.
Experimental setup
Datasets: COCO CrowdPose MPII AIC COCO-Stuff gRefCOCO RefCOCO GoPro REDS SIDD CLWD InstructPix2Pix dataset
Benchmarks: COCO val2017 HumanArt AP-10K gRefCOCO RefCOCO RefCOCO+ G-Ref RefClef COCO-Stuff ADE20K Pascal Context Pascal VOC
Metrics: average precision (OKS) cIoU mcIoU CLIP-Sim aesthetic predictor score PSNR
Baselines: Unified-IO Painter PromptDiffusion InstructPix2Pix MagicBrush EDICT Null-text Inversion PCT ViTPose LAVT ReLA SAN
Builds on: Stable Diffusion InstructPix2Pix Painter Unified-IO
Closest papers by meaning
- InstructCV: Instruction-Tuned Text-to-Image Diffusion Models as Vision Generalists (ICLR 2024 · similarity 0.90)
- Generate Subgoal Images before Act: Unlocking the Chain-of-Thought Reasoning in Diffusion Model for Robot Manipulation with Multimodal Prompts (CVPR 2024 · similarity 0.86)
- Instruct-Imagen: Image Generation with Multi-modal Instruction (CVPR 2024 · similarity 0.85)
- Explanatory Instructions: Towards Unified Vision Tasks Understanding and Zero-shot Generalization (ICML 2025 · similarity 0.85)
- Seeing the Unseen: Visual Common Sense for Semantic Placement (CVPR 2024 · similarity 0.84)
- Mind the GAP: Glimpse-based Active Perception improves generalization and sample efficiency of visual reasoning (ICLR 2025 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.