Research map / Image generation / Text-to-image diffusion
InstructCV: Instruction-Tuned Text-to-Image Diffusion Models as Vision Generalists
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Casts computer vision tasks (segmentation, detection, depth, classification) as instruction-guided text-to-image generation by instruction-tuning Stable Diffusion on a pooled multi-task dataset with LLM-rephrased prompts.
Problem
Standard vision recognition tasks rely on task-specific architectures and losses, and prior generalist models use visual or learned prompts that are uninterpretable and generalize poorly to new categories or phrasings.
Approach
Represent each vision task output as an RGB image (masks, drawn boxes, depth-coded images, color blocks) and instruction-tune a pretrained Stable Diffusion conditioned on the input image and a natural language task instruction, with instructions paraphrased by a T5-based LLM to increase diversity.
Key results
- On NYUv2 depth, InstructCV-FP reaches RMSE 0.275, better than BinsFormer's 0.330 and Painter's 0.288 (about 10% better than second best).
- On FSS-1000 open-vocabulary segmentation, InstructCV-RP reaches 69.8 mIoU vs 62.3 for Generalist Painter and 58.5 for Inpainting.
- InstructCV-FP scores 49.1 mAP@0.5 on COCO detection and 80.4% on Oxford-Pets classification, trailing task-specific models on classification (ViT 98.4).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use fixed prompts instead of rephrased prompts during training (ablation) | InstructCV · NYUv2 | RMSE: 0.297 -> 0.275 | Helped |
| Use fixed prompts instead of rephrased prompts during training (ablation) | InstructCV · ADE-20K | mIoU: 47.235 -> 52.3 | Helped |
| Use fixed prompts instead of rephrased prompts during training (ablation) | InstructCV · Oxford-Pets | accuracy: 82.135 -> 80.4 | Hurt |
| Use rephrased prompts during training (ablation) | InstructCV · ADE-20K with unseen user-written instructions | mIoU: 36.315 -> 48.739 | Helped |
| Use rephrased prompts during training (ablation) | InstructCV · ADE-20K with varied unseen instructions | mIoU: The FP model had a 51.8% drop; the RP model had a 6% reduction. | Helped |
| Use InstructCV-RP for open-vocabulary segmentation | InstructCV · FSS-1000 | mIoU: +3.5 mIoU over Generalist Painter; +7.3 in mIoU over Inpainting | Helped |
| Use InstructCV-FP rather than the strongest task-specific baseline | InstructCV · NYUv2 | RMSE: 10% improvement in RMSE | Helped |
| Evaluate task-specific depth models on external SUNRGB-D data | DepthFormer and BinsFormer · SUNRGB-D | RMSE: BinsFormer: ≈ 31.2%; DepthFormer: ≈ 26.0% | Hurt |
| Evaluate InstructCV on external depth data | InstructCV · SUNRGB-D | RMSE: performance improved by 6% | Helped |
| Evaluate Pix2SeqV2 on external VOC detection data | Pix2SeqV2 · VOC | mAP@0.5: 32.2% | Hurt |
| Use InstructCV instead of Pix2SeqV2 on external VOC detection data | InstructCV · VOC | mAP@0.5: +23.2 in mAP@0.5 | Helped |
| Use InstructCV instead of Unified-IO for semantic segmentation | InstructCV · ADE-20K | mIoU: +24.401 in mIOU | Helped |
Surprising findings
- Task-specific depth models degrade substantially on external data (BinsFormer ~31% drop) while InstructCV improves on SUNRGB-D.
- Fixed-prompt instruction tuning is highly sensitive to phrasing; one rephrased instruction caused a 51.8% mIoU drop, whereas the rephrased-prompt model dropped only about 6%.
Limitations
- Inference is slow relative to task-specific models and not real-time.
- Pix2Pix formulation can yield inadmissible outputs, such as non-box-shaped detection outputs.
- Instruction diversity is limited by rephrasing a small set of templates; classification is weakest and only handles one category per prompt.
Experimental setup
Datasets: MS-COCO ADE20K Oxford-IIIT Pets NYUv2 SUNRGB-D VOC ImageNet FSS-1000
Benchmarks: ADE20K COCO NYUv2 Oxford-IIIT Pets SUNRGB-D VOC ImageNet-sub FSS-1000
Metrics: RMSE mIoU Acc mAP@0.5
Baselines: DepthFormer BinsFormer UviM Mask2Former SSA ResNet ViT DETR Mask R-CNN Unified-IO Pix2SeqV2 Painter
Builds on: InstructPix2Pix Stable Diffusion latent diffusion models Unified-IO Painter
Closest papers by meaning
- Instruct-Imagen: Image Generation with Multi-modal Instruction (CVPR 2024 · similarity 0.91)
- InstructDiffusion: A Generalist Modeling Interface for Vision Tasks (CVPR 2024 · similarity 0.90)
- Explanatory Instructions: Towards Unified Vision Tasks Understanding and Zero-shot Generalization (ICML 2025 · similarity 0.89)
- Vision‑Language‑Vision Auto‑Encoder: Scalable Knowledge Distillation from Diffusion Models (NeurIPS 2025 · similarity 0.89)
- Dual Diffusion for Unified Image Generation and Understanding (CVPR 2025 · similarity 0.89)
- AP-Adapter: Improving Generalization of Automatic Prompts on Unseen Text-to-Image Diffusion Models (NeurIPS 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.