Research map / Vision-language and multimodal / CLIP and zero-shot
Explanatory Instructions: Towards Unified Vision Tasks Understanding and Zero-shot Generalization
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Explanatory Instructions that describe vision tasks as linguistic input-to-output transformations, builds the 12M-triplet DECVT dataset, and fine-tunes a 7B Lumina-mGPT-based AR VLM to show instruction-level and task-level zero-shot generalization.
Problem
Computer vision lacks the task-level zero-shot generalization seen in NLP because vision tasks are defined by discrete terminological labels (e.g., "image segmentation") that do not convey task objectives.
Approach
Replace terminological task names with explanatory instructions that describe the detailed transformation from input image to output image, and build a bidirectional image-instruction-image dataset so an autoregressive VLM can follow these instructions and generalize to unseen tasks.
Key results
- On unseen Canny-to-Image (MultiGen-20M), the model reaches F1 20.69 and FID 26.93, versus 10.09 F1 and 61.65 FID for Lumina-mGPT, but trails ControlNet-SD1.5 (F1 34.65) and OmniGen (F1 35.54).
- On the MagicBrush test set the model reports CLIP_out 0.292 and DINO 0.831, slightly above Lumina-mGPT but below Emu Edit and PixWizard on most metrics.
- On low-light enhancement (LOLv2), the model reaches PSNR 14.48, well below PromptIR (21.23) and PixWizard (20.29).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Fine-tune Lumina-mGPT on DECVT using explanatory instructions | Lumina-mGPT-7B-768 · MultiGen-20M Canny-to-Image | F1: 10.09 -> 20.69 | Helped |
| Fine-tune Lumina-mGPT on DECVT using explanatory instructions | Lumina-mGPT-7B-768 · MultiGen-20M HED-to-Image | SSIM: 0.1834 -> 0.6561 | Helped |
| Fine-tune Lumina-mGPT on DECVT using explanatory instructions | Lumina-mGPT-7B-768 · Places Inpainting | FID: 42.69 -> 15.16 | Helped |
| Fine-tune Lumina-mGPT on DECVT using explanatory instructions | Lumina-mGPT-7B-768 · Places Outpainting | FID: 42.66 -> 15.15 | Helped |
| Fine-tune Lumina-mGPT on DECVT using explanatory instructions | Lumina-mGPT-7B-768 · MagicBrush test set | CLIP_out: 0.282 -> 0.292 | Helped |
| Fine-tune Lumina-mGPT on DECVT using explanatory instructions | Lumina-mGPT-7B-768 · Emu Edit test set | CLIP_out: 0.283 -> 0.286 | Helped |
| Fine-tune Lumina-mGPT on DECVT using explanatory instructions | Lumina-mGPT-7B-768 · NYU-Depth V2 | RMSE: null | Hurt |
| Use explanatory instructions to infer the unseen Image-to-Canny or Image-to-Depth task | DECVT-fine-tuned AR-based VLM · Unseen vision tasks | task-level zero-shot capability | Hurt |
Surprising findings
- The model fails to show task-level zero-shot ability on Image-to-Canny and Image-to-Depth, which the authors attribute to weak image-tokenizer/text alignment in the pretrained initialization.
- Off-the-shelf recaption models (InternVL2-76B, Qwen2.5, Claude-3.5-Sonnet) produced instructions with hallucinated or non-conforming content, leading the authors to select GPT-4o.
Limitations
- Task-level zero-shot ability is limited: it fails on Image-to-Canny and Image-to-Depth and is weaker on low-level tasks.
- Generation quality lags task-specific and diffusion-based generalist models (e.g., ControlNet, OmniGen, PixWizard) on most benchmarks.
- Relies on GPT-4o for generating explanatory instructions, with potential biases and inaccuracies.
Experimental setup
Datasets: DECVT AllWeather NYU-Rain D-HAZE NH-HAZE UIEB Adobe-5k LVIS CSGO ADE20K Depth in the Wild MagicBrush
Benchmarks: MultiGen-20M Places NYU-Depth V2 ADE20K Rain100L SOTS UDC(T-OLED) LOLv2 Emu Edit test set MagicBrush test set
Metrics: F1 FID CLIP-Score SSIM LPIPS IS RMSE mIoU mean angle error PSNR
Baselines: ControlNet-SD1.5 T2I-Adapter-SD1.5 LDM-4 LaMa DeepFill v2 MaskGIT OmniGen PixWizard Lumina-mGPT Emu Edit InstructPix2Pix MagicBrush
Builds on: Lumina-mGPT Chameleon LLaMA-2 ControlNet++
Closest papers by meaning
- InstructCV: Instruction-Tuned Text-to-Image Diffusion Models as Vision Generalists (ICLR 2024 · similarity 0.89)
- On the Test-Time Zero-Shot Generalization of Vision-Language Models: Do We Really Need Prompt Learning? (CVPR 2024 · similarity 0.89)
- Learning by Correction: Efficient Tuning Task for Zero-Shot Generative Vision-Language Reasoning (CVPR 2024 · similarity 0.89)
- TPR: Topology-Preserving Reservoirs for Generalized Zero-Shot Learning (NeurIPS 2024 · similarity 0.88)
- Interpreting and Analysing CLIP's Zero-Shot Image Classification via Mutual Knowledge (NeurIPS 2024 · similarity 0.88)
- Enhancing Target-unspecific Tasks through a Features Matrix (ICML 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.