Research map / Detection and segmentation / 3D detection and driving
Virtual Task Prompting for Multi-Task Scene Understanding
Zedong Wang, Yucheng Wang, Dan Xu
Read the paperCheck your idea against 137,217 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Virtual Task Prompting (VTP), which casts multi-task prompting as retrieving task-adaptive subsets from a shared virtual state to enable multi-task scene understanding on off-the-shelf vision foundation models.
Problem
Multi-task scene understanding models typically need modified vision encoders or complex decoders for cross-task interaction, limiting flexibility and compatibility with off-the-shelf vision foundation models.
Approach
Frames multi-task prompting as retrieving task-adaptive subsets from a shared virtual state, where both the state and access mechanism are learned, so cross-task interaction happens without modifying the vision encoder or using complex decoders.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace TaskPrompter prompting with Virtual Task Prompting (ImageNet ViT-L) | TaskPrompter · Pascal-Context | Delta_p: 0.00 -> +2.77 | Improved |
| use DINOv3 backbone with VTP | VTP-timm · Pascal-Context | Delta_p: +2.77 -> +5.15 | Improved |
| use DINOv3 instead of ImageNet ViT with VTP | VTP-timm · Pascal-Context normal | mErr: 13.56 -> 13.85 | Hurt |
| add virtual tasks V=2 to layer-wise routing (V=0) (ablation) | VTP · Pascal-Context | Delta_p: +1.52 -> +2.77 | Improved |
| increase virtual tasks beyond V=2 (V=4, V=8) (ablation) | VTP · Pascal-Context | Delta_p: +2.77 -> +2.41 / +2.43 | No clear effect |
| double decoder parameters (ablation) | TaskPrompter · Pascal-Context | Delta_p: -0.18 | Hurt |
| increase task vectors per task (2x, 4x) (ablation) | TaskPrompter · Pascal-Context | Delta_p: +0.09 / -0.14 | No clear effect |
| use VTP-timm instead of TaskPrompter on 13 Taskonomy tasks | TaskPrompter · Taskonomy | Delta_p: +4.50 overall; Edge 3D L1 0.0819 -> 0.0948 | Mixed |
| use ViT-5 backbone with VTP on Taskonomy normals | VTP · Taskonomy normal | mErr: 29.046 -> 33.060 | Hurt |
| use VTP instead of TaskPrompter for efficiency | TaskPrompter · efficiency | parameters/throughput: 401M -> 342M; 14.394 -> 16.318 imgs/s | Improved |
Closest papers by meaning
- MmAP: Multi-Modal Alignment Prompt for Cross-Domain Multi-Task Learning (AAAI 2024 · similarity 0.86)
- Joint Scheduling of Causal Prompts and Tasks for Multi-Task Learning (CVPR 2025 · similarity 0.86)
- Understanding Model Reprogramming for CLIP via Decoupling Visual Prompts (ICML 2025 · similarity 0.85)
- AutoVP: An Automated Visual Prompting Framework and Benchmark (ICLR 2024 · similarity 0.85)
- Exploring Task-Level Optimal Prompts for Visual In-Context Learning (AAAI 2025 · similarity 0.85)
- VMT-Adapter: Parameter-Efficient Transfer Learning for Multi-Task Dense Scene Understanding (AAAI 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
