Research map / Detection and segmentation / Open-set and few-shot detection
VSCode: General Visual Salient and Camouflaged Object Detection with 2D Prompt Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes VSCode, a single generalist model for salient and camouflaged object detection across RGB, RGB-D, RGB-T, video, and VCOD tasks, using VST as backbone plus 2D (domain- and task-specific) prompts and a prompt discrimination loss.
Problem
Existing SOD and COD methods rely on task-specific specialist models that are redundant and generalize poorly; joint training across these multimodal tasks suffers from conflicts between commonalities and peculiarities.
Approach
Decompose task peculiarities along two dimensions, domain (RGB/depth/thermal/flow) and task (SOD/COD), using learnable prompts inserted into a shared VST encoder-decoder, with a prompt discrimination loss that decorrelates prompts so the foundation model focuses on shared knowledge; combining domain and task prompts enables zero-shot RGB-D COD.
Key results
- VSCode-S outperforms specialist and generalist baselines across all six tasks on 26 datasets, e.g., COD10K Sm 0.869 vs EVP 0.845 (VSCode-S vs EVP).
- Zero-shot RGB-D COD on CAMO: VSCode-T Sm 0.885 vs PopNet 0.808 (specialist trained on RGB-D COD).
- 2D prompts add only about 0.03M parameters per task over the shared model, with a total of 323.46M parameters for all six tasks in specialized training.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use the VSCode-S generalist model rather than the strongest listed specialist baseline | VSCode-S · DUTS | Sm: .917 -> .926 | Helped |
| Use the VSCode-S generalist model rather than the strongest listed specialist baseline | VSCode-S · NJUD | Sm: .928 -> .944 | Helped |
| Use the VSCode-S generalist model rather than the strongest listed specialist baseline | VSCode-S · VT5000 | Sm: .895 -> .925 | Helped |
| Use the VSCode-S generalist model rather than the strongest listed specialist baseline | VSCode-S · SegV2 | Sm: .874 -> .946 | Helped |
| Use the VSCode-S generalist model rather than the strongest listed specialist baseline | VSCode-S · COD10K | Sm: .845 -> .869 | Helped |
| Use the VSCode-S generalist model rather than the strongest listed specialist baseline | VSCode-S · CAD | Sm: .715 -> .790 | Helped |
| Combine depth-domain and COD-task prompts for the unseen RGB-D COD task | VSCode-T · COD10K (zero-shot RGB-D COD) | Sm: .851 -> .882 | Helped |
| Add encoder task-specific prompts (ablation) | VSCode general-training model · DUTS, NJUD, VT5000, SegV2, CAMO, CAD | Sm: DUTS .908 -> .909; NJUD .932 -> .935; VT5000 .909 -> .912; SegV2 .933 -> .943; CAMO .800 -> .811; CAD .743 -> .736 | Helped |
| Add decoder task-specific prompts (ablation) | VSCode general-training model · DUTS, NJUD, VT5000, SegV2, CAMO, CAD | Sm: DUTS .902 -> .909; NJUD .929 -> .935; VT5000 .904 -> .912; SegV2 .940 -> .943; CAMO .799 -> .811; CAD .740 -> .736 | Mixed |
| Add prompt discrimination loss (ablation) | VSCode with domain and task prompts · DUTS, NJUD, VT5000, SegV2, CAMO, CAD | Sm: DUTS .904 -> .909; NJUD .931 -> .935; VT5000 .906 -> .912; SegV2 .925 -> .943; CAMO .804 -> .811; CAD .759 -> .736 | Mixed |
| Use deep rather than shallow domain-specific prompts (ablation) | VSCode · DUTS, NJUD, VT5000 | Sm: DUTS .900 -> .902; NJUD .926 -> .931; VT5000 .905 -> .909 | Mixed |
| Use deep rather than shallow encoder task-specific prompts (ablation) | VSCode · CAMO and CAD | Sm: CAMO .793 -> .804; CAD .747 -> .759 | Helped |
Surprising findings
- Joint training on SOD tasks gave no significant gain for RGB SOD and RGB-D SOD, while RGB-T SOD and VSOD improved.
- Adding more prompt tokens does not monotonically help; increasing decoder task prompts beyond 10 degraded performance.
Limitations
- Joint training provided limited benefit for RGB and RGB-D SOD, which already have ample data.
- Zero-shot evaluation covers only one unseen task (RGB-D COD).
Experimental setup
Datasets: DUTS NJUD NLPR DUTLF-Depth VT5000 DAVIS DAVSOD COD10K CAMO MoCA-Mask
Benchmarks: DUTS ECSSD HKU-IS PASCAL-S DUT-O SOD NJUD NLPR DUTLF-Depth ReDWeb-S STERE SIP
Metrics: Sm Fm Em MAE-free structure-measure max F-measure max E-measure
Baselines: VST ICON-R VST-T++ MENet EVP CMINet CAVER MIDD TNet CGMDR DCFNet FSNet
Builds on: VST VPT Swin Transformer EVP
Closest papers by meaning
- Relax Image-Specific Prompt Requirement in SAM: A Single Generic Prompt for Segmenting Camouflaged Objects (AAAI 2024 · similarity 0.83)
- Endow SAM with Keen Eyes: Temporal-spatial Prompt Learning for Video Camouflaged Object Detection (CVPR 2024 · similarity 0.82)
- CP-DETR: Concept Prompt Guide DETR Toward Stronger Universal Object Detection (AAAI 2025 · similarity 0.82)
- DoGA: Enhancing Grounded Object Detection via Grouped Pre-Training with Attributes (AAAI 2025 · similarity 0.82)
- Shift the Lens: Environment-Aware Unsupervised Camouflaged Object Detection (CVPR 2025 · similarity 0.81)
- Learning Endogenous Attention for Incremental Object Detection (CVPR 2025 · similarity 0.81)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.