Research map / Video / Video segmentation and tracking
Endow SAM with Keen Eyes: Temporal-spatial Prompt Learning for Video Camouflaged Object Detection
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes TSP-SAM, an end-to-end prompt-learning framework that adapts frozen SAM to video camouflaged object detection by learning motion-driven self-prompts with long-range temporal-spatial consistency.
Problem
Applying SAM to video camouflaged object detection is hard because camouflaged objects are hard to see, so user-provided prompts are unreliable, and existing VCOD methods use two-stage pipelines that accumulate errors and generalize poorly.
Approach
Learns self-prompts (dense and sparse) from inter-frame motion perceived in the frequency domain, refines them with long-range windowed-sequence consistency from a 3D ResNet, and injects that consistency into SAM's image encoder, training everything end-to-end while SAM stays frozen.
Key results
- On MoCA-Mask, TSP-SAM (M+B) reaches mIoU 0.388 and mDice 0.458, versus 0.310 mIoU for SLT-Net, a 7.8-point mIoU gain as claimed.
- On CAD2016 (trained on MoCA-Mask), TSP-SAM (M+B) reaches mIoU 0.496 versus 0.401 for SLT-Net.
- Ablation on MoCA-Mask: motion-driven self-prompt learning raises mDice from 36.7% to 40.7%, and temporal-spatial injection adds a further 2.9%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use TSP-SAM with mask and box prompts instead of SLT-Net | VCOD · MoCA-Mask test | mIoU: 0.310 -> 0.388 | Helped |
| Use TSP-SAM with mask and box prompts instead of SLT-Net | VCOD · CAD2016 test | mIoU: 0.401 -> 0.496 | Helped |
| Use mask and box prompts instead of mask and point prompts | TSP-SAM · MoCA-Mask test | mIoU: 0.345 -> 0.388 | Helped |
| Use mask and box prompts instead of mask and point prompts | TSP-SAM · CAD2016 test | mIoU: 0.422 -> 0.496 | Helped |
| Add temporal-spatial injection (TSI) (ablation) | TSP-SAM with MSP and LCP · MoCA-Mask | mIoU: 0.366 -> 0.388 | Helped |
| Add long-range consistency-based robust prompt learning (LCP) (ablation) | TSP-SAM with MSP · MoCA-Mask | mIoU: 0.340 -> 0.366 | Helped |
| Add motion-driven self-prompt learning (MSP) instead of spatial-based self-prompt learning (SSP) (ablation) | TSP-SAM · MoCA-Mask | mIoU: 0.311 -> 0.340 | Helped |
| Use frequency-based motion perception instead of spatial-based motion perception (ablation) | Motion-driven self-prompt learning · MoCA-Mask | mIoU: 0.324 -> 0.340 | Helped |
| Use frequency-based appearance perception instead of spatial-based appearance perception (ablation) | Self-prompt learning · MoCA-Mask | mIoU: 0.311 -> 0.328 | Helped |
| Replace frequency-based appearance perception with frequency-based motion perception (ablation) | Self-prompt learning · MoCA-Mask | mIoU: 0.328 -> 0.340 | Helped |
| Vary the windowed sequence size T (ablation) | TSP-SAM · MoCA-Mask | mIoU | Mixed |
| Set the loss balance factors to (α, β) = (1, 0.6) (ablation) | TSP-SAM · MoCA-Mask | mIoU | Helped |
Surprising findings
- Box prompts are more reliable than point prompts for SAM on camouflaged video; point prompts often land off-target or only segment local regions.
Limitations
- Two-stage-like cost: SAM dominates inference time, giving only 2.53 fps versus 2.94 fps for SLT-Net.
- Point prompts fail on irregular shapes and unclear boundaries, so performance depends on the prompt type chosen.
- Relies on a windowed sequence length T that was tuned (best at T=7) on MoCA-Mask.
Experimental setup
Datasets: MoCA-Mask CAD2016
Benchmarks: MoCA-Mask CAD2016 DAVIS2016 SegTrack-v2
Metrics: S-measure weighted F-measure E-measure MAE mDice mIoU
Baselines: SINet SINet-v2 ZoomNet BGNet FEDERNet FSPNet PUENet RCRNet PNS-Net MG SLT-Net
Builds on: Segment Anything Model SLTNet PVT 3D ResNet SAM-Adapter
Closest papers by meaning
- Relax Image-Specific Prompt Requirement in SAM: A Single Generic Prompt for Segmenting Camouflaged Objects (AAAI 2024 · similarity 0.91)
- SAM-I2V: Upgrading SAM to Support Promptable Video Segmentation with Less than 0.2% Training Cost (CVPR 2025 · similarity 0.88)
- CamSAM2: Segment Anything Accurately in Camouflaged Videos (NeurIPS 2025 · similarity 0.88)
- Shift the Lens: Environment-Aware Unsupervised Camouflaged Object Detection (CVPR 2025 · similarity 0.86)
- SdalsNet: Self-Distilled Attention Localization and Shift Network for Unsupervised Camouflaged Object Detection (AAAI 2025 · similarity 0.86)
- AoP-SAM: Automation of Prompts for Efficient Segmentation (AAAI 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.