Research map / Video / Video-language understanding
SAMWISE: Infusing Wisdom in SAM2 for Text-Driven Video Segmentation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Adapts a frozen SAM2 for streaming referring video object segmentation by adding a Cross-Modal Temporal Adapter (CMT) and a Conditional Memory Encoder (CME) that corrects SAM2's tracking bias, training under 5M parameters.
Problem
Referring video object segmentation (RVOS) must work in a streaming fashion with global context, but existing methods either process short independent clips or require the whole video offline; SAM2 lacks text understanding and suffers from tracking bias when the referred object appears later.
Approach
Keep SAM2 and the text encoder frozen, insert a lightweight adapter into both encoders that performs hierarchical selective spatio-temporal attention on visual features and bidirectional visual-text cross-attention, prompt SAM2's mask decoder with CLS and verb embeddings, and add a learnable memory-correction module that switches SAM2's tracked object when a text-aligned candidate appears.
Key results
- On MeViS, SAMWISE with RoBERTa reaches 49.5 J&F versus 46.4 for the prior best DsHmp (Swin T), and 48.3 with CLIP-B using 150M total parameters.
- On Ref-YouTube-VOS, SAMWISE (RoBERTa) reaches 69.2 J&F, improving on DsHmp (V-Swin B) at 67.1.
- On Ref-DAVIS17, SAMWISE (RoBERTa) reaches 70.6 J&F, outperforming VISA's 69.4 and DsHmp's 64.9 under the reported setting.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use SAMWISE with RoBERTa rather than DsHmp | RVOS methods · MeViS | J &F: +3.1% | Helped |
| Use SAMWISE with RoBERTa rather than MUTR | RVOS methods · Ref-Youtube-VOS | J &F: +2.1% | Helped |
| Use SAMWISE with RoBERTa rather than the strongest reported baseline | RVOS methods · Ref-DAVIS17 | J &F: +5.7% | Helped |
| Add Text-to-Visual attention to MLP-only prompting (ablation) | SAMWISE · MeViS | J &F | Helped |
| Add Visual-to-Text attention to MLP-only prompting (ablation) | SAMWISE · MeViS | J &F | Helped |
| Add both cross-modal attention directions (ablation) | SAMWISE · MeViS | J &F | Helped |
| Add HSA temporal feature modeling to cross-modal attention (ablation) | SAMWISE · MeViS | J &F | Helped |
| Add the Conditional Memory Encoder (ablation) | SAMWISE with CMT · MeViS | J &F: +1.3% | Helped |
| Use hierarchical rather than fixed HSA patch sizes (ablation) | SAMWISE HSA · MeViS | J &F: +1.1% | Helped |
| Increase fixed HSA patch size from 1 to 4 or 8 (ablation) | SAMWISE HSA · MeViS | J &F | Helped |
| Apply the Conditional Memory Encoder correction at every frame rather than using its predictions (ablation) | SAMWISE · MeViS | J &F | Hurt |
| Use random correction once every 4 frames rather than CME predictions (ablation) | SAMWISE · MeViS | J &F | Hurt |
Surprising findings
- Sending each frame's prompt-based detections at every frame degrades performance (50.7 J&F for Always vs 54.2 for Never), showing that frequent correction adds noise to tracking.
Limitations
- Relies on frozen SAM2 and text encoder, so performance is bounded by the frozen backbone's capabilities.
- Still trails large VLM-based methods on some benchmarks in absolute terms, which the authors note are not directly comparable.
- Ablations are conducted only on MeViS valid_u with the CLIP-B encoder.
Experimental setup
Datasets: RefCOCO RefCOCO+ RefCOCOg Ref-YouTube-VOS MeViS Ref-DAVIS17
Benchmarks: MeViS Ref-YouTube-VOS Ref-DAVIS17
Metrics: J&F J F
Baselines: DsHmp LMPM OnlineRefer RefSAM GroundingDINO+SAM2 VISA LISA One-Token-Seg-All ReferFormer SOC MUTR LoRA
Builds on: Segment Anything 2 Adapter (Houlsby) AdaptFormer LoRA OnlineRefer
Closest papers by meaning
- LoSh: Long-Short Text Joint Prediction Network for Referring Video Object Segmentation (CVPR 2024 · similarity 0.90)
- SAM 2: Segment Anything in Images and Videos (ICLR 2025 · similarity 0.90)
- Semantic and Sequential Alignment for Referring Video Object Segmentation (CVPR 2025 · similarity 0.89)
- SAM-I2V: Upgrading SAM to Support Promptable Video Segmentation with Less than 0.2% Training Cost (CVPR 2025 · similarity 0.88)
- The Devil is in Temporal Token: High Quality Video Reasoning Segmentation (CVPR 2025 · similarity 0.87)
- SANSA: Unleashing the Hidden Semantics in SAM2 for Few-Shot Segmentation (NeurIPS 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.