Research map / Video / Video-language understanding
TSAM: Temporal SAM Augmented with Multimodal Prompts for Referring Audio-Visual Segmentation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Extends SAM with a parallel temporal modeling branch and data-driven sparse/dense multimodal prompts to segment objects in videos from audio-visual-text cues on the Ref-AVS benchmark.
Problem
Referring audio-visual segmentation (Ref-AVS) requires segmenting objects in video using multimodal text expressions, but SAM is single-frame and only accepts user-interactive visual prompts, so it cannot exploit audio/text cues or temporal context.
Approach
Adds a trainable temporal branch (initialized from SAM's last M encoder blocks) alongside the frozen SAM image encoder with cached memory and adapters for audio cues, and replaces SAM's user prompts with language-guided audio query sparse prompts and cross-attention dense prompts derived from audio, visual and text cues.
Key results
- On the Ref-AVS Seen test set TSAM reaches J 43.43% and F 0.568, improving on EEMC by 9.23 J points.
- On the Unseen test set TSAM reaches J 54.58% and F 0.664, improving on EEMC by 5.04 J points.
- Removing the temporal branch drops the mixed-set J from 49.01% to 41.77%; TSAM is slightly worse than EEMC on the Null split (S 0.017 vs 0.007).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use TSAM instead of EEMC | Ref-AVS · Ref-AVS Seen | J (%): 34.20 -> 43.43 | Helped |
| add the temporal branch (TB) (ablation) | TSAM · Ref-AVS Mix(S+U) | J (%): 41.77 -> 49.01 | Helped |
| add the temporal modality fusion layer (TMFL) (ablation) | TSAM · Ref-AVS Mix(S+U) | J (%): 42.95 -> 49.01 | Helped |
| add the dense prompting module (DPM) (ablation) | TSAM · Ref-AVS Mix(S+U) | J (%): 45.91 -> 49.01 | Helped |
| add the sparse prompting module (SPM) (ablation) | TSAM · Ref-AVS Mix(S+U) | J (%): 46.40 -> 49.01 | Helped |
| add sparse and dense prompting modules (ablation) | TSAM · Ref-AVS Mix(S+U) | J (%): 41.59 -> 49.01 | Helped |
| add cached memory for audio and visual modalities (ablation) | TSAM · Ref-AVS Mix(S+U) | J (%): 45.59 -> 49.01 | Helped |
| add visual cached memory (ablation) | TSAM · Ref-AVS Mix(S+U) | J (%): 46.97 -> 49.01 | Helped |
| add the adapter module (AM) (ablation) | TSAM · Ref-AVS Mix(S+U) | J (%): 41.96 -> 49.01 | Helped |
| add intersection over union loss (LIoU) (ablation) | TSAM · Ref-AVS Mix(S+U) | J (%): 40.22 -> 49.01 | Helped |
| use k=3 rather than k=5 audio queries (ablation) | TSAM · Ref-AVS Mix(S+U) | J (%): 49.01 -> 47.01 | Hurt |
| use k=7 rather than k=5 audio queries (ablation) | TSAM · Ref-AVS Mix(S+U) | J (%): 49.01 -> 45.10 | Hurt |
Surprising findings
- TSAM is slightly worse than EEMC on the Null split (S 0.017 vs 0.007), attributed to SAM always producing a mask.
- Omitting the IoU loss gives the best Null-set score (S 0.008) even though it hurts Seen and Unseen performance.
Limitations
- SAM always produces a segmentation mask even when no target is present, hurting Null-split performance.
- No direct comparison with SAM2 yet, which lacks multimodal support.
- Audio is encoded offline with VGGish and text with RoBERTa, rather than trained end-to-end.
Experimental setup
Datasets: Ref-AVS dataset
Benchmarks: Ref-AVS dataset (Seen, Unseen, Null splits)
Metrics: Jaccard index J F-score F S (null-set mask area ratio, lower is better)
Baselines: AVSBench AVSegFormer GAVS SAMA ReferFormer R2-VOS EEMC
Builds on: Segment Anything Model EEMC RefSAM GAVS SAMA
Closest papers by meaning
- Think Before You Segment: An Object-aware Reasoning Agent for Referring Audio-Visual Segmentation (AAAI 2026 · similarity 0.90)
- SAM2-LOVE: Segment Anything Model 2 in Language-aided Audio-Visual Scenes (CVPR 2025 · similarity 0.90)
- AURORA: Augmented Understanding via Structured Reasoning and Reinforcement Learning for Reference Audio-Visual Segmentation (AAAI 2026 · similarity 0.88)
- To Think or Not to Think: Pre-Decisional Reasoning Budgets for Referring Audio-Visual Segmentation (NeurIPS 2026 · similarity 0.88)
- FAVE: A Structured Benchmark for Fine-Grained Audio-Visual Temporal Evaluation in Multimodal LLMs (CVPR 2026 · similarity 0.87)
- Prompting Segmentation with Sound Is Generalizable Audio-Visual Source Localizer (AAAI 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.