Research map / Vision-language and multimodal / Cross-modal learning

SAM2-LOVE: Segment Anything Model 2 in Language-aided Audio-Visual Scenes

CVPR 2025 · Vision-language and multimodal › Cross-modal learning · approach: Audio-visual source localization · 3 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes SAM2-LOVE, which fuses text, audio and visual features into a learnable [seg] token that prompts SAM2 for referring audio-visual video segmentation, with token propagation and [cls] token accumulation for temporal consistency.

Problem

Reference audio-visual segmentation (Ref-AVS) in language-aided audio-visual scenes requires segmenting objects referred to by text and audio across video frames, while existing methods lack triple-modality fusion and spatio-temporal consistency.

Approach

Compress text, audio and visual embeddings into a learnable [seg] token via a fusion transformer that prompts SAM2 on the first frame, then rely on SAM2's zero-shot video propagation; the [seg] token is propagated frame to frame and a history of [cls] tokens is accumulated to avoid forgetting earlier frames.

Key results

• SAM2-LOVE reaches 47.7% J&F on Ref-AVS seen and 69.4% on unseen, and 58.5% on the mixed split, versus 50.0% for prior SOTA EEMC on mix (+8.5%).
• On the NULL split SAM2-LOVE has S-score 0.23 versus EEMC's 0.007, so it is weaker at rejecting null references in the paper's own table (the text says it lags EEMC by 0.2).
• Ablation: [cls] token accumulation improves mixed J&F from 46.7 to 47.7 on seen split (with backbone change) and 69.0 to 69.4 unseen; learnable token beats mean fusion (46.7 vs 45.8 seen J&F).

Measured effects reported in the paper

Surprising findings

• Using a single [seg] token outperforms multiple tokens on seen splits, while more tokens help unseen splits.
• Swapping CLIP text/vision encoders for RoBERTa and ViT improves results, which the authors attribute to CLIP focusing on category names rather than control-signal expressions.

Limitations

• Weak at rejecting null references (NULL split), since SAM2 tends to segment something regardless of the expression.
• Relies on the first-frame prompt; misaligned triple-modality references are not handled.
• Performance on unseen categories is lower than on seen for some configurations.

Experimental setup

Datasets: Ref-AVS

Benchmarks: Ref-AVS

Metrics: J F J&F S-score

Baselines: AVSBench AVSegFormer GAVS ReferFormer R2VOS EEMC

Builds on: SAM2 EEMC SAM VGGish RoBERTa

Closest papers by meaning

• TSAM: Temporal SAM Augmented with Multimodal Prompts for Referring Audio-Visual Segmentation (CVPR 2025 · similarity 0.90)
• SAM 2: Segment Anything in Images and Videos (ICLR 2025 · similarity 0.88)
• Improving Audio-Visual Segmentation with Bidirectional Generation (AAAI 2024 · similarity 0.87)
• Unraveling Instance Associations: A Closer Look for Audio-Visual Segmentation (CVPR 2024 · similarity 0.87)
• Think Before You Segment: An Object-aware Reasoning Agent for Referring Audio-Visual Segmentation (AAAI 2026 · similarity 0.87)
• Unsupervised Audio-Visual Segmentation with Modality Alignment (AAAI 2025 · similarity 0.86)