Research map / Vision-language and multimodal / Multimodal LLMs

Think Before You Segment: An Object-aware Reasoning Agent for Referring Audio-Visual Segmentation

AAAI 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Multimodal LLM benchmarks · 1 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes TGS-Agent, a mask-free Think-Ground-Segment agent where an audio-visual MLLM (Ref-Thinker) explicitly identifies the referred object, Grounding-DINO grounds it, and frozen SAM2 segments it, plus a new reasoning-intensive R²-AVSBench.

Problem

Referring audio-visual segmentation requires segmenting the object referred to by a text expression using audio and visual cues; prior methods rely on pixel-level mask supervision and implicit fusion, which lack interpretability and may shortcut on simple references.

Approach

Decompose Ref-AVS into an explicit think step (an audio-visual LLM generates an object-aware reasoning chain and object description), followed by grounding with Grounding-DINO and segmentation with frozen SAM2, requiring no pixel-level supervision; also introduces a benchmark with linguistically diverse references.

Key results

• On Ref-AVSBench, TGS-Agent reaches 54.9 J&F on Seen and 76.9 J&F on Unseen, versus 47.7 and 69.4 for SAM2-LOVE.
• On R²-AVSBench (mix), TGS-Agent achieves 60.1 J&F versus 37.4 for Grounded-SAM2 with the same videos' original references (in the mix-split comparison, TGS-Agent 60.1 vs 59.0 on original refs for Grounded-SAM2).
• Ablation shows the simplified object category prompt (S-object) outperforms the fine-grained description (F-object) on Ref-AVSBench mix (J&F 54.9 vs 49.0 on Seen).

Surprising findings

• The simplified category-only object description outperforms the fine-grained description as a Grounding-DINO prompt.
• Grounded-SAM2 drops by about 20 points on R²-AVSBench references versus the original references on Seen and Unseen splits.

Limitations

• Reasoning over temporal audio in mixed audio streams is error-prone (e.g., longest sound duration).
• Grounding-DINO may produce imprecise boxes for rare objects, limiting segmentation.
• Ref-Thinker output only matches the annotated category exactly for about 55% (Seen) and 49% (Unseen) of samples.

Experimental setup

Datasets: Ref-AVSBench AudioSet-style caption datasets (unspecified)

Benchmarks: Ref-AVSBench R²-AVSBench

Metrics: J F J&F S

Baselines: AVSBench AVSegFormer GAVS SAMA ReferFormer R2VOS EEMC Grounded-SAM2 Crab TSAM SAM2-LOVE

Builds on: SAM2 Grounding-DINO LLaMA-2 Ref-AVSBench Crab

Closest papers by meaning

• To Think or Not to Think: Pre-Decisional Reasoning Budgets for Referring Audio-Visual Segmentation (NeurIPS 2026 · similarity 0.93)
• AURORA: Augmented Understanding via Structured Reasoning and Reinforcement Learning for Reference Audio-Visual Segmentation (AAAI 2026 · similarity 0.91)
• TSAM: Temporal SAM Augmented with Multimodal Prompts for Referring Audio-Visual Segmentation (CVPR 2025 · similarity 0.90)
• Unleashing the Temporal-Spatial Reasoning Capacity of GPT for Training-Free Audio and Language Referenced Video Object Segmentation (AAAI 2025 · similarity 0.89)
• JointAVBench: A Benchmark for Joint Audio-Visual Reasoning Evaluation (ICLR 2026 · similarity 0.88)
• Prompting Segmentation with Sound Is Generalizable Audio-Visual Source Localizer (AAAI 2024 · similarity 0.88)