Research map / Vision-language and multimodal / Multimodal LLMs

AURORA: Augmented Understanding via Structured Reasoning and Reinforcement Learning for Reference Audio-Visual Segmentation

AAAI 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Multimodal LLM benchmarks · 1 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes AURORA, a SAM + VideoLLaMA2 framework for Ref-AVS that combines structured CoT SFT with a segmentation feature distillation loss, corrective reflective-style training, and GRPO with IoU and class rewards.

Problem

Reference audio-visual segmentation requires understanding textual references and audio-visual cues to segment sounding objects, but existing LLM-based methods memorize fixed reasoning templates and joint reasoning-segmentation training can degrade pixel-level precision.

Approach

Uses structured four-step CoT generated by Qwen-Omni, a segmentation feature distillation loss from a frozen segmentation-only teacher's [SEG] feature, a reflective-style training stage that splices flawed reasoning, a self-correction trigger and Gemini-corrected reasoning, and GRPO with format, IoU and class rewards.

Key results

• On Ref-AVS Mix (S+U) split, AURORA reaches J&F 70.1, versus 55.3 for TSAM and 46.2 for CRAB in the table.
• On the Unseen split, AURORA achieves J&F 73.0 versus 69.4 for SAM-LAVS.
• On AVSBench S4, AURORA reaches J 77.3 versus 73.3 for Crab, with F of 86.7 versus 86.8.

Surprising findings

• Learnable-token variants of the segmentation head perform far worse than the [SEG] token approach (e.g., MLLM+seg reaches J&F 17.1 on seen), and the fixed CoT format underperforms the flexible format.

Limitations

• Single [SEG] token suits only single-object segmentation, so MS3 multi-source evaluation is omitted.
• Class reward does not consider synonyms in evaluation.
• Relies on proprietary Gemini 2.5-Pro for reflective corrections.

Experimental setup

Datasets: Ref-AVS AVSBench

Benchmarks: Ref-AVS AVSBench S4

Metrics: J F J&F

Baselines: AVSBench AVSegFormer GAVS ReferFormer R2VOS EEMC SAM-LAVS TSAM CRAB

Builds on: GRPO LISA Chain-of-Thought prompting SAM VideoLLaMA2

Closest papers by meaning

• Think Before You Segment: An Object-aware Reasoning Agent for Referring Audio-Visual Segmentation (AAAI 2026 · similarity 0.91)
• To Think or Not to Think: Pre-Decisional Reasoning Budgets for Referring Audio-Visual Segmentation (NeurIPS 2026 · similarity 0.90)
• Unraveling Instance Associations: A Closer Look for Audio-Visual Segmentation (CVPR 2024 · similarity 0.88)
• TSAM: Temporal SAM Augmented with Multimodal Prompts for Referring Audio-Visual Segmentation (CVPR 2025 · similarity 0.88)
• Dynamic Derivation and Elimination: Audio Visual Segmentation with Enhanced Audio Semantics (CVPR 2025 · similarity 0.88)
• Robust Audio-Visual Segmentation via Audio-Guided Visual Convergent Alignment (CVPR 2025 · similarity 0.88)