Research map / Video / Video-language understanding
Unleashing the Temporal-Spatial Reasoning Capacity of GPT for Training-Free Audio and Language Referenced Video Object Segmentation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes AL-Ref-SAM 2, a training-free pipeline that uses GPT-4 chain-of-thought temporal-spatial reasoning to pick pivot frames and boxes for SAM 2, and converts audio to language references for unified AVS and RVOS.
Problem
Training-free audio- and language-referenced video object segmentation (AVS and RVOS) is weak because GroundingDINO on a single frame, often the first frame, gives poor initial prompts for SAM 2 under spatiotemporal variation.
Approach
A GPT-assisted Pivot Selection module uses two CoT prompts so GPT-4 first selects a pivot frame (temporal reasoning) and then a pivot box among GroundingDINO candidates (spatial reasoning), with the box prompting SAM 2. A Language-Binded Reference Unification module turns audio into language categories via BEATs plus GPT-4 visual filtering so both tasks share one pipeline.
Key results
- On Ref-DAVIS17 the training-free AL-Ref-SAM 2 reaches 74.2 J&F, above the best fully-supervised entry shown in the table (DsHmp 64.9) and the Grounded-SAM 2 baseline (66.2).
- On Ref-YouTube-VOS AL-Ref-SAM 2 reaches 67.9 J&F versus 64.8 for the G-DINO + SAM 2 baseline, matching or beating several fully-supervised methods such as LoSh (67.2).
- On AVSBench S4, AL-Ref-SAM 2 reaches 70.5 M_J, well above the training-free SAM-BIND (42.0) and OWOD-BIND (58.0) baselines, though below fully-supervised COMBO (84.7).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use AL-Ref-SAM 2 instead of the strongest listed baseline | RVOS · Ref-YouTube-VOS validation | J&F: 67.2 -> 67.9 | Helped |
| Use AL-Ref-SAM 2 instead of the strongest listed baseline | RVOS · Ref-DAVIS17 validation | J&F: 70.6 -> 74.2 | Helped |
| Use AL-Ref-SAM 2 instead of the strongest listed baseline | RVOS · MeViS validation | J&F: 46.4 -> 42.8 | Hurt |
| Use AL-Ref-SAM 2 instead of the strongest listed baseline | AVS · S4 | M_J: 84.7 -> 70.5 | Hurt |
| Use AL-Ref-SAM 2 instead of the strongest listed baseline | AVS · MS3 | M_J: 63.7 -> 48.6 | Hurt |
| Use AL-Ref-SAM 2 instead of the strongest listed baseline | AVS · AVSS | M_J: 42.1 -> 36.0 | Hurt |
| Use AL-Ref-SAM 2 instead of the strongest listed baseline | AVS · AVSS-V2-Binary | M_J: 67.7 -> 59.2 | Hurt |
| Add pivot-frame selection (ablation) | GPT-PS on Ref-YouTube-VOS · Ref-YouTube-VOS | J&F: 64.8 -> 65.6 | Helped |
| Add pivot-frame selection (ablation) | GPT-PS on Ref-DAVIS17 · Ref-DAVIS17 | J&F: 66.2 -> 71.1 | Helped |
| Add pivot-box selection after pivot-frame selection (ablation) | GPT-PS on Ref-YouTube-VOS · Ref-YouTube-VOS | J&F: 65.6 -> 67.9 | Helped |
| Add pivot-box selection after pivot-frame selection (ablation) | GPT-PS on Ref-DAVIS17 · Ref-DAVIS17 | J&F: 71.1 -> 74.2 | Helped |
| Use LBRU instead of BEATs-only audio categories (ablation) | AVS reference generation · AVSS-V2-Binary | M_J: 53.7 -> 56.9 | Helped |
Surprising findings
- Naively asking GPT to select the box directly without description lowers performance (63.3 vs 64.8 J&F baseline on Ref-YouTube-VOS), showing that abstract instructions hurt.
- Choosing the last frame as the starting frame degrades results substantially (58.5 J&F vs 64.8 for the first frame) on Ref-YouTube-VOS.
Limitations
- Depends on closed-source GPT-4 API calls and on GroundingDINO candidate quality.
- Still well below fully-supervised methods on AVS (e.g., 70.5 vs 84.7 M_J on S4 with COMBO).
- Audio handling relies on top-k audio classifier categories and visual filtering, which may miss sounds not visible in frames.
Experimental setup
Datasets: Ref-YouTube-VOS Ref-DAVIS17 MeViS AVSBench
Benchmarks: Ref-YouTube-VOS Ref-DAVIS17 MeViS AVSBench
Metrics: J&F J F M_J M_F
Baselines: G-DINO + SAM 2 Grounded-SAM 2 G-L + SAM 2 SAM-BIND OWOD-BIND AT-GDINO-SAM ReferFormer LoSh COMBO AVSegFormer
Builds on: SAM 2 GroundingDINO GPT-4 Grounded-SAM
Closest papers by meaning
- Think Before You Segment: An Object-aware Reasoning Agent for Referring Audio-Visual Segmentation (AAAI 2026 · similarity 0.89)
- The Devil is in Temporal Token: High Quality Video Reasoning Segmentation (CVPR 2025 · similarity 0.87)
- AURORA: Augmented Understanding via Structured Reasoning and Reinforcement Learning for Reference Audio-Visual Segmentation (AAAI 2026 · similarity 0.87)
- Semantic and Sequential Alignment for Referring Video Object Segmentation (CVPR 2025 · similarity 0.86)
- SAMWISE: Infusing Wisdom in SAM2 for Text-Driven Video Segmentation (CVPR 2025 · similarity 0.86)
- Motion-Grounded Video Reasoning: Understanding and Perceiving Motion at Pixel Level (CVPR 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.