Research map / Video / Video-language understanding
GLUS: Global-Local Reasoning Unified into A Single Large Language Model for Video Segmentation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes GLUS, a single MLLM (LISA-based) that unifies global reasoning (sparse context frames) and local reasoning (continuous query frames) with an end-to-end SAM-2 memory bank, object contrastive loss, and self-refined key-frame selection for referring video object segmentation.
Problem
Referring video object segmentation (RefVOS) with MLLMs struggles with a dilemma: models either reason globally over a few key frames or track objects locally, and rely on external VOS or frame selectors to cover the other aspect, within a limited context window.
Approach
Split the MLLM's frame budget into uniformly sampled context frames for global reasoning and continuous query frames for local tracking, train end-to-end with a pre-trained SAM-2 memory bank so no external VOS is needed, add a segmentation-token object contrastive loss, and use the model's own IoU predictions to fine-tune a key-frame selector.
Key results
- On MeViS, GLUS-A reaches 51.3 J&F versus 44.5 for VISA-13B and 45.2 for VideoGLaMM among MLLM methods, and 43.5 for VISA-7B.
- On Ref-Youtube-VOS, GLUS-A achieves 67.3 J&F, the best among reported methods in the table, with GLUS-S at 66.6 J&F.
- On ReVOS, GLUS-A achieves 51.4 J&F versus 44.3 for VISA-13B.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| unify global context frames and local query frames (ablation) | GLUS · MeViS (valid) | J &F: 47.2 -> 47.4; 46.2 -> 47.4 | Helped |
| add an end-to-end memory bank (ablation) | GLUS with global-local reasoning · MeViS (valid) | J &F: 47.4 -> 49.5 | Helped |
| add object contrastive loss (ablation) | GLUS with global-local reasoning and memory bank · MeViS (valid) | J &F: 49.5 -> 50.3 | Helped |
| add key frame selection (ablation) | GLUS with global-local reasoning, memory bank, and object contrastive loss · MeViS (valid) | J &F: 50.3 -> 50.3 | No clear effect |
| add a key frame selector (ablation) | GLUS · MeViS (valid u) | J &F: 60.9 -> 61.6 | Helped |
| add a key frame selector (ablation) | GLUS · MeViS (valid) | J &F: 50.3 -> 50.3 | No clear effect |
| add a key frame selector (ablation) | GLUS · RefYTB (valid) | J &F: 65.5 -> 66.6 | Helped |
| fine-tune Chat-UniVi 7B with GLUS pseudo-labels for key frame selection (ablation) | Chat-UniVi 7B key frame selector · MeViS (valid u) | J &F: 61.5 -> 61.6 | No clear effect |
| fine-tune Chat-UniVi 7B with GLUS pseudo-labels for key frame selection (ablation) | Chat-UniVi 7B key frame selector · MeViS (valid) | J &F: 49.9 -> 50.3 | Helped |
| fine-tune Chat-UniVi 7B with GLUS pseudo-labels for key frame selection (ablation) | Chat-UniVi 7B key frame selector · RefYTB (valid) | J &F: 65.4 -> 66.6 | Helped |
| use GLUS itself for propagation instead of an external VOS model (ablation) | GLUS · MeViS (valid u) | J &F: 58.3 -> 61.6; 56.9 -> 61.6 | Helped |
| use GLUS itself for propagation instead of an external VOS model (ablation) | GLUS · MeViS (valid) | J &F: 45.0 -> 50.3; 47.1 -> 50.3 | Helped |
Surprising findings
- Object contrastive loss applied only on MeViS samples also improves Ref-Youtube-VOS performance.
- Using GLUS itself for propagation outperforms external VOS models Cutie and SAM-2 on MeViS (61.6 vs 58.3 and 56.9 on MeViS valid-u).
Limitations
- Limited computational resources forced Additional-SFT to omit object contrastive loss and key frame selector.
- Context window limited to 8 frames (4 context + 4 query), so global information is sparse.
Experimental setup
Datasets: MeViS Ref-Youtube-VOS ReVOS LV-VIS DAVIS-17 ReasonVOS
Benchmarks: MeViS Ref-Youtube-VOS ReVOS ReasonVOS
Metrics: J&F J F
Baselines: VISA-7B VISA-13B VideoLISA-3.8B ViLLa TrackGPT-7B TrackGPT-13B LISA-7B LISA-13B DsHmp VideoGLaMM Cutie SAM-2
Builds on: LISA SAM-2 VISA VideoLISA SimCLR Chat-UniVi
Closest papers by meaning
- One Token to Seg Them All: Language Instructed Reasoning Segmentation in Videos (NeurIPS 2024 · similarity 0.86)
- The Devil is in Temporal Token: High Quality Video Reasoning Segmentation (CVPR 2025 · similarity 0.86)
- ViCaS: A Dataset for Combining Holistic and Pixel-level Video Understanding using Captions with Grounded Segmentation (CVPR 2025 · similarity 0.84)
- GSVA: Generalized Segmentation via Multimodal Large Language Models (CVPR 2024 · similarity 0.84)
- LISA: Reasoning Segmentation via Large Language Model (CVPR 2024 · similarity 0.84)
- Unleashing the Temporal-Spatial Reasoning Capacity of GPT for Training-Free Audio and Language Referenced Video Object Segmentation (AAAI 2025 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.