Research map / Video / Video-language understanding
The Devil is in Temporal Token: High Quality Video Reasoning Segmentation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes VRS-HQ, an end-to-end video reasoning segmentation model that uses frame-level <SEG> and temporal <TAK> tokens from an MLLM (Chat-UniVi, LoRA-tuned), fuses them via similarity weighting, and uses SAM2 for keyframe selection (via occlusion scores) and mask propagation.
Problem
Video reasoning segmentation relies on a single special token for keyframe-based segmentation, which captures limited spatial detail and inter-frame motion, and existing pipelines depend on external models for keyframe selection and propagation, preventing end-to-end inference.
Approach
Encode frame-level and temporal-level information into hierarchical tokens (<SEG> and <TAK>), fuse frame tokens into the temporal token via cosine-similarity-weighted aggregation, and use SAM2 occlusion scores combined with token similarity to select keyframes at inference, so SAM2 both segments and propagates masks end-to-end.
Key results
- On ReVOS, VRS-HQ-13B surpasses VISA-13B by 5.9% J&F on the referring subset and 12.5% on the reasoning subset (overall 9.1%).
- On MeViS, VRS-HQ-13B reaches 50.9 J&F versus VISA-13B's 44.5; on Ref-YouTube-VOS and Ref-DAVIS17 it improves J&F over VideoLISA and VISA-13B by 7.3% and 5.6%.
- Ablation: removing TDA drops ReVOS reasoning J&F from 56.1 to 54.1; removing TKS drops it to 53.5.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add TDA and TKS (ablation) | VRS-HQ · ReVOS referring | J &F: 56.8 -> 62.1 | Helped |
| Add TDA and TKS (ablation) | VRS-HQ · ReVOS reasoning | J &F: 51.2 -> 56.1 | Helped |
| Add TDA (ablation) | VRS-HQ · ReVOS referring | J &F: 58.8 -> 62.1 | Helped |
| Add TDA (ablation) | VRS-HQ · ReVOS reasoning | J &F: 54.1 -> 56.1 | Helped |
| Add TKS (ablation) | VRS-HQ · ReVOS referring | J &F: 60.1 -> 62.1 | Helped |
| Add TKS (ablation) | VRS-HQ · ReVOS reasoning | J &F: 53.5 -> 56.1 | Helped |
| Set TDA fusion coefficient α to 0 instead of 0.1 (ablation) | VRS-HQ · ReVOS referring | J &F: 62.1 -> 58.8 | Hurt |
| Set TDA fusion coefficient α to 0.25 instead of 0.1 (ablation) | VRS-HQ · ReVOS referring | J &F: 62.1 -> 61.3 | Hurt |
| Set TDA fusion coefficient α to 0.5 instead of 0.1 (ablation) | VRS-HQ · ReVOS referring | J &F: 62.1 -> 60.6 | Hurt |
| Add occlusion score to keyframe selection (ablation) | VRS-HQ TKS · ReVOS referring | J &F: 60.1 -> 62.1 | Helped |
| Add occlusion score to keyframe selection (ablation) | VRS-HQ TKS · ReVOS reasoning | J &F: 53.6 -> 56.1 | Helped |
| Add CLIP score to token similarity and occlusion scores (ablation) | VRS-HQ TKS · ReVOS referring | J &F: 62.1 -> 61.9 | Hurt |
Surprising findings
- Adding CLIP scores to keyframe selection slightly degrades performance, which the authors attribute to CLIP's limited spatiotemporal understanding.
- Image segmentation performance is marginally lower than video results, which the authors attribute to video-focused training and reliance on SAM2's memory mechanism.
Limitations
- Image segmentation results are slightly below video-focused baselines such as VideoLISA on ReasonSeg and RIS benchmarks.
- Keyframe selection relies on SAM2 occlusion scores and token similarity; CLIP scores did not help.
Experimental setup
Datasets: LLaVA-Instruct-150k Video-ChatGPT ADE20K COCO-Stuff PACO-LVIS PASCAL-Part refCLEF refCOCO refCOCO+ refCOCOg ReasonSeg Ref-YouTube-VOS
Benchmarks: ReVOS Ref-YouTube-VOS Ref-DAVIS17 MeViS refCOCO refCOCO+ refCOCOg ReasonSeg
Metrics: J&F J F robustness score R gIoU cIoU
Baselines: VISA VideoLISA LISA TrackGPT ReferFormer MTTR LMPM OnlineRefer DsHmp ReLA LAVT CRIS
Builds on: LISA VISA SAM2 Chat-UniVi VideoLISA
Closest papers by meaning
- Reinforcing Video Object Segmentation to Think before it Segments (CVPR 2026 · similarity 0.91)
- One Token to Seg Them All: Language Instructed Reasoning Segmentation in Videos (NeurIPS 2024 · similarity 0.89)
- VideoEspresso: A Large-Scale Chain-of-Thought Dataset for Fine-Grained Video Reasoning via Core Frame Selection (CVPR 2025 · similarity 0.89)
- Logic-in-Frames: Dynamic Keyframe Search via Visual Semantic-Logical Verification for Long Video Understanding (NeurIPS 2025 · similarity 0.88)
- Unleashing the Temporal-Spatial Reasoning Capacity of GPT for Training-Free Audio and Language Referenced Video Object Segmentation (AAAI 2025 · similarity 0.87)
- VideoReasonBench: Can MLLMs Perform Vision-Centric Complex Video Reasoning? (ICLR 2026 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.