Research map / Video / Video-language understanding
One Token to Seg Them All: Language Instructed Reasoning Segmentation in Videos
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces VideoLISA, a video MLLM (LLaVA-Phi-3-V 3.8B with SAM decoder) that segments and tracks objects from language instructions using Sparse Dense Sampling and a One-Token-Seg-All <TRK> token, plus a new ReasonVOS benchmark.
Problem
Language-instructed reasoning segmentation in videos requires temporal understanding and temporally consistent masks, which image-based reasoning segmentation models like LISA cannot provide.
Approach
Sparse Dense Sampling keeps full-resolution tokens for a few uniformly sampled dense frames and pools the rest to one token per frame, and a single <TRK> token is trained to segment multiple frames at once to enable cross-frame tracking via SAM's decoder.
Key results
- VideoLISA-3.8B with post-optimization reaches 63.7 J&F on Refer-YouTube-VOS and 68.8 J&F on Ref-DAVIS-17, surpassing LISA-13B and TrackGPT-13B.
- On MeViS, VideoLISA-3.8B (post-optimization) achieves 44.4 J&F versus 37.2 for LMPM.
- On ReasonVOS, VideoLISA-3.8B reaches 47.5 J&F (post-optimization) versus 31.1 for LISA; on ReasonSeg test it reaches 53.8 gIoU overall.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add XMem++ post-optimization | VideoLISA-3.8B · MeViS validation | J&F: 42.3 -> 44.4 | Helped |
| Use Sparse Dense Sampling instead of alternative temporal strategies (ablation) | VideoLISA-3.8B · MeViS valid_u | J&F: 49.9, 50.8, 50.2 -> 51.7 | Helped |
| Fine-tune LISA-7B on video segmentation data (ablation) | LISA-7B · ReasonSeg val | gIoU: 51.7 -> 48.6 | Hurt |
| Fine-tune LISA-7B on video segmentation data (ablation) | LISA-7B · MeViS valid_u | J&F: 43.2 -> 44.8 | Helped |
| Replace n-frame concatenation with Sparse Dense Sampling (ablation) | VideoLISA-3.8B · ReasonSeg val | gIoU: 55.6 -> 58.9 | Helped |
| Replace spatial and temporal pooling with Sparse Dense Sampling (ablation) | VideoLISA-3.8B · MeViS valid_u | J&F: 50.8 -> 51.7 | Helped |
| Replace slow-fast pooling with Sparse Dense Sampling (ablation) | VideoLISA-3.8B · Ref-DAVIS-17 | J&F: 65.7 -> 67.8 | Helped |
| Add XMem tracking to LISA-7B (ablation) | LISA-7B · MeViS valid_u | J&F: 43.2 -> 45.6 | Helped |
| Train one <TRK> token to segment all frames rather than one frame (ablation) | VideoLISA-3.8B · MeViS valid_u | J&F: 46.1 -> 51.7 | Helped |
| Add XMem++ post-optimization using dense-frame masks (ablation) | VideoLISA-3.8B · Ref-DAVIS-17 | J&F: 67.8 -> 68.7 | Helped |
| Train with video segmentation data instead of image segmentation data only (ablation) | VideoLISA-3.8B · MeViS valid_u | J&F: 46.0 -> 49.3 | Helped |
| Train with video segmentation data instead of image segmentation data only (ablation) | VideoLISA-3.8B · ReasonSeg val | gIoU: 57.2 -> 41.4 | Hurt |
Surprising findings
- Simply finetuning LISA-7B on video segmentation data does not improve video performance and hurts image reasoning segmentation.
- Pooling-based temporal strategies (spatial-temporal and slow-fast) underperform the full-resolution Sparse Dense Sampling, indicating detailed visual tokens matter for dense video segmentation.
Limitations
- Relatively high computational cost compared with prior video segmentation methods.
- No dedicated video backbone; relies on image vision encoder from SAM and LLaVA.
- One-Token-Seg-All masks for non-dense frames have lower quality, requiring XMem++ post-optimization.
Experimental setup
Datasets: ADE20K COCO-Stuff PACO-LVIS PASCAL-Part refCLEF refCOCO refCOCO+ refCOCOg ReasonSeg YouTube-VOS Refer-YouTube-VOS MeViS
Benchmarks: Ref-YouTube-VOS Ref-DAVIS-17 MeViS ReasonVOS ReasonSeg refCOCO refCOCO+ refCOCOg
Metrics: J&F J F gIoU cIoU
Baselines: LISA-7B LISA-13B TrackGPT-7B TrackGPT-13B ReferFormer SgMg OnlineRefer MTTR OVSeg GRES X-Decoder SEEM
Builds on: LISA LLaVA SAM XMem++ DeepSpeed
Closest papers by meaning
- LISA: Reasoning Segmentation via Large Language Model (CVPR 2024 · similarity 0.90)
- ViCaS: A Dataset for Combining Holistic and Pixel-level Video Understanding using Captions with Grounded Segmentation (CVPR 2025 · similarity 0.89)
- The Devil is in Temporal Token: High Quality Video Reasoning Segmentation (CVPR 2025 · similarity 0.89)
- Seg-ReSearch: Segmentation with Interleaved Reasoning and External Search (ICML 2026 · similarity 0.89)
- HourVideo: 1-Hour Video-Language Understanding (NeurIPS 2024 · similarity 0.88)
- Look, Remember and Reason: Grounded Reasoning in Videos with Language Models (ICLR 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.