Research map / Video / Video-language understanding
DTOS: Dynamic Time Object Sensing with Large Multimodal Model
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes DTOS, a two-stage MLLM framework (Text-guided Clip Sampler plus Text-guided Clip Detector, built on VILA1.5-8B with LoRA and SAM2) that uses special <reg>/<seg> tokens and text-guided clip sampling for referring video object segmentation and moment retrieval.
Problem
MLLMs struggle with referring video object segmentation (RVOS) because they regress numerical timestamps and coordinates poorly as text, produce repetitive templates when multiple targets are referenced, and lose visual information when video sampling ignores the query.
Approach
Use special tokens whose hidden states are decoded into multiple event boundaries and boxes (one-to-many querying with bipartite matching), and a Text-guided Clip Sampler that selects query-relevant clips while keeping uniform temporal resolution via an interleaved original-frame and image-grid multimodal sequence.
Key results
- DTOS achieves J&F of 48.86 on MeViS, 74.88 on Ref-DAVIS17 and 68.52 on Ref-YT-VOS, improvements of +4.36, +4.48 and +3.02 over prior SOTA.
- TCS reaches R1@0.7 of 43.2 on Charades-STA, improving over prior LLM-based methods by about +6.71.
- One-to-many special-token design improves J&F by 8.73 over the one-to-one setting on MeViS validation.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add DTOS instead of the previous best baseline | RVOS on MeViS · MeViS | J&F: +4.36 | Helped |
| add DTOS instead of the previous best baseline | RVOS on Ref-DAVIS17 · Ref-DAVIS17 | J&F: +4.48 | Helped |
| add DTOS instead of the previous best baseline | RVOS on Ref-YT-VOS · Ref-YT-VOS | J&F: +3.02 | Helped |
| use TCS instead of random sampling (ablation) | DTOS on Ref-DAVIS17 · Ref-DAVIS17 | J&F: 66.42 -> 74.88 | Helped |
| use TCS instead of uniform sampling (ablation) | DTOS on Ref-DAVIS17 · Ref-DAVIS17 | J&F: 72.56 -> 74.88 | Helped |
| use TCS instead of single-frame clips (ablation) | DTOS on Ref-DAVIS17 · Ref-DAVIS17 | J&F: 67.28 -> 74.88 | Helped |
| use TCS without adding uniform sampling (ablation) | DTOS on Ref-DAVIS17 · Ref-DAVIS17 | J&F: 72.24 -> 74.88 | Helped |
| regress boundaries with <reg> rather than output absolute boundary values as text (ablation) | TCS on Charades-STA · Charades-STA | mIoU: 34.03 -> 58 | Helped |
| regress boundaries with <reg> rather than output relative boundary values as text (ablation) | TCS on Charades-STA · Charades-STA | mIoU: 50.94 -> 58 | Helped |
| use one <reg> token instead of separate <st> and <ed> tokens (ablation) | TCS on Charades-STA · Charades-STA | mIoU: 24.34 -> 58 | Helped |
| use 7 <seg> tokens instead of 1 (ablation) | DTOS on MeViS val · MeViS val | J&F: 47.35 -> 56.08 | Helped |
| use 7 <seg> tokens instead of 4 (ablation) | DTOS on MeViS val · MeViS val | J&F: 52.88 -> 56.08 | Helped |
Surprising findings
- Text-only absolute-timestamp output drops mIoU by 24.01 on Charades-STA, and relative-value text output also drops performance, showing numeric tokenization is a bottleneck.
- Uniform sampling performs nearly as well as the TCS sampler on Ref-DAVIS17 (J&F 72.56 vs 74.88 with TCS plus frame-level setup), suggesting RVOS descriptions often refer to whole-video targets.
Limitations
- Bounding-box decoding of special tokens lacks detailed target descriptions and depends heavily on SAM2.
- Final propagation target is chosen from one or more frames without explicit temporal tracking across frames.
Experimental setup
Datasets: MeViS Ref-YouTube-VOS Ref-DAVIS17 Charades-STA DiDeMo Activity-Captions QVHighlights
Benchmarks: MeViS Ref-DAVIS17 Ref-YT-VOS Charades-STA ActivityNet Captions QVHighlights DiDeMo
Metrics: J&F J F R1@0.3 R1@0.5 R1@0.7 mIoU mAP
Baselines: URVOS MTTR ReferFormer HTR DsHmp LISA TrackGPT ViLLa VISA VTimeLLM Momentor HawkEye
Builds on: VISA LISA SAM2 VILA1.5 MomentDETR
Closest papers by meaning
- LoSh: Long-Short Text Joint Prediction Network for Referring Video Object Segmentation (CVPR 2024 · similarity 0.87)
- Transferable Video Moment Localization by Moment-Guided Query Prompting (AAAI 2024 · similarity 0.86)
- TSAM: Temporal SAM Augmented with Multimodal Prompts for Referring Audio-Visual Segmentation (CVPR 2025 · similarity 0.86)
- The Devil is in Temporal Token: High Quality Video Reasoning Segmentation (CVPR 2025 · similarity 0.86)
- Semantic and Sequential Alignment for Referring Video Object Segmentation (CVPR 2025 · similarity 0.86)
- TimeChat: A Time-sensitive Multimodal Large Language Model for Long Video Understanding (CVPR 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.