Research map / Video / Video segmentation and tracking
Referred by Multi-Modality: A Unified Temporal Transformer for Video Object Segmentation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes MUTR, a unified DETR-style temporal transformer that segments video objects referred by either text or audio, using multi-scale temporal aggregation before the transformer and multi-object temporal interaction after it.
Problem
Referring video object segmentation with multi-modal references (language or audio) requires aligning semantics across modalities and capturing inter-frame correspondence, but existing methods use separate architectures per modality and ignore temporal interaction with the references.
Approach
A single DETR-style transformer handles both text and audio references; the MTA module aggregates multi-scale visual features from adjacent frames via cascaded cross-attention from the reference tokens to initialize object queries, and the MTI module shares object-query information across frames with video-wise queries.
Key results
- On Ref-YouTube-VOS with Swin-L backbone, MUTR reaches 68.4 J&F vs. 64.2 for ReferFormer (+4.2).
- On AVSBench S4 with ResNet-50, MUTR reaches 83.0 J&F vs. 78.8 for the prior best baseline.
- Ablation on Ref-YouTube-VOS (ResNet-50): removing MTA and MTI drops J&F from 61.9 to 60.2, with a marginal FPS change (19.64 to 19.37).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add MTA and MTI (ablation) | MUTR · Ref-YouTube-VOS | J&F: 60.2% to 61.9% | Helped |
| add MTA (ablation) | MUTR with MTI · Ref-YouTube-VOS | J&F: 60.8 -> 61.9 | Helped |
| add MTI (ablation) | MUTR with MTA · Ref-YouTube-VOS | J&F: 61.5 -> 61.9 | Helped |
| use single-scale temporal aggregation (ablation) | MUTR · Ref-YouTube-VOS | J&F: 60.4 | Hurt |
| use multi-scale aggregation at image level (ablation) | MUTR · Ref-YouTube-VOS | J&F: 61.3 | Hurt |
| increase the number of MTA blocks (ablation) | MUTR · Ref-YouTube-VOS | J&F | No clear effect |
| increase the number of MTI blocks (ablation) | MUTR · Ref-YouTube-VOS | J&F | Helped |
| use only the MTI encoder or only the MTI decoder (ablation) | MUTR · Ref-YouTube-VOS | J&F | Hurt |
| use MUTR instead of ReferFormer | Ref-YouTube-VOS with Swin-L · Ref-YouTube-VOS | J&F: +4.2% | Helped |
| use MUTR instead of ReferFormer | Ref-DAVIS 2017 · Ref-DAVIS 2017 | J&F: +4.1% | Helped |
| use MUTR instead of Baseline | AVSBench with audio reference · AVSBench | J&F: +8.7% | Helped |
Limitations
- Gains from more MTA blocks saturate, attributed to limited training videos and the 256-dimensional embedding space.
- Ref-DAVIS 2017 results for ReferFormer are re-reported from official checkpoints, not from the original paper.
Experimental setup
Datasets: Ref-YouTube-VOS AVSBench S4 AVSBench MS3
Benchmarks: Ref-YouTube-VOS Ref-DAVIS 2017 AVSBench S4 AVSBench MS3
Metrics: J&F J F FPS parameters
Baselines: CMSA URVOS LBDT YOFO ReferFormer CITD MTTR MANet VLT LVS SST LGVT
Builds on: ReferFormer DETR Video Swin Transformer Hungarian matching
Closest papers by meaning
- Dual Prototype Attention for Unsupervised Video Object Segmentation (CVPR 2024 · similarity 0.88)
- MUVR: A Multi-Modal Untrimmed Video Retrieval Benchmark with Multi-Level Visual Correspondence (NeurIPS 2025 · similarity 0.87)
- Audio-Visual Segmentation via Unlabeled Frame Exploitation (CVPR 2024 · similarity 0.87)
- M^3-VOS: Multi-Phase, Multi-Transition, and Multi-Scenery Video Object Segmentation (CVPR 2025 · similarity 0.87)
- ViCaS: A Dataset for Combining Holistic and Pixel-level Video Understanding using Captions with Grounded Segmentation (CVPR 2025 · similarity 0.87)
- OmniSegmentor: A Flexible Multi-Modal Learning Framework for Semantic Segmentation (NeurIPS 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.