Research map / Video / Video segmentation and tracking
LiVOS: Light Video Object Segmentation with Gated Linear Matching
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes LiVOS, a space-time memory network for semi-supervised video object segmentation that replaces softmax matching with gated linear matching (linear attention with a data-dependent forget gate), giving constant-size state and enabling 4096p inference on a 32G GPU.
Problem
Softmax-attention space-time memory (STM) networks for video object segmentation have quadratic spatial and linear temporal memory/compute costs, limiting use on long and high-resolution videos.
Approach
Reformulates memory matching as linear attention computed recurrently into a constant-size 2D state, and adds a data-dependent gate (low-rank, from image-encoder features) that multiplicatively decays the state to improve selectivity, inspired by gated linear attention in language modeling.
Key results
- Achieves 64.8 J&F on MOSE (with MOSE training) and 85.1 J&F on DAVIS (ResNet-50 variant), surpassing all non-STM methods.
- On LVOS val, LiVOS reaches 50.6 J&F with ~503M memory, outperforming RDE (47.2) and downgraded Cutie variants.
- At 2048p LiVOS uses 7.7G GPU memory versus OOM for most STM methods, and is the only method running at 4096p (30.4G).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add the gate mechanism to linear matching (ablation) | LiVOS · DAVIS 2017 val | J &F: 83.0 -> 84.4 | Helped |
| add object memory (ablation) | LiVOS · DAVIS 2017 val | J &F: 80.4 -> 84.4 | Helped |
| add sensory memory (ablation) | LiVOS · DAVIS 2017 val | J &F: 79.1 -> 84.4 | Helped |
| use LiVOS instead of Cutie-base | video object segmentation · MOSE val | J &F: 68.3 -> 64.8 | Hurt |
| use LiVOS instead of Cutie-base | video object segmentation · DAVIS-17 val | J &F: 88.8 -> 84.4 | Hurt |
| use LiVOS instead of Cutie-base | video object segmentation · YouTube-VOS 2019 val | G: 86.5 -> 79.9 | Hurt |
| use LiVOS instead of Cutie-base | video object segmentation · LVOS val | J &F: 63.5 -> 50.6 | Hurt |
| use LiVOS instead of Cutie-base | video object segmentation · LVOS test | J &F: 63.6 -> 44.6 | Hurt |
| use LiVOS for high-resolution inference | video object segmentation · DAVIS 2017 val, 2048p | GPU memory: 7.7G | Helped |
| use LiVOS for high-resolution inference | video object segmentation · DAVIS 2017 val, 4096p | GPU memory: 30.4G | Helped |
Surprising findings
- Gate mechanism adds only 1.4 J&F gain with minor speed/memory trade-off, while sensory and object memories contribute more.
Limitations
- Single recurrent state may yield suboptimal results on longer, high-resolution videos.
- Not optimized for high-resolution videos; relies on no dedicated high-res modules.
- Gap remains to Cutie (about 3.5 J&F on MOSE) and STM-based methods overall.
Experimental setup
Datasets: DAVIS 2017 YouTube-VOS 2019 MOSE ImageNet
Benchmarks: MOSE DAVIS-17 YouTube-VOS 2019 LVOS DAVIS 2017 upsampled to 1024p-4096p
Metrics: J&F J F G FPS GPU memory
Baselines: STCN AOT XMem DeAOT DEVA Cutie-small Cutie-base CFBI CFBI+ SwiftNet RDE MobileVOS
Builds on: space-time memory networks Cutie gated linear attention linear attention (Shen et al.) XMem
Closest papers by meaning
- Efficient Regional Memory Network for Video Object Segmentation (CVPR 2021 · similarity 0.89)
- RVOS: End-To-End Recurrent Network for Video Object Segmentation (CVPR 2019 · similarity 0.89)
- Learning Position and Target Consistency for Memory-Based Video Object Segmentation (CVPR 2021 · similarity 0.89)
- Recurrent Dynamic Embedding for Video Object Segmentation (CVPR 2022 · similarity 0.89)
- Long-RVOS: A Comprehensive Benchmark for Long-term Referring Video Object Segmentation (CVPR 2026 · similarity 0.88)
- RMem: Restricted Memory Banks Improve Video Object Segmentation (CVPR 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.