Research map / Video / Video segmentation and tracking
RMem: Restricted Memory Banks Improve Video Object Segmentation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Restricts the memory bank of memory-based video object segmentation models to a fixed small number of frames, with UCB-inspired updates and learnable temporal positional embedding, achieving state-of-the-art on VOST and Long Videos.
Problem
Expanding memory banks in memory-based video object segmentation (VOS) may add redundant information that confuses the decoder, hurting accuracy on challenging long videos with object state changes.
Approach
Bound the memory bank to a small constant number of frames (frame 0 plus closest frames), update it by balancing attention-based relevance and freshness in a UCB-style rule, and use learnable temporal positional embeddings enabled by the reduced train-inference memory-length gap.
Key results
- On VOST, AOT+RMem reaches J 50.5 and Jtr 39.8 versus AOT's 49.2 and 37.0 (own implementation); DeAOT+RMem reaches J 51.8 and Jtr 40.4.
- On the Long Videos dataset, DeAOT+RMem achieves J&F 91.5 versus XMem's 89.8, a new state of the art in the paper's comparison.
- On DAVIS2017, RMem keeps accuracy roughly unchanged (J&F 85.3 vs 85.2 for DeAOT) while reducing max memory from 2.24G to 1.53G and raising FPS from 25.11 to 27.42.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add restricted memory (8 frames) and RMem components | AOTΨ · VOST | Jtr: 36.4 -> 39.8 | Helped |
| add restricted memory (8 frames) and RMem components | AOTΨ · VOST | J: 49.2 -> 50.5 | Helped |
| add restricted memory (8 frames) and RMem components | AOTΨ · Long Videos dataset | J &F: 86.7 -> 90.3 | Helped |
| restrict memory banks (ablation) | AOT · VOST | Jtr: 37.0 -> 38.6 | Helped |
| add temporal positional embedding to restricted memory (ablation) | AOT · VOST | Jtr: 38.6 -> 39.7 | Helped |
| add UCB memory update to restricted memory (ablation) | AOT · VOST | Jtr: 38.6 -> 39.4 | Helped |
| remove the 0-th frame from memory (ablation) | AOT restricted-memory system · VOST | Jtr: 38.6 -> 35.9 | Hurt |
| remove the latest frame from memory (ablation) | AOT restricted-memory system · VOST | Jtr: 38.6 -> 35.7 | Hurt |
| remove a random frame instead of the 1-st frame (ablation) | AOT restricted-memory system · VOST | Jtr: 38.6 -> 38.0 | Hurt |
| use relevance plus freshness for frame removal instead of relevance alone (ablation) | AOT restricted-memory system · VOST | Jtr: 39.1 -> 39.4 | Helped |
| use learnable temporal PE instead of SinCos PE with restricted memory (ablation) | AOT · VOST | Jtr: 37.9 -> 39.7 | Helped |
| restrict the memory bank to 3, 5, 8, 12, or 20 frames rather than unlimited memory (ablation) | AOT · VOST | J: 48.7; 49.8; 50.3; 49.6; 49.3 | Mixed |
Surprising findings
- Expanding memory banks degrades decoding of frame 0 masks in the memory deciphering study, despite later frames containing strictly more information.
- Random frame removal from the memory bank performs surprisingly well compared with heuristic selection.
Limitations
- Method is a simple heuristic adaptation; more sophisticated memory strategies are left to future work.
- Gains are reported mainly on challenging datasets; on regular DAVIS2017 accuracy is essentially unchanged.
- Restricting memory targets the capacity of current VOS decoders; improving decoding of huge memory banks is not addressed.
Experimental setup
Datasets: YoutubeVOS DAVIS VOST
Benchmarks: VOST Long Videos dataset DAVIS2017
Metrics: J Jtr J&F F max memory FPS
Baselines: AOT DeAOT XMem CFBI CFBI+ HODOR OSMN CRW STCN AFB-URR
Builds on: AOT XMem DeAOT UCB algorithm
Closest papers by meaning
- Point-VOS: Pointing Up Video Object Segmentation (CVPR 2024 · similarity 0.88)
- Recurrent Dynamic Embedding for Video Object Segmentation (CVPR 2022 · similarity 0.88)
- Test-time Training for Matching-based Video Object Segmentation (NeurIPS 2023 · similarity 0.88)
- LiVOS: Light Video Object Segmentation with Gated Linear Matching (CVPR 2025 · similarity 0.87)
- M^3-VOS: Multi-Phase, Multi-Transition, and Multi-Scenery Video Object Segmentation (CVPR 2025 · similarity 0.86)
- Putting the Object Back into Video Object Segmentation (CVPR 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.