Research map / Video / Video segmentation and tracking
MatAnyone: Stable Video Matting with Consistent Memory Propagation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes MatAnyone, a memory-based target-assigned human video matting framework with region-adaptive memory fusion, a new dataset VM800, and a training scheme that uses segmentation data to supervise the matting head.
Problem
Auxiliary-free video matting degrades with ambiguous backgrounds, and mask-guided matting suffers from semantic instability because existing matting data is small, synthetic, and of limited quality.
Approach
Stores alpha mattes in a memory bank and softly merges current-frame memory readout with previous-frame values using a learned per-token change probability, so core regions stay stable while boundaries update; also supervises the matting head directly with segmentation data via region-specific losses including a scaled DDC loss.
Key results
- On the real-world benchmark, MatAnyone reaches MAD 0.14 and MSE 0.10 versus 0.95 and 0.50 for the best prior baseline (RVM-Large).
- On VideoMatte at 1920x1080, MatAnyone achieves MAD 4.24 versus 4.42 for AdaM and MaGGIe, and best dtSSD 1.19.
- Ablation on the real benchmark: adding new training scheme drops MAD from 1.85 to 0.42 on top of CMP and new data.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add VM800 training data (ablation) | MatAnyone · Real-world benchmark [30] | MAD#: 3.16 -> 2.55 | Helped |
| add consistent memory propagation (CMP) (ablation) | MatAnyone trained with New Data · Real-world benchmark [30] | dtSSD#: 1.36 -> 1.25 | Helped |
| add the segmentation-data New Training scheme (ablation) | MatAnyone trained with New Data and CMP · Real-world benchmark [30] | MAD#: 1.85 -> 0.42 | Helped |
| use scaled DDC loss instead of vanilla DDC loss (ablation) | MatAnyone · Video matting boundary regions | matting quality | Helped |
| increase first-frame warm-up repetitions (ablation) | MatAnyone · first-frame matting | matting detail quality | Helped |
Limitations
- Real-world evaluation uses core-region GT masks generated from VOS and verified manually, not true alpha mattes.
- Training data is synthetic green-screen composites, limiting generalization to real alpha labels.
Experimental setup
Datasets: VM800 VideoMatte240K COCO SPD YouTubeVIS D646 AIM VideoMatte YouTubeMatte real-world benchmark from RVM
Benchmarks: VideoMatte YouTubeMatte real-world video matting benchmark (25 videos from RVM)
Metrics: MAD MSE Grad dtSSD Conn
Baselines: MODNet RVM RVM-Large AdaM FTP-VM MaGGIe
Builds on: STCN Cutie RVM DDC loss XMem
Closest papers by meaning
- MaGGIe: Masked Guided Gradual Human Instance Matting (CVPR 2024 · similarity 0.88)
- Memory Efficient Matting with Adaptive Token Routing (AAAI 2025 · similarity 0.88)
- MP-Mat: A 3D-and-Instance-Aware Human Matting and Editing Framework with Multiplane Representation (ICLR 2025 · similarity 0.88)
- Training Matting Models Without Alpha Labels (AAAI 2025 · similarity 0.87)
- Revisiting Context Aggregation for Image Matting (ICML 2024 · similarity 0.87)
- Unifying Automatic and Interactive Matting with Pretrained ViTs (CVPR 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.