Research map / Video / Video segmentation and tracking
MaGGIe: Masked Guided Gradual Human Instance Matting
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes MaGGIe, an efficient mask-guided instance matting framework that predicts all human instance alpha mattes in one forward pass using transformer attention, sparse convolution, and feature-plus-matte temporal consistency, along with new synthesized image and video instance matting datasets.
Problem
Human instance matting from binary instance masks in images and videos is costly when scaling to many instances and frames, and prior methods lack temporal consistency of alpha mattes across frames.
Approach
Combines query-based transformer instance decoding at the coarsest scale with progressive refinement via sparse convolution (instance guidance module) so that inference cost stays roughly constant with instance count, and adds bidirectional Conv-GRU feature fusion plus a learned alpha-matte discrepancy module for output-level temporal consistency.
Key results
- On M-HIM2K composition set, MaGGIe achieves MAD 12.93 and Grad 8.91, versus retrained InstMatt MAD 16.99 and MGM? 14.31; the public InstMatt weights (trained on private external data) get the best MAD 12.85.
- On V-HIM60 hard set, MaGGIe reports MAD 21.23 and dtSSD 29.90, best among mask-guided methods, with dtSSD best on easy and hard sets.
- Memory and inference time of MaGGIe stay steady as instance count rises to 10, while InstMatt, MGM, and SparseMat grow linearly.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use MaGGIe instead of InstMatt | human instance matting · HIM2K+M-HIM2K Composition set | MAD: 12.85 (0.2) -> 12.93 (0.3) | No clear effect |
| use MaGGIe instead of InstMatt | human instance matting · HIM2K+M-HIM2K Natural set | MAD: 26.76 (2.5) -> 27.17 (3.3) | Hurt |
| use MaGGIe instead of MGM?-TCVOM | video instance matting · V-HIM60 Easy | MAD: 10.97 -> 10.12 | Helped |
| use MaGGIe instead of MGM?-TCVOM | video instance matting · V-HIM60 Medium | MAD: 13.76 -> 13.85 | No clear effect |
| use MaGGIe instead of MGM-TCVOM | video instance matting · V-HIM60 Hard | MAD: 22.16 -> 21.23 | Helped |
| replace stacked mask input with mask embedding (Ce=3) (ablation) | MaGGIe · M-HIM2K Composition set | MAD: 27.01 -> 17.75 | Helped |
| replace stacked mask input with mask embedding (Ce=3) (ablation) | MaGGIe · M-HIM2K Natural set | MAD: 39.29 -> 33.06 | Helped |
| add attention loss Latt (ablation) | MaGGIe · HIM2K+M-HIM2K Composition set | MAD: 31.77 -> 17.56 | Helped |
| add customized coarse-scale loss weights W8 alongside Latt (ablation) | MaGGIe · HIM2K+M-HIM2K Natural set | MAD: 32.95 -> 32.03 | Helped |
| add single-direction Conv-GRU temporal feature consistency (ablation) | MaGGIe · V-HIM60 Easy | MAD: 10.26 -> 10.15 | Helped |
| add backward Conv-GRU feature fusion (ablation) | MaGGIe · V-HIM60 Easy | MAD: 10.15 -> 10.14 | Helped |
| use forward matte fusion without backward propagation (ablation) | MaGGIe · V-HIM60 Easy | MAD: 10.14 -> 11.32 | Hurt |
Surprising findings
- The publicly released InstMatt weights (trained on private external data) outperform the proposed model on the composition set, which the authors attribute partly to unfair training data.
Limitations
- Requires one-hot instance assignment per pixel, which can misalign when masks come from varied sources with overlaps.
- Training on composite datasets may limit generalization to natural real-world scenes.
- Public InstMatt weights trained on external private data outperform the model on some metrics, so comparisons are not fully fair.
Experimental setup
Datasets: HHM50K HIM2K VM108 VideoMatte240K CRGNN COCO
Benchmarks: M-HIM2K V-HIM60
Metrics: MAD MSE Grad Conn MADf MADu dtSSD MESSDdt GPU memory inference time
Baselines: InstMatt MGM MGM-in-the-wild SparseMat OTVM FTP-VM TCVOM MGM-TCVOM
Builds on: MGM InstMatt OTVM AOT SAM Spconv
Closest papers by meaning
- $\alpha$Matte4K & $\mu$Matting: Dataset and Model for Ultra-Micro Precision Alpha Video Matting (CVPR 2026 · similarity 0.91)
- MP-Mat: A 3D-and-Instance-Aware Human Matting and Editing Framework with Multiplane Representation (ICLR 2025 · similarity 0.90)
- MatAnyone: Stable Video Matting with Consistent Memory Propagation (CVPR 2025 · similarity 0.88)
- Training Matting Models Without Alpha Labels (AAAI 2025 · similarity 0.88)
- Unifying Automatic and Interactive Matting with Pretrained ViTs (CVPR 2024 · similarity 0.86)
- In-Context Matting (CVPR 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.