Research map / 3D vision / 3D generation
MP-Mat: A 3D-and-Instance-Aware Human Matting and Editing Framework with Multiplane Representation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes MP-Mat, a 3D- and instance-aware human matting framework using two multiplane representations (feature-level scene geometry planes and instance-level matte-plus-color planes with background as an instance), which also enables training-free instance-level image editing.
Problem
Multi-instance human matting struggles to disentangle mingled pixels of overlapping instances along hairy and thin boundaries, especially under occlusion; existing methods are weakly instance-aware and matting-focused methods are not evaluated for editing.
Approach
Builds two multiplane representations: a feature-level scene geometry plane representation (depth-split planes via a plane generation network) and an instance-level representation where each instance (and background as an instance) has an alpha matte and color, with rendering I = sum c_i alpha_i, which lets instance edits be done by manipulating alpha planes without retraining.
Key results
- On HIM-100K, MP-Mat reaches 26.75 SAD and 0.49 MSE versus 29.48 SAD for the next best (Maggie), a gain of at least 2.76 SAD.
- On SMPMat, MP-Mat reaches 33.25 SAD versus 37.41 for Maggie (at least 4.16 SAD improvement).
- For instance removal, MP-Mat (zero-shot) achieves 9.37% mean L1 and 23.58 dB PSNR versus 12.13% and 22.17 dB for DragonDiffusion, with 0.0003 s per editing after building the representation.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add SG-MP to the model with Inst-MP (ablation) | MP-Mat · HIM-100K | SAD: 33.78 to 26.75 | Helped |
| add Inst-MP to the model with SG-MP (ablation) | MP-Mat · HIM-100K | SAD: 29.46 to 26.75 | Helped |
| add depth input (ablation) | MP-Mat SG-MP extraction · HIM-100K | SAD: 33.78 to 31.82 | Helped |
| add PGN-based feature-level multiplane representation (ablation) | MP-Mat SG-MP extraction with depth input · HIM-100K | SAD: 31.82 to 26.75 | Helped |
| add explicit background estimation (ablation) | MP-Mat Inst-MP extraction · HIM-100K | SAD: 29.46 to 28.53 | Helped |
| add color estimation for each instance (ablation) | MP-Mat Inst-MP extraction with background estimation · HIM-100K | SAD: 28.53 to 27.79 | Helped |
| add uncertainty-guided refinement (ablation) | MP-Mat Inst-MP extraction with background and color estimation · HIM-100K | SAD: 27.79 to 26.75 | Helped |
| vary SG-MP plane number from 8 to 14 (ablation) | MP-Mat · HIM-100K | SAD: 27.49; 26.96; 26.75; 26.73; 26.47; 26.82; 27.28 | Mixed |
| vary uncertainty threshold T from 0% to 15% (ablation) | MP-Mat refinement · HIM-100K | SAD: 27.79; 27.24; 26.75; 26.82 | Helped |
| increase query number K from 15 to 40 (ablation) | MP-Mat · HIM-100K | SAD: 27.94; 27.55; 26.75; 26.36; 26.08 | Helped |
Surprising findings
- Removing the SG-MP module causes the largest drop (SAD 26.75 to 33.78), larger than removing Inst-MP.
- The method reportedly surpasses human-labeled ground truth in some qualitative rows.
Limitations
- Focuses only on human instances; generalization to other categories is limited by data availability.
- Depends on an off-the-shelf depth estimator for input depth.
Experimental setup
Datasets: HIM-100K SMPMat GQA-inpaint ORHuman P3M Composition-1k AM-2K BG-20K
Benchmarks: HIM-100K SMPMat GQA-inpaint ORHuman P3M Composition-1k AM-2K
Metrics: SAD MSE Grad Conn IMQ Mean L1 Mean L2 PSNR speed
Baselines: InstMatte E2E-HIM Maggie FBA MG Inst-inpaint DragonDiffusion
Builds on: Multiplane Image (MPI) InstMatte DETR-style bipartite matching
Closest papers by meaning
- MaGGIe: Masked Guided Gradual Human Instance Matting (CVPR 2024 · similarity 0.90)
- Unifying Automatic and Interactive Matting with Pretrained ViTs (CVPR 2024 · similarity 0.88)
- MatAnyone: Stable Video Matting with Consistent Memory Propagation (CVPR 2025 · similarity 0.88)
- In-Context Matting (CVPR 2024 · similarity 0.88)
- Training Matting Models Without Alpha Labels (AAAI 2025 · similarity 0.87)
- Revisiting Context Aggregation for Image Matting (ICML 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.