Research map / Detection and segmentation / Open-set and few-shot detection
Unifying Automatic and Interactive Matting with Pretrained ViTs
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Smart Matting (SMat), a single pretrained-ViT model that unifies automatic and interactive image matting via a candidate-feature mechanism (class token or prompt average feature) that produces a probabilistic similarity map guiding alpha matte decoding.
Problem
Automatic matting lacks instance completeness guidance and detail focus, while interactive matting relies on prompts and over-relies on binary SAM masks; existing systems also require switching between separate models for the two settings.
Approach
Use an adaptable candidate feature—the ViT class token when no prompt is given, or the mask-average prompt feature otherwise—updated via cross-attention with image features to form a cosine similarity map as probabilistic guidance, decoded to alpha by a lightweight ViTMatte-style decoder; plus a foreground duplication training strategy.
Key results
- Automatic mode on AIM-500 reaches 34.30 SAD and 0.0129 MSE, improving on AIM (48.09 SAD) with fewer parameters (26.9M).
- Interactive mode on RefMatte-RW100 with best prompt achieves 25.60 SAD, outperforming MAM (29.23) without using SAM.
- Inference speed of 0.0246 s (auto) and 0.0302 s (interactive) per 512x512 image, faster than MAM (0.2940 s) and MatAny (0.5408 s).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use the class token rather than a random embedding as automatic saliency guidance (ablation) | SMat automatic mode · AIM-500 | SAD: 413.86 -> 34.30 | Helped |
| use the class token rather than a learnable embedding as automatic saliency guidance (ablation) | SMat automatic mode · AIM-500 | SAD: 36.62 -> 34.30 | Helped |
| replace binary similarity guidance with probabilistic similarity guidance (ablation) | SMat interactive mode · RefMatte-RW100 | SAD: 231.28 -> 44.18 | Helped |
| update the candidate feature with cross-attention (ablation) | SMat interactive mode · RefMatte-RW100 | SAD: 44.18 -> 36.37 | Helped |
| add foreground duplication (ablation) | SMat interactive mode · RefMatte-RW100 | MSE: 0.0210 -> 0.0172 | Helped |
| train with mixed automatic and interactive samples instead of automatic-only training (ablation) | SMat · AIM-500 | SAD: 35.32 -> 34.30 | Helped |
| train with mixed automatic and interactive samples instead of interactive-only training (ablation) | SMat · RefMatte-RW100 | MSE: 0.2042 -> 0.0172 | Helped |
| use SMat rather than AIM for automatic matting | automatic natural image matting · AIM-500 | SAD: 48.09 -> 34.30 | Helped |
| use SMat rather than MAM for interactive matting | interactive matting · RefMatte-RW100 | SAD: 29.23 -> 25.60 | Helped |
Surprising findings
- Learnable embedding in place of class token gives 36.62 SAD vs 34.30, while random embedding fails badly (413.86 SAD), showing class token carries saliency information.
- Training automatic and interactive modes jointly improves transparency prediction over automatic-only training (34.30 vs 35.32 SAD on AIM-500).
Limitations
- Interactive mode with box prompts is weaker on RefMatte-RW100 than point prompts (34.86 vs 25.60 SAD), and automatic mode lags category-specific specialists on their domains (e.g., AM-2k animal, P3M human).
Experimental setup
Datasets: Distinction-646 AM-2k Composition-1k DUTS RefMatte
Benchmarks: AIM-500 RefMatte-RW100 AM-2k P3M-500-NP RWP-636
Metrics: SAD MSE MAD Grad Conn
Baselines: AIM GFM HATT LF MatAny MAM PPM PPM-ViTAE
Builds on: ViTMatte DINOv2 AIM MAM SAM
Closest papers by meaning
- In-Context Matting (CVPR 2024 · similarity 0.90)
- MP-Mat: A 3D-and-Instance-Aware Human Matting and Editing Framework with Multiplane Representation (ICLR 2025 · similarity 0.88)
- Training Matting Models Without Alpha Labels (AAAI 2025 · similarity 0.87)
- Revisiting Context Aggregation for Image Matting (ICML 2024 · similarity 0.87)
- MatAnyone: Stable Video Matting with Consistent Memory Propagation (CVPR 2025 · similarity 0.87)
- Privileged Prior Information Distillation for Image Matting (AAAI 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.