Research map / Image generation / Autoregressive and latent generation
In-Context Matting
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces in-context matting, where one reference image with a user prior guides automatic alpha estimation on a batch of target images, and proposes IconMatting, a Stable Diffusion-based model using inter- and intra-similarity, plus the ICM-57 test set.
Problem
Auxiliary-input matting (trimaps, masks, scribbles) requires user input per image, while automatic matting is limited to specific categories; the paper seeks a middle ground that gives customization with automation.
Approach
Treat in-context matting as region-to-region correspondence: use Stable Diffusion features to match a reference RoI to target images (inter-similarity), then propagate the sparse matches with self-attention maps (intra-similarity) before a matting head predicts the alpha matte.
Key results
- On ICM-57, IconMatting reaches MSE 0.0081, SAD 19.12, versus SegGPT 0.0198 / 38.81 and VitMatte 0.0030 / 16.16 (trimap upper bound).
- On AIM-500, IconMatting achieves MSE 0.0062 and SAD 18.65, outperforming automatic AIM (MSE 0.0183, SAD 48.09) and MGMiW (MSE 0.0030, SAD 16.72 with per-image masks).
- Ablation shows removing intra-similarity raises ICM-57 SAD from 19.12 to 24.15, and removing both inter and intra similarity raises it to 40.32.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use IconMatting instead of automatic AIM | Image matting · ICM-57 | MSE: 0.0265 -> 0.0081 | Helped |
| Use IconMatting instead of trimap-based VitMatte | Image matting · ICM-57 | SAD: 16.16 -> 19.12 | Hurt |
| Use IconMatting instead of mask-guided MGMiW | Image matting · AIM-500 | SAD: 16.72 -> 18.65 | Hurt |
| Add intra-similarity (ablation) | IconMatting · ICM-57 | MSE: 0.0099 -> 0.0081 | Helped |
| Add inter- and intra-similarity (ablation) | IconMatting · ICM-57 | MSE: 0.0315 -> 0.0081 | Helped |
| Add multi-scale features (ablation) | IconMatting · ICM-57 | MSE: 0.0054 -> 0.0081 | Hurt |
| Add segmentation-dataset training (ablation) | IconMatting · ICM-57 | MSE: 0.0071 -> 0.0081 | No clear effect |
| Increase reference inputs from 1 to 4 (ablation) | IconMatting · ICM-57 | MSE: 0.0085 -> 0.0068 | Helped |
Surprising findings
- Performance on ICM-57 with one reference mask per group is roughly on par with trimap-based VitMatte, despite using far less user input.
- Table 3 shows MGMiW has lower MSE on AIM-500 (0.0030) than IconMatting (0.0062), so IconMatting does not win on every metric.
Limitations
- Reference-target matching is sparse, so intra-similarity propagation is required to cover the full foreground.
- Point and scribble prompts yield weaker in-context queries than masks, motivating the reference-prompt extension.
- Evaluated only on a single new test set (ICM-57) and AIM-500, with the model built on a single base diffusion backbone.
Experimental setup
Datasets: RM-1K Open Images DreamBooth instance-segmented dataset AIM-500
Benchmarks: ICM-57 AIM-500
Metrics: SAD MSE Grad Conn
Baselines: SegGPT SEEM LF AIM MGM MGMiW VitMatte MatAny MAM ICMP
Builds on: Stable Diffusion DIFT ViTMatte HIM
Closest papers by meaning
- Referring Image Matting (CVPR 2023 · similarity 0.91)
- Natural Image Matting via Guided Contextual Attention (AAAI 2020 · similarity 0.91)
- Training Matting Models Without Alpha Labels (AAAI 2025 · similarity 0.90)
- Unifying Automatic and Interactive Matting with Pretrained ViTs (CVPR 2024 · similarity 0.90)
- Revisiting Context Aggregation for Image Matting (ICML 2024 · similarity 0.90)
- MP-Mat: A 3D-and-Instance-Aware Human Matting and Editing Framework with Multiplane Representation (ICLR 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.