Research map / Image generation / Autoregressive and latent generation
DRIP: Unleashing Diffusion Priors for Joint Foreground and Alpha Prediction in Image Matting
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Drip repurposes a pre-trained Stable Diffusion v2 latent diffusion model to jointly predict foreground color and alpha matte from an image and trimap, adding a switcher, cross-domain attention, and a latent transparency decoder.
Problem
Image matting requires recovering both foreground color and alpha from a single image, but two-stage methods (alpha prediction then post-processing) accumulate errors and yield poor foregrounds, and limited matting data hurts generalization to semi-transparent and rare objects.
Approach
Use the generative data priors of a pre-trained latent diffusion model to jointly denoise foreground and alpha latents in one U-Net, with a domain switcher embedding and cross-domain self-attention for mutual guidance, plus a trainable latent transparency decoder that corrects VAE reconstruction errors.
Key results
- On Composition-1k, Drip reaches alpha SAD 20.8 and MSE 3.1, improving SAD by 3.3% and MSE by 6.1% over ViTMatte.
- On Composition-1k foreground, Drip achieves SAD 34.1 and MSE 4.3 versus FBA's 38.8 and 6.0 (12.1% and 28.33% relative gains).
- On AIM-500, Drip reaches alpha SAD 17.3 and MSE 1.5, improving SAD by 26.6 over AIM.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use Drip instead of ViTMatte | Alpha prediction · Composition-1k | SAD: 21.5 -> 20.8 | Helped |
| Use Drip instead of ViTMatte | Alpha prediction · Composition-1k | MSE: 3.3 -> 3.1 | Helped |
| Use Drip instead of AIM | Alpha prediction · AIM-500 | SAD: 43.9 -> 17.3 | Helped |
| Use Drip instead of AIM | Alpha prediction · AIM-500 | MSE: 16.1 -> 1.5 | Helped |
| Use Drip instead of FBA | Foreground prediction · Composition-1k | SAD: 38.8 -> 34.1 | Helped |
| Use Drip instead of FBA | Foreground prediction · Composition-1k | MSE: 6.0 -> 4.3 | Helped |
| Add cross-domain attention (ablation) | Drip · AIM-500 | SAD: 17.8 -> 17.3 | Helped |
| Add cross-domain attention (ablation) | Drip · AIM-500 | Foreground SAD: 21.3 -> 20.6 | Helped |
| Add cross-domain attention (ablation) | Drip · AIM-500 | RGBA SAD: 26.7 -> 23.3 | Helped |
| Add the foreground and alpha switcher (ablation) | Drip · AIM-500 | SAD: 18.1 -> 17.3 | Helped |
| Add the transparent latent decoder (ablation) | Drip · AIM-500 | SAD: 21.3 -> 17.3 | Helped |
| Add the transparent latent decoder (ablation) | Drip · AIM-500 | Foreground SAD: 25.3 -> 20.6 | Helped |
Limitations
- LDM-based architecture adds complexity and may hinder real-time or resource-constrained deployment.
- Inherits biases from generative priors, which may hurt images with complex lighting, shadows, or fine textures.
- Training uses only Composition-1k and results are from fixed random seeds without error bars.
Experimental setup
Datasets: Composition-1k COCO VOC2012 LAION-5B
Benchmarks: Composition-1k AIM-500
Metrics: SAD MSE Grad Conn foreground SAD foreground MSE
Baselines: DIM IndexNet FBA HATT AIM GFM MFormer ViTMatte DiffMat Global-Matting KNN-Matting ContextAware-Matting
Builds on: Stable Diffusion ControlNet LayerDiffusion Marigold DiffMatte
Closest papers by meaning
- Learning-Based Sampling for Natural Image Matting (CVPR 2019 · similarity 0.90)
- Training Matting Models Without Alpha Labels (AAAI 2025 · similarity 0.87)
- PixelDiT2: Representation-Grounded Pixel Diffusion Transformers (NeurIPS 2026 · similarity 0.87)
- Privileged Prior Information Distillation for Image Matting (AAAI 2024 · similarity 0.87)
- Neural LightRig: Unlocking Accurate Object Normal and Material Estimation with Multi-Light Diffusion (CVPR 2025 · similarity 0.86)
- In-Context Matting (CVPR 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.