Research map / Image generation / Text-to-image diffusion
MAGICK: A Large-scale Captioned Dataset from Matting Generated Images using Chroma Keying
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Builds MAGICK, a 150,000-object dataset of generated images with alpha mattes produced by generating subjects on chroma-key backgrounds and extracting alphas with multiple matting methods, and shows an alpha-to-RGB ControlNet trained on it beats canny/sketch baselines in a user study.
Problem
Text-to-image models cannot generate objects with accurate alpha mattes (hair, fur, transparency), largely because large-scale ground-truth alpha datasets do not exist; existing matting datasets contain only hundreds to a few thousand objects.
Approach
Generate subjects on a solid, least-common-hue background using a DeepFloyd-then-SDXL-SDEdit pipeline so they are keyable, then extract alphas with three complementary matting methods and select mattes by a multi-scale SSIM agreement score (with manual selection for hard cases).
Key results
- Alpha-to-RGB ControlNet trained on MAGICK was preferred 82% of the time over SD1.5 canny-edge ControlNet in a 52-participant study.
- Against SD1.5 sketch-edge ControlNet, the MAGICK-trained model was preferred 77% of the time.
- The pipeline produced 150,000 captioned RGB images with alpha mattes; median keyability similarity score was 0.984.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Train ControlNet on MAGICK for alpha-to-RGB generation instead of using Canny-edge guidance | SD1.5 alpha-to-RGB generation · alphamatting.com training set (user study) | user preference: .16 -> .82 | Helped |
| Train ControlNet on MAGICK for alpha-to-RGB generation instead of using sketch-edge guidance | SD1.5 alpha-to-RGB generation · alphamatting.com training set (user study) | user preference: .23 -> .77 | Helped |
Surprising findings
- Off-the-shelf ControlNet v1.1 trained on segmentation masks did not follow fine alpha details, motivating the dataset.
Limitations
- Pipeline is imperfect: some images retain background color gradients or foreground tinting, and Photoshop subject selection can leave green pixels in hair.
- Automatically selected subset favors solid objects; transparent objects required manual selection (40,000 images).
- Alpha-to-RGB evaluation is limited to 27 test masks and a user study of 52 participants.
Experimental setup
Datasets: MAGICK alphamatting.com training set
Benchmarks: alphamatting.com training set (27 alpha masks) for alpha-to-RGB
Metrics: user preference rate
Baselines: ControlNet v1.1 canny edges ControlNet v1.1 sketch edges SegGen
Builds on: ControlNet SDEdit DeepFloyd SDXL Stable Diffusion 1.5
Closest papers by meaning
- Boosting Semantic Human Matting With Coarse Annotations (CVPR 2020 · similarity 0.86)
- Training Matting Models Without Alpha Labels (AAAI 2025 · similarity 0.85)
- DRIP: Unleashing Diffusion Priors for Joint Foreground and Alpha Prediction in Image Matting (NeurIPS 2024 · similarity 0.84)
- In-Context Matting (CVPR 2024 · similarity 0.84)
- Revisiting Context Aggregation for Image Matting (ICML 2024 · similarity 0.83)
- TKG-DM: Training-free Chroma Key Content Generation Diffusion Model (CVPR 2025 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.