Research map / Vision-language and multimodal / Human and embodied modeling
Decomposition of Graphic Design with Unified Multimodal Model
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces the Layer Decomposition of Graphic Designs (LDGD) task and DeaM, a unified multimodal model that outputs ordered RGB-A layers with metadata from a poster image, using an RGB-A VQ-GAN, conjoined encoder, and condition-aware decoder.
Problem
Decomposing composite graphic designs (posters) into ordered RGB-A layers with metadata (position, text attributes, order), including recovering occluded regions, for editing and reuse.
Approach
A unified multimodal LLM generates layer metadata and discrete VQ-GAN codebook tokens for each image layer, which an RGB-A VQ decoder conditioned on the input poster crop turns into pixel-accurate layers; it also supports text- and point-guided interactive decomposition.
Key results
- DeaM reaches FID 70.629 on Crello reconstruction versus 99.603 for the OCR/matting/inpainting baseline.
- Mean matched-box IoU of 0.7128 for DeaM versus 0.7069 for the baseline.
- Ablation: adding the condition-aware RGB-A decoder reduces FID from 95.488 to 70.629 when combined with the other components.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use DeaM instead of the baseline | Layer decomposition of graphic designs · Crello test | FID: 99.603 -> 70.629 | Helped |
| use DeaM instead of the baseline | Layer decomposition of graphic designs · Crello test | Locα: 0.7069 -> 0.7128 | Helped |
| add the conjoined visual encoder (ablation) | DeaM · Crello test | FID: 105.524 -> 101.132 | Helped |
| add enhancing prediction regularity (ablation) | DeaM with the conjoined visual encoder · Crello test | FID: 101.132 -> 95.488 | Helped |
| add the condition-aware RGB-A decoder (ablation) | DeaM with the conjoined visual encoder and enhancing prediction regularity · Crello test | FID: 95.488 -> 70.629 | Helped |
Surprising findings
- The paper reports that the RGB-A decoder gives the largest FID gain, while the conjoined encoder mainly reduces missed detections.
Limitations
- Font prediction accuracy is not satisfactory, per qualitative results.
- Tokenizer quality limits decoded image clarity; the model tends to merge decorative elements into the same layer when hierarchy is unclear.
- Evaluation is limited to the Crello dataset with FID and box IoU, and the main training dataset CreatiLD is in-house and not released.
Experimental setup
Datasets: CreatiLD Crello ImageNet LAION
Benchmarks: Crello
Metrics: FID mean IoU of matched boxes (Loc_alpha)
Baselines: Baseline (OCR + matting + inpainting pipeline) LIVE MGIE
Builds on: VQ-GAN LLaVA Shikra DINOv2 CLIP
Closest papers by meaning
- PixelART: Image-to-Layer Decomposition without Latents or Text-to-Image Pretraining (NeurIPS 2026 · similarity 0.87)
- Detect Anything in Graphic Design: Element-Level Rewards for Autoregressive Detection (NeurIPS 2026 · similarity 0.86)
- Graphic Design with Large Multimodal Model (AAAI 2025 · similarity 0.86)
- Generative Image Layer Decomposition with Visual Effects (CVPR 2025 · similarity 0.86)
- Can Vision–Language Models Assess Graphic Design Aesthetics? A Benchmark, Evaluation, and Dataset Perspective. (ICLR 2026 · similarity 0.83)
- Chat2SVG: Vector Graphics Generation with Large Language Models and Image Diffusion Models (CVPR 2025 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.