Research map / Image generation / Autoregressive and latent generation
Graphic Design with Large Multimodal Model
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces Hierarchical Layout Generation (HLG) from unordered RGB-A design elements and Graphist, an LMM (Qwen1.5 with CLIP ViT-L/14 RGBA encoder and visual shrinker) that outputs a JSON layout with coordinates, size and layer order.
Problem
Graphic layout generation requires a predefined layer order, limiting flexibility; the paper targets generating multi-layer graphic compositions from unordered design elements, jointly predicting spatial layout and z-order.
Approach
Reframes hierarchical layout as sequence generation with a large multimodal model: RGB-A element images are encoded and compressed to 5 tokens, coordinates are written as plain digits in natural language (Shikra-style), and the model emits a JSON draft; training shuffles element order so the model learns layer ordering.
Key results
- On Crello HLG, Graphist (Qwen1.5-7B) reaches IOPR_avg 0.96 and GPT-4V rating S_DL 5.66, versus Gemini-1.5-Pro GLG-based IOPR 0.70 in the comparison table.
- On CGL-V2 content-agnostic layout, Graphist* achieves R_com 7.074 versus 10.260 for RADM, the best among the reported methods.
- Ablation: Qwen1.5-7B outperforms Qwen1.5-0.5B and InternLM2-7B on Crello in GPT-4V rating; using 5 visual tokens matches or beats 17 tokens and RGB-A beats RGB.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use Graphist* instead of Flex-DM | Crello GLG · Crello GLG | S_GI: 6.13 -> 6.70 | Helped |
| Use Graphist* for hierarchical layout generation | Crello HLG · Crello HLG | S_GI | Helped |
| Use Graphist* instead of RADM | CGL-V2 content-aware · CGL-V2 content-aware | R_ove: 0.046 -> 0.002 | Helped |
| Use Graphist* instead of RADM | CGL-V2 content-agnostic · CGL-V2 content-agnostic | R_com: 10.260 -> 7.074 | Helped |
| Use Qwen1.5-7B instead of Qwen1.5-0.5B (ablation) | Graphist · Crello | S_GI: 6.35 -> 6.90 | Helped |
| Replace Qwen1.5-7B with InternLM2-7B (ablation) | Graphist · Crello | S_DL: 5.85 -> 5.60 | Hurt |
| Replace Qwen1.5-7B with InternLM2-7B (ablation) | Graphist · CGL-V2 content-agnostic | R_com: 7.074 -> 9.510 | Hurt |
| Increase visual token length from 5 to 17 tokens (ablation) | Graphist-Tiny · Crello | S_DL: 5.44 -> 5.47 | No clear effect |
| Increase visual token length from 5 to 17 tokens (ablation) | Graphist-Tiny · Crello | S_GI: 6.35 -> 6.28 | Hurt |
| Shuffle input design element order rather than use a structured sequence (ablation) | Graphist · Crello | S_DL | Helped |
| Shuffle input design element order rather than use a structured sequence (ablation) | Graphist · Crello | S_TV | Hurt |
| Use RGB-A inputs instead of RGB inputs (ablation) | Graphist · Crello | layer ordering accuracy | Helped |
Surprising findings
- Increasing visual tokens from 5 to 17 did not improve layout performance, suggesting 5 tokens suffice.
- Unordered (shuffled) input improved S_DL and S_IO scores but reduced S_TV text readability.
Limitations
- Graphist does not generate complete sets of high-quality design materials matching design intent.
- Evaluation relies on GPT-4V judgments and a small Crello test set of 242 compositions.
- Proprietary in-house data is used for Graphist*, limiting reproducibility.
Experimental setup
Datasets: Crello CGL-Dataset V2 ShareGPT4V Flickr30k IH-RGBA (in-house) IH-Design (in-house)
Benchmarks: Crello CGL-Dataset V2
Metrics: IOPR GPT-4V rating (S_DL, S_GI, S_IO, S_TV) GPT-4V voting R_com R_occ R_ali R_ove R_und human rating
Baselines: FlexDM GPT-4V Gemini-1.5-Pro ContentGAN CGL-GAN RADM BLT LayoutDM
Builds on: Shikra Qwen1.5 CLIP FlexDM COLE
Closest papers by meaning
- Towards Aligned Layout Generation via Diffusion Model with Aesthetic Constraints (ICLR 2024 · similarity 0.87)
- AesthetiQ: Enhancing Graphic Layout Design via Aesthetic-Aware Preference Alignment of Multi-modal Large Language Models (CVPR 2025 · similarity 0.87)
- Seeing is Improving: Visual Feedback for Iterative Text Layout Refinement (CVPR 2026 · similarity 0.86)
- Decomposition of Graphic Design with Unified Multimodal Model (ICML 2025 · similarity 0.86)
- Desigen: A Pipeline for Controllable Design Template Generation (CVPR 2024 · similarity 0.86)
- Visual Layout Composer: Image-Vector Dual Diffusion Model for Design Layout Generation (CVPR 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.