Research map / Image generation / Text-to-image diffusion
Ranni: Taming Text-to-Image Diffusion for Accurate Instruction Following
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Ranni introduces a semantic panel (LLM-parsed visual concepts with boxes, colors, keypoints, descriptions) as middleware between text and image, enabling a text-to-panel LLM plus a panel-conditioned diffusion model for better prompt following and panel-based or chat-based editing.
Problem
Text-to-image diffusion models struggle with complex prompts involving quantity, object-attribute binding, spatial relations and multiple subjects, and offer limited fine-grained editing controllability.
Approach
Decompose T2I into text-to-panel (an LLM, fine-tuned with LoRA, parses text into structured visual concepts with bounding boxes, main colors, keypoints) and panel-to-image (a latent diffusion model conditioned on the panel encoded as a control map), with editing done by updating the panel via six unit operations.
Key results
- On T2I-CompBench, Ranni reaches 0.6893 color, 0.6325 texture, 0.4934 shape, 0.3167 spatial, outperforming the strongest listed baseline (GORS, 0.6603 color) and its base model (0.5446 color).
- Ranni achieves 0.2720 on the quantity benchmark versus 0.2337 for the base model and 0.2086 for SDXL (b2r).
- Ranni scores 0.6400 on the multi-object benchmark versus 0.5905 for SDXL (b2r) and 0.5579 for the base model.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add semantic panel control to the base model | Ranni · T2I-CompBench Color | score: 0.5446 -> 0.6893 | Helped |
| Add semantic panel control to the base model | Ranni · T2I-CompBench Texture | score: 0.5970 -> 0.6325 | Helped |
| Add semantic panel control to the base model | Ranni · T2I-CompBench Shape | score: 0.4732 -> 0.4934 | Helped |
| Add semantic panel control to the base model | Ranni · T2I-CompBench Spatial | score: 0.1833 -> 0.3167 | Helped |
| Add semantic panel control to the base model | Ranni · Quantity | score: 0.2337 -> 0.2720 | Helped |
| Add semantic panel control to the base model | Ranni · Multi-object | score: 0.5579 -> 0.6400 | Helped |
Limitations
- Relies on the quality of external recognition models (Grounding DINO, SAM) for dataset construction
- Quantitative gains on quantity and multi-object are modest in absolute terms (e.g., 0.27 quantity score)
- Evaluation is mostly qualitative visualizations with custom quantitative benchmarks
Experimental setup
Datasets: LAION WebVision T2I-CompBench
Benchmarks: T2I-CompBench custom quantity benchmark (300 prompts) custom multi-object benchmark
Metrics: BLIP-VQA score UniDet score proportion of correct object counts
Baselines: SD v1.5 SD v2.1 Composable diffusion Structured diffusion Attn-Exct GORS SDXL LMD DALL-E 3 Midjourney InstructPix2Pix MagicBrush
Builds on: Latent Diffusion Model LoRA Grounding DINO SAM LLaVA
Closest papers by meaning
- T2I-Adapter: Learning Adapters to Dig Out More Controllable Ability for Text-to-Image Diffusion Models (AAAI 2024 · similarity 0.91)
- DreamDistribution: Learning Prompt Distribution for Diverse In-distribution Generation (ICLR 2025 · similarity 0.90)
- Mastering Text-to-Image Diffusion: Recaptioning, Planning, and Generating with Multimodal LLMs (ICML 2024 · similarity 0.89)
- Prompt-Free Diffusion: Taking "Text" out of Text-to-Image Diffusion Models (CVPR 2024 · similarity 0.89)
- Progressive Text-to-Image Diffusion with Soft Latent Direction (AAAI 2024 · similarity 0.89)
- R&B: Region and Boundary Aware Zero-shot Grounded Text-to-image Generation (ICLR 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.