Research map / Image generation / Text-to-image diffusion
Rare-to-Frequent: Unlocking Compositional Generation Power of Diffusion Models on Rare Concepts with LLM Guidance
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes R2F, a training-free method where an LLM maps rare concepts to frequent alternatives and alternates rare and frequent prompts during early diffusion steps (with an R2F+ region-guided extension), evaluated on the new RareBench plus DVMP and T2I-CompBench.
Problem
Text-to-image diffusion models struggle to compose rare concept combinations (e.g., unusual attribute-object pairs) that are underrepresented in training data.
Approach
Use an LLM to find frequent concepts relevant to rare ones and alternately expose the diffusion sampler to rare and frequent prompts during early steps, with stop points set by LLM-predicted visual detail levels; theory shows interpolating rare and frequent score functions can reduce Wasserstein distance to the rare target.
Key results
- R2F outperforms the best baselines on RareBench by 3.1 to 28.1 points in GPT-4o evaluation and 0.6 to 19.4 points in human evaluation.
- On DVMP, R2F exceeds the best baseline by 2.7 to 5.5 points in GPT-4o evaluation.
- R2F with LLaMA3-8B-Instruct as the LLM also improves over SD3.0 on RareBench (e.g., Property 81.9 vs 49.4), though GPT-4o gives better results.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use R2F rare-to-frequent alternating guidance instead of SD3.0 alone | SD3.0 · RareBench | GPT-4o T2I alignment: up to 28.1%p | Helped |
| Use R2F rare-to-frequent guidance instead of the baseline | Diffusion models · DVMP | GPT-4o T2I alignment: from $2.7\%p$ to $5.5\%p$ | Helped |
| Use R2F rare-to-frequent guidance instead of the baseline | Diffusion models · T2I-CompBench | GPT-4o T2I alignment: from $0.1\%p$ to $3.6\%p$ | Helped |
| Use alternating guidance instead of latent interpolation, Composable Diffusion, or Prompt-to-Prompt (ablation) | R2F · RareBench | T2I alignment and image quality | Helped |
| Use adaptive visual-detail-aware guidance stop points instead of a fixed stop point (ablation) | R2F · RareBench single-object cases | T2I alignment | Helped |
| Increase fixed guidance stop point for property and texture concepts (ablation) | R2F · RareBench property and texture | T2I alignment | Helped |
| Increase fixed guidance stop point for shape concepts (ablation) | R2F · RareBench shape | T2I alignment | Hurt |
| Use R2F with LLaMA3 rather than SD3.0 without R2F | SD3.0 · RareBench single-object categories | GPT-4o T2I alignment | Helped |
| Use R2F with GPT-4o rather than R2F with LLaMA3 | R2F · RareBench single-object categories | GPT-4o T2I alignment | Helped |
| Apply R2F after GPT-4o prompt paraphrasing rather than paraphrasing alone | SD3.0 · Rare concepts | T2I alignment | Helped |
Surprising findings
- Regional LLM-grounded models LMD and RPG score below their SDXL backbone on RareBench, since they bind attributes poorly within regions.
- Even careful GPT-4o paraphrasing leaves some rare prompts (e.g., violin with ax-shaped body) nearly impossible for SD3.0 to synthesize, while R2F succeeds.
Limitations
- R2F+ fails on complex prompts where multiple objects are intertwined in a region, likely due to overlapping bounding boxes.
- Effectiveness is tied to prompt rareness; gains on T2I-CompBench are smaller and auto metrics there are sometimes inaccurate.
- Relies on LLM quality for rare concept detection and frequent concept selection, with GPT-4o outperforming LLaMA3-8B.
Experimental setup
Datasets: DVMP T2I-CompBench
Benchmarks: RareBench DVMP T2I-CompBench
Metrics: T2I alignment accuracy (GPT-4o judged) human alignment score BLIP/UniDet/CLIP/3in1 auto metrics on T2I-CompBench
Baselines: SD1.5 SDXL PixArt-alpha SD3.0 FLUX-schnell SynGen LMD RPG ELLA IterComp
Builds on: Attend-and-Excite Prompt-to-Prompt Composable Diffusion RPG LMD
Closest papers by meaning
- Generating Images of Rare Concepts Using Pre-trained Diffusion Models (AAAI 2024 · similarity 0.89)
- How Compositional Generalization and Creativity Improve as Diffusion Models are Trained (ICML 2025 · similarity 0.88)
- Unveiling Concept Attribution in Diffusion Models (NeurIPS 2025 · similarity 0.88)
- Concept Reachability in Diffusion Models: Beyond Dataset Constraints (ICML 2025 · similarity 0.88)
- SemanticDLM+: Improving Diffusion LLMs through Bias-variance Trade-off in Transition Kernel Design (NeurIPS 2026 · similarity 0.88)
- LLM4GEN: Leveraging Semantic Representation of LLMs for Text-to-Image Generation (AAAI 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.