Research map / Image generation / Text-to-image diffusion
Alchemist: Turning Public Text-to-Image Data into Generative Gold
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces Alchemist, a compact 3,350-sample general-purpose SFT dataset curated by a multi-stage image-only pipeline whose final stage scores images with cross-attention activations of a pre-trained diffusion model, and fine-tunes five SD models on it.
Problem
Public general-purpose supervised fine-tuning datasets for text-to-image models are scarce, and manual or heuristic curation is costly and fails to identify high-impact samples.
Approach
Use a pre-trained diffusion model as a quality estimator: cross-attention activation norms for a multi-keyword aesthetic prompt, with top-K indices chosen to separate manually scored high vs low quality images, select the top 3,350 images from ~300M candidates; then re-caption them with user-like prompts.
Key results
- Alchemist-tuned SD models win human side-by-side comparisons on aesthetics and complexity against baselines, with win rates up to about 0.64 and 0.78 on those criteria for SD1.5.
- Alchemist outperforms size-matched LAION-Aesthetics-tuned models on aesthetics and complexity by roughly +12% to +20% win rate across base models.
- Relevance is largely unchanged; fidelity drops slightly for SD3.5 Large (win rate 0.41 vs baseline).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Fine-tune Stable Diffusion models on the 3,350-sample Alchemist dataset | SD1.5, SD2.1, SDXL, SD3.5 Medium, and SD3.5 Large · PartiPrompts test | aesthetic quality: 0.64; 0.69; 0.61; 0.57; 0.62 win rates vs baseline | Helped |
| Fine-tune Stable Diffusion models on the 3,350-sample Alchemist dataset | SD1.5, SD2.1, SDXL, SD3.5 Medium, and SD3.5 Large · PartiPrompts test | image complexity: 0.78; 0.81; 0.78; 0.67; 0.72 win rates vs baseline | Helped |
| Fine-tune Stable Diffusion models on the 3,350-sample Alchemist dataset | Stable Diffusion models · PartiPrompts test | image-text relevance: null | No clear effect |
| Fine-tune newer architectures on Alchemist | SD3.5 Large · PartiPrompts test | fidelity: 0.41 win rate vs baseline | Hurt |
| Use a larger 7k or 19k Alchemist variant instead of 3,350 samples (ablation) | Stable Diffusion models fine-tuned on Alchemist · PartiPrompts test | human-perceived generation quality | Hurt |
| Replace diffusion-based filtering with TOPIQ/IAA and IC-9600, TAD-66k, and KonIQ-10k classifiers (ablation) | Stable Diffusion models fine-tuned on Alchemist · PartiPrompts test | image-text relevance: 0.78; 0.80; 0.84; 0.94; 0.91 Alchemist win rates vs classifier-filtered variants | Helped |
| Replace diffusion-based filtering with TOPIQ/IAA and IC-9600, TAD-66k, and KonIQ-10k classifiers (ablation) | Stable Diffusion models fine-tuned on Alchemist · PartiPrompts test | fidelity: 0.62; 0.68; 0.68; 0.79; 0.72 Alchemist win rates vs classifier-filtered variants | Helped |
| Use the full 31k LAION-Aesthetics v2 dataset instead of Alchemist (ablation) | Stable Diffusion models fine-tuned on LAION-Aesthetics v2 · PartiPrompts test | aesthetic quality: Alchemist win rates: 0.59; 0.62; 0.66; 0.65; 0.62 | Hurt |
| Use the full 31k LAION-Aesthetics v2 dataset instead of Alchemist (ablation) | Stable Diffusion models fine-tuned on LAION-Aesthetics v2 · PartiPrompts test | image complexity: Alchemist win rates: 0.77; 0.76; 0.86; 0.82; 0.72 | Hurt |
Surprising findings
- Larger datasets (7k and 19k samples) from relaxed filtering performed worse than the 3,350-sample Alchemist, indicating sample quality matters more than volume.
- Increased image complexity from Alchemist SFT tends to reduce fidelity for later, highly optimized models such as SD3.5.
Limitations
- Fidelity decreases slightly for some highly optimized models (SDXL, SD3.5) as complexity increases.
- No significant improvement in image-text relevance.
- Re-captioning uses a proprietary captioning model, and the diffusion estimator relies on manually calibrated thresholds and a hand-picked prompt.
Experimental setup
Datasets: LAION-Aesthetics v2 PartiPrompts MJHQ-30K public IQA and IAA datasets TAD-66k KonIQ-10k IC-9600
Benchmarks: MJHQ-30K PartiPrompts
Metrics: human side-by-side win rate FD-DINOv2 CLIP Score ImageReward HPS-v2
Baselines: LAION-Aesthetics v2 size-matched SFT FLUX.1-dev
Builds on: Stable Diffusion LAION-Aesthetics V2 TOPIQ Diffusers
Closest papers by meaning
- OneIG-Bench: Omni-dimensional Nuanced Evaluation for Image Generation (NeurIPS 2025 · similarity 0.88)
- Stylebreeder: Exploring and Democratizing Artistic Styles through Text-to-Image Models (NeurIPS 2024 · similarity 0.87)
- Image Regeneration: Evaluating Text-to-Image Model via Generating Identical Image with Multimodal Large Language Models (AAAI 2025 · similarity 0.87)
- DEFT: Decompositional Efficient Fine-Tuning for Text-to-Image Models (NeurIPS 2025 · similarity 0.87)
- CSGO: Content-Style Composition in Text-to-Image Generation (NeurIPS 2025 · similarity 0.87)
- FineStyle: Fine-grained Controllable Style Personalization for Text-to-image Models (NeurIPS 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.