Research map / Image generation / Text-to-image diffusion
CommonCanvas: Open Diffusion Models Trained on Creative-Commons Images
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Trains open latent diffusion text-to-image models (CommonCanvas) on ~70M Creative-Commons images captioned synthetically with BLIP-2 ("telephoning"), reaching human-preference parity with SD2 for the largest variant.
Problem
High-quality text-to-image models are trained on web-scraped LAION data with uncertain copyright status and link rot, and CC-licensed images lack captions and are scarce relative to LAION.
Approach
Synthesize captions for CC-licensed images with a pretrained BLIP-2 captioner (a "telephoning" transfer-learning step), then train SD2-style and SDXL-UNet latent diffusion models on this CC-image/synthetic-caption dataset, with systems optimizations giving 2.71x training speedup.
Key results
- CommonCanvas-L-NC shows no statistically significant human preference difference from SD2 on Parti Prompts, while the small models reach 37-38% preference rates versus SD2.
- SD2-90M (trained on ~3% of LAION-2B) matches the 1.1B-example model on FID and KID on MS COCO.
- The training optimizations give a cumulative 2.71x A100-hour speedup over the SD2 baseline.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use the larger SDXL U-Net (ablation) | CommonCanvas-L-NC · PartiPrompts | user preference versus SD2 | No clear effect |
| Use commercial-only CommonCatalog-C rather than CommonCatalog-NC (ablation) | CommonCanvas-S · PartiPrompts | user preference rate over SD2: 37% vs 38% | No clear effect |
| Train on 90M rather than 1.1B LAION examples (ablation) | SD2 · MS COCO | FID | No clear effect |
| Train on 10M rather than 1.1B LAION examples (ablation) | SD2 · MS COCO | FID | No clear effect |
| Train on 1M rather than larger LAION subsets (ablation) | SD2 · MS COCO | FID | Hurt |
| Replace LAION captions with BLIP-2 synthetic captions (ablation) | SD2 trained on 10M or 90M LAION images · MS COCO | CLIP-FID | Helped |
| Replace LAION captions with BLIP-2 synthetic captions (ablation) | SD2 trained on 10M or 90M LAION images · MS COCO | FID | No clear effect |
| Use synthetic captions instead of LAION captions (ablation) | CommonCanvas · Conceptual Captions | CLIP-FID | Hurt |
| Use synthetic captions instead of LAION captions (ablation) | CommonCanvas · MS COCO | CLIP-FID | Helped |
| Apply training optimizations, including Flash Attention, latent precomputation, low-precision normalization, FSDP, and later EMA (ablation) | SD2 training pipeline · SD2 training | training speedup: 2.71× | Helped |
| Use CommonCatalog rather than LAION training data | CommonCanvas · Human evaluation | user preference versus SD2 | Mixed |
Surprising findings
- Models trained on 10M and 90M LAION subsets perform comparably to the full 1.1B dataset on FID/KID, suggesting SD2 may be underparameterized rather than data-limited.
- Synthetic BLIP-2 captions give nearly identical FID/KID to original LAION captions and better CLIP-FID on the MS COCO benchmark.
Limitations
- Underperforms SD2 on faces, general photography, and paintings, inherited from Conceptual Captions-style domains.
- YFCC100M-sourced images are about a decade old and less current than LAION-2B.
- Synthetic BLIP-2 captions are less diverse than human/web captions.
Experimental setup
Datasets: YFCC100M CommonCatalog-C CommonCatalog-NC LAION-2B LAION-1.1B MS COCO Conceptual Captions Parti Prompts
Benchmarks: MS COCO Parti Prompts Conceptual Captions
Metrics: FID KID CLIP-FID CLIP-Score human pairwise preference rate
Baselines: Stable Diffusion 2 SD2-90M SD2 trained on LAION subsets
Builds on: Stable Diffusion BLIP-2 SDXL FSDP
Closest papers by meaning
- On improved Conditioning Mechanisms and Pre-training Strategies for Diffusion Models (NeurIPS 2024 · similarity 0.90)
- FlashEval: Towards Fast and Accurate Evaluation of Text-to-image Diffusion Generative Models (CVPR 2024 · similarity 0.89)
- Image Copy Detection for Diffusion Models (NeurIPS 2024 · similarity 0.88)
- Faster Diffusion: Rethinking the Role of the Encoder for Diffusion Model Inference (NeurIPS 2024 · similarity 0.88)
- Ambient Diffusion Omni: Training Good Models with Bad Data (NeurIPS 2025 · similarity 0.88)
- Shielded Diffusion: Generating Novel and Diverse Images using Sparse Repellency (ICML 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.