Research map / Image generation / Autoregressive and latent generation
Auto Cherry-Picker: Learning from High-quality Generative Data Driven by Language
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Auto Cherry-Picker (ACP) generates language-driven synthetic detection, segmentation, and VQA training data from object combinations using LLMs and a controllable diffusion model, then filters it with a composite layout-and-image score (CLIS).
Problem
Using diffusion-generated images to improve downstream perception and multimodal tasks is hampered by annotation dependence, distribution misalignment, scaling limits, and lack of metrics that assess layout and image quality for multi-instance samples.
Approach
Generate scene graphs, captions and layouts with LLMs conditioned only on object combinations drawn from real-data priors, render them with a controllable diffusion model, and cherry-pick samples using CLIS-L (layout similarity to data-prior layouts) and CLIS-I (caption-based image quality and alignment), which correlates with downstream gains.
Key results
- ACP lifts Mask R-CNN LVIS AP mask rare from 9.3 to 15.3 (+6.0) with 100K synthetic samples.
- LLaVA-v1.5 with ACP improves MME perception by +80.1 (1434.4 to 1514.5) and GQA by +0.4.
- CLIS reduces FID for InstanceDiffusion from 53.5 to 48.9 and raises YOLO score from 45.6 to 47.9 on COCO.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Apply CLIS filtering to InstanceDiffusion generations | InstanceDiffusion · COCO val | FID: 53.5 -> 48.9 | Helped |
| Apply CLIS filtering to InstanceDiffusion generations | InstanceDiffusion · COCO val | YOLO score: 45.6 -> 47.9 | Helped |
| Add ACP-generated training data | Mask R-CNN, ResNet-50 · LVIS long-tailed instance segmentation | APmask r: 9.3 -> 14.5 | Helped |
| Add ACP-generated training data | Grounding-DINO, Swin-T · LVIS open-vocabulary object detection | APbox novel: 31.7 -> 33.0 | Helped |
| Add ACP-generated instruction data | LLaVA-1.5, Vicuna-7B · MME | score: 1434.4 -> 1514.5 | Helped |
| Use synthetic layouts instead of ground-truth layouts (ablation) | ACP with Mask R-CNN · LVIS long-tailed instance segmentation | APmask r: 11.3 -> 12.2 | Helped |
| Add LLM-based image-text alignment to CLIP-text filtering (ablation) | CLIS-I on Mask R-CNN · LVIS long-tailed instance segmentation | APmask r: 11.3 -> 11.5 | Helped |
| Add image captioning to CLIP filtering (ablation) | CLIS-I on Mask R-CNN · LVIS long-tailed instance segmentation | APmask r: 10.3 -> 11.5 | Helped |
| Replace CLIS-I with CLIP filtering (ablation) | synthetic data for Mask R-CNN · LVIS long-tailed instance segmentation | APmask r: 11.5 -> 10.3 | Hurt |
| Increase ACP synthetic training data to 100K samples | Mask R-CNN · LVIS long-tailed instance segmentation | APmask r: +6.0 | Helped |
| Select high-CLIS-I synthetic samples | Mask R-CNN and Grounding-DINO · LVIS long-tailed and open-vocabulary settings | performance | Helped |
| Select high-CLIS-L synthetic samples | LLaVA-v1.5 · MME and GQA | performance | Helped |
Surprising findings
- CLIP-based selection gives the best generation metrics but yields suboptimal downstream gains compared with CLIS-I.
- Synthetic layouts from the LLM outperform ground-truth layouts from annotations for rare-category AP (+0.9).
Limitations
- Scaling synthetic data is resource-intensive, so only a limited volume is used to ensure quality.
- Low-quality synthetic data is not exploited.
Experimental setup
Datasets: LVIS v1.0 COCO MME GQA Objects365 GoldG GRIT V3Det
Benchmarks: LVIS v1.0 COCO MME GQA
Metrics: AP mask rare AP mask AP box novel AP box FID CLIP score YOLO score MME perception score GQA accuracy
Baselines: GLIGEN BoxDiff InstanceDiffusion Stable Diffusion X-Paste MosaicFusion CLIP filtering
Builds on: InstanceDiffusion LMD MosaicFusion LLaVA-v1.5 Grounding-DINO
Closest papers by meaning
- DEEM: Diffusion models serve as the eyes of large language models for image perception (ICLR 2025 · similarity 0.88)
- Diff-2-in-1: Bridging Generation and Dense Perception with Diffusion Models (ICLR 2025 · similarity 0.87)
- What Matters When Repurposing Diffusion Models for General Dense Perception Tasks? (ICLR 2025 · similarity 0.87)
- Boosting Generative Image Modeling via Joint Image-Feature Synthesis (NeurIPS 2025 · similarity 0.87)
- Kaleido Diffusion: Improving Conditional Diffusion Models with Autoregressive Latent Modeling (NeurIPS 2024 · similarity 0.86)
- The Unmet Promise of Synthetic Training Images: Using Retrieved Real Images Performs Better (NeurIPS 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.