Research map / Image generation / Editing and style transfer
Constantly Improving Image Models Need Constantly Improving Benchmarks
Jiaxin Ge, Grace Luo, Heekyung Lee, Nishant Malpani, Long Lian, XuDong Wang, Aleksander Holynski, Trevor Darrell, Sewon Min, David M. Chan
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use ECHO social-media-derived prompts instead of older benchmark prompts | image generation benchmark · ECHO vs prior benchmarks | unique first bigrams: 2.3x more unique first bigrams | Improved |
| use ECHO prompts instead of older benchmark prompts | image generation benchmark · ECHO vs prior benchmarks | LLM perplexity: 1.2x lower in LLM perplexity | Improved |
| ensemble three VLM judges by majority vote instead of one judge | ECHO automatic evaluation · ECHO (human vs judge, 200 samples) | Kendall tau_b with humans: GPT 0.117, Gemini 0.083, Qwen 0.045 | Mixed |
| add LLM-based relevance filtering to keyword query results | ECHO post collection · ECHO collection | share of posts retained: 47% of 68k posts retained | Improved |
| add reply-tree traversal to keyword-based retrieval | ECHO post collection · ECHO collection | additional posts discovered: 19k new posts | Improved |
| use ECHO prompts for model comparison instead of a recent image editing benchmark | model separation · 4o Image Gen vs open-source unified model | win-rate gap | Improved |
Closest papers by meaning
- Social Reward: Evaluating and Enhancing Generative AI through Million-User Feedback from an Online Creative Community (ICLR 2024 · similarity 0.88)
- IDEA-Bench: How Far are Generative Models from Professional Designing? (CVPR 2025 · similarity 0.86)
- What's Producible May Not Be Reachable: Measuring the Steerability of Generative Models (NeurIPS 2025 · similarity 0.86)
- ArtWhisperer: A Dataset for Characterizing Human-AI Interactions in Artistic Creations (ICML 2024 · similarity 0.86)
- Revisiting text-to-image evaluation with Gecko: on metrics, prompts, and human rating (ICLR 2025 · similarity 0.86)
- UnlearnCanvas: Stylized Image Dataset for Enhanced Machine Unlearning Evaluation in Diffusion Models (NeurIPS 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
