Research map / Large language models / Agents, code and math
AtelierEval: Agentic Evaluation of Humans & LLMs as Text-to-Image Prompters
Hanjun Luo, Zhimu Huang, Sylvia Chung, Yiran Wang, Yingbin Jin, Jialin Li, Jiang Li, Xinfeng Li, Hanan Salam
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add memory-augmented agentic judging (AtelierJudge) to a GPT-5.4 judge | GPT-5.4 subjective judge · Subjective verification | Spearman rho vs human experts: 0.55 -> 0.81 | Improved |
| add memory-augmented agentic judging | GPT-5.4 subjective judge · Subjective verification | MAE: 0.74 -> 0.33 | Improved |
| MLLM prompter writes prompts instead of passing the task text directly | GI-1 (T2I with MLLM middleware), CO tasks · CO objective score | objective score (%): 69.6 -> 47.1 | Hurt |
| MLLM prompter writes prompts instead of passing the task text directly | Flux Pro (no middleware), CO tasks · CO objective score | objective score (%): 32.2 -> 37.5 | Improved |
| use Skilled (structured reasoning) prompting strategy instead of Novice for MLLMs (ablation) | MLLM prompters, OE tasks · OE tasks, generated image quality | subjective image score | No clear effect |
| image-based imitation prompting versus text-only constraint planning | T0 MLLMs with skilled prompts on GI-1 · IM vs CO objective score | objective score (%): 76.1% vs human experts 70.4% | Improved |
| use T2I backend with MLLM middleware (GI-1) in OE tasks | T2I backends · OE tasks | variance among prompters | Improved |
| human skilled users prompting on GI-1 CO tasks | Human prompters · CO objective score on GI-1 | objective score (%): 62.8 -> 81.5 | Improved |
Closest papers by meaning
- Revisiting text-to-image evaluation with Gecko: on metrics, prompts, and human rating (ICLR 2025 · similarity 0.90)
- ChatEval: Towards Better LLM-based Evaluators through Multi-Agent Debate (ICLR 2024 · similarity 0.88)
- DynT2I-Eval: A Dynamic Evaluation Framework for Text-to-Image Models (NeurIPS 2026 · similarity 0.88)
- Who Evaluates the Evaluations? Objectively Scoring Text-to-Image Prompt Coherence Metrics with T2IScoreScore (TS2) (NeurIPS 2024 · similarity 0.88)
- FineGRAIN: Evaluating Failure Modes of Text-to-Image Models with Vision Language Model Judges (NeurIPS 2025 · similarity 0.88)
- Evaluating Semantic Variation in Text-to-Image Synthesis: A Causal Perspective (ICLR 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
