Research map / Image generation / Text-to-image diffusion

Arena-T2I Hard: Benchmarking and Improving Faithfulness with Dependency-Aware Checklist Rewards

NeurIPS 2026 · Image generation › Text-to-image diffusion · approach: T2I evaluation and controllable adapters

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces Arena-T2I Hard, a 310-prompt stress benchmark of long compositional arena prompts scored with a dependency-aware checklist, and uses that checklist as a faithfulness reward combined with a PickScore aesthetic reward via GDPO to fine-tune SD3.5-Medium and FLUX.1-dev.

Problem

Existing T2I faithfulness benchmarks use simple atomic prompts that top models already saturate, and holistic Bradley-Terry preference rewards conflate aesthetics with prompt adherence, so they fail to measure or train fine-grained faithfulness on complex multi-constraint prompts.

Approach

Decompose each prompt into a DAG of yes/no questions and zero out descendants of failed parents (dependency-aware scoring), use this as a per-constraint faithfulness reward, and combine it with a BT aesthetic reward using group-decoupled normalization so each reward is standardized within its rollout group before weighted summation.

Key results

• The strongest closed-source system, gemini-3-pro-image-preview-2k, reaches 0.855 on Arena-T2I Hard with a 33 pp spread across 11 systems, while DPG-Bench and DSG scores are saturated (0.89-0.97 and 0.84-0.95).
• On SD3.5-M, Faith+Pick GDPO reaches 0.405 on Arena-T2I Hard vs 0.328 base, and wins against every single-reward and ensemble baseline in MMRB2 pairwise comparisons.
• Human A/B study on 1,899 votes: Faith+Pick wins 64.1% overall against the other Matrix-1 SD3.5-M models.

Surprising findings

• Single BT preference rewards (HPSv3, ImageReward) can drive faithfulness below the base model on SD3.5-M, and high public-arena rank does not predict faithfulness (recraft-v4 rises from #29 to 4th).
• Linear ensembling of four open-source BT rewards yields no gain over base (38.2% win rate against base on SD3.5-M).

Limitations

• Arena-T2I Hard scores depend on the choice of decomposer and judge; per-prompt decomposer agreement is only moderate (Pearson 0.67).
• Strongest open-source fine-tune (FLUX Faith+Pick at 0.529) is below the bottom of the closed-source ladder, so the RL gains do not reach frontier performance.
• Main RL results are judged by Gemini-3-flash under MMRB2 rubric rather than a fully independent metric; human validation covers only a subset.

Experimental setup

Datasets: Arena Text-to-Image prompt pool (Jan-Mar 2026) Pick-a-Pic HPDv3

Benchmarks: Arena-T2I Hard DPG-Bench DSG MMRB2 pairwise protocol on 1k arena test set

Metrics: faithfulness yes-ratio pairwise net win rate Pearson correlation Spearman correlation human A/B win rate

Baselines: PickScore HPSv3 ImageReward UnifiedReward-2.0 4-reward ensemble DreamSync RubricRL DPG-Bench DSG

Builds on: FlowGRPO GDPO DSG DPG-Bench RubricRL DreamSync

Closest papers by meaning

• Who Evaluates the Evaluations? Objectively Scoring Text-to-Image Prompt Coherence Metrics with T2IScoreScore (TS2) (NeurIPS 2024 · similarity 0.90)
• OneIG-Bench: Omni-dimensional Nuanced Evaluation for Image Generation (NeurIPS 2025 · similarity 0.90)
• DynT2I-Eval: A Dynamic Evaluation Framework for Text-to-Image Models (NeurIPS 2026 · similarity 0.90)
• Revisiting text-to-image evaluation with Gecko: on metrics, prompts, and human rating (ICLR 2025 · similarity 0.89)
• OVERT: A Benchmark for Over-Refusal Evaluation on Text-to-Image Models (NeurIPS 2025 · similarity 0.89)
• FineGRAIN: Evaluating Failure Modes of Text-to-Image Models with Vision Language Model Judges (NeurIPS 2025 · similarity 0.88)