Research map / Image generation / Text-to-image diffusion

DynT2I-Eval: A Dynamic Evaluation Framework for Text-to-Image Models

NeurIPS 2026 · Image generation › Text-to-image diffusion · approach: T2I evaluation and controllable adapters · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes DynT2I-Eval, a fully automated dynamic prompt-generation and Bayesian pairwise ranking framework that maintains an online leaderboard for text-to-image models across alignment, perceptual quality, and aesthetics.

Problem

Fixed T2I prompt-set benchmarks overfit and become contaminated once released; dynamic evaluation needs both high-quality fresh prompts and stable ranking when prompt distributions and model sets change.

Approach

Decomposes long-form DOCCI descriptions into controllable semantic dimensions to sample fresh difficulty-tiered prompts, unifies heterogeneous evaluators into prompt-conditioned pairwise outcomes, and uses UCB-style scheduling with micro-batch confidence-weighted Bayesian updates and an uncertainty penalty warmed up for late entrants.

Key results

• Human validation on 450 pairwise cases gives 92.22% relaxed and 80.66% strict agreement with automatic evaluators, overall kappa 0.662.
• In Monte Carlo simulations the method achieves best SRCC@500 and SRCC@1500 across all six environments, with late-entry discovery within 5.7-6.1 batches and 100% discovery success.
• Top configurations selected on prompt set A transfer weakly to independent prompt sets B/C/D; for SD3.5 Medium the A-selected top-1 config drops to average private ranks of 22.0/26 (VQAScore).

Surprising findings

• SDXL-Base-1.0 ranks 12th in alignment but 2nd in aesthetic quality.
• Configurations top-ranked on the exposed prompt set often fall to low ranks on private prompt streams, e.g. SD3.5 Medium top-1 drops to rank 23/26 on B under VQAScore.

Limitations

• Evaluation bias from imperfect automatic evaluators remains; results depend on evaluator backend per dimension.
• Aesthetic judgments are subjective and strongly conditioned on the evaluator's training distribution.
• Human validation is small (450 cases) and borderline Tie cases drive most disagreement.

Experimental setup

Datasets: DOCCI

Benchmarks: DynT2I-Eval dynamic leaderboard

Metrics: SRCC posterior mean and uncertainty conservative score late-entry discovery batches transfer ratio relaxed and strict agreement Cohen kappa

Baselines: ELO TrueSkill TrueSkill2 Glicko-2 Rank Centrality K-Sort Arena Active Ranking RUCB VQAScore LMM4LMM

Builds on: Elo Glicko-2 TrueSkill K-Sort Arena DOCCI

Closest papers by meaning

• Revisiting text-to-image evaluation with Gecko: on metrics, prompts, and human rating (ICLR 2025 · similarity 0.93)
• OneIG-Bench: Omni-dimensional Nuanced Evaluation for Image Generation (NeurIPS 2025 · similarity 0.91)
• Who Evaluates the Evaluations? Objectively Scoring Text-to-Image Prompt Coherence Metrics with T2IScoreScore (TS2) (NeurIPS 2024 · similarity 0.91)
• FineGRAIN: Evaluating Failure Modes of Text-to-Image Models with Vision Language Model Judges (NeurIPS 2025 · similarity 0.90)
• Arena-T2I Hard: Benchmarking and Improving Faithfulness with Dependency-Aware Checklist Rewards (NeurIPS 2026 · similarity 0.90)
• Image Regeneration: Evaluating Text-to-Image Model via Generating Identical Image with Multimodal Large Language Models (AAAI 2025 · similarity 0.88)