Research map / Image generation / Text-to-image diffusion
DreamBench++: A Human-Aligned Benchmark for Personalized Image Generation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes DreamBench++, a benchmark for personalized image generation that uses GPT-4o with human-aligned meta-prompts (internal thinking, summary and planning) to automatically score concept preservation and prompt following, and evaluates 7 personalized T2I methods on 150 images and 1,350 prompts.
Problem
Existing evaluations of personalized image generation are either automated metrics (DINO, CLIP) that misalign with human judgment or require costly human evaluation; DreamBench is also small and low-diversity.
Approach
Use multimodal GPT-4o as an evaluator via a structured meta-prompt (task description, criteria, range, format) with self-aligned internal thinking and summary/planning steps, rating on a 0-4 integer scale, and pair it with a larger, more diverse image and prompt dataset.
Key results
- GPT-4o-based scoring reaches 83.31% mean Pearson agreement with humans on concept preservation and 98.71% on prompt following, 32.59% and 37.23% above DINO and CLIP respectively (as stated in the Discussion section).
- Across 7 personalized T2I methods, DreamBooth LoRA (SDXL) ranks top in overall CP·PF (0.517), while IP-Adapter-Plus ViT-H has the highest concept preservation (0.833).
- The human category has the lowest concept preservation average (0.482), with the human-to-animal gap of 0.204.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use GPT-4o with the designed DreamBench++ evaluation prompts instead of traditional metrics | personalized image-generation evaluation · DreamBench++ concept preservation | human-alignment consistency: +32.59% | Helped |
| Use GPT-4o with the designed DreamBench++ evaluation prompts instead of traditional metrics | personalized image-generation evaluation · DreamBench++ prompt following | human-alignment consistency: +37.23% | Helped |
| Remove internal thinking (ablation) | DreamBench++ GPT-4o evaluator · concept preservation alignment across seven methods | mean Krippendorff’s alpha: -0.040; -0.023; -0.012; +0.001; -0.045; -0.038; -0.008 | Mixed |
| Remove internal thinking (ablation) | DreamBench++ GPT-4o evaluator · prompt following alignment across seven methods | mean Krippendorff’s alpha: -0.013; +0.004; -0.032; -0.002; -0.014; +0.012; -0.002 | Mixed |
| Remove scoring criteria (ablation) | DreamBench++ GPT-4o evaluator · concept preservation alignment across seven methods | mean Krippendorff’s alpha: -0.125; -0.116; -0.093; -0.158; -0.103; -0.227; -0.166 | Hurt |
| Remove scoring criteria (ablation) | DreamBench++ GPT-4o evaluator · prompt following alignment across seven methods | mean Krippendorff’s alpha: -0.025; -0.012; -0.009; -0.012; -0.018; -0.017; -0.013 | Hurt |
| Remove scoring range (ablation) | DreamBench++ GPT-4o evaluator · concept preservation alignment across seven methods | mean Krippendorff’s alpha: -0.038; -0.017; -0.027; -0.006; -0.016; -0.009; -0.017 | Hurt |
| Remove scoring range (ablation) | DreamBench++ GPT-4o evaluator · prompt following alignment across seven methods | mean Krippendorff’s alpha: -0.010; -0.013; -0.011; +0.043; -0.038; +0.060; +0.036 | Mixed |
| Add human prior knowledge (ablation) | DreamBench++ GPT-4o evaluator · concept preservation alignment across seven methods | mean Krippendorff’s alpha: -0.033; -0.022; -0.006; -0.015; -0.022; +0.009; -0.019 | Hurt |
| Replace GPT-4o with GPT4V (ablation) | DreamBench++ evaluator · concept preservation alignment across seven methods | mean Krippendorff’s alpha: -0.105; -0.067; -0.131; -0.180; -0.016; -0.301; -0.250 | Hurt |
| Replace GPT-4o with GPT4V (ablation) | DreamBench++ evaluator · prompt following alignment across seven methods | mean Krippendorff’s alpha: -0.010; +0.012; 0.000; -0.111; -0.007; -0.161; -0.134 | Mixed |
| Add chain-of-thought prompting (ablation) | DreamBench++ GPT evaluator · concept preservation | human alignment | Hurt |
Surprising findings
- In-context learning with human-written few-shot examples consistently lowered human alignment of GPT-4o scoring.
- Chain-of-thought improved prompt-following alignment but did not help concept preservation evaluation.
Limitations
- Evaluation results rely heavily on the capabilities of multimodal LLMs and carefully designed system prompts.
- Only preliminary attempts at personalized image generation; human-aligned evaluation is limited to this domain.
- Chain-of-thought may shift GPT attention away from the subject, hurting concept preservation scoring.
Experimental setup
Datasets: DreamBench CustomConcept101 Unsplash Rawpixel Google Image Search PartiPrompts
Benchmarks: DreamBench++
Metrics: Pearson correlation Krippendorff's alpha human-MLLM consistency ratio DINO-I CLIP-I CLIP-T
Baselines: Textual Inversion DreamBooth DreamBooth LoRA BLIP-Diffusion Emu2 IP-Adapter-Plus ViT-H IP-Adapter ViT-G DINO-I CLIP-I CLIP-T
Builds on: DreamBench Self-Align chain-of-thought DINO CLIP
Closest papers by meaning
- MMIG-Bench: Towards Comprehensive and Explainable Evaluation of Multi-Modal Image Generation Models (NeurIPS 2025 · similarity 0.88)
- IDEA-Bench: How Far are Generative Models from Professional Designing? (CVPR 2025 · similarity 0.88)
- Video-Bench: Human-Aligned Video Generation Benchmark (CVPR 2025 · similarity 0.87)
- MM-R1: Unleashing the Power of Unified Multimodal Large Language Models for Personalized Image Generation (AAAI 2026 · similarity 0.87)
- GenAI Arena: An Open Evaluation Platform for Generative Models (NeurIPS 2024 · similarity 0.87)
- MultiHuman-Testbench: Benchmarking Image Generation for Multiple Humans (NeurIPS 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.