Research map / Image generation / Text-to-image diffusion
GPT-4V(ision) is a Human-Aligned Evaluator for Text-to-3D Generation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes GPT-4V-based evaluator for text-to-3D models that generates prompts, compares pairs of multi-view renderings under user criteria with ensembling, and ranks models via Elo, showing better human alignment than CLIP/Aesthetic/PickScore metrics.
Problem
Existing text-to-3D metrics (CLIP similarity, aesthetic, PickScore) each target a single criterion and align poorly with human preferences, while user studies are costly and hard to scale.
Approach
Use GPT-4V as a pairwise judge of 3D assets (via concatenated RGB and surface-normal multi-view renderings with criterion-specific instructions), perturbation-based ensembling for robustness, a GPT-4V meta-prompt generator for evaluation prompts, and Elo aggregation to rank models across customizable criteria.
Key results
- Ours achieves the best Kendall's tau average of 0.710 across five criteria, versus 0.671 for the best baseline (Aesthetic-S).
- Ours attains top Kendall's tau on 4 of 5 criteria, e.g., 0.641 on 3D plausibility vs 0.484 for PickScore.
- Pairwise rating agreement with humans averages 0.307 (lower is better) vs 0.326 for Aesthetic-S.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use the GPT-4V evaluator instead of the best listed baseline | Text-to-3D model ranking · Alignment | Kendall's tau: 0.813 -> 0.821 | Helped |
| Use the GPT-4V evaluator instead of the best listed baseline | Text-to-3D model ranking · Plausibility | Kendall's tau: 0.484 -> 0.641 | Helped |
| Use the GPT-4V evaluator instead of the best listed baseline | Text-to-3D model ranking · T-G Coherency | Kendall's tau: 0.564 -> 0.564 | No clear effect |
| Use the GPT-4V evaluator instead of the best listed baseline | Text-to-3D model ranking · Texture Details | Kendall's tau: 0.813 -> 0.821 | Helped |
| Use the GPT-4V evaluator instead of the best listed baseline | Text-to-3D model ranking · Geometry Details | Kendall's tau: 0.744 -> 0.795 | Helped |
| Use the GPT-4V evaluator instead of the strongest listed baseline | Text-to-3D model ranking · Average across evaluation criteria | Kendall's tau: 0.671 -> 0.710 | Helped |
Surprising findings
- Using world-space surface normal renders improves GPT-4V comparisons, providing geometric information that aids correspondence across views.
- Existing CLIP-based metrics perform well on at most two criteria, while the GPT-4V evaluator is consistent across criteria.
Limitations
- Small-scale experiments and user studies due to limited resources
- GPT-4V hallucinations and position bias
- Metric may be gamed by adversarial patterns; quadratic number of comparisons
Experimental setup
Datasets: Pick-a-Pic
Benchmarks: Expert human annotations on text-to-3D model outputs (13 models)
Metrics: Kendall's tau pairwise rating agreement (L1 distance)
Baselines: PickScore CLIP-S CLIP-E Aesthetic-S Aesthetic-E
Builds on: Elo rating system (Nichol et al. adaptation) Chain-of-thought style explanation prompting DreamFusion
Closest papers by meaning
- DreamDPO: Aligning Text-to-3D Generation with Human Preferences via Direct Preference Optimization (ICML 2025 · similarity 0.88)
- Revisiting text-to-image evaluation with Gecko: on metrics, prompts, and human rating (ICLR 2025 · similarity 0.88)
- Gen3DEval: Using vLLMs for Automatic Evaluation of Generated 3D Objects (CVPR 2025 · similarity 0.88)
- DreamAlign: Dynamic Text-to-3D Optimization with Human Preference Alignment (AAAI 2025 · similarity 0.87)
- GenAI Arena: An Open Evaluation Platform for Generative Models (NeurIPS 2024 · similarity 0.87)
- EvalCrafter: Benchmarking and Evaluating Large Video Generation Models (CVPR 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.