Research map / 3D vision / 3D generation
Gen3DEval: Using vLLMs for Automatic Evaluation of Generated 3D Objects
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces Gen3DEval, a LLaVA-style vision LLM (8.35B, Llama3 backbone, CLIP encoder) fine-tuned on synthetic and human-preference 3D comparisons to pairwise-judge text-to-3D outputs from multi-view renderings, aggregated into ELO rankings.
Problem
Text-to-3D generation lacks scalable, human-aligned evaluation metrics; existing metrics like PSNR, Chamfer, FID need ground truth, and CLIP covers only text fidelity.
Approach
Fine-tune a vision LLM on pairwise 3D comparison VQA data (human preferences plus synthetically perturbed artist meshes) so it chooses the better of two objects on appearance, surface quality, or text fidelity from multi-view RGB and normal renderings, then rank methods via ELO without ground truth.
Key results
- Gen3DEval (CLIP) reaches 0.90 accuracy on the in-domain human appearance set, versus 0.50-0.59 for other vLLMs and 0.30-0.73 for classical metrics (CLIP, ImageReward, PickScore) in the table's appearance-human column.
- On the out-of-domain text fidelity benchmark, Gen3DEval (CLIP) achieves 0.86 vs ImageReward 0.85 and PickScore 0.85, and GPT-4o 0.55.
- Gen3DEval ranks 10 text-to-3D methods on Gen3DEval-Bench, with Trellis ranked first overall.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| fine-tune a vLLM specifically for 3D quality assessment instead of using GPT-4o | Gen3DEval · Appearance, human (in-domain) | accuracy: 0.59 -> 0.9 | Helped |
| replace CLIP with Fit3D as the image encoder (ablation) | Gen3DEval · Appearance, human (in-domain), synthetic, and OOD | accuracy: 0.9 -> 0.81; 0.85 -> 0.8; 0.89 -> 0.55 | Hurt |
| replace CLIP with DinoV2 as the image encoder (ablation) | Gen3DEval · Appearance, human (in-domain), synthetic, and OOD | accuracy: 0.9 -> 0.77; 0.85 -> 0.87; 0.89 -> 0.54 | Mixed |
| add Fit3D to the CLIP image encoder (ablation) | Gen3DEval · Appearance, human (in-domain), synthetic, and OOD | accuracy: 0.9 -> 0.9; 0.85 -> 0.88; 0.89 -> 0.78 | Mixed |
| add DinoV2 to the CLIP image encoder (ablation) | Gen3DEval · Appearance, human (in-domain), synthetic, and OOD | accuracy: 0.9 -> 0.86; 0.85 -> 0.89; 0.89 -> 0.78 | Mixed |
| replace CLIP with Fit3D as the image encoder (ablation) | Gen3DEval · Text fidelity, synthetic and OOD | accuracy: 0.98 -> 0.93; 0.86 -> 0.44 | Mixed |
| replace CLIP with DinoV2 as the image encoder (ablation) | Gen3DEval · Text fidelity, synthetic and OOD | accuracy: 0.98 -> 0.75; 0.86 -> 0.58 | Mixed |
| add Fit3D to the CLIP image encoder (ablation) | Gen3DEval · Text fidelity, synthetic and OOD | accuracy: 0.98 -> 1; 0.86 -> 0.53 | Mixed |
| add DinoV2 to the CLIP image encoder (ablation) | Gen3DEval · Text fidelity, synthetic and OOD | accuracy: 0.98 -> 0.98; 0.86 -> 0.74 | Mixed |
Surprising findings
- GPT-4o performs much worse than the fine-tuned Gen3DEval at aligning with human 3D quality judgments, despite being a much larger general model.
- Pairing Fit3D or DinoV2 encoders with CLIP did not beat CLIP alone on the out-of-domain benchmarks; CLIP was the most consistent encoder.
Limitations
- Performance on Janus artifacts is somewhat erratic.
- Out-of-domain surface evaluation is weaker due to limited annotated surface comparison data.
- Image-to-3D methods are affected by the upstream text-to-image pipeline, so the benchmark is text-only.
Experimental setup
Datasets: Artist-created 3D meshes (internal, ~140K) Human preference annotations (over 5K comparisons across 13 3D generative methods) Synthetic perturbation comparison data Gen3DEval-Bench
Benchmarks: Gen3DEval-Bench Human evaluation dataset (appearance, OOD surface, OOD text fidelity) Synthetic appearance/surface/text-fidelity datasets
Metrics: accuracy ELO rating ranking
Baselines: CLIP score ImageReward PickScore Phi-3.5-Vision LLaVA-Qwen-7B LLaVA-Llama3-8b Llama3.2-Vision-11B BLIP GPT-4o PaliGemma
Builds on: LLaVA Llama3 CLIP GPT4V-Eval
Closest papers by meaning
- Eval3D: Interpretable and Fine-grained Evaluation for 3D Generation (CVPR 2025 · similarity 0.90)
- Hi3DEval: Advancing 3D Generation Evaluation with Hierarchical Validity (NeurIPS 2025 · similarity 0.89)
- GPT-4V(ision) is a Human-Aligned Evaluator for Text-to-3D Generation (CVPR 2024 · similarity 0.88)
- Let Language Constrain Geometry: Vision–Language Models as Semantic and Spatial Critics for 3D Generation (ICML 2026 · similarity 0.88)
- MARVEL-40M+: Multi-Level Visual Elaboration for High-Fidelity Text-to-3D Content Creation (CVPR 2025 · similarity 0.87)
- Unveiling the Mist over 3D Vision-Language Understanding: Object-centric Evaluation with Chain-of-Analysis (CVPR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.