Research map / 3D vision / 3D generation
Hi3DEval: Advancing 3D Generation Evaluation with Hierarchical Validity
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Hi3DEval, a hierarchical (object-, part-, and material-level) 3D generation evaluation framework with the Hi3DBench dataset, a multi-agent MLLM annotation pipeline (M²AP), and hybrid video-based and 3D-feature-based automated scorers.
Problem
Existing 3D generation evaluation relies on image-based 2D metrics and operates only at object level, failing to capture spatial coherence, material realism, and local part-level quality.
Approach
Combine object-, part-, and material-level absolute scoring with a multi-agent multi-modal annotation pipeline (with reflection and physical plausibility checks), and score using video representations (InternVideo2.5 with contrastive alignment) for object/material dimensions and PartField-based 3D features with attention for part-level dimensions.
Key results
- On object-level pairwise alignment (text-to-3D GP), the method reaches 0.774 vs. 0.690 for GPTEval3D and 0.556 for CLIP Score.
- On material-level pairwise alignment, the method reaches 0.767 (DC), 0.773 (CS), 0.733 (CA), and 0.745 (MP) for text-to-3D, outperforming all baselines.
- The M²AP full pipeline reaches L1 loss 0.257 against human annotations, vs. 0.702 for the best single agent (o3/o4-mini).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use the multi-agent M²AP annotation pipeline instead of a single-agent annotator | 3D asset annotation · Human alignment study | L1 loss: 0.702 -> 0.257 | Helped |
| Add a physical plausibility check (ablation) | M²AP annotation pipeline · Human alignment study | L1 loss: 0.568 -> 0.257 | Helped |
| Add the reflection mechanism (ablation) | M²AP annotation pipeline · Human alignment study | L1 loss: 0.476 -> 0.257 | Helped |
| Use DINOv2 instead of CLIP as the prompt image encoder (ablation) | Video-based scoring model · Material DC | L1 loss: 0.312 -> 0.550 | Hurt |
| Increase dropout ratio from 0.1 to 0.5 (ablation) | Video-based scoring model · Material DC | L1 loss: 0.426 -> 0.312 | Helped |
| Use SmoothL1 plus ranking loss instead of MSE or MAE (ablation) | Video-based scoring model · Material DC | L1 loss: 0.332 -> 0.312; 0.397 -> 0.312 | Helped |
| Add ranking loss to the objective (ablation) | Video-based scoring model · Material DC | pairwise accuracy: 0.758 -> 0.798 | Helped |
| Increase video frames from 4 to 16 (ablation) | Video-based scoring model · Material evaluation dimensions | L1 loss: 0.338 -> 0.312 (DC); 0.330 -> 0.291 (CS); 0.341 -> 0.288 (CA); 0.343 -> 0.287 (MP) | Helped |
| Increase video frames from 16 to 32 (ablation) | Video-based scoring model · Material evaluation dimensions | L1 loss: 0.312 -> 0.311 (DC); 0.291 -> 0.296 (CS); 0.288 -> 0.294 (CA); 0.287 -> 0.291 (MP) | Mixed |
| Add global cross-attention (ablation) | 3D-based part scoring model · Part-level test parts | L1 loss: 0.0866 -> 0.0850 | Helped |
| Add intra-part self-attention (ablation) | 3D-based part scoring model · Part-level test parts | L1 loss: 0.0935 -> 0.0850 | Helped |
| Add both attention modules (ablation) | 3D-based part scoring model · Part-level test parts | L1 loss: 0.0913 -> 0.0850 | Helped |
Surprising findings
- Using a 3-layer or 2-layer MLP prediction head in the part-level scorer performs worse than the single linear layer, with 2-layer performing worst among tested settings.
Limitations
- Focuses on static, object-centric assets; does not cover compositional scenes or dynamic content.
- Part-level annotation requires high-quality mesh segmentation and assumes consistent part semantics across instances, which may fail for deformable or abstract shapes.
Experimental setup
Datasets: OmniObjects 3DGen-Bench
Benchmarks: Hi3DBench
Metrics: pairwise rating accuracy L1 loss L2 loss
Baselines: CLIP Score ViCLIP Score Aesthetic Score ImageReward GPTEval3D T3Bench
Builds on: GPTEval3D PartField 3DGen-Bench InternVideo2.5 CLIP
Closest papers by meaning
- Eval3D: Interpretable and Fine-grained Evaluation for 3D Generation (CVPR 2025 · similarity 0.92)
- Gen3DEval: Using vLLMs for Automatic Evaluation of Generated 3D Objects (CVPR 2025 · similarity 0.89)
- DreamCraft3D: Hierarchical 3D Generation with Bootstrapped Diffusion Prior (ICLR 2024 · similarity 0.89)
- 3DTopia-XL: Scaling High-quality 3D Asset Generation via Primitive Diffusion (CVPR 2025 · similarity 0.88)
- Cycle3D: High-quality and Consistent Image-to-3D Generation via Generation-Reconstruction Cycle (AAAI 2025 · similarity 0.87)
- Make-it-Real: Unleashing Large Multimodal Model for Painting 3D Objects with Realistic Materials (NeurIPS 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.