Research map / 3D vision / 3D generation
MARVEL-40M+: Multi-Level Visual Elaboration for High-Fidelity Text-to-3D Content Creation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Builds MARVEL-40M+, a 40M-caption dataset for 8.9M 3D assets using InternVL2-40B multi-view captioning and Qwen2.5 multi-level compression with filtered human metadata, and MARVEL-FX3D, a LoRA-finetuned Stable Diffusion 3.5 + Stable Fast 3D text-to-3D pipeline.
Problem
Text-to-3D generation is limited by small, low-diversity, and shallowly annotated 3D caption datasets, and existing automated captioners rely on single-view VLMs or GPT-4 and produce inconsistent or generic captions.
Approach
A multi-stage captioning pipeline that feeds four multi-view renders plus filtered human metadata into an open-source VLM (InternVL2-40B), then compresses the dense description into five hierarchical levels with Qwen2.5-72B, and uses the resulting captions to LoRA-finetune Stable Diffusion for a fast two-stage text-to-3D pipeline.
Key results
- MARVEL Level 4 captions win 72.41% vs GPT-4 judge and 73.40% with human evaluators in image-text alignment, versus Cap3D's 14.55% and 9.50%.
- MARVEL-FX3D achieves prompt fidelity 7.71 and overall preference 6.94 in human rating, above LucidDreamer (6.59 overall) and HiFA (6.44), while generating in 15s.
- MARVEL Level 1 caption accuracy reaches 84.70% (GPT-4) and 82.80% (human), above Kabra (83.40%, 78.20%) and Cap3D (76.00%, 72.80%).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use MARVEL Level 4 captions instead of existing dataset captions | 3D caption annotations · Objaverse image-text alignment | GPT-4 win rate: 14.55 -> 72.41 | Helped |
| Use MARVEL Level 4 captions instead of existing dataset captions | 3D caption annotations · Objaverse image-text alignment | Human win rate: 14.00 -> 73.40 | Helped |
| Use MARVEL Level 1 captions instead of the strongest baseline captions | 3D caption annotations · Objaverse caption accuracy | GPT-4 accuracy: 83.40 -> 84.70 | Helped |
| Use MARVEL Level 1 captions instead of the strongest baseline captions | 3D caption annotations · Objaverse caption accuracy | Human accuracy: 78.20 -> 82.80 | Helped |
| Fine-tune Stable Diffusion 3.5 on MARVEL captions rather than use the pretrained model | MARVEL-FX3D · Objaverse test set | Overall preference: 2.41 ± 1.69 -> 6.94 ± 0.71 | Helped |
| Fine-tune Stable Diffusion 3.5 on MARVEL rather than Cap3D captions | MARVEL-FX3D · Objaverse test set | Prompt fidelity: 6.54 ± 1.68 -> 7.71 ± 0.68 | Helped |
| Fine-tune Stable Diffusion 3.5 on MARVEL captions rather than use the pretrained model | MARVEL-FX3D · Objaverse test set | Prompt fidelity: 2.58 ± 1.86 -> 7.71 ± 0.68 | Helped |
| Use MARVEL-FX3D instead of Lucid-Dreamer | Text-to-3D generation · Objaverse test set | Geometric consistency: 7.25 ± 0.60 -> 7.20 ± 0.91 | Hurt |
| Compress Level 1 annotations to Level 2 (ablation) | MARVEL annotations · Objaverse | Semantic similarity: 0.91 | No clear effect |
| Compress Level 2 annotations to Level 3 (ablation) | MARVEL annotations · Objaverse | Semantic similarity: 0.92 | No clear effect |
| Compress Level 3 annotations to Level 4 (ablation) | MARVEL annotations · Objaverse | Semantic similarity: 0.88 | No clear effect |
| Convert Level 4 descriptions to concise tags at Level 5 (ablation) | MARVEL annotations · Objaverse | Semantic similarity: 0.88 -> 0.72 | Hurt |
Surprising findings
- Level 4 captions gain a higher win rate than the much longer Level 1 captions in alignment tests, while Level 5 tags show the lowest semantic retention (0.72).
Limitations
- VLMs and LLMs struggle with numerical precision and directional understanding in complex multi-object scenes.
- InternVL2 misidentifies very thin objects, treating side views as separate entities.
- MARVEL-FX3D sometimes produces flat 3D objects due to depth ambiguity in the input image.
Experimental setup
Datasets: ShapeNet Pix3D OmniObject3D Toys4K GSO ABO Objaverse Objaverse-XL
Benchmarks: Objaverse
Metrics: MTLD unigram count bigram count win rate caption accuracy geometric consistency visual quality prompt fidelity overall preference semantic similarity
Baselines: Cap3D 3D-Topia Kabra et al. Shap-E DreamFusion HiFA LucidDreamer
Builds on: CAP3D Stable Diffusion 3.5 Stable Fast 3D DreamFusion Objaverse
Closest papers by meaning
- CG-MLLM: Captioning and Generating 3D content via Multi-modal Large Language Models (ICML 2026 · similarity 0.88)
- Instant3D: Fast Text-to-3D with Sparse-view Generation and Large Reconstruction Model (ICLR 2024 · similarity 0.88)
- MMScan: A Multi-Modal 3D Scene Dataset with Hierarchical Grounded Language Annotations (NeurIPS 2024 · similarity 0.88)
- VP3D: Unleashing 2D Visual Prompt for Text-to-3D Generation (CVPR 2024 · similarity 0.88)
- PI3D: Efficient Text-to-3D Generation with Pseudo-Image Diffusion (CVPR 2024 · similarity 0.87)
- IT3D: Improved Text-to-3D Generation with Explicit View Synthesis (AAAI 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.