Research map / 3D vision / 3D generation

Hallo3D: Multi-Modal Hallucination Detection and Mitigation for Consistent 3D Content Generation

NeurIPS 2024 · 3D vision › 3D generation · approach: Feed-forward 3D asset generation and editing · 1 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Hallo3D is a tuning-free 3D generation method that uses a large multi-modal model (LLaVA) to detect hallucinated geometry in rendered views, turns the findings into enhanced negative prompts for re-consistent image targets, and adds attention-based multi-view appearance alignment to SDS-based text-to-3D and image-to-3D pipelines.

Problem

Score distillation sampling with 2D diffusion priors causes view-specific overfitting and hallucinations such as the Janus problem, yielding multi-view inconsistent 3D content.

Approach

Use an LMM to detect 3D inconsistencies in renderings and convert them into enhanced negative prompts that drive DDIM-based re-generation of renderings, whose difference from the originals supervises the 3D model (L_CG), combined with focal-view attention that aligns appearance across the four simultaneously denoised views.

Key results

• Text-to-3D CLIP-Score L/14 improves from 23.70 to 30.12 for GaussianDreamer and from 18.19 to 26.72 for DreamFusion-IF with Hallo3D.
• Image-to-3D on 60 GSO/Objaverse objects: Chamfer Distance drops from 0.0370 to 0.0283 for Zero-1-to-3 and from 0.0185 to 0.0171 for DreamGaussian; LPIPS improves to 0.1726 for DreamGaussian.
• User study (58 volunteers) multi-view consistency rises from 6.00 to 8.87 for GaussianDreamer-based text-to-3D.

Measured effects reported in the paper

Surprising findings

• Hallo3D with LLaVA detects 3D inconsistencies in 2D renderings even though LMMs were not trained with geometric constraints.
• Ablation shows the enhanced negative prompt used in LSDS alone (w/o C) is worse than with the LCG re-consistency loss, and removing the prompt entirely (w/o C & P-E) is worse still.

Limitations

• Quality is bounded by the underlying baseline 3D generation models.
• Code and data are not yet released; user study is the main evaluation with no error bars.
• Potential misuse of advanced 3D generation technology is noted as a risk.

Experimental setup

Datasets: GSO Objaverse

Benchmarks: GSO Objaverse

Metrics: CLIP-Score user study rating Chamfer Distance Volume IoU PSNR SSIM LPIPS time consumption

Baselines: GaussianDreamer Score Jacobian Chain DreamFusion-IF Magic3D DreamGaussian Zero-1-to-3 Perp-Neg Debias

Builds on: DreamFusion Score Distillation Sampling Zero-1-to-3 DDIM classifier-free guidance

Closest papers by meaning

• SAVER: Mitigating Hallucinations in Large Vision-Language Models via Style-Aware Visual Early Revision (AAAI 2026 · similarity 0.90)
• Learning to Decode Against Compositional Hallucination in Video Multimodal Large Language Models (ICML 2026 · similarity 0.90)
• GHOST: Hallucination-Inducing Image Generation for Multimodal LLMs (ICLR 2026 · similarity 0.89)
• Mitigating Object Hallucination in Large Vision-Language Models via False Discovery Controlled Visual Data Splitting (NeurIPS 2026 · similarity 0.89)
• Reducing Hallucinations in Large Vision-Language Models via Latent Space Steering (ICLR 2025 · similarity 0.89)
• Causally-Grounded Dual-Path Attention Intervention for Object Hallucination Mitigation in LVLMs (AAAI 2026 · similarity 0.89)