Research map / 3D vision / 3D generation
MetaFind: Scene-Aware 3D Asset Retrieval for Coherent Metaverse Scene Generation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes MetaFind, a tri-modal dual-tower retriever for 3D assets that fuses text/image/point-cloud queries with an SE(3)-equivariant scene-graph encoder (ESSGNN) for layout-aware, iterative scene composition.
Problem
Retrieving 3D assets from large repositories for scene generation ignores spatial, semantic and stylistic context, and existing retrievers rely on general-purpose shape models and do not support arbitrary multimodal query combinations.
Approach
A dual-tower ULIP-2-based retriever with modality-aware fusion for arbitrary query modality subsets, plus a plug-and-play equivariant scene-graph encoder (ESSGNN, an EGNN extension with LLM-generated semantic edge embeddings) that adds a layout vector to the query, applied iteratively one object at a time.
Key results
- On Objaverse-LVIS, MetaFind w/o ESSGNN reaches 13.8 R@1 text-only and 51.7 R@1 on T+I+PC, versus 5.3 and 11.9 for the best baseline OmniBind (Full).
- MetaFind w/o ESSGNN has lower PC-only accuracy (75.1 vs ~98 for baselines), which the authors attribute to dual-tower design rather than a gain.
- With ESSGNN, GPT-4o scene coherence rises to 4.10 versus 3.01 for OpenShape and 3.26 for MetaFind without ESSGNN, on 200 sampled scenes.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add ESSGNN layout encoder | MetaFind · Scene-level composition | Scene Coherence (GPT-4o): 3.26 -> 4.10 | Helped |
| Add ESSGNN layout encoder | MetaFind · Objaverse-LVIS, Text Only | R@1 / R@5: 13.8 / 23.1 -> 11.3 / 21.5 | Hurt |
| Add ESSGNN layout encoder | MetaFind · Objaverse-LVIS, T + I + PC | R@1 / R@5: 51.7 / 76.5 -> 48.2 / 74.9 | Hurt |
| Add iterative retrieval (ablation) | MetaFind with ESSGNN · Text Only scene construction | Aesthetic (GPT-4o): 4.0 -> 4.1 | Helped |
| Add layout context (ablation) | MetaFind · Text Only scene construction | Scene Coherence (GPT-4o): 3.3 -> 4.2 | Helped |
| Use GAT instead of ESSGNN for layout context (ablation) | MetaFind · Text Only scene construction | Scene Coherence (GPT-4o): 3.7 -> 4.2 | Hurt |
| Replace selected fusion with mean fusion (ablation) | MetaFind · Text Only | R@1: 9.4 -> 11.4 | Hurt |
| Replace selected fusion with MLP fusion (ablation) | MetaFind · Text Only | R@1: 9.9 -> 11.4 | Hurt |
| Set modality dropout to 10% (ablation) | MetaFind · Text Only | R@1: 7.3 -> 11.4 | Hurt |
| Set modality dropout to 50% (ablation) | MetaFind · Text Only scene construction | Aesthetic (GPT-4o): 3.1 -> 4.1 | Hurt |
| Train only the fuser (ablation) | MetaFind query encoder · Text Only | R@1: 8.7 -> 11.4 | Hurt |
| Zero-pad missing modalities instead of masking them (ablation) | MetaFind · Text Only | R@1: 10.5 -> 11.4 | Hurt |
Surprising findings
- Adding ESSGNN lowers object-level retrieval accuracy (e.g., text-only R@1 from 13.8 to 11.3) while improving scene-level quality.
- Baselines report near-98% PC-only accuracy only because they use identical query and gallery embeddings, so the numbers are not directly comparable to the dual-tower model.
Limitations
- Annotations depend on GPT-4o and may carry language bias, hallucinations, and mislabeling; debiasing not done.
- Experiments use only single-room indoor scenes from ProcTHOR; open-world generalization is not shown.
- Iterative composition adds latency, and the single shared fusion head suffers from attribution drift that lowers object-level accuracy.
Experimental setup
Datasets: Objaverse-LVIS ProcTHOR-10K
Benchmarks: Objaverse-LVIS ProcTHOR-10K
Metrics: R@1 R@5 aesthetic score color and material score scene coherence score realism and geometry score
Baselines: ULIP OpenShape SCA3D Uni3DL Uni3D OmniBind I-Design with OpenShape retrieval
Builds on: ULIP-2 EGNN I-Design OpenShape
Closest papers by meaning
- Retrieval-Augmented Score Distillation for Text-to-3D Generation (ICML 2024 · similarity 0.85)
- MMScan: A Multi-Modal 3D Scene Dataset with Hierarchical Grounded Language Annotations (NeurIPS 2024 · similarity 0.85)
- AmaraSpatial-10K: A Spatially and Semantically Aligned 3D Dataset for Spatial Computing and Embodied AI (NeurIPS 2026 · similarity 0.85)
- 3DCoMPaT200: Language Grounded Large-Scale 3D Vision Dataset for Compositional Recognition (NeurIPS 2024 · similarity 0.85)
- ArtiScene: Language-Driven Artistic 3D Scene Generation Through Image Intermediary (CVPR 2025 · similarity 0.85)
- Disentangled 3D Scene Generation with Layout Learning (ICML 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.