Research map / Vision-language and multimodal / CLIP and zero-shot
SceneForge: Enhancing 3D-text alignment with Structured Scene Compositions
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SceneForge, which composes multiple 3D shapes into multi-object scenes with spatial relations, pairs them with LLM-refined captions, and adds them to contrastive 3D-text training to improve zero-shot, few-shot, and QA performance across several 3D encoders.
Problem
Scaling contrastive 3D-text alignment is limited by the scarcity of large-scale 3D-text data, and single-object training samples lack compositional and spatial diversity.
Approach
Construct multi-object 3D scenes by placing individual point clouds with explicit spatial relations ("over", "under", "next to"), generate matching multi-object captions by concatenating object captions and refining them with an LLM, and mix these samples into contrastive batches with a loss that keeps 2D-3D terms on single objects only.
Key results
- SF-Uni3D reaches 54.7 LVIS top-1 and 49.4 ScanNet top-1 with LVIS training, with average top-1 gain of +1.75 pp over Uni3D.
- On ScanQA with BLIP2-FlanT5, SF-Uni3D improves BLEU-4 by +2.9, CIDEr by +8.4 and EM by +4.1 over Uni3D.
- Best SF-variant one-shot ShapeNetPart mIoU 78.5 (SF-Uni3D), up from 75.9 for Uni3D.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add SceneForge multi-object composition during Uni3D pretraining | Uni3D · Objaverse-LVIS, ModelNet40, ScanObjNN, and ScanNet zero-shot (with LVIS) | Top-1 accuracy: +1.2, +0.9, +1.3, and +3.6 pp | Helped |
| Add SceneForge composition during pretraining | Uni3D · ShapeNetPart, 1-shot and 2-shot | mIoU: +2.6 / +3.0 pp | Helped |
| Add SceneForge composition during pretraining | Uni3D + BLIP2-FlanT5 · ScanQA | CIDEr: +8.4 pp | Helped |
| Add SceneForge composition during pretraining | Uni3D · ModelNet40, ScanObjectNN, and ScanNet Instances supervised fine-tuning with PointGST | accuracy: 94.83 -> 94.95; 97.68 -> 98.09; 83.04 -> 84.29 | Helped |
| Increase the maximum number of composed objects from N=1 to N=3 (ablation) | SceneForge variants · LVIS, ScanObjNN, ScanNet, and ModelNet40 zero-shot | Top-1 accuracy: LVIS +0.8-1.3 pp; ScanObjNN +0.8-1.3 pp; ScanNet +1.4-3.6 pp; ModelNet40 about +0.8 pp | Helped |
| Increase the maximum number of composed objects to N=5 (ablation) | SceneForge variants · Zero-shot classification datasets | Top-1 accuracy | Hurt |
| Increase the fraction of multi-object samples; performance peaks around alpha=0.5, then declines at larger alpha (ablation) | SceneForge training · Zero-shot datasets (average) | Top-1 accuracy | Mixed |
| Replace structured scene composition with PointCutMix-R (ablation) | Uni3D, N=2 composition comparison · LVIS, ModelNet, ScanObjNN, and ScanNet zero-shot | Top-1 accuracy: 53.5, 87.1, 64.1, 47.5 vs SF-Uni3D 53.9, 87.6, 64.5, 48.2 | Mixed |
| Replace structured scene composition with PointCutMix-K (ablation) | Uni3D, N=2 composition comparison · LVIS, ModelNet, ScanObjNN, and ScanNet zero-shot | Top-1 accuracy: 44.7, 83.0, 45.1, 34.8 vs Uni3D 53.5, 87.3, 63.9, 45.8 | Hurt |
| Replace structured scene composition with PointMixup (ablation) | Uni3D, N=2 composition comparison · LVIS, ModelNet, ScanObjNN, and ScanNet zero-shot | Top-1 accuracy: 39.2, 78.7, 41.4, 30.2 vs Uni3D 53.5, 87.3, 63.9, 45.8 | Hurt |
| Train with SceneForge rather than the single-object baseline | OpenShape, ViT-Lens, and Uni3D · ScanQA, unseen spatial relations | CIDEr: Attached To +7.0/+6.2/+8.7; Sitting On +6.6/+6.1/+9.1; Between +7.1/+6.1/+9.3; Closest To +6.8/+6.0/+8.5; In Front Of +6 | Helped |
| Train with SceneForge rather than the single-object baseline | SceneForge variants · N-LVIS cross-modal retrieval, increasing scene complexity | averaged top-1 retrieval accuracy: over 70% at N=6 and around 60% at N=7 for the N=3 model | Mixed |
Surprising findings
- Multi-object compositions improve single-object zero-shot classification despite being a distribution shift.
- Increasing the number of composed objects beyond 3 degrades accuracy on LVIS and ScanNet under a fixed 10k-point budget.
Limitations
- Only three basic spatial relations, which do not capture natural scene complexity.
- Fixed 10k-point budget degrades accuracy for densely populated scenes with many objects.
- LLM refinement adds non-negligible overhead; images are not composed due to rendering cost.
Experimental setup
Datasets: OpenShape Objaverse-LVIS ModelNet40 ScanObjectNN ScanNet ShapeNetPart ScanQA
Benchmarks: Objaverse-LVIS ModelNet40 ScanObjectNN ScanNet ShapeNetPart ScanQA N-LVIS
Metrics: top-1 accuracy top-5 accuracy mIoU BLEU-4 CIDEr exact match
Baselines: OpenShape ViT-Lens Uni3D ULIP-2 TAMM MixCon3D OmniBind PointCutMix PointMixUp
Builds on: CLIP OpenShape Uni3D ViT-Lens CutMix
Closest papers by meaning
- MMScan: A Multi-Modal 3D Scene Dataset with Hierarchical Grounded Language Annotations (NeurIPS 2024 · similarity 0.88)
- Chat-Scene: Bridging 3D Scene and Large Language Models with Object Identifiers (NeurIPS 2024 · similarity 0.88)
- Object-centric binding in Contrastive Language-Image Pretraining (NeurIPS 2025 · similarity 0.88)
- Aligning Forest and Trees in Images & Long Captions for Visually Grounded Understanding (NeurIPS 2026 · similarity 0.87)
- More Text, Less Point: Towards 3D Data-Efficient Point-Language Understanding (AAAI 2025 · similarity 0.87)
- CLIP-Driven Open-Vocabulary 3D Scene Graph Generation via Cross-Modality Contrastive Learning (CVPR 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.