Research map / Point clouds / Point cloud classification
GPT4Point: A Unified Framework for Point-Language Understanding and Generation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces GPT4Point, a point-cloud-language MLLM (Point-QFormer + OPT/FlanT5) for 3D object understanding plus controllable text-to-3D generation via Point-E, backed by the Pyramid-XL annotation engine on Objaverse-XL.
Problem
MLLMs lack understanding of 3D objects; existing 3D MLLMs rely on 2D images or scene-level coordinates and lose geometric detail, and object-level point-text data is scarce for both understanding and generation evaluation.
Approach
Align point clouds directly with text via a BERT-based Point Q-Former (contrast, matching, captioning losses), feed the aligned tokens to an LLM for text tasks and to a frozen Point-E diffusion model as conditioning for controllable text-to-3D, and generate scalable 1M-object captions with a progressive three-level annotation pipeline (Pyramid-XL).
Key results
- Zero-shot ModelNet40 accuracy of 43.90 vs PointLLM 41.33 and InstructBLIP 31.48.
- Objaverse-XL-LVIS QA accuracy of 27.1 (OPT6.7B) and 27.6 (FlanT5-XL), surpassing InstructBLIP by about 11.7 points.
- Controllable text-to-3D reaches FID 31.6 and CLIP score 76.2, beating direct text-to-3D (34.7, 74.9) on the Cap3D 2K test set.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use GPT4Point instead of InstructBLIP for point-to-text retrieval | 3D point-text retrieval · ObjaverseXL-LVIS retrieval, Point → Text | R@1: 20.4 -> 32.2 | Helped |
| use GPT4Point instead of PointLLM | zero-shot 3D classification · ModelNet40 | Acc@1: 41.33 -> 43.90 | Helped |
| use GPT4Point (OPT6.7B) instead of Qwen-VL | 3D point-cloud captioning · ObjaverseXL-LVIS caption | CIDEr: 63.8 -> 78.7 | Helped |
| use GPT4Point (FLANT5XL) instead of PointLLM | 3D point-cloud question answering · ObjaverseXL-LVIS QA | Acc: 23.4 -> 27.6 | Helped |
| use controllable text-to-3D instead of direct text-to-3D | 3D generation · Cap3D, 2K test set | FID: 34.7 -> 31.6 | Helped |
| add Level 1 annotations to Level 2 training data (ablation) | GPT4Point (OPT2.7B) · ObjaverseXL-LVIS QA test | Acc: 22.1 -> 25.4 | Helped |
| add Level 3 annotations (30%) to Level 1 + Level 2 (ablation) | GPT4Point (OPT2.7B) · ObjaverseXL-LVIS QA test | Acc: 25.4 -> 27.1 | Helped |
| increase Level 3 annotations from 30% to 70% (ablation) | GPT4Point (OPT2.7B) · ObjaverseXL-LVIS QA test | Acc: 27.1 -> 28.2 | Helped |
| increase Level 3 annotations from 70% to 100% (ablation) | GPT4Point (OPT2.7B) · ObjaverseXL-LVIS QA test | Acc: 28.2 -> 28.4 | Helped |
Limitations
- Text-to-3D evaluation relies on rendering-based metrics and a small user study.
- Pyramid-XL level-3 data is only 70K objects, and benchmark test sets were partly manually revised.
Experimental setup
Datasets: Objaverse-XL Objaverse ModelNet40 Cap3D
Benchmarks: Objaverse-XL-LVIS retrieval ModelNet40 zero-shot classification Objaverse-XL-LVIS captioning Objaverse-XL-LVIS QA
Metrics: R@1 R@5 R@10 ACC@1 BLEU1 BLEU4 METEOR ROUGE-L CIDEr QA accuracy
Baselines: BLIP-2 InstructBLIP Qwen-VL PointLLM Point-E Cap3D
Builds on: BLIP-2 Point-BERT Point-E BLIP-Diffusion Objaverse-XL
Closest papers by meaning
- More Text, Less Point: Towards 3D Data-Efficient Point-Language Understanding (AAAI 2025 · similarity 0.90)
- 3D-PLOT-LLM: Part-Level Object Tokens for 3D Large Language Models (NeurIPS 2026 · similarity 0.88)
- GPT4Scene: Understand 3D Scenes from Videos with Vision-Language Models (ICLR 2026 · similarity 0.88)
- CG-MLLM: Captioning and Generating 3D content via Multi-modal Large Language Models (ICML 2026 · similarity 0.88)
- Part-X-MLLM: Part-aware 3D Multimodal Large Language Model (ICLR 2026 · similarity 0.88)
- Learning from Videos for 3D World: Enhancing MLLMs with 3D Vision Geometry Priors (NeurIPS 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.