Research map / Vision-language and multimodal / Multimodal LLMs
Leveraging VLM-Based Pipelines to Annotate 3D Objects
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes ScoreAgg, a visually grounded score-based aggregation of VLM responses across views and prompts (using image-text likelihoods instead of GPT-4 summarization) to annotate 3D Objaverse objects with type and material labels.
Problem
Captioning and annotating unlabeled 3D objects at scale using VLMs from multi-view renderings; existing GPT-4-based summarization (CAP3D) hallucinates by merging contradictory per-view descriptions.
Approach
Aggregate VLM responses across views and prompts by deduplicating candidate strings and combining their sampled log-likelihoods with log-sum-exp, yielding a probability distribution over responses without an LLM summarizer; also uses conditional prompting with inferred type and an unsupervised visual sensitivity metric (Hellinger distance between VLM and LLM mode outputs).
Key results
- PaLI VQA type annotations place the exact expected type in top-k for about two-thirds of Objaverse-LVIS objects, with soft accuracy 19%.
- ScoreAgg with PaLI-X material inference from type and appearance reaches top-1 accuracy 0.60 and soft accuracy 0.44, versus 0.46 and 0.36 for CAP3D captions alone in type-only LLM mode.
- CAP3D GPT-4 aggregates reach caption blow-up ratios up to 5.6, while ScoreAgg outputs always have ratio 1.0 by construction.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| provide object type along with image appearance (ablation) | PaLI-X 55B VQA material inference · Material test set | Top-1 accuracy: 0.56 ± 0.50 -> 0.61 ± 0.49 | Helped |
| provide object type along with image appearance (ablation) | BLIP-2 T5 XL material inference · Material test set | Top-3 accuracy: 0.68 ± 0.47 -> 0.69 ± 0.46 | Helped |
| remove image appearance input (ablation) | PaLI-X 55B VQA material inference · Material test set | Top-1 accuracy: 0.61 ± 0.49 -> 0.46 ± 0.50 | Hurt |
| remove type annotation input (ablation) | PaLI-X 55B VQA material inference · Material test set | Top-1 accuracy: 0.61 ± 0.49 -> 0.56 ± 0.50 | Hurt |
| remove image appearance input (ablation) | BLIP-2 T5 XL material inference · Material test set | Top-1 accuracy: 0.56 ± 0.50 -> 0.21 ± 0.41 | Hurt |
| replace log-sum-exp aggregation with max-score aggregation (ablation) | ScoreAgg type prediction · Objaverse-LVIS | type similarity score | Hurt |
| aggregate across more object views or sample more responses per probe (ablation) | ScoreAgg type prediction · Objaverse-LVIS | type similarity score | Helped |
| render without textures (ablation) | PaLI VQA type prediction · Objaverse-LVIS | type similarity score: 0.593 ± 0.290 -> 0.549 ± 0.289 | Hurt |
| use a camera backlight instead of surround area lights (ablation) | PaLI VQA type prediction · Objaverse-LVIS | type similarity score: 0.580 ± 0.291 -> 0.593 ± 0.290 | Helped |
| use dark grey rather than white image background (ablation) | PaLI VQA type prediction · Objaverse-LVIS | type similarity score: 0.593 ± 0.290 -> 0.612 ± 0.294 | Helped |
| use CAP3D camera poses with top and bottom views instead of constant polar angle (ablation) | PaLI VQA type prediction · Objaverse-LVIS | type similarity score: 0.593 ± 0.290 -> 0.588 ± 0.290 | Hurt |
| vary camera polar angle (ablation) | PaLI VQA type prediction · Objaverse-LVIS | type similarity score: 0.591 ± 0.291; 0.593 ± 0.290; 0.593 ± 0.290 | No clear effect |
Surprising findings
- Material prediction from appearance alone with BLIP-2 (0.57 top-1) outperforms its type-conditioned variants using CAP3D captions (0.47), contrary to expectation that more text context helps.
- Untextured rendering only mildly reduces PaLI type similarity (0.549 vs 0.593), suggesting shape-driven recognition.
Limitations
- Reliability depends on overlapping queries; disjoint coverage drops infrequent details.
- Log-sum-exp aggregation may need replacement with max for feature-detection goals.
- Material test labels derived from noisy Objaverse tags and one-hot, underestimating correlations.
Experimental setup
Datasets: Objaverse Objaverse-LVIS
Benchmarks: Objaverse-LVIS material test set
Metrics: text-embedding cosine similarity top-k accuracy soft accuracy caption blow-up ratio Hellinger distance
Baselines: CAP3D Objaverse tags BLIP-2 T5 XL
Builds on: CAP3D PaLI-X BLIP-2 Objaverse
Closest papers by meaning
- Unveiling the Mist over 3D Vision-Language Understanding: Object-centric Evaluation with Chain-of-Analysis (CVPR 2025 · similarity 0.89)
- Synthesize Diagnose and Optimize: Towards Fine-Grained Vision-Language Understanding (CVPR 2024 · similarity 0.89)
- DepthLM: Metric Depth from Vision Language Models (ICLR 2026 · similarity 0.89)
- Seeing Clearly, Reasoning Confidently: Plug-and-Play Remedies for Vision Language Model Blindness (CVPR 2026 · similarity 0.89)
- From Flatland to Space: Teaching Vision-Language Models to Perceive and Reason in 3D (NeurIPS 2025 · similarity 0.88)
- VLM-3R: Vision-Language Models Augmented with Instruction-Aligned 3D Reconstruction (CVPR 2026 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.