Research map / Vision-language and multimodal / Multimodal LLMs
TVDRNet: Text-driven Viewpoint Optimization via Differentiable Rendering for 3D Reasoning Segmentation
Zaiyang Yu, Changshuo Wang, Pinjie Xu, Zhang Huang, Yuan Shi, Linjun Sun, Weijun Li
Read the paperCheck your idea against 137,217 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes TVDRNet, a text-driven viewpoint optimization method using differentiable rendering to bridge MLLMs and 3D point clouds for 3D reasoning segmentation.
Problem
3D reasoning segmentation from text instructions and 3D spatial cues; MLLMs trained mainly on text-image pairs struggle to capture implicit semantic intent when applied directly to unstructured point clouds.
Approach
Uses text-driven viewpoint optimization through differentiable rendering so that the MLLM views the 3D scene from text-relevant perspectives, instead of directly ingesting raw point clouds. Details beyond the abstract are not stated in the available text.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add AVPL and MGL to concat baseline (ablation) | TVDRNet baseline · Reason3D | mIoU: 25.35 -> 43.92 | Improved |
| add MGL module to baseline (ablation) | baseline · Reason3D | mIoU: 25.35 -> 26.47 | Improved |
| add AVPL module to baseline (ablation) | baseline · Reason3D | mIoU: 25.35 -> 36.57 | Improved |
| add MGL on top of AVPL (ablation) | AVPL · Reason3D | mIoU: 36.57 -> 43.92 | Improved |
| add AVPL on top of MGL (ablation) | MGL · Reason3D | Acc@0.50: 24.12 -> 42.13 | Improved |
| use adaptive text-guided viewpoints (8) instead of best fixed viewpoints (16) (ablation) | AVPL · Reason3D | Acc@0.50: 39.29% -> 42.13% | Improved |
| increase adaptive views from 8 to 16 (ablation) | AVPL · Reason3D | Acc@0.50: -2.36% | Hurt |
| reduce adaptive views from 8 to 2 (ablation) | AVPL · Reason3D | Acc@0.50: -5.14% | Hurt |
| use semantic-coherence grouping instead of K-means grouping (ablation) | MGL · Reason3D | mIoU: +1.27% | Improved |
| increase group number G from 6 to 8 or 10 (ablation) | MGL · Reason3D | mIoU: 43.92% at G=6 | No clear effect |
| use LLaVA-1.5-7B with Sparse 3D U-Net rather than InternVL2.5, PointNet++ or DGCNN (ablation) | TVDRNet · Reason3D | mIoU | Improved |
| TVDRNet versus OpenMaskDINO3D | TVDRNet · Reason3D | mIoU: 39.81 -> 43.92 | Improved |
Closest papers by meaning
- MLLM-For3D: Adapting Multimodal Large Language Model for 3D Reasoning Segmentation (NeurIPS 2025 · similarity 0.89)
- VLM-3R: Vision-Language Models Augmented with Instruction-Aligned 3D Reconstruction (CVPR 2026 · similarity 0.88)
- S$^2$-MLLM: Boosting Spatial Reasoning Capability of MLLMs for 3D Visual Grounding with Structural Guidance (CVPR 2026 · similarity 0.88)
- Perception Tokens Enhance Visual Reasoning in Multimodal Language Models (CVPR 2025 · similarity 0.87)
- Actial: Activate Spatial Reasoning Ability of Multimodal Large Language Models (NeurIPS 2025 · similarity 0.87)
- Pursuing Minimal Sufficiency in Spatial Reasoning (ICLR 2026 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
