AI Paper Atlas

Research map / Vision-language and multimodal / Multimodal LLMs

TVDRNet: Text-driven Viewpoint Optimization via Differentiable Rendering for 3D Reasoning Segmentation

Zaiyang Yu, Changshuo Wang, Pinjie Xu, Zhang Huang, Yuan Shi, Linjun Sun, Weijun Li

ICML 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Instruction-tuned multimodal assistants

Read the paperCheck your idea against 137,217 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes TVDRNet, a text-driven viewpoint optimization method using differentiable rendering to bridge MLLMs and 3D point clouds for 3D reasoning segmentation.

Problem

3D reasoning segmentation from text instructions and 3D spatial cues; MLLMs trained mainly on text-image pairs struggle to capture implicit semantic intent when applied directly to unstructured point clouds.

Approach

Uses text-driven viewpoint optimization through differentiable rendering so that the MLLM views the 3D scene from text-relevant perspectives, instead of directly ingesting raw point clouds. Details beyond the abstract are not stated in the available text.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
add AVPL and MGL to concat baseline (ablation)TVDRNet baseline · Reason3DmIoU: 25.35 -> 43.92Improved
add MGL module to baseline (ablation)baseline · Reason3DmIoU: 25.35 -> 26.47Improved
add AVPL module to baseline (ablation)baseline · Reason3DmIoU: 25.35 -> 36.57Improved
add MGL on top of AVPL (ablation)AVPL · Reason3DmIoU: 36.57 -> 43.92Improved
add AVPL on top of MGL (ablation)MGL · Reason3DAcc@0.50: 24.12 -> 42.13Improved
use adaptive text-guided viewpoints (8) instead of best fixed viewpoints (16) (ablation)AVPL · Reason3DAcc@0.50: 39.29% -> 42.13%Improved
increase adaptive views from 8 to 16 (ablation)AVPL · Reason3DAcc@0.50: -2.36%Hurt
reduce adaptive views from 8 to 2 (ablation)AVPL · Reason3DAcc@0.50: -5.14%Hurt
use semantic-coherence grouping instead of K-means grouping (ablation)MGL · Reason3DmIoU: +1.27%Improved
increase group number G from 6 to 8 or 10 (ablation)MGL · Reason3DmIoU: 43.92% at G=6No clear effect
use LLaVA-1.5-7B with Sparse 3D U-Net rather than InternVL2.5, PointNet++ or DGCNN (ablation)TVDRNet · Reason3DmIoUImproved
TVDRNet versus OpenMaskDINO3DTVDRNet · Reason3DmIoU: 39.81 -> 43.92Improved

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy