Research map / Vision-language and multimodal / Multimodal LLMs

Localizing, Structuring, and Rendering: Bridging 3D and 2D Vision-Language-Action Models for Robotic Manipulation

CVPR 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Spatial reasoning in VLMs

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes DiffRender-VLA, which localizes a 3D end-effector target, colors surrounding geometry with world-aligned differentiable beams, and renders adaptive viewpoints so a 2D VLA (OpenVLA-based) can consume spatially enriched images with gradients flowing back to 3D representations.

Problem

2D VLA models lack explicit 3D spatial grounding while 3D VLA models lack intuitive image-level visual semantics; existing RGB-D fusion is view-dependent and feature-level, offering limited interpretability for robotic manipulation.

Approach

Embed 3D spatial relations into images via differentiable point-cloud rendering: a world-aligned cube marker at the predicted target, color-coded distance-gradient beams around it, and learned camera viewpoints, with end-to-end gradients from the action loss flowing through the rendered images into the 3D encoder and camera parameters.

Key results

• Achieves 80.5% average success over 12 RLBench tasks, vs. 68.4% for GWM (best 3D baseline) and 63.9% for VLA-Adapter.
• Achieves 78.3% average success on six real-world AgileX PIPER tasks, +17.5% over best baseline VLA-Adapter (60.8%).
• Ablations show removing adaptive viewpoints drops average success to 73.6%-range variants and non-differentiable beams drop it by 5.7 points.

Surprising findings

• Two-stage training (spatial module first, then VLA fine-tuning) reduces performance by 4.3 points, supporting the claim that end-to-end gradient flow matters.

Limitations

• Relies on a voxel-based 503 grid and point-cloud rendering, which may be costly and depends on RGB-D point clouds.
• Real-world evaluation is limited to six tasks on one robot platform with 20 trials each.

Experimental setup

Datasets: RLBench RH20T Open X-Embodiment

Benchmarks: RLBench AgileX PIPER real-world tasks

Metrics: success rate translation error rotation error

Baselines: RT-2 UniVLA OpenVLA-OFT VLA-Adapter SmolVLA Pi0 TraceVLA PerAct DP3 ManiGaussian Act3D 3D Diffuser

Builds on: OpenVLA PerAct Perceiver IO RVT-2

Closest papers by meaning

• OmniManip: Towards General Robotic Manipulation via Object-Centric Interaction Primitives as Spatial Constraints (CVPR 2025 · similarity 0.92)
• ConsisVLA-4D: Advancing Spatiotemporal Consistency in Efficient 3D-Perception and 4D-Reasoning for Robotic Manipulation (CVPR 2026 · similarity 0.91)
• Seeing Across Views: Benchmarking Spatial Reasoning of Vision-Language Models in Robotic Scenes (ICLR 2026 · similarity 0.90)
• CoT-VLA: Visual Chain-of-Thought Reasoning for Vision-Language-Action Models (CVPR 2025 · similarity 0.90)
• Robotic Visual Instruction (CVPR 2025 · similarity 0.90)
• BridgeVLA: Input-Output Alignment for Efficient 3D Manipulation Learning with Vision-Language Models (NeurIPS 2025 · similarity 0.90)