Research map / 3D vision / 3D generation
PDFactor: Learning Tri-Perspective View Policy Diffusion Field for Multi-Task Robotic Manipulation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes PDFactor, a multi-task robotic manipulation agent that encodes point features into tri-perspective view triplanes forming a latent diffusion field, decoded by a small denoising MLP that samples 6-DoF actions via score matching.
Problem
Multi-task language-conditioned robotic manipulation needs action distributions that are both accurate and efficient to sample; explicit voxel/categorical policies scale poorly with resolution, while diffusion policies with large denoising networks are slow.
Approach
Model the 6-DoF action distribution as a hybrid triplane latent field: a transformer produces three orthogonal feature planes, the latent at any 3D location is obtained by bilinear sampling and summing the three planes, and a small MLP trained with a DDPM-style score matching loss decodes action scores, so the diffusion network is small and anchored to observation space.
Key results
- PDFactor-2 reaches 87.3% average success over 18 RLBench tasks, a 5.9% absolute gain over RVT-2 (81.4%).
- PDFactor-B (83M params) reaches 84.0% average success, and the paper reports roughly tenfold faster inference than existing diffusion-based methods.
- On 6 real-world UR5 tasks with 15 demonstrations each, PDFactor-B achieves 47/60 successes versus 38/60 for RVT-2 and 31/60 for 3D Diffuser Actor.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace 2D ViT frequency positional embedding with 3D rotary position embedding (ablation) | PDFactor-B triplane transformer · RLBench 18 tasks | Avg. success rate (%): +6.9 points (reported) | Helped |
| replace learnable feature projection with point renderer (RVT-style re-rendering) (ablation) | PDFactor-B · RLBench 18 tasks | Avg. success rate (%): -9 points (reported as drop of 9%) | Hurt |
| replace score matching latent-field decoder with vanilla diffusion transformer conditioned on triplane features (ablation) | PDFactor-B · RLBench 18 tasks | Avg. success rate (%): -15.2 points (reported) | Hurt |
| increase denoising MLP depth from 3 to 1 or 5 blocks (ablation) | PDFactor-B · RLBench 18 tasks | Avg. success rate (%): 81.6 (1 layer) and 82.5 (5 layers) vs 84.0 (3 layers) | No clear effect |
| scale model from tiny to base size | PDFactor · RLBench 18 tasks | Avg. success rate (%): 77.9 (T) -> 84.0 (B) | Helped |
| use two-stage zoom-in prediction (PDFactor-2) vs single-stage PDFactor-B | PDFactor · RLBench 18 tasks | Avg. success rate (%): 84.0 -> 87.3 | Helped |
| PDFactor-B versus RVT-2 on real-world multi-task tasks | PDFactor-B · Real world 6 tasks | Avg. success (out of 60): 47/60 vs 38/60 | Helped |
| reduce number of diffusion inference steps | PDFactor · RLBench 18 tasks | Success rate and FPS | Mixed |
Surprising findings
- A single-layer denoising MLP performs competitively (81.6% vs 84.0% with 3 layers), suggesting the small decoder suffices.
- A plain diffusion transformer conditioned on triplane features without score matching loss drops 15.2% in success and runs at 0.2 FPS versus 3.8 FPS.
Limitations
- Still needs a considerable number of diffusion steps to reach higher precision.
- Best RLBench result (PDFactor-2) relies on a two-stage zoom-in strategy borrowed from RVT-2.
- Real-world evaluation is limited to 6 tasks with a single front-view camera and 10 episodes per task.
Experimental setup
Datasets: RLBench (PerAct dataset) real-world demonstrations collected on UR5
Benchmarks: RLBench (18 tasks) real-world UR5 manipulation (6 tasks)
Metrics: task success rate average rank inference speed (FPS)
Baselines: C2F-ARM-BC PerAct Act3D 3D Diffuser Actor RVT RVT-2
Builds on: DDPM PerAct RVT-2 3D Diffuser Actor CLIP
Closest papers by meaning
- DynaRend: Learning 3D Dynamics via Masked Future Rendering for Robotic Manipulation (NeurIPS 2025 · similarity 0.88)
- Prediction with Action: Visual Policy Learning via Joint Denoising Process (NeurIPS 2024 · similarity 0.88)
- Predictive Inverse Dynamics Models are Scalable Learners for Robotic Manipulation (ICLR 2025 · similarity 0.88)
- OASIS: Observation-Action Space Alignment via SE(3) Trajectory Prediction for Robotic Manipulation (NeurIPS 2026 · similarity 0.88)
- GravMAD: Grounded Spatial Value Maps Guided Action Diffusion for Generalized 3D Manipulation (ICLR 2025 · similarity 0.88)
- Localizing, Structuring, and Rendering: Bridging 3D and 2D Vision-Language-Action Models for Robotic Manipulation (CVPR 2026 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.