Research map / 3D vision / 3D generation
G3Flow: Generative 3D Semantic Flow for Pose-aware and Generalizable Object Manipulation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes G3Flow, which builds an object-centric 3D semantic flow from a generated digital twin, DINOv2 features and pose tracking, and feeds it into a diffusion policy for pose-aware and cross-object manipulation on RoboTwin tasks.
Problem
Geometry-only 3D diffusion policies lack semantic understanding needed for pose-aware (terminal-constrained) manipulation and generalization across object variants; prior semantic-field methods need manual keypoints, multi-view setups, and break under occlusion.
Approach
Generate a digital twin of the object from multi-view wrist-camera observations with a 3D generative model, extract DINOv2 features rendered in virtual space (PCA-reduced), and keep the semantic point cloud aligned in real time by transforming it with FoundationPose pose tracking, then condition a diffusion policy on it.
Key results
- G3Flow reaches 68.3% average success on terminal-constrained tasks vs 46.2% for the best baseline (3D DP).
- G3Flow reaches 50.1% average success on cross-object generalization tasks vs 31.7% for 3D DP.
- G3Flow runs at 34.04 Hz decision frequency, about 6x faster than GenDP (D3Fields-based).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add G3Flow semantic flow to DP | DP · terminal-constraint manipulation tasks (average) | success rate (%): 46.2 -> 68.3 | Helped |
| add G3Flow semantic flow to DP | DP · cross-object generalization tasks (average) | success rate (%): 31.7 -> 50.1 | Helped |
| replace scene-level features with G3Flow semantic flow (ablation) | DP · Shoe Place (T) | success rate (%): 67.7±1.5 -> 83.0±3.6 | Helped |
| replace scene-level features with G3Flow semantic flow (ablation) | DP · Dual Shoes Place (T) | success rate (%): 17.0±1.7 -> 24.0±3.6 | Helped |
| replace D3Fields with G3Flow semantic flow (ablation) | DP · Shoe Place (T) | success rate (%): 73.7±2.5 -> 83.0±3.6 | Helped |
| replace D3Fields with G3Flow semantic flow (ablation) | DP · Dual Shoes Place (T) | success rate (%): 20.3±6.8 -> 24.0±3.6 | Helped |
| use DINOv2 features instead of CLIP features (ablation) | G3Flow · Shoe Place (T) | success rate (%): 72.3±2.4 -> 83.0±3.6 | Helped |
| use DINOv2 features instead of SAM features (ablation) | G3Flow · Shoe Place (T) | success rate (%): 74.7±5.0 -> 83.0±3.6 | Helped |
| use G3Flow for field generation instead of scene-level DINOv2 (ablation) | DP · Field generation | frequency: 10.52 Hz -> 42.92 Hz | Helped |
| use G3Flow for decision-making instead of scene-level DINOv2 (ablation) | DP · Decision-making | frequency: 9.52 Hz -> 34.04 Hz | Helped |
| use G3Flow for field generation instead of D3Fields (ablation) | DP · Field generation | frequency: 8.14 Hz -> 42.92 Hz | Helped |
| use G3Flow for decision-making instead of D3Fields (ablation) | DP · Decision-making | frequency: 6.89 Hz -> 34.04 Hz | Helped |
Surprising findings
- Replacing DINOv2 with CLIP or SAM as the feature extractor lowers success on Shoe Place (T), indicating CLIP and SAM features are less suited to part-level pose awareness.
Limitations
- Single-object focus; articulated-object and multi-object interactions left to future work.
- Relies on quality of 3D generative digital twins, which can be wrong from single-view reconstruction (Fig. 3).
Experimental setup
Datasets: RoboTwin expert demonstrations
Benchmarks: RoboTwin Shoe Place RoboTwin Dual Shoes Place RoboTwin Tool Adjust RoboTwin Bottle Adjust RoboTwin Diverse Bottles Pick
Metrics: success rate decision frequency (Hz)
Baselines: DP DP3 DP3 w/ color GenDP (D3Fields) DP w/ Scene-Level DINOv2
Builds on: Diffusion Policy 3D Diffusion Policy D3Fields FoundationPose Grounded-SAM
Closest papers by meaning
- FlowPolicy: Enabling Fast and Robust 3D Flow-Based Policy via Consistency Flow Matching for Robot Manipulation (AAAI 2025 · similarity 0.88)
- FlowHOI: Flow-based Semantics-Grounded Generation of Hand-Object Interactions for Dexterous Robot Manipulation (NeurIPS 2026 · similarity 0.87)
- GravMAD: Grounded Spatial Value Maps Guided Action Diffusion for Generalized 3D Manipulation (ICLR 2025 · similarity 0.87)
- OmniManip: Towards General Robotic Manipulation via Object-Centric Interaction Primitives as Spatial Constraints (CVPR 2025 · similarity 0.87)
- DynaFLIP: Rethinking Robotics Perception via Tri-Modal-Dynamics Guided Representation (NeurIPS 2026 · similarity 0.86)
- Localizing, Structuring, and Rendering: Bridging 3D and 2D Vision-Language-Action Models for Robotic Manipulation (CVPR 2026 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.