Research map / Vision-language and multimodal / Robot action and navigation
OmniManip: Towards General Robotic Manipulation via Object-Centric Interaction Primitives as Spatial Constraints
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes OmniManip, a zero-shot open-vocabulary manipulation system that represents VLM-derived interaction points and directions in object canonical space as spatial constraints, with a planning loop (resample-render-check) and a 6D pose-tracking execution loop, without fine-tuning the VLM.
Problem
General robotic manipulation in unstructured environments requires fine-grained 3D spatial understanding that VLMs lack, while VLA fine-tuning is costly and yields robot-specific representations; existing primitive-based methods suffer from task-agnostic proposals, manual post-processing rules, and
Approach
Describe interaction primitives (points and directions) in an object's canonical space via 6D pose estimation and 3D generation, turn VLM reasoning into analytic distance and angular spatial constraints, and close the loop twice: rendering-based self-correction of plans (RRC) and object pose tracking during execution.
Key results
- OmniManip achieves 68.3% success on 6 rigid-object tasks and 61.7% on 6 articulated-object tasks, versus 45.0% and 30.0% for the best baseline (ReKep and CoPa respectively).
- Disabling closed-loop planning reduces success by over 15% for both rigid and articulated tasks (auto 51.7% vs closed-loop 68.3% on rigid).
- Object-centric sampling reaches 80% success on Recycle Battery with 1.7 iterations versus 50% with 1.8 iterations for uniform SO(3) sampling.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use OmniManip instead of ReKep | Rigid object manipulation tasks · 6-task total | success rate: 45.0% -> 68.3% | Helped |
| Use OmniManip instead of CoPa | Articulated object manipulation tasks · 6-task total | success rate: 26.7% -> 61.7% | Helped |
| Add closed-loop planning with RRC (ablation) | OmniManip · Rigid object manipulation tasks | success rate: 51.7% -> 68.3% | Helped |
| Add closed-loop planning with RRC (ablation) | OmniManip · Articulated object manipulation tasks | success rate: 45.0% -> 61.7% | Helped |
| Sample directions along canonical principal axes rather than uniformly in SO(3) (ablation) | OmniManip · Recycle battery | success rate: 50% -> 80% | Helped |
| Sample directions along canonical principal axes rather than uniformly in SO(3) (ablation) | OmniManip · Pour tea | success rate: 30% -> 70% | Helped |
| Sample directions along canonical principal axes rather than uniformly in SO(3) (ablation) | OmniManip · Recycle battery | iterations: 1.8 -> 1.7 | Helped |
| Sample directions along canonical principal axes rather than uniformly in SO(3) (ablation) | OmniManip · Pour tea | iterations: 3.4 -> 1.8 | Helped |
| Change viewpoint across 0°, 25°, 45°, 75°, and 90° (ablation) | OmniManip · Recycle the battery | success rate: 7/10, 8/10, 8/10, 7/10, 7/10 | No clear effect |
| Change viewpoint across 0°, 25°, 45°, 75°, and 90° (ablation) | ReKep · Recycle the battery | success rate: 0/10, 1/10, 3/10, 5/10, 7/10 | Helped |
Surprising findings
- ReKep's success rate on the Recycle battery task rises from 0/10 at 0° view to 7/10 at 90° view, while OmniManip stays at 7-8/10 across viewpoints.
Limitations
- Cannot model deformable objects because of the pose-based representation.
- Effectiveness depends on the mesh quality of single-view 3D generation.
- Multiple VLM calls create computational cost, even with parallel processing.
Experimental setup
Benchmarks: 12 real-world manipulation tasks on Franka Panda
Metrics: success rate
Baselines: VoxPoser CoPa ReKep
Builds on: ReKep CoPa VoxPoser SCAFFOLD Omni6DPose
Closest papers by meaning
- Localizing, Structuring, and Rendering: Bridging 3D and 2D Vision-Language-Action Models for Robotic Manipulation (CVPR 2026 · similarity 0.92)
- ManipLLM: Embodied Multimodal Large Language Model for Object-Centric Robotic Manipulation (CVPR 2024 · similarity 0.91)
- Object-Centric Prompt-Driven Vision-Language-Action Model for Robotic Manipulation (CVPR 2025 · similarity 0.91)
- PAC Bench: Do Foundation Models Understand Prerequisites for Executing Manipulation Policies? (NeurIPS 2025 · similarity 0.91)
- Robotic Visual Instruction (CVPR 2025 · similarity 0.90)
- Seeing Across Views: Benchmarking Spatial Reasoning of Vision-Language Models in Robotic Scenes (ICLR 2026 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.