Research map / Reinforcement learning / Visual and representation RL
Self-Correcting Robot Manipulation via Gaussian-Splatted Foresight
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes a foresight-driven failure detection and self-correction module that predicts future 3D Gaussian Splatting scene representations conditioned on actions, rolling back when predicted and observed point clouds diverge, and integrates it with PerAct.
Problem
Learned language-conditioned manipulation policies fail under partial observation or imprecise actions and can enter untrained states after failures, with no mechanism to detect and recover from execution-level errors.
Approach
Represent the current scene with 3D Gaussians, train a prediction network that transforms the Gaussians conditioned on the planned action to forecast the future scene, and flag failure when the Chamfer distance between predicted Gaussian centers and the observed point cloud exceeds a threshold, triggering a rollback to the previous action with a small perturbation.
Key results
- On ten PerAct/RLBench tasks, the method reaches 56.4% average success rate versus 44.4% for ManiGaussian (the prior best), a 12.0-point gain.
- On six HiveFormer tasks, average success rate is 59.3% versus 46.0% for ManiGaussian.
- Ablation: adding self-correction raises average success from 38.0% to 56.4% on PerAct tasks.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add the self-correction scheme (ablation) | PerAct · PerAct dataset (10-task average) | success rate: 38.0 (36.0±1.8) -> 56.4 (55.6±0.7) | Helped |
| add the self-correction scheme (ablation) | PerAct · PerAct dataset, Planning | success rate: 32.0 -> 64.0 | Helped |
| add the self-correction scheme (ablation) | PerAct · PerAct dataset, Long | success rate: 20.0 -> 22.0 | Helped |
| add the self-correction scheme (ablation) | PerAct · PerAct dataset, Tools | success rate: 58.7 -> 70.7 | Helped |
| add the self-correction scheme (ablation) | PerAct · PerAct dataset, Motion | success rate: 52.0 -> 68.0 | Helped |
| add the self-correction scheme (ablation) | PerAct · PerAct dataset, Screw | success rate: 12.0 -> 44.0 | Helped |
| add the self-correction scheme (ablation) | PerAct · PerAct dataset, Occlusion | success rate: 36.0 -> 68.0 | Helped |
| vary the Chamfer distance threshold τ from 0.25 to 0.50 (ablation) | PerAct self-correcting policy · PerAct dataset (10 representative tasks) | success rate: 43.9, 51.2, 52.0, 56.4, 50.0, 53.6 | No clear effect |
Limitations
- Failure detection is limited to execution-level errors; long-horizon planning failures are not addressed.
- Performance depends on the distance threshold tau, with success dropping at some values (e.g., 43.9% at 0.25).
- Some tasks (e.g., stack wine, put in drawer) show lower success than baselines.
Experimental setup
Datasets: PerAct dataset (10 tasks, 166 variations) HiveFormer dataset (6 tasks) Colosseum benchmark
Benchmarks: RLBench (PerAct dataset, 10 tasks) HiveFormer dataset (6 tasks) Colosseum
Metrics: success rate Chamfer distance
Baselines: PerAct GNFactor ManiGaussian ACT3D
Builds on: PerAct ManiGaussian 3D Gaussian Splatting Stable Diffusion
Closest papers by meaning
- AHA: A Vision-Language-Model for Detecting and Reasoning Over Failures in Robotic Manipulation (ICLR 2025 · similarity 0.88)
- Zero-Shot Robotic Manipulation with Pre-Trained Image-Editing Diffusion Models (ICLR 2024 · similarity 0.86)
- Grasp-Then-Plan with Failure Attribution: A Closed Two-Stage Framework for Precise and Generalizable Robotic Manipulation (NeurIPS 2026 · similarity 0.86)
- Phoenix: A Motion-based Self-Reflection Framework for Fine-grained Robotic Action Correction (CVPR 2025 · similarity 0.86)
- Failing Forward: Adaptive Failure-Informed Learning for Vision-Language-Action Models (NeurIPS 2026 · similarity 0.86)
- RoboProcessBench: Benchmarking Process-Aware Understanding in Visual-Language Robotic Manipulation (NeurIPS 2026 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.