Research map / Image generation / Editing and style transfer
Stem-OB: Generalizable Visual Imitation Learning with Stem-Like Convergent Observation through Diffusion Inversion
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Stem-OB applies partial DDPM diffusion inversion to training observations before training Diffusion Policy, so the policy becomes robust to lighting and texture perturbations at test time without inversion or extra training.
Problem
Visual imitation learning policies are brittle to low-level visual perturbations such as lighting and texture changes, which hinders real-world deployment.
Approach
Partially invert training observations through a pretrained image diffusion model (DDPM inversion for a few steps) so that low-level appearance differences converge toward a shared high-level structure; the policy is trained on these inverted observations and tested on raw observations with no inference-time overhead.
Key results
- Real-world: Stem-OB achieves 92.6% overall success on C2P and 90.7% on ToF versus Org at 79.6% and 35.2%, with an average improvement of 22.2% over the best baseline reported.
- SAPIEN 3: Stem-OB reaches 99.1% on PushCube versus 61.8% for Org, and 25.1% on PickCube versus 8.6% for Org.
- MimicGen: Stem-OB does not win most settings; e.g., Mix-Spectrum scores 22.3% on MugCleanup versus Stem-OB's 19.5%, which the authors attribute to the texture-poor, low-resolution simulation environments.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Apply DDPM inversion preprocessing | Diffusion Policy · Real-world tasks, Gen. (C2P, D2B, OD, ToF) | success rate (%): 77.8 -> 93.3; 22.2 -> 44.4; 88.9 -> 100.0; 82.2 -> 88.9 | Helped |
| Apply DDPM inversion preprocessing | Diffusion Policy · SAPIEN 3 (PushCube, PickCube, StackCube) | success rate (%): 61.8 -> 99.1; 8.6 -> 25.1; 4.3 -> 27.2 | Helped |
| Apply DDPM inversion preprocessing | Diffusion Policy · MimicGen (MugCleanup, Threading, ThreePieceAssembly, Coffee) | success rate (%): 16.4 -> 19.5; 18.1 -> 16.4; 14.8 -> 13.1; 43.3 -> 50.5 | Mixed |
| Increase inversion steps from 0/50 to 15/50 (ablation) | Stem-OB on Diffusion Policy · SAPIEN 3 PushCube | success rate (%): 61.8 -> 99.1 | Helped |
| Vary inversion steps from 5 to 25 while holding total steps at 50 (ablation) | Stem-OB on Diffusion Policy · SAPIEN 3 PushCube, PickCube, StackCube | success rate (%): PushCube: 97.1, 91.1, 99.1, 98.5, 96.3; PickCube: 12.0, 28.0, 25.1, 12.7, 8.29; StackCube: 20.7, 27.2, 19.6, 11.6, 2.0 | Mixed |
| Vary total steps at a fixed 30% inversion ratio (ablation) | Stem-OB on Diffusion Policy · SAPIEN 3 PushCube, PickCube, StackCube | success rate (%): PushCube: 98.1, 99.1, 98.3; PickCube: 23.7, 25.1, 25.6; StackCube: 19.5, 19.6, 20.2 | No clear effect |
| Replace DDPM inversion with DDIM inversion (ablation) | Stem-OB on Diffusion Policy · SAPIEN 3 PushCube and PickCube | success rate (%): 99.1 -> 81.3; 25.1 -> 5.4 | Hurt |
| Omit the DDPM inversion error-reduction step (ablation) | Stem-OB · Visual imitation learning experiments | performance | No clear effect |
Surprising findings
- On the MimicGen benchmark, Stem-OB does not perform best in most settings, which the authors attribute to texture-free, low-resolution environments.
- DDPM inversion outperforms DDIM inversion for this purpose (PickCube 25.1% vs 5.4%).
Limitations
- Evaluated only with Diffusion Policy as the visual IL backbone.
- Underperforms baselines on MimicGen's low-resolution, texture-free environments.
- Requires choosing inversion step count per task (Table of task-specific steps).
Experimental setup
Datasets: ManiSkill 3 MimicGen
Benchmarks: SAPIEN 3 (ManiSkill 3) PushCube, PegInsertionSide, PickCube, StackCube MimicGen MugCleanup, Threading, ThreePieceAssembly, Coffee Real-world C2P, D2B, OD, ToF on Franka Panda
Metrics: success rate (%)
Baselines: Org SRM Mix-Spectrum Random Overlay
Builds on: Diffusion Policy DDPM inversion DDIM inversion Spectrum Random Masking Mix-Spectrum
Closest papers by meaning
- What's the Move? Hybrid Imitation Learning via Salient Points (ICLR 2025 · similarity 0.88)
- OSVI-WM: One-Shot Visual Imitation for Unseen Tasks using World-Model-Guided Trajectory Generation (NeurIPS 2025 · similarity 0.87)
- Prediction with Action: Visual Policy Learning via Joint Denoising Process (NeurIPS 2024 · similarity 0.86)
- Robust Visual Imitation Learning with Inverse Dynamics Representations (AAAI 2024 · similarity 0.86)
- Diffusion Renderer: Neural Inverse and Forward Rendering with Video Diffusion Models (CVPR 2025 · similarity 0.86)
- Unleashing Large-Scale Video Generative Pre-training for Visual Robot Manipulation (ICLR 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.