Research map / Reinforcement learning / Visual and representation RL

SPOC: Imitating Shortest Paths in Simulation Enables Effective Navigation and Manipulation in the Real World

CVPR 2024 · Reinforcement learning › Visual and representation RL · approach: Skill learning and diffusion policies · 26 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Trains an RGB-only transformer agent (SPOC) by behavior cloning on shortest-path expert trajectories collected in ~200k procedurally generated AI2-THOR houses, and transfers it zero-shot to real Stretch robot navigation and manipulation.

Problem

Training embodied navigation and manipulation agents usually requires slow, reward-shaped RL or expensive human demonstrations, and sim-to-real transfer is hard; it is unclear whether cheap shortest-path planner data at scale can yield capable agents.

Approach

Imitate heuristic shortest-path planners at very large scale (millions of frames over ~200k procedural houses with ~41k Objaverse assets) using an end-to-end transformer with two RGB cameras, a goal-conditioned transformer visual encoder, a causal transformer action decoder, and strong SigLIP encoders plus augmentation, with no RL, depth, GPS, or maps.

Key results

• Multi-task SPOC reaches 49.9% average success on CHORES-S versus 31.2% for a single-task RL EmbSigLIP baseline, and 65.0% with ground-truth detection.
• Real-world evaluation over 88 trials gives SPOC 39.5% average success, rising to 56.1% when paired with the DETIC detector, with no real-world fine-tuning.
• Scaling diversity from 100 to 10k houses at equal episode count improves OBJNAV success by 13.5 points.

Measured effects reported in the paper

Surprising findings

• A shortest-path-only OBJNAV agent spontaneously explores, peeks into rooms, and backtracks despite never seeing such behavior in training data.
• Exploration-based expert trajectories gave no gain over shortest-path trajectories, contrary to prior work (PIRLNav).

Limitations

• Performance is gated by object perception; failures are mainly perception errors, and GT detection is needed for high success.
• Manipulation in the real world uses a heuristic grasping model invoked by PickUp.
• Real-world evaluation is small (88 trials across two settings).

Experimental setup

Datasets: ProcTHOR generated houses Objaverse assets AI2-THOR shortest-path expert trajectories

Benchmarks: CHORES-S CHORES-L CHORESNAV-S CHORESNAV-L real-world Stretch RE-1 trials

Metrics: Success rate SEL (episode-length weighted success) percentage rooms visited

Baselines: EmbSigLIP (EmbCLIP with SigLIP backbone, single-task RL) CLIP-RN50 encoder DINOv2 encoder GRU action decoder Explore-OBJNAV expert

Builds on: EmbCLIP ProcTHOR AI2-THOR Decision Transformer

Closest papers by meaning

• Learning Interactive Real-World Simulators (ICLR 2024 · similarity 0.86)
• Reasoning in Visual Navigation of End-to-end Trained Agents: A Dynamical Systems Approach (CVPR 2025 · similarity 0.85)
• Zero-Shot Robotic Manipulation with Pre-Trained Image-Editing Diffusion Models (ICLR 2024 · similarity 0.85)
• OSVI-WM: One-Shot Visual Imitation for Unseen Tasks using World-Model-Guided Trajectory Generation (NeurIPS 2025 · similarity 0.85)
• Overcoming the Sim-to-Real Gap: Leveraging Simulation to Learn to Explore for Real-World RL (NeurIPS 2024 · similarity 0.85)
• DiffTORI: Differentiable Trajectory Optimization for Deep Reinforcement and Imitation Learning (NeurIPS 2024 · similarity 0.85)