Research map / Reinforcement learning / Visual and representation RL
RT-Trajectory: Robotic Task Generalization via Hindsight Trajectory Sketches
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes RT-Trajectory, which conditions an RT-1-style transformer policy on 2D (and 2.5D) hindsight trajectory sketches so robots can generalize to new tasks specified by drawings, human videos, LLM code, or image generators.
Problem
Robot policies conditioned on language or goal images fail to generalize to new tasks whose motions differ from training tasks, even when the arm trajectory resembles seen motions.
Approach
Represent the task as a coarse 2D end-effector trajectory sketch projected into the camera view (with time encoded as color, optional height, and grasp/release markers), automatically extracted from demonstrations via hindsight labeling, and use it as the policy conditioning signal.
Key results
- RT-Trajectory (2.5D) achieves 67% overall success on 7 unseen skills vs. 26% for RT-1-Goal, 17% for RT-1, and 11% for RT-2.
- RT-Trajectory (2D) achieves 50% overall success on the same unseen skills.
- With human videos, RT-Trajectory (2D/2.5D) reaches 75% on Fold Towel vs. 25% for the IK Planner baseline.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use trajectory sketches instead of language conditioning | RT-Trajectory (2D) vs RT-1 · Unseen tasks (overall) | success rate: 17% -> 50% | Helped |
| Use trajectory sketches instead of goal-image conditioning | RT-Trajectory (2.5D) vs RT-1-Goal · Unseen tasks (overall) | success rate: 26% -> 67% | Helped |
| Add height information to the 2D trajectory (ablation) | RT-Trajectory · Unseen tasks (overall) | success rate: 50% -> 67% | Helped |
| Add height information to the 2D trajectory (ablation) | RT-Trajectory · Unseen tasks | success rate: Place Fruit: 75% -> 75%; Upright and Move: 33% -> 50%; Move within Drawer: 67% -> 100%; Restock Drawer: 92% -> 67%; Pick | Mixed |
| Condition on a human-video-derived trajectory sketch instead of executing it with an IK planner | RT-Trajectory (2D) · Human video trajectories: Pick | success rate: 42% -> 94% | Helped |
| Condition on a human-video-derived trajectory sketch instead of executing it with an IK planner | RT-Trajectory (2.5D) · Human video trajectories: Fold Towel | success rate: 25% -> 75% | Helped |
| Add height information to the 2D trajectory (ablation) | RT-Trajectory · Human video trajectories: Pick | success rate: 94% -> 100% | Helped |
| Condition on an LLM-generated trajectory sketch instead of executing it with an IK planner | RT-Trajectory (2D) · LLM-generated trajectories: Pick | success rate: 83% -> 89% | Helped |
| Condition on an LLM-generated trajectory sketch instead of executing it with an IK planner | RT-Trajectory (2D) · LLM-generated trajectories: Open Drawer | success rate: 71% -> 60% | Hurt |
| Add height information to the 2D trajectory (ablation) | RT-Trajectory · LLM-generated trajectories | success rate: Pick: 89% -> 89%; Open Drawer: 60% -> 60% | No clear effect |
Surprising findings
- Language-conditioned RT-2, which uses internet-scale VQA data, scored lower (11%) than RT-1 (17%) on unseen tasks.
- Changing only the trajectory sketch, not the scene or weights, can switch the policy's behavior, suggesting 'visual prompt engineering'.
Limitations
- Assumes a stationary robot using only the end-effector; no mobile manipulation.
- Policy follows sketch guidance on a best-effort basis and cannot strictly enforce spatial constraints.
- Image-generated trajectory sketches are noisy and differ from training sketches, yielding only qualitative results.
Experimental setup
Datasets: RT-1 demonstration dataset first-person human demonstration videos (22 videos)
Benchmarks: 7 unseen real-world manipulation skills (Place Fruit, Upright and Move, Move within Drawer, Restock Drawer, Pick from Ch
Metrics: success rate
Baselines: RT-1 RT-2 RT-1-Goal IK Planner
Builds on: RT-1 RT-2 Behavior Cloning Code as Policies
Closest papers by meaning
- STRAP: Robot Sub-Trajectory Retrieval for Augmented Policy Learning (ICLR 2025 · similarity 0.88)
- RoboGround: Robotic Manipulation with Grounded Vision-Language Priors (CVPR 2025 · similarity 0.88)
- TraceVLA: Visual Trace Prompting Enhances Spatial-Temporal Awareness for Generalist Robotic Policies (ICLR 2025 · similarity 0.87)
- Robotic Visual Instruction (CVPR 2025 · similarity 0.87)
- Zero-Shot Robotic Manipulation with Pre-Trained Image-Editing Diffusion Models (ICLR 2024 · similarity 0.87)
- Efficient Robotic Policy Learning via Latent Space Backward Planning (ICML 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.