Research map / Reinforcement learning / Visual and representation RL
Chain-of-Thought Predictive Control
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes CoTPC, a Transformer-based hierarchical imitation learning method that unsupervisedly discovers subskill chain-of-thought from action segmentation of sub-optimal demos and jointly predicts subskills and actions via learnable CoT prompt tokens with hybrid masking.
Problem
Generalizable imitation learning for low-level contact-rich manipulation from sub-optimal, noisy, non-Markovian demonstrations, where behavior cloning generalizes poorly.
Approach
Discover subskill boundaries offline by change-point detection (PELT) over cosine-distance action segments, then train a BeT-style Transformer with learnable CoT prompt tokens that predict next/last subskill observations and guide action decoding, with hybrid (all-to-all CoT, causal action/state) masking and random CoT attention masking.
Key results
- On Moving Maze, CoTPC reaches 44.0% success rate versus 33.0% for BeT, 23.0% for DT and 9.0% for vanilla BC.
- On ManiSkill2 state-observation tasks, CoTPC achieves best unseen/0-shot SR on Stack Cube (86.0), Peg Insertion (59.3), Turn Faucet 0-shot (39.3) and Push Chair 0-shot (41.0).
- Franka Kitchen full-task success rises to 25.6% for CoTPC versus 14.4% for BeT.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use coupled action and subskill prediction instead of decoupled prediction (ablation) | CoTPC · Peg Insertion (0-shot) and Push Chair (unseen), state observations | success rate (%): 47.0 -> 59.3; 36.0 -> 41.0 | Helped |
| Predict both next and last subskills instead of only the last subskill (ablation) | CoTPC · Peg Insertion (0-shot) and Push Chair (unseen), state observations | success rate (%): 52.0 -> 59.3; 36.0 -> 41.0 | Helped |
| Predict both next and last subskills instead of only the next subskill (ablation) | CoTPC · Peg Insertion (0-shot) and Push Chair (unseen), state observations | success rate (%): 49.0 -> 59.3; 37.0 -> 41.0 | Helped |
| Use discovered subskills instead of a randomly selected future observation as guidance (ablation) | CoTPC · Peg Insertion (0-shot) and Push Chair (unseen), state observations | success rate (%): 41.0 -> 59.3; 31.0 -> 41.0 | Helped |
| Use multiple shared CoT tokens rather than the vanilla single-token design (ablation) | CoTPC · Peg Insertion (0-shot) and Push Chair (unseen), state observations | success rate (%): 45.0 -> 59.3; 35.0 -> 41.0 | Helped |
| Avoid sharing CoT tokens with both action center and offset decoders (ablation) | CoTPC · Peg Insertion (0-shot) and Push Chair (unseen), state observations | success rate (%): 39.0 -> 59.3; 29.0 -> 41.0 | Helped |
| Use the original decoder-to-token assignment instead of swapping center and offset inputs (ablation) | CoTPC · Peg Insertion (0-shot) and Push Chair (unseen), state observations | success rate (%): 46.0 -> 59.3; 32.0 -> 41.0 | Helped |
| Add CoT supervision and the CoT decoder rather than adding prompt tokens alone (ablation) | CoTPC · Peg Insertion (0-shot) and Push Chair (unseen), state observations | success rate (%): 43.0 -> 59.3; 32.7 -> 41.0 | Helped |
| Apply random attention masking during training (ablation) | CoTPC · Peg Insertion (0-shot) and Push Chair (unseen), state observations | success rate (%): 48.5 -> 59.3; 34.2 -> 41.0 | Helped |
Surprising findings
- Adding extra prompt tokens to BeT without the CoT auxiliary loss gives almost no improvement, indicating gains do not come from extra capacity.
- Predicting random future observations as subskills performs worst, and a more expressive CoT decoder harms action decoding.
Limitations
- Assumes similar and nearby actions form one subskill, which fails for complex manipulations with many micro-adjustments.
- Purely offline and not robust to sub-optimal demos; online fine-tuning suggested to help.
- CoT decoder is not used at inference, so the method is not strictly hierarchical.
Experimental setup
Datasets: ManiSkill2 demonstrations Franka Kitchen human demonstrations Moving Maze demonstrations (generated with heuristics and RRT-style planner)
Benchmarks: Moving Maze Franka Kitchen (variant) ManiSkill2
Metrics: success rate average completed sub-tasks
Baselines: Vanilla BC Decision Transformer Behavior Transformer Decision Diffuser Diffusion Policy
Builds on: Behavior Transformer Decision Transformer PELT Detection Transformer
Closest papers by meaning
- Multi-Stage Manipulation with Demonstration-Augmented Reward, Policy, and World Model Learning (ICML 2025 · similarity 0.87)
- Chain-of-Action: Trajectory Autoregressive Modeling for Robotic Manipulation (NeurIPS 2025 · similarity 0.86)
- Zero-Shot Robotic Manipulation with Pre-Trained Image-Editing Diffusion Models (ICLR 2024 · similarity 0.86)
- Generalizable Domain Adaptation for Sim-and-Real Policy Co-Training (NeurIPS 2025 · similarity 0.85)
- Meta-Controller: Few-Shot Imitation of Unseen Embodiments and Tasks in Continuous Control (NeurIPS 2024 · similarity 0.85)
- Predictive Inverse Dynamics Models are Scalable Learners for Robotic Manipulation (ICLR 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.