Research map / Reinforcement learning / Visual and representation RL
Privileged Sensing Scaffolds Reinforcement Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Scaffolder, a model-based RL method that uses training-only privileged sensors in a scaffolded world model, critic, reward, and exploration policy to train a target policy with limited sensing, and introduces the S3 suite of ten robotic tasks.
Problem
Training agents to act from cheap, limited target sensors while exploiting expensive privileged sensors available only at training time (sensory scaffolding).
Approach
Train a second, scaffolded world model on privileged-plus-target observations and use it for imagined rollouts, scaffolded reward and critic estimates, and an exploration policy, while mapping scaffolded latents back to target latents via a transdecoder (nested latent imagination), so the deployed target policy needs only target sensors.
Key results
- Scaffolder bridges 79% of the gap between target-only and privileged-observation DreamerV3 performance (normalized median score) across ten S3 tasks.
- Scaffolder learns faster in 8 of 10 tasks and matches or beats AAC and RMA+ trained with 100M samples in 9 of 10 tasks while using 250K to 5M samples.
- Scaffolded TD(lambda) return estimates have lower mean absolute error against Monte-Carlo returns than target-only estimates in Blind Pick.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use privileged observations to scaffold world modeling, credit assignment, exploration, and representation learning | Scaffolder · S3 (10 tasks) | final performance: 79% | Helped |
| Use privileged observations to scaffold RL components | Scaffolder · S3 (10 tasks) | learning speed | Helped |
| Use privileged observations to scaffold RL components | Scaffolder · S3 (10 tasks), compared with AAC and RMA | task performance: 9 out of 10 tasks | Helped |
| Add a scaffolded exploration policy that collects data alongside the target policy (ablation) | Scaffolder · Blind Pick | task performance | Helped |
| Add a privileged critic using scaffolded information (ablation) | Scaffolder · Blind Cube Rotation | task performance | Helped |
| Add a scaffolded world model trained with privileged observations (ablation) | Scaffolder · Blind Locomotion | task performance | Helped |
| Train the target representation to reconstruct privileged observations (ablation) | Scaffolder · RGB Cube Rotation | task performance | Helped |
| Use scaffolded world-model dynamics, reward estimates, and critic for TD(λ) returns | Scaffolder · Blind Pick | TD(λ) return estimation error (MAE) | Helped |
Surprising findings
- DreamerV3 operating only on target sensors is a surprisingly strong baseline, outperforming some privileged methods such as AAC.
- Privileged baselines show high variance and are often worse than unprivileged DreamerV3.
Limitations
- Scaffolder trains an extra world model and exploration policy, adding substantial compute and resource cost.
- Using additional training-time sensors adds computation, bandwidth, and handling of incomplete multimodal data.
- Lacks a theoretical characterization of how privileged information affects the full RL process.
Experimental setup
Datasets: S3 Sensory Scaffolding Suite (10 simulated robotic tasks)
Benchmarks: S3 Sensory Scaffolding Suite
Metrics: normalized median score task return success rate number of handholds mean absolute error of return estimates
Baselines: DreamerV3 AAC DreamerV3+BC Informed Dreamer RMA+ Guided Observability
Builds on: DreamerV3 Asymmetric Actor Critic RMA Informed Dreamer Guided Observability
Closest papers by meaning
- Real-World Reinforcement Learning of Active Perception Behaviors (NeurIPS 2025 · similarity 0.85)
- Enhancing Tactile-based Reinforcement Learning for Robotic Control (NeurIPS 2025 · similarity 0.85)
- CaP-X: A Framework for Benchmarking and Improving Coding Agents for Robot Manipulation (ICML 2026 · similarity 0.84)
- Self-Improving Embodied Foundation Models (NeurIPS 2025 · similarity 0.83)
- Scaffolding Dexterous Manipulation with Vision-Language Models (NeurIPS 2025 · similarity 0.83)
- SOLE-R1: Video-Language Reasoning as the Sole Reward for On-Robot Reinforcement Learning (NeurIPS 2026 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.