Research map / Reinforcement learning / Visual and representation RL
Real-World Reinforcement Learning of Active Perception Behaviors
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes asymmetric advantage weighted regression (AAWR), an offline-to-online RL recipe that uses privileged training-time sensors in critics to train real-world active perception policies from suboptimal demonstrations.
Problem
Learning active perception behaviors under partial observability on real robots is hard: imitation is limited by demonstrator quality, and RL is sample-inefficient and hard to sim2real transfer for sensor-dependent tasks.
Approach
Extends advantage weighted regression to POMDPs by training IQL critics and value functions with privileged observations/state available only at training time, while the policy uses only partial observations; theory shows the asymmetric advantage yields the correct POMDP policy-improvement objective whereas symmetric advantages do not.
Key results
- AAWR outperforms AWR and BC on simulated tasks by approximately 2x (Camouflage Pick) and 3x (Fully Obs. Pick).
- On Active Perception Koch, online AAWR reaches 100% while distillation plateaus at 80% and VIB collapses at evaluation.
- In handoff tasks, AAWR achieves 92.4 Search and 44.4 Completion on Bookshelf-P, versus 11.0 Search and 16.7 Completion for pi0 alone.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use privileged observations for value and critic inputs instead of unprivileged AWR | Blind Pick · Blind Pick | Grasp: 71 -> 94 | Helped |
| Use privileged observations for value and critic inputs instead of unprivileged AWR | Blind Pick · Blind Pick | Pick: 55 -> 89 | Helped |
| Use privileged observations for value and critic inputs instead of unprivileged AWR | Bookshelf-P AAWR · Bookshelf-P | Search: 79.6 -> 92.4 | Helped |
| Use privileged observations for value and critic inputs instead of unprivileged AWR | Bookshelf-P AAWR · Bookshelf-P | Completion: 0.0 -> 44.4 | Helped |
| Use privileged observations for value and critic inputs instead of unprivileged AWR | Bookshelf-D AAWR · Bookshelf-D | Search: 62.6 -> 81.3 | Helped |
| Use privileged observations for value and critic inputs instead of unprivileged AWR | Bookshelf-D AAWR · Bookshelf-D | Completion: 16.7 -> 44.4 | Helped |
| Use privileged observations for value and critic inputs instead of unprivileged AWR | Shelf-Cabinet AAWR · Shelf-Cabinet | Search: 52.3 -> 78.2 | Helped |
| Use privileged observations for value and critic inputs instead of unprivileged AWR | Shelf-Cabinet AAWR · Shelf-Cabinet | Completion: 10.0 -> 40.0 | Helped |
| Use privileged observations for value and critic inputs instead of unprivileged AWR | Complex AAWR · Complex | Search: 33.2 -> 73.2 | Helped |
| Use privileged observations for value and critic inputs instead of unprivileged AWR | Complex AAWR · Complex | Completion: 40.0 -> 50.0 | Helped |
| Use privileged observations for value and critic inputs instead of unprivileged AWR | Complex AAWR · Complex | Steps: 67.0 -> 43.0 | Helped |
| Train with a privileged expert and distill into a partially observed policy | Active Perception Koch · Active Perception Koch | success: 80% | Hurt |
Surprising findings
- Privileged critics improved performance even in a fully observable task, which the authors attribute to the non-privileged critic needing to learn object position extraction from pixels.
- Distillation from a privileged expert plateaus because the expert ignores the camera's limited field of view.
Limitations
- Relies on switching between a helper active perception policy and pi0 rather than end-to-end fine-tuning of generalist policies.
- Scalability to longer-horizon tasks is untested.
- Privileged information is prespecified rather than learned.
Experimental setup
Datasets: teleoperated demonstrations collected with SpaceMouse simulated demonstrations (100 suboptimal per task)
Benchmarks: Camouflage Pick Fully Obs. Pick Active Perception Koch Blind Pick Bookshelf-P Bookshelf-D Shelf-Cabinet Complex
Metrics: success rate Search score Completion rate Steps
Baselines: AWR BC Distillation VIB pi0 VLM+pi0 Exhaustive
Builds on: advantage weighted regression implicit q-learning DROID pi0
Closest papers by meaning
- ACTIVE-o3 : Empowering MLLMs with Active Perception via Pure Reinforcement Learning (ICML 2026 · similarity 0.88)
- Active Fine-Tuning of Multi-Task Policies (ICML 2025 · similarity 0.86)
- PIGDreamer: Privileged Information Guided World Models for Safe Partially Observable Reinforcement Learning (ICML 2025 · similarity 0.85)
- Privileged Sensing Scaffolds Reinforcement Learning (ICLR 2024 · similarity 0.85)
- CaP-X: A Framework for Benchmarking and Improving Coding Agents for Robot Manipulation (ICML 2026 · similarity 0.85)
- Robot-R1: Reinforcement Learning for Enhanced Embodied Reasoning in Robotics (NeurIPS 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.