Research map / Reinforcement learning / Imitation learning
Policy Decorator: Model-Agnostic Online Refinement for Large Policy Model
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Policy Decorator, a model-agnostic residual SAC policy with controlled exploration (bounded residual actions and progressive exploration schedule) that refines frozen offline-trained Behavior Transformer and Diffusion Policy models online on ManiSkill and Adroit tasks.
Problem
Offline-trained large imitation learning policies are limited by demonstration quantity, quality and diversity, and existing RL fine-tuning is hard for multimodal IL models and computationally costly; the paper asks how to improve such policies through online interaction with sparse rewards.
Approach
Freeze the large base policy and learn a small Gaussian residual policy with SAC whose output is bounded by a scale alpha and whose exploration is gradually introduced via an epsilon-greedy-like linear schedule over H steps, so the agent keeps receiving success signals while staying close to the base trajectory.
Key results
- Policy Decorator improves both Behavior Transformer and Diffusion Policy to near-perfect success on all ManiSkill and Adroit tasks with state observations, per the paper's main results.
- Among baselines, JSRL performs best but exceeds the base policy only on about half of scenarios; Cal-QL and RLPD fail on ManiSkill tasks with Behavior Transformer.
- On DPPO's Transport task, Policy Decorator reaches 80%+ success in about 4M steps versus roughly 16M for DPPO, reported as nearly 4x sample efficiency.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add tanh to the regression head and rescale actions | Behavior Transformer · ManiSkill tasks (StackCube, PegInsertionSide, TurnFaucet, PushChair) | success rate: 71% -> 67%; 15% -> 13%; 41% -> 35%; 18% -> 23% | Mixed |
| Add tanh to the regression head and rescale actions | Behavior Transformer · Adroit tasks (Door, Pen, Hammer, Relocate) | success rate: 78% -> 77%; 65% -> 63%; 23% -> 21%; 20% -> 13% | Hurt |
| Add bounded residual actions (ablation) | Policy Decorator · StackCube and PushChair | success rate | Helped |
| Add progressive exploration schedule (ablation) | Policy Decorator · StackCube and PushChair | sample efficiency | Helped |
| Add residual policy learning (ablation) | Policy Decorator · StackCube and PushChair | success rate | Mixed |
| Use observation alone rather than observation and base action as residual-policy input (ablation) | Policy Decorator · Design-choice experiments | performance | Helped |
| Use the sum of base and residual actions as critic input (ablation) | Policy Decorator · Design-choice experiments | performance | Helped |
| Set residual action bound α too small (ablation) | Policy Decorator · PushChair and StackCube | final success rate | Hurt |
| Set residual action bound α too large (ablation) | Policy Decorator · PushChair and StackCube | sample efficiency | Hurt |
| Use α values in a broad range (0.1 to 0.5 for PushChair; 0.03 to 0.1 for StackCube) (ablation) | Policy Decorator · PushChair and StackCube | success rate: 0.1 to 0.5; 0.03 to 0.1 | No clear effect |
| Use a too-small H in the progressive exploration schedule (ablation) | Policy Decorator · Progressive exploration experiments | success rate | Hurt |
| Use a large H in the progressive exploration schedule (ablation) | Policy Decorator · Progressive exploration experiments | sample efficiency | Hurt |
Surprising findings
- Naive DIPO fine-tuning of Diffusion Policy obtained no success signals on any task.
- Residual RL performs poorly with Behavior Transformer but improves Diffusion Policy on some tasks, attributed to the stronger base policy yielding more random-exploration success.
Limitations
- Enhancing large models through online interaction requires significant training time and resources, especially for slow-inference diffusion policies.
- Performance depends on hyperparameters alpha and H, which must be set per task.
- Evaluated mainly on simulated benchmarks with sparse rewards; no real-robot results reported.
Experimental setup
Datasets: ManiSkill demonstrations Adroit demonstrations
Benchmarks: ManiSkill StackCube ManiSkill PegInsertionSide ManiSkill TurnFaucet ManiSkill PushChair Adroit Door Adroit Pen Adroit Hammer Adroit Relocate
Metrics: success rate sample efficiency
Baselines: SAC fine-tuning DIPO RLPD ROT Cal-QL JSRL Residual RL FISH DPPO
Builds on: Soft Actor-Critic Residual RL DIPO JSRL ROT
Closest papers by meaning
- OLLIE: Imitation Learning from Offline Pretraining to Online Finetuning (ICML 2024 · similarity 0.90)
- Reward-free World Models for Online Imitation Learning (ICML 2025 · similarity 0.89)
- Is Behavior Cloning All You Need? Understanding Horizon in Imitation Learning (NeurIPS 2024 · similarity 0.89)
- GO-DICE: Goal-Conditioned Option-Aware Offline Imitation Learning via Stationary Distribution Correction Estimation (AAAI 2024 · similarity 0.89)
- Agnostic Interactive Imitation Learning: New Theory and Practical Algorithms (ICML 2024 · similarity 0.89)
- Efficient Imitation under Misspecification (ICLR 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.