Research map / Reinforcement learning / Imitation learning
Uncertainty-Based Smooth Policy Regularisation for Reinforcement Learning with Few Demonstrations
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SPReD, which uses critic-ensemble Q-value distributions to produce continuous, uncertainty-proportional weights for a behaviour cloning term from demonstrations in sparse-reward robotics, via a probabilistic (SPReD-P) and an exponential advantage (SPReD-E) variant.
Problem
Deciding when an RL agent should imitate demonstrations versus follow its own policy in sparse-reward continuous control, especially with few or suboptimal demonstrations, where binary Q-filter decisions rely on point estimates.
Approach
Replace Q-filter's binary indicator with continuous weights derived from ensemble critic Q-value distributions: SPReD-P uses the Gaussian probability that the demonstration action beats the policy action, and SPReD-E uses a clipped exponential of the mean advantage scaled by the IQR of the Q-distributions.
Key results
- On FetchStack2, SPReD-E reaches 0.920 success vs 0.064 for RLPD after 10M steps (about 14x).
- SPReD-E achieves 0.240 on FetchSlide versus 0.160 for the next best method.
- SPReD-P and SPReD-E outperform Q-filter and RLPD on most of the eight tasks, and remain robust when demonstration quality drops, where standard Q-filter falls below TD3 on PickAndPlace with one expert demo among 99 random ones.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use SPReD-P instead of the strongest baseline | TD3 with demonstrations · FetchPush | success rate: 0.968 ± 0.016 -> 0.976 ± 0.020 | Helped |
| Use SPReD-E instead of the strongest baseline | TD3 with demonstrations · FetchSlide | success rate: 0.160 ± 0.057 -> 0.240 ± 0.044 | Helped |
| Use SPReD-E instead of the strongest baseline | TD3 with demonstrations · FetchPickAndPlace | success rate: 0.688 ± 0.069 -> 0.888 ± 0.064 | Helped |
| Use SPReD-E instead of the strongest baseline | TD3 with demonstrations · FetchStack2 | success rate: 0.144 ± 0.103 -> 0.920 ± 0.057 | Helped |
| Use SPReD-E instead of the strongest baseline | TD3 with demonstrations · FetchStack3 | success rate: 0.080 ± 0.062 -> 0.384 ± 0.125 | Helped |
| Use SPReD-P instead of the strongest baseline | TD3 with demonstrations · ManipulateBlock | success rate: 0.856 ± 0.103 -> 0.864 ± 0.065 | Helped |
| Use SPReD-P instead of the strongest baseline | TD3 with demonstrations · ManipulateEgg | success rate: 0.096 ± 0.041 -> 0.208 ± 0.082 | Helped |
| Use SPReD-E instead of the strongest baseline | TD3 with demonstrations · ManipulatePen | success rate: 0.264 ± 0.093 -> 0.288 ± 0.053 | Helped |
| Use SPReD-P instead of SPReD-E | SPReD · ManipulateBlock | success rate: 0.832 ± 0.089 -> 0.864 ± 0.065 | Helped |
| Use SPReD-P instead of SPReD-E | SPReD · ManipulateEgg | success rate: 0.16 ± 0.076 -> 0.208 ± 0.082 | Helped |
| Use SPReD-P rather than binary Q-filtering | demonstration-based RL with severely suboptimal data · FetchPickAndPlace | learning performance | Helped |
| Use SPReD rather than binary Q-filtering | demonstration-based RL across demonstration quality levels · FetchPickAndPlace and FetchPush | learning performance | Helped |
Surprising findings
- Standard Q-filter performs worse than TD3 without demonstrations in the extreme low-quality PickAndPlace setting, amplifying misleading demonstrations.
- AWAC performs poorly in this few-demonstration sparse-reward setting despite being a strong offline-to-online method.
Limitations
- Demonstration influence can slow later-stage learning in highly dexterous tasks.
- Hyperparameter sensitivity, notably the scaling constant alpha for SPReD-E (set to 10 from preliminary experiments)
- Evaluation only in simulation; no sim-to-real study.
Experimental setup
Datasets: OpenAI Gym Fetch and Shadow Dexterous Hand demonstrations
Benchmarks: FetchPush FetchSlide FetchPickAndPlace FetchStack2 FetchStack3 ManipulateBlock ManipulateEgg ManipulatePen
Metrics: success rate standard deviation over seeds gradient variance
Baselines: TD3 EnsTD3 Q-filter EnsQ-filter RLPD AWAC AWAC-p AWAC-r
Builds on: Q-filter TD3 Hindsight Experience Replay REDQ RLPD
Closest papers by meaning
- Demonstration-Regularized RL (ICLR 2024 · similarity 0.91)
- Limited Preference Aided Imitation Learning from Imperfect Demonstrations (ICML 2024 · similarity 0.89)
- Learning from Demonstrations via Capability-Aware Goal Sampling (NeurIPS 2025 · similarity 0.89)
- GO-DICE: Goal-Conditioned Option-Aware Offline Imitation Learning via Stationary Distribution Correction Estimation (AAAI 2024 · similarity 0.89)
- Conformalized Interactive Imitation Learning: Handling Expert Shift and Intermittent Feedback (ICLR 2025 · similarity 0.88)
- Efficient Imitation under Misspecification (ICLR 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.