Research map / Reinforcement learning / Imitation learning
Stealthy Imitation: Reward-guided Environment-free Policy Stealing
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Stealthy Imitation, an environment-free, input-range-agnostic black-box attack that steals deep RL control policies by iteratively refining a Gaussian query distribution using a GAIL-style reward model.
Problem
Stealing deep RL control policies via black-box queries when the attacker has no access to the environment and does not know the valid input range or state distribution of the victim.
Approach
Fit a diagonal Gaussian query distribution, train a GAIL-style reward model to discriminate victim from attacker actions, and use the reward as weights to refine the Gaussian toward the victim's state distribution; the victim's hard-to-imitate regions indicate in-distribution states.
Key results
- In Hopper, the method reaches a 97% return ratio with 5M queries, while the best baseline (Random N(0,10^2)) reaches only 70% under the same budget.
- On average, KL(S_v || S_a) is reduced by 81% across MuJoCo environments.
- A defense that returns random actions for out-of-range queries drops the return ratio to 0% (Hopper), 7% (Walker2D), and 4% (HalfCheetah) at 50M queries.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use reward-guided distribution refinement instead of fixed random query distributions | policy stealing · Hopper, 5 million queries | return ratio: 97\% vs 70\% | Helped |
| Use reward-guided distribution refinement | Stealthy Imitation · Mujoco control tasks | KL divergence: 81\% reduction | Helped |
| Add a reward model for distribution refinement instead of directly weighting samples by validation loss (ablation) | Stealthy Imitation · Mujoco control tasks | KL divergence | Helped |
| Use a fixed dataset size to train the distribution evaluator (ablation) | Stealthy Imitation · Mujoco control tasks | return ratio | Helped |
| Prune extreme-action samples before training the reward model (ablation) | Stealthy Imitation · Mujoco control tasks | stability | Helped |
| Use a dynamic transfer-dataset budget proportional to the estimated standard deviation (ablation) | Stealthy Imitation · Hopper | return ratio | Helped |
| Use a dynamic transfer-dataset budget proportional to the estimated standard deviation (ablation) | Stealthy Imitation · Hopper | distribution stability | Helped |
| Increase reserved query budget from 1M to 3M | Stealthy Imitation · PandaPickAndPlace | return ratio | Helped |
| Randomize actions for queries outside the valid input range | victim policy · Hopper, 50M queries | attacker return ratio: 98\% -> 0\% | Hurt |
| Randomize actions for queries outside the valid input range | victim policy · Walker2D, 50M queries | attacker return ratio: 101\% -> 7\% | Hurt |
| Randomize actions for queries outside the valid input range | victim policy · HalfCheetah, 50M queries | attacker return ratio: 47\% -> 4\% | Hurt |
Surprising findings
- Validation loss of the distribution evaluator is negatively correlated with KL divergence (Spearman rho about -0.8), meaning harder-to-imitate regions indicate closer distributions.
- DFME adapted to unknown input ranges fails to steal the policy in all tested variants.
Limitations
- Performance depends on the initial distribution; the tolerance to initial divergence is not characterized.
- Results may vary across victim RL algorithms beyond SAC and TQC.
- Return ratio has high variance across attacker architectures due to compounding errors.
Experimental setup
Datasets: MuJoCo Hopper-v3 Walker2D-v3 HalfCheetah-v3 PandaPickAndPlace-v3 PandaReach-v3 PandaSlide-v3
Benchmarks: MuJoCo Hopper-v3 Walker2D-v3 HalfCheetah-v3 PandaPickAndPlace-v3 PandaReach-v3 PandaSlide-v3
Metrics: KL divergence between victim and attacker state distribution return ratio
Baselines: Random normal query distributions DFME
Builds on: GAIL DFME SAC Huber loss behavioral cloning
Closest papers by meaning
- EvIL: Evolution Strategies for Generalisable Imitation Learning (ICML 2024 · similarity 0.88)
- PolicyGuard: Towards Test-time and Step-level Adversary Defense for Reinforcement Learning Agent (ICML 2026 · similarity 0.87)
- Reward-free World Models for Online Imitation Learning (ICML 2025 · similarity 0.85)
- Optimal Attack and Defense for Reinforcement Learning (AAAI 2024 · similarity 0.85)
- Blending Imitation and Reinforcement Learning for Robust Policy Improvement (ICLR 2024 · similarity 0.85)
- Policy Regularization on Globally Accessible States in Cross-Dynamics Reinforcement Learning (ICML 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.