Research map / Adversarial robustness and security / Attacks on text and graphs
RobustZero: Enhancing MuZero Reinforcement Learning Robustness to State Perturbations
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes RobustZero, a MuZero-class RL method that stays robust to worst-case and random-case state perturbations via a contrastive self-supervised representation network, a dual-buffer loss, and adaptive loss weighting.
Problem
MuZero-class planning agents degrade under state perturbations (adversarial attacks or noisy sensors), and existing robust DRL methods rely on gradient-based policies or do not target MCTS-based planning.
Approach
Use contrastive learning (projector/predictor with stop-gradient) so the initial hidden state of perturbed and unperturbed observations matches, add worst-case and random-case loss terms trained in parallel environments with separate buffers, and adaptively set the loss weight w1 (from contrastive loss) and weight-decay w2 (from gradient norms) for flat-minima convergence.
Key results
- On Racetrack, RobustZero achieves worst-case reward 477.00 vs. 429.36 for PROTECTED (best baseline) and random-case 494.00 vs. 459.10.
- On IEEE 34-bus, RobustZero worst-case reward -12.16 ± 1.19 vs. -12.89 for PROTECTED, and random-case -11.43 vs. -12.26.
- Ablation shows removing the self-supervised representation network lowers worst-case reward on Racetrack from 477.00 to 360.33.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add the self-supervised representation network (ablation) | RobustZero · Pendulum | worst-case reward: -91.15 ± 58.15 -> -87.92 ± 53.24 | Helped |
| use dynamic w1 instead of w1 = 1 (ablation) | RobustZero · Pendulum | worst-case reward: -89.41 ± 49.42 -> -87.92 ± 53.24 | Helped |
| use adaptive w2 instead of fixed w2 (ablation) | RobustZero · Pendulum | worst-case reward: -90.23 ± 57.00 -> -87.92 ± 53.24 | Helped |
| use both worst-case and random-case buffers instead of only the worst-case buffer (ablation) | RobustZero · Pendulum | worst-case reward: -90.63 ± 58.02 -> -87.92 ± 53.24 | Helped |
| use both worst-case and random-case buffers instead of only the random-case buffer (ablation) | RobustZero · Pendulum | worst-case reward: -92.26 ± 57.66 -> -87.92 ± 53.24 | Helped |
Surprising findings
- S-MuZero-worst scores higher on worst-case than on random-case reward, attributed to over-specialization to the particular attack pattern.
Limitations
- Evaluated only in single-agent settings; multi-agent extension left to future work.
- Attacks are black-box and restricted to lp-ball budgets; random-case perturbations are uniform noise.
Experimental setup
Datasets: CartPole Pendulum IEEE 34-bus IEEE 123-bus IEEE 8500-node Highway Intersection Racetrack Hopper Walker2d HalfCheetah Ant
Benchmarks: CartPole Pendulum IEEE 34-bus IEEE 8500-node Racetrack Highway Intersection Hopper Walker2d HalfCheetah Ant
Metrics: average episodic reward (natural, worst-case, random-case)
Baselines: ATLA-PPO PROTECTED S-MuZero S-MuZero-worst S-MuZero-random
Builds on: MuZero Sampled MuZero EfficientZero PROTECTED ATLA-PPO
Closest papers by meaning
- Improve Robustness of Reinforcement Learning against Observation Perturbations via l∞ Lipschitz Policy Networks (AAAI 2024 · similarity 0.88)
- State Entropy Regularization for Robust Reinforcement Learning (NeurIPS 2025 · similarity 0.88)
- Robust Gymnasium: A Unified Modular Benchmark for Robust Reinforcement Learning (ICLR 2025 · similarity 0.87)
- Game-Theoretic Robust Reinforcement Learning Handles Temporally-Coupled Perturbations (ICLR 2024 · similarity 0.87)
- Beyond Worst-case Attacks: Robust RL with Adaptive Defense via Non-dominated Policies (ICLR 2024 · similarity 0.87)
- Breaking the Barrier: Enhanced Utility and Robustness in Smoothed DRL Agents (ICML 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.