Research map / Reinforcement learning / Policy gradient and actor-critic
RN-D: Discretized Categorical Actors for On-Policy Reinforcement Learning
Yuexin Bian, Jie Feng, Tao Wang, Yijiang Li, Sicun Gao, Yuanyuan Shi
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace Gaussian actor with discretized categorical actor and add regularized residual network (RN-D) | PPO · Gym locomotion and ManiSkill | sample efficiency: 1.3--1.9x fewer interaction steps | Improved |
| add regularized residual pre-LayerNorm network to Gaussian actor (ablation) | MLP-C · Gym locomotion and ManiSkill-State | return / success | Improved |
| discretize actions with categorical actor on a standard MLP (ablation) | MLP-C · Gym locomotion and ManiSkill-State | return / success | Improved |
| pair categorical actor with regularized network on vision tasks | MLP-C · ManiSkill RGB | success rate | Mixed |
| swap categorical likelihood for Gaussian squared-error loss on discretized actor (ablation) | RN-D · Gym locomotion / StackCube | return | Hurt |
| use discretized categorical actor instead of Gaussian | RN-C / MLP-C · MuJoCo tasks | policy-gradient variance | Improved |
| add residual connections and layer normalization to actor (ablation) | MLP actor · Gym locomotion | gradient SNR | Improved |
| increase actor hidden width from 64 to 512 (ablation) | RN-D · Humanoid-v4 and StackCube-v1 | convergence speed | Mixed |
| increase number of action bins beyond about 100 (ablation) | RN-D · StackCube-v1 | success rate | Hurt |
| increase action bins from 11 to 1001 (ablation) | RN-D · Humanoid-v4 | wall-clock time (h): 0.76 -> 1.95 | Hurt |
| use discretized instead of continuous actor with standard MLP | MLP-C · Gym locomotion | wall-clock time (h): 1.65 -> 1.56 | Improved |
| use discretized actor with regularized network | RN-C · Gym locomotion | SPS: 811.1 -> 873.8 | Improved |
Closest papers by meaning
- What Matters for On-Policy Deep Actor-Critic Methods? A Large-Scale Study (ICLR 2021 · similarity 0.89)
- Policy Gradient and Actor-Critic Learning in Continuous Time and Space: Theory and Algorithms (JMLR 2022 · similarity 0.89)
- Breaking the Computational Barrier: Provably Efficient Actor–Critic for Low-Rank MDPs (ICML 2026 · similarity 0.88)
- Geometry of Neural Reinforcement Learning in Continuous State and Action Spaces (ICLR 2025 · similarity 0.87)
- The Ladder in Chaos: Improving Policy Learning by Harnessing the Parameter Evolving Path in A Low-dimensional Space (NeurIPS 2024 · similarity 0.87)
- ReLU to the Rescue: Improve Your On-Policy Actor-Critic with Positive Advantages (ICML 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
