Research map / Reinforcement learning / Policy gradient and actor-critic
Truncated Gaussian Policy for Debiased Continuous Control
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes a truncated Gaussian policy and a scale-adjusted variant for PPO that reduce boundary action bias in bounded continuous control, outperforming the clipped Gaussian on MuJoCo, HumanoidBench and DMC.
Problem
Gaussian policies with action clipping in PPO over-sample boundary actions in bounded continuous control tasks, which can degrade learning.
Approach
Replace the unbounded Gaussian with a truncated Gaussian that samples directly within action bounds (no clipping), then discount the scale as the location approaches a boundary (semi-ellipse scale-adjustment) so that boundary actions are not under- or over-penalized.
Key results
- SA-TGaussian achieves the highest normalized score (0.747 at sigma_init=0.5) versus 0.609 for Gaussian and 0.679 for CAPG on MuJoCo.
- Gaussian policies show roughly 10-30% boundary action rate in locomotion tasks, while truncated variants keep aBAR near 0-2%.
- At sigma_init=1.0, SA-TGaussian reaches normalized score 0.812 versus 0.373 for the Gaussian baseline.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use scale-adjusted truncated Gaussian instead of Gaussian with action clipping | PPO · MuJoCo 8 tasks | normalization score: 0.609 -> 0.747 | Helped |
| use scale-adjusted truncated Gaussian instead of plain truncated Gaussian | PPO · MuJoCo 8 tasks | normalization score: 0.554 -> 0.747 | Helped |
| replace Gaussian with scale-adjusted truncated Gaussian | PPO · HalfCheetah-v4 | episodic return: 5165 ± 173.2 -> 5554 ± 115.3 | Helped |
| replace Gaussian with scale-adjusted truncated Gaussian | PPO · Walker2d-v4 | episodic return: 3153 ± 437.6 -> 3619 ± 376.2 | Helped |
| replace Gaussian with scale-adjusted truncated Gaussian | PPO · Ant-v4 | episodic return: 4001 ± 447.8 -> 3972 ± 452.3 | No clear effect |
| replace Gaussian with scale-adjusted truncated Gaussian | PPO · Hopper-v4 | episodic return: 1705 ± 267.3 -> 1488 ± 239.6 | Hurt |
| replace Gaussian with scale-adjusted truncated Gaussian | PPO · Humanoid-v4 | episodic return: 3059 ± 553.5 -> 3599 ± 577.0 | Helped |
| replace Gaussian with scale-adjusted truncated Gaussian | PPO · Swimmer-v4 | episodic return: 77.23 ± 1.49 -> 72.54 ± 0.55 | Hurt |
| replace Gaussian with scale-adjusted truncated Gaussian | PPO · Pusher-v4 | episodic return: -38.20 ± 2.50 -> -41.06 ± 2.92 | Hurt |
| replace Gaussian with scale-adjusted truncated Gaussian | PPO · Reacher-v4 | episodic return: -6.04 ± 0.92 -> -4.88 ± 0.61 | Helped |
| increase initial scale from σinit = 0.5 to σinit = 1.0 (ablation) | SA-TGaussian · MuJoCo 8 tasks | normalization score: 0.747 -> 0.812 | Helped |
| increase initial scale from σinit = 0.5 to σinit = 1.0 (ablation) | TGaussian · MuJoCo 8 tasks | normalization score: 0.554 -> 0.435 | Hurt |
Surprising findings
- Plain truncated Gaussian overcompensates and can underperform the clipped Gaussian, indicating a counter-bias toward interior actions.
- Forcing boundary-only actions degrades performance in most tasks, though not in Swimmer and Hopper.
Limitations
- Mechanism by which suppressing boundary actions improves performance is not fully explored.
- Performance varies irregularly across tasks; optimal hyperparameters (k, d_min) are task-dependent.
- Evaluated only with PPO on continuous control tasks.
Experimental setup
Datasets: MuJoCo HumanoidBench DeepMind Control Suite
Benchmarks: MuJoCo HalfCheetah MuJoCo Walker2d MuJoCo Ant MuJoCo Hopper MuJoCo Humanoid MuJoCo Swimmer MuJoCo Pusher MuJoCo Reacher HumanoidBench h1hand-walk HumanoidBench h1hand-reach DMC dog-walk DMC dog-run
Metrics: episodic return normalized score average boundary action rate (aBAR)
Baselines: Gaussian with clipping CAPG Beta policy Logit-normal (tanh-squashed) policy TGaussian
Builds on: PPO CAPG SAC
Closest papers by meaning
- Smoothed Action Value Functions for Learning Gaussian Policies (ICML 2018 · similarity 0.87)
- Improving Stochastic Action-Constrained Reinforcement Learning via Truncated Distributions (AAAI 2026 · similarity 0.87)
- On the Hidden Biases of Policy Mirror Ascent in Continuous Action Spaces (ICML 2022 · similarity 0.87)
- Relaxed Stationary Distribution Correction Estimation for Improved Offline Policy Optimization (AAAI 2024 · similarity 0.86)
- Implicit Bias of Policy Gradient in Linear Quadratic Control: Extrapolation to Unseen Initial States (ICML 2024 · similarity 0.85)
- Exclusively Penalized Q-learning for Offline Reinforcement Learning (NeurIPS 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.