Research map / Reinforcement learning / Policy gradient and actor-critic
$q$-exponential family for policy optimization
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes using the general q-exponential family (Student's t, heavy- and light-tailed q-Gaussians) as policy parameterizations inside actor-critic algorithms and empirically shows heavy-tailed policies, especially Student's t, often beat the Gaussian online and offline.
Problem
Gaussian policies are the default for continuous-action policy optimization but are thin-tailed, unstable when sigma is small, and cannot represent sparse/heavy-tailed behavior; the impact of the policy parametrization on actor-critic performance is understudied.
Approach
Use the q-exponential family as the policy class inside existing actor-critic algorithms (SAC, GreedyAC, TAWAC, AWAC, IQL, InAC, TD3BC), including Student's t with learnable degrees of freedom and heavy-tailed (1<q<3) and light-tailed (q<1) q-Gaussians, and handle out-of-support actions for sparse q-Gaussians by nearest in-support resampling.
Key results
- Student's t policy outperformed the Gaussian in about 75% of algorithm-environment settings in online classic control and was best in 5 of 12 settings, while the Gaussian was best only once.
- On offline D4RL MuJoCo, TAWAC with heavy-tailed q-Gaussian was often the top performer and Student's t consistently outperformed the Squashed Gaussian, with improvements up to ~20%.
- Light-tailed q-Gaussian performed poorly offline due to out-of-support action evaluation, and the proposed nearest in-support resampling did not help much offline.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace Gaussian with Student's t | Online actor-critic policies · Online classic control | AUC: settings better than Gaussian: 75% | Helped |
| Replace Gaussian with light-tailed q-Gaussian | Online actor-critic policies · Online classic control | AUC: settings better than Gaussian: 66% | Helped |
| Replace Gaussian with Student's t | Online actor-critic policies · Online classic control | AUC: number of settings ranked best: 1 time -> 5/12 settings | Helped |
| Replace Gaussian with Beta | SAC · Mountain Car | AUC rank | Helped |
| Replace Gaussian with light-tailed q-Gaussian | TAWAC(0) · Mountain Car | AUC rank | Helped |
| Use a q-Gaussian policy instead of Gaussian | TAWAC(0) and TAWAC(2) · Online classic control | AUC | Helped |
| Replace Squashed Gaussian with Student's t | Offline actor-critic algorithms · D4RL MuJoCo Medium-Replay | normalized score: ~20% | Helped |
| Replace Squashed Gaussian with heavy-tailed q-Gaussian | TAWAC and IQL · D4RL MuJoCo | normalized score: ~20% | Helped |
| Replace Squashed Gaussian with heavy-tailed q-Gaussian | AWAC · D4RL MuJoCo | normalized score | Hurt |
| Replace Gaussian with Beta | Policy optimization methods · Experiments overall | performance | Hurt |
| Replace Squashed Gaussian with light-tailed q-Gaussian | Offline algorithms · D4RL MuJoCo | performance | Hurt |
| Replace out-of-support behavior actions with the nearest in-support sampled action | Light-tailed q-Gaussian in offline experiments · Offline experiments | learning performance | No clear effect |
Surprising findings
- The Gaussian was the best policy parametrization in only 1 of 12 online settings despite being the default for all algorithms.
- The light-tailed q-Gaussian led to poor offline performance even with random online sampling.
Limitations
- Light-tailed q-Gaussian has finite support, so off-policy/offline actions can fall outside it; the nearest in-support replacement heuristic did not help much offline.
- Heavy-tailed q-Gaussian fixed at q=2 and light-tailed at q=0 rather than tuned.
- Experiments are limited to small classic control and MuJoCo tasks.
Experimental setup
Datasets: D4RL MuJoCo halfcheetah, hopper, walker2d (medium, medium-replay, medium-expert) Mountain Car Pendulum Acrobot
Benchmarks: D4RL MuJoCo Mountain Car Pendulum Acrobot
Metrics: normalized score area under learning curve (AUC) percentage of settings better than Gaussian
Baselines: Gaussian Squashed Gaussian Beta policy SAC GreedyAC AWAC IQL InAC TD3BC
Builds on: Soft Actor-Critic GreedyAC TAWAC AWAC Implicit Q-Learning
Closest papers by meaning
- Smoothed Action Value Functions for Learning Gaussian Policies (ICML 2018 · similarity 0.87)
- Actor-Free Continuous Control via Structurally Maximizable Q-Functions (NeurIPS 2025 · similarity 0.86)
- Actor-Critics Can Achieve Optimal Sample Efficiency (ICML 2025 · similarity 0.86)
- On the Convergence of Single-Timescale Actor-Critic (NeurIPS 2025 · similarity 0.86)
- Non-Asymptotic Analysis for Single-Loop (Natural) Actor-Critic with Compatible Function Approximation (ICML 2024 · similarity 0.85)
- Augmented Lagrangian Method for Last-Iterate Convergence for Constrained MDPs (NeurIPS 2026 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.