Research map / Reinforcement learning / Policy gradient and actor-critic
General Munchausen Reinforcement Learning with Tsallis Kullback-Leibler Divergence
Lingwei Zhu, Zheng Chen, Matthew Kyle Schlegel, Martha White
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use Tsallis KL regularization with q=2 (MVI(q=2)) instead of standard KL (q=1) | Munchausen Value Iteration on QR-DQN · Atari 2600 (about 35 games, 50M frames) | game score | Improved |
| Use MVI(q=2) instead of MVI(q=1) | Munchausen Value Iteration · PrivateEye, Chopper and Seaquest | game score | Hurt |
| Add Munchausen (KL) term, alpha greater than 0, to Tsallis-VI (ablation) | Tsallis-VI (entropy only) · 35 Atari games | game score | Improved |
| Add Munchausen (KL) term to Tsallis-VI (ablation) | Tsallis-VI · 10 Atari games | game score: more than 400% improvement | Improved |
| Use q-logarithm of policy with action gap Q_k - M_q Q_k as Munchausen term instead of ln_q pi (ablation) | MVI(q) · CartPole-v1 | learning stability | Improved |
| Increase q from 2 to 5 in the Munchausen approximation (ablation) | MVI(q) · CartPole-v1 | residual term magnitude | Hurt |
| Simply replace ln pi with ln_q pi in MVI (ablation) | MVI · CartPole-v1 | learning curve | Hurt |
| Use Tsallis entropy alone (q=2 sparsemax) without KL | policy optimization · prior work comparison | performance | Hurt |
Closest papers by meaning
- Achieving Logarithmic Regret in KL-Regularized Zero-Sum Markov Games (ICML 2026 · similarity 0.89)
- Leverage the Average: an Analysis of KL Regularization in Reinforcement Learning (NeurIPS 2020 · similarity 0.88)
- Categorical Distributional Reinforcement Learning with Kullback-Leibler Divergence: Convergence and Asymptotics (ICML 2025 · similarity 0.87)
- Logarithmic Regret for Online KL-Regularized Reinforcement Learning (ICML 2025 · similarity 0.86)
- Fast Rates for Offline Contextual Bandits with Forward-KL Regularization under Single-Policy Concentrability (NeurIPS 2026 · similarity 0.86)
- Sharp Analysis for KL-Regularized Contextual Bandits and RLHF (NeurIPS 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
