Research map / Reinforcement learning / Offline RL
Q-SAM: Unlocking Sharpness-Aware Minimization for Generalization in Offline Reinforcement Learning
Da Wang, Yi Ma, Ting Guo, Lin Li, Wei Wei, Jiye Liang
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace Adam with plain SAM for the policy | TD3BC · D4RL gym MuJoCo-v2 total | Total normalized score: 688.0 -> 693.8 | No clear effect |
| replace Adam with Q-SAM for the policy | TD3BC · D4RL AntMaze-v2 total | Total normalized score: 116.2 -> 163.4 | Improved |
| replace Adam with Q-SAM for the policy | IQL · D4RL gym MuJoCo-v2 total | Total normalized score: 734.6 -> 772.1 | Improved |
| replace Adam with Q-SAM for the policy | CQL · D4RL AntMaze-v2 total | Total normalized score: 304.5 -> 335.0 | Improved |
| add weighted sharpness (lambda-weighted SAM) (ablation) | Q-SAM on CQL · D4RL gym MuJoCo-v2 total | Total normalized score: 730.8 -> 745.0 | Improved |
| add Q-bound weighted samples (ablation) | Q-SAM on CQL · D4RL gym MuJoCo-v2 total | Total normalized score: 736.5 -> 745.0 | Improved |
| add Q-bound weighted samples on halfcheetah datasets (ablation) | Q-SAM on CQL · halfcheetah-mr / halfcheetah-me | normalized score: drop -0.1 and -0.3 without weighted samples | No clear effect |
| use SAM instead of Adam for the policy | TD3BC · hopper-medium-replay | Local Action Sensitivity: 1.36 -> 0.67 | Improved |
| double TD3BC training steps from 1M to 2M instead of using Q-SAM (ablation) | TD3BC · hopper-medium-replay | normalized score: 64.4 -> 65.4 (TD3BC + Q-SAM at 1M: 87.3) | No clear effect |
| apply Q-SAM with small-sample data (5% of dataset) | TD3BC · halfcheetah-medium-replay 5% data | normalized score | Improved |
| add weighted sharpness to reduce Bellman error (ablation) | Q-SAM · critic loss (small-sample) | Log(1 + Critic Loss) | Improved |
| Q-SAM on TD3BC for hard antmaze tasks | TD3BC · antmaze-large-play and large-diverse | normalized score: 0.0 -> 0.0 | No clear effect |
Closest papers by meaning
- Doubly Mild Generalization for Offline Reinforcement Learning (NeurIPS 2024 · similarity 0.90)
- Offline RL with Smooth OOD Generalization in Convex Hull and its Neighborhood (ICLR 2025 · similarity 0.90)
- On the Complexity of Offline Reinforcement Learning with Q*-Approximation and Partial Coverage (NeurIPS 2026 · similarity 0.90)
- $Q\sharp$: Provably Optimal Distributional RL for LLM Post-Training (NeurIPS 2025 · similarity 0.90)
- A Perspective of Q-value Estimation on Offline-to-Online Reinforcement Learning (AAAI 2024 · similarity 0.89)
- Adaptive Neighborhood-Constrained Q Learning for Offline Reinforcement Learning (NeurIPS 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
