Research map / Reinforcement learning / Offline RL
Penalizing Infeasible Actions and Reward Scaling in Reinforcement Learning with Offline Data
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes PARS, which combines reward scaling with layer normalization (RS-LN) and a penalty on Q-values of infeasible actions (PA) on top of TD3+BC to reduce Q-extrapolation error in offline and offline-to-online RL on D4RL.
Problem
Q-value extrapolation errors in offline RL and offline-to-online fine-tuning, where ReLU MLP critics extrapolate linearly outside the data range and overestimate out-of-distribution actions.
Approach
Increasing the reward scale together with layer normalization reduces the NTK similarity between in-data and far out-of-data actions, so Q-values beyond the data range fall relative to in-data values; an extra TD-style loss pushes Q-values of sampled infeasible actions far outside the feasible region down to a minimum value Q_min.
Key results
- Offline training: PARS averages 81.6 on AntMaze vs 60.6 for the best prior (ReBRAC/MSG), 79.9 on Adroit vs 75.0 for ReBRAC, and 77.3 on MuJoCo vs 76.3 for SAC-N.
- Online fine-tuning with 300k samples: PARS averages 95.2 on AntMaze vs 78.4 for RLPD, and 113.2 on Adroit vs 102.5 for RLPD.
- On AntMaze ultra-play and ultra-diverse, PARS reaches 86.5 and 86.4 after online fine-tuning, where most baselines score below 25.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add RS-LN to IQL (ablation) | IQL · AntMaze umaze-diverse, large-play, and ultra-play offline | normalized score: 66.5$\pm$5.5 -> 83.0$\pm$6.2; 45.4$\pm$5.8 -> 60.4$\pm$6.2; 13.3$\pm$5.7 -> 36.6$\pm$13.3 | Helped |
| add PA to IQL with RS-LN (ablation) | IQL with RS-LN · AntMaze umaze-diverse, large-play, and ultra-play offline | normalized score: 83.0$\pm$6.2 -> 81.1$\pm$7.9; 60.4$\pm$6.2 -> 60.6$\pm$5.3; 36.6$\pm$13.3 -> 37.3$\pm$10.7 | Mixed |
| combine LN and PA with reward scaling (ablation) | PARS offline training · D4RL tasks | normalized score | Helped |
| remove PA during online fine-tuning after using it offline (ablation) | PARS · Online fine-tuning | normalized score | Hurt |
| keep PA active during online fine-tuning (ablation) | PARS · Online fine-tuning | learning stability | Helped |
| move infeasible-action samples near the feasible action region (ablation) | PARS · Offline training | normalized score | Hurt |
| increase reward scale with LN (ablation) | PARS critic · AntMaze-medium-diverse offline | normalized return | Helped |
| use reward scaling without LN (ablation) | PARS critic · Offline training | normalized score | No clear effect |
| remove the activation function while using LN and reward scale 100 (ablation) | Q-function regression · Toy dataset OOD extrapolation | OOD Q-value | Hurt |
| use GELU activation instead of ReLU (ablation) | Q-function regression · Toy dataset OOD extrapolation | OOD Q-value | Mixed |
| use Sigmoid activation instead of ReLU (ablation) | Q-function regression · Toy dataset OOD extrapolation | OOD Q-value | Mixed |
Surprising findings
- Penalizing infeasible actions was not particularly effective when applied to in-sample IQL, while RS-LN gave noticeable gains for IQL.
- Max-Q actions lie at the boundary of the feasible region even with layer normalization, indicating LN alone does not fix linear extrapolation.
Limitations
- Most effective when the target Q-value is positive; negative returns require reward shifting.
- Still benefits from a critic ensemble in MuJoCo and Adroit, so it is not a pure double-critic method.
- Performance is sensitive to the infeasible-region distance when it is set too close to the feasible region.
Experimental setup
Datasets: D4RL AntMaze D4RL Adroit D4RL MuJoCo
Benchmarks: D4RL AntMaze D4RL Adroit D4RL MuJoCo
Metrics: normalized score average normalized score
Baselines: TD3+BC IQL CQL SAC-N EDAC MCQ MSG SPOT SVR SAC-RND ReBRAC Off2On
Builds on: TD3+BC layer normalization neural tangent kernel SAC-N CORL
Closest papers by meaning
- A Perspective of Q-value Estimation on Offline-to-Online Reinforcement Learning (AAAI 2024 · similarity 0.89)
- Exclusively Penalized Q-learning for Offline Reinforcement Learning (NeurIPS 2024 · similarity 0.89)
- Iteratively Refined Behavior Regularization for Offline Reinforcement Learning (NeurIPS 2024 · similarity 0.89)
- Adaptive Advantage-Guided Policy Regularization for Offline Reinforcement Learning (ICML 2024 · similarity 0.89)
- Doubly Mild Generalization for Offline Reinforcement Learning (NeurIPS 2024 · similarity 0.89)
- Adaptive Neighborhood-Constrained Q Learning for Offline Reinforcement Learning (NeurIPS 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.