Research map / Reinforcement learning / Offline RL
A Perspective of Q-value Estimation on Offline-to-Online Reinforcement Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Studies Q-value estimation as the bottleneck in offline-to-online RL and proposes SO2, which adds target-action noise (perturbed value update) and more Q-value updates per online sample on a Q-ensemble SAC base, improving finetuning on D4RL MuJoCo and Adroit.
Problem
Offline-to-online RL finetuning is slow and unstable because Q-value estimates inherited from offline pretraining are biased and poorly ranked, which yields misleading policy-update signals.
Approach
Two simple techniques on a Q-ensemble SAC base: perturb the target action with clipped Gaussian noise to smooth sharp Q-value peaks, and increase the number of Q-value (and policy) updates per online collected sample (N_upc), improving Q-value accuracy without explicit distribution-shift estimation or balanced replay.
Key results
- SO2 reaches an average normalized return of 98.9 on D4RL MuJoCo after online finetuning, versus 75.9 for OFF2ON and 58.0 for AWAC.
- On Adroit, SO2 reaches 136.8 total after 400k online steps, versus 124.0 for IQL after 1M steps.
- Combining SO2 with OFF2ON raises its 100k-step average from 75.89 to 88.64 on MuJoCo.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| apply SO2 instead of OFF2ON | Offline-to-online RL · D4RL MuJoCo Gym tasks, Average | normalized average return: 48.4 → 75.9; 76.1 → 98.9 | Helped |
| apply SO2 rather than IQL | Offline-to-online RL · D4RL Adroit, Total | normalized average score: 38.1→124.0; 67.9→ 136.8 | Helped |
| add target-action perturbation (σ=0.3) rather than no noise (σ=0.0) (ablation) | SO2 · D4RL MuJoCo, Walker2d, Average | normalized average return: 76.5 ±19.1 → 95.4 ±4.7 | Helped |
| increase target-action perturbation from σ=0.3 to σ=0.45 (ablation) | SO2 · D4RL MuJoCo, Walker2d, Average | normalized average return: 95.4 ±4.7 → 82.5 ± 5.3 | Hurt |
| increase updates per collection from N_upc=1 to N_upc=10 (ablation) | SO2 · D4RL MuJoCo, Walker2d, Average | normalized average return: 81.3 ± 5.8 → 95.4 ± 4.7 | Helped |
| increase updates per collection from N_upc=5 to N_upc=10 (ablation) | SO2 · D4RL MuJoCo, Walker2d, Average | normalized average return: 84.5 ± 8.7 → 95.4 ± 4.7 | Helped |
| combine SO2 with OFF2ON | OFF2ON · MuJoCo environments, 100k online samples | normalized average return: 75.89 ± 14.26 → 88.64 ± 4.33 | Helped |
| combine SO2 with PEX | PEX · MuJoCo environments, 100k online samples | normalized average return: 67.68 ± 8.83 → 74.26 ± 4.26 | Helped |
| use only increased Q-value update frequency (ablation) | SO2 · MuJoCo environments | policy performance | No clear effect |
Surprising findings
- SO2 with only increased Q-value update frequency performs comparably to increasing both Q and policy update frequency.
- SO2 underperforms MBPO on Hopper, which the authors attribute to pessimism in that environment.
Limitations
- Results depend on hand-tuned noise scale sigma and clip c and on N_upc, with variance across sigma settings visible in ablations.
- Baselines are re-run with different starting points, and the paper notes starting performance differences across methods.
- Evaluation limited to MuJoCo and Adroit locomotion/manipulation tasks.
Experimental setup
Datasets: D4RL MuJoCo v2 D4RL Adroit
Benchmarks: D4RL MuJoCo (HalfCheetah, Hopper, Walker2d) D4RL Adroit (pen-human, door-human)
Metrics: normalized average return standard deviation over seeds Kendall's tau coefficient normalized Q-value difference
Baselines: OFF2ON IQL AWAC ODT PEX SAC MBPO CQL TD3-BC EDAC
Builds on: SAC EDAC CQL TD3-BC OFF2ON
Closest papers by meaning
- Exclusively Penalized Q-learning for Offline Reinforcement Learning (NeurIPS 2024 · similarity 0.93)
- Is Value Learning Really the Main Bottleneck in Offline RL? (NeurIPS 2024 · similarity 0.92)
- A Clean Slate for Offline Reinforcement Learning (NeurIPS 2025 · similarity 0.92)
- Optimistic Critic Reconstruction and Constrained Fine-Tuning for General Offline-to-Online RL (NeurIPS 2024 · similarity 0.91)
- Online Pre-Training for Offline-to-Online Reinforcement Learning (ICML 2025 · similarity 0.91)
- Q-Distribution guided Q-learning for offline reinforcement learning: Uncertainty penalized Q-value via consistency model (NeurIPS 2024 · similarity 0.91)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.