Research map / Reinforcement learning / Offline RL
SUF: Stabilized Unconstrained Fine-Tuning for Offline-to-Online Reinforcement Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SUF, which removes offline constraints during online fine-tuning and stabilizes policy collapse by raising critic UTD (20) and lowering actor UTD (1/4), without ensembles.
Problem
Offline-to-online RL faces an efficiency-stability dilemma: constrained fine-tuning is stable but inefficient, while unconstrained fine-tuning is efficient but suffers severe policy collapse from distribution shift and value bias on OOD data.
Approach
Eliminate policy collapse without any offline constraints by modifying the update-to-data ratio: increase critic UTD to fit value on OOD data faster and decrease actor UTD to make policy updates more accurate, avoiding ensembles because the offline dataset and pre-trained initialization suppress the value bias usually caused by high UTD.
Key results
- On IQL backbone, SUF reaches a total final return of 1381 at 1M steps versus 1217 for PROTO, 1032 for PEX and 808 for IQL.
- SUF-TD3-BC total return of 799 vs APL 618 and SUNE 684 on the 9 non-ant tasks; SUF-CQL total 782 with 1336 overall for SUF-IQL (Table 1).
- Modifying critic UTD to 20 and actor UTD to 1/4 almost eliminates policy collapse during unconstrained fine-tuning, shown via ANB/SNB and NCD heat maps.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| remove offline constraints during online fine-tuning (ablation) | SUF-IQL · Walker2d-medium | normalized return | Helped |
| increase Critic UTD (ablation) | unconstrained fine-tuning · Walker2d-medium and Ant-medium | normalized return | Helped |
| decrease Actor UTD (ablation) | unconstrained fine-tuning · Walker2d-medium and Ant-medium | normalized return | Helped |
| set Critic UTD to 20 and Actor UTD to 1/4 (ablation) | SUF · Walker2d-medium and Ant-medium | normalized return | Helped |
| increase offline-data sampling ratio to 0.7 (ablation) | SUF · 12 MuJoCo tasks | IQM normalized return | Hurt |
| remove access to offline datasets (sampling ratio p=0) (ablation) | SUF · 12 MuJoCo tasks | IQM normalized return | No clear effect |
| use SUF instead of PROTO | IQL fine-tuning · 12 MuJoCo tasks | total final return: 1217 -> 1381 | Helped |
Surprising findings
- Increasing critic UTD to 20 without ensembles normally harms value estimates in online RL, but offline data or offline initialization mitigates this bias in offline-to-online RL.
- Offline data access is not required for robustness: SUF performs well across sampling ratios of offline data including p=0 (no offline data).
Limitations
- Uses a single fixed UTD configuration (Gc=20, Ga=1/4) across environments and datasets, which may not be optimal.
- Evaluated only on MuJoCo D4RL locomotion tasks; APL and SUNE results are taken from their reported numbers rather than rerun.
Experimental setup
Datasets: D4RL halfcheetah-random D4RL halfcheetah-medium D4RL halfcheetah-medium-replay D4RL hopper-random D4RL hopper-medium D4RL hopper-medium-replay D4RL walker2d-random D4RL walker2d-medium D4RL walker2d-medium-replay D4RL ant-random D4RL ant-medium D4RL ant-medium-replay
Benchmarks: D4RL MuJoCo halfcheetah, hopper, walker2d, ant (v2)
Metrics: normalized return IQM normalized return total return ANB SNB NCD
Baselines: IQL PEX PROTO APL SUNE
Builds on: IQL TD3-BC CQL SAC REDQ
Closest papers by meaning
- Efficient Online Reinforcement Learning Fine-Tuning Need Not Retain Offline Data (ICLR 2025 · similarity 0.92)
- A Clean Slate for Offline Reinforcement Learning (NeurIPS 2025 · similarity 0.91)
- Optimistic Critic Reconstruction and Constrained Fine-Tuning for General Offline-to-Online RL (NeurIPS 2024 · similarity 0.91)
- Online Pre-Training for Offline-to-Online Reinforcement Learning (ICML 2025 · similarity 0.89)
- Uni-O4: Unifying Online and Offline Deep Reinforcement Learning with Multi-Step On-Policy Optimization (ICLR 2024 · similarity 0.89)
- A Perspective of Q-value Estimation on Offline-to-Online Reinforcement Learning (AAAI 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.