Research map / Reinforcement learning / Offline RL
Uncertainty-based Offline Variational Bayesian Reinforcement Learning for Robustness under Diverse Data Corruptions
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes TRACER, a variational Bayesian action-value distribution method for offline RL that uses an entropy-based uncertainty measure to down-weight corrupted transitions, evaluated under random and adversarial corruptions of states, actions, rewards and dynamics on D4RL.
Problem
Offline RL degrades when the dataset contains diverse corruptions (states, actions, rewards, dynamics) from random noise or adversarial attacks, and existing robust methods either target only some elements or assume testing-time robustness.
Approach
Model all corruptions as uncertainty in the action-value distribution, approximate its posterior via variational Bayesian inference using all offline data elements as observations, and weight each sample's loss by the reciprocal of exponential entropy so corrupted (higher-entropy) data is down-weighted.
Key results
- Under simultaneous random corruption, TRACER averages 24.54 across Halfcheetah, Walker2d and Hopper versus 20.27 for RIQL (the strongest baseline).
- Under individual random corruptions, TRACER averages 61.1 versus 60.0 for RIQL; under individual adversarial corruptions it averages 58.1 versus 56.2 for RIQL.
- The paper claims an average improvement of +33.6% across corruption levels under random simultaneous corruption.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use TRACER instead of RIQL | MuJoCo offline RL under simultaneous random corruptions · MuJoCo simultaneous random corruptions | average score: 20.27 -> 24.54 | Helped |
| Use TRACER instead of RIQL | MuJoCo offline RL under simultaneous adversarial corruptions · MuJoCo simultaneous adversarial corruptions | average score: 56.2 -> 58.1 | Helped |
| Use TRACER instead of RIQL | MuJoCo offline RL under individual random corruptions · MuJoCo individual random corruptions | average score: 60.0 -> 61.1 | Helped |
| Use TRACER instead of RIQL | MuJoCo offline RL under individual adversarial corruptions · MuJoCo individual adversarial corruptions | average score: 56.2 -> 58.1 | Helped |
| Add entropy-based uncertainty weighting to TRACER (ablation) | TRACER under simultaneous corruptions · MuJoCo simultaneous corruptions | average score | Helped |
| Remove entropy constraints (do not use entropy weighting) (ablation) | TRACER · MuJoCo simultaneous corruptions | corrupted-versus-clean entropy distinction | No clear effect |
Surprising findings
- Several baselines such as EDAC and MSG collapse to near-zero scores under simultaneous corruption, indicating uncertainty-based ensembles alone are not robust to corruption.
- The paper reports that TRACER's advantage over RIQL on individual corruptions is small (e.g., 61.1 vs 60.0), with RIQL winning several settings.
Limitations
- Gains over RIQL on individual corruption settings are modest and TRACER does not win every setting.
- Evaluated only on MuJoCo and CARLA with a single medium-replay dataset and fixed corruption rate 0.3 and scale 1.0 in main results.
- Hyperparameters (N, K, alpha, kappa) are tuned per environment and corruption type, with extra tuning on Hopper.
Experimental setup
Datasets: D4RL medium-replay-v2 CARLA-lane_v0
Benchmarks: D4RL MuJoCo Halfcheetah D4RL MuJoCo Walker2d D4RL MuJoCo Hopper CARLA-lane_v0
Metrics: normalized average return standard error
Baselines: BC EDAC MSG UWMSG CQL IQL RIQL
Builds on: IQL RIQL UWMSG quantile regression generalized Bayesian inference
Closest papers by meaning
- Towards Robust Offline Reinforcement Learning under Diverse Data Corruption (ICLR 2024 · similarity 0.92)
- Tackling Data Corruption in Offline Reinforcement Learning via Sequence Modeling (ICLR 2025 · similarity 0.89)
- A Unified Principle of Pessimism for Offline Reinforcement Learning under Model Mismatch (NeurIPS 2024 · similarity 0.89)
- Bayesian Design Principles for Offline-to-Online Reinforcement Learning (ICML 2024 · similarity 0.89)
- Robust Policy Expansion for Offline-to-Online RL under Diverse Data Corruption (NeurIPS 2025 · similarity 0.89)
- Exact Policy Recovery in Offline RL with Both Heavy-Tailed Rewards and Data Corruption (AAAI 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.