Research map / Reinforcement learning / Offline RL
Test-time Adapted Reinforcement Learning with Action Entropy Regularization
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes TARL, an unsupervised test-time adaptation method for offline RL that minimizes action entropy (discrete) or action variance (continuous) on test states, updating only layer-norm parameters with a KL debiasing term.
Problem
Offline RL policies suffer from distribution shift between offline datasets and online test environments, producing OOD state-action pairs and unreliable decisions when no environment reward is available at test time.
Approach
Adapt the offline policy at test time using only unlabeled test states by minimizing policy output entropy (discrete) or a normal-distribution uncertainty (continuous) on confident samples, updating only layer-norm parameters, and regularizing with a KL divergence to the frozen original policy to prevent bias.
Key results
- On D4RL hopper-expert-v2, CQL (TARL) reaches 113.34 vs CQL 99.72, a 13.62-point gain (about 13.6% relative).
- On Atari Asterix, CQL (TARL) achieves 524.76 vs CQL 504.59, and QR-DQN (TARL) reaches 819.24 vs QR-DQN 503.38.
- On Walker2d-medium-v2, IQL (TARL) achieves 82.43 vs IQL 79.92.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add TARL test-time adaptation to CQL | CQL · Atari: Qbert, Seaquest, Asterix, Pong, Breakout | average episode return: 4334.64 ± 259.67 -> 4602.67 ± 264.11; 257.80 ± 9.25 -> 270.40 ± 8.42; 504.59 ± 13.98 -> 524.76 ± 13.98; 1.02 ± 1.22 -> 2 | Helped |
| Add TARL test-time adaptation to CQL | CQL · hopper-expert-v2 | average episode return: 99.72 -> 113.34 | Helped |
| Add TARL test-time adaptation to CQL | CQL · walker2d-fully-replay-v2 | average episode return: 95.31 -> 97.99 | Helped |
| Add TARL test-time adaptation to CQL | CQL · walker2d-expert-v2 | average episode return: 113.25 -> 113.57 | Helped |
| Add TARL test-time adaptation to IQL | IQL · Walker2d-medium-v2 | average episode return: 79.92 -> 82.43 | Helped |
| Add TARL test-time adaptation to IQL | IQL · Walker2d-expert-v2 | average episode return: 110.31 -> 110.49 | Helped |
| Set entropy threshold E0 to 0.9, 0.7, 0.5, or 0.3 rather than 0.1 (ablation) | TARL on CQL · Pong | average episode return: 1.70 ± 0.82 -> −19.98 ± 0.11; 1.70 ± 0.82 -> −19.61 ± 0.22; 1.70 ± 0.82 -> −18.79 ± 0.42; 1.70 ± 0.82 -> −13.54 ± 1.22 | Hurt |
| Use low-entropy selective training instead of global entropy minimization (ablation) | TARL · Walker2d-medium-v2 | average episode return: 77.74 -> 82.95 | Helped |
| Use high-entropy selective training instead of low-entropy selective training (ablation) | TARL · Walker2d-medium-v2 | average episode return: 82.95 -> 72.91 | Hurt |
| Set KL divergence constraint λ to 0.5 rather than 1.5 (ablation) | TARL on CQL · Pong | average episode return: 2.08 ± 0.92 -> 0.92 ± 1.08 | Hurt |
| Set KL divergence constraint λ to 1.0 rather than 1.5 (ablation) | TARL on CQL · Pong | average episode return: 2.08 ± 0.92 -> 1.32 ± 1.39 | Hurt |
| Set KL divergence constraint λ to 2.0 rather than 1.5 (ablation) | TARL on CQL · Pong | average episode return: 2.08 ± 0.92 -> 0.62 ± 1.73 | Hurt |
Surprising findings
- Test-time entropy minimization without selection degrades Pong performance; a suitable entropy threshold is needed, and very high thresholds reduce returns to near-random levels.
Limitations
- Gains on several Atari tasks are small and within reported variance (e.g., Pong, Breakout).
- Hyperparameters such as learning rate, entropy threshold and KL weight are sensitive and tuned per domain.
- Evaluation uses offline datasets and test environments that are not truly shifted in a controlled way beyond standard benchmarks.
Experimental setup
Datasets: Atari (Qbert, Seaquest, Asterix, Pong, Breakout) D4RL (hopper-expert-v2, walker2d-fully-replay-v2, walker2d-expert-v2, walker2d-medium-v2)
Benchmarks: Atari D4RL
Metrics: average episode return
Baselines: REM QR-DQN CQL IQL
Builds on: CQL Tent (entropy minimization test-time adaptation) REM QR-DQN IQL
Closest papers by meaning
- Efficient Test-Time Adaptation For Robot Policies (NeurIPS 2026 · similarity 0.88)
- From Static Policies to Adaptive Priors in Offline Reinforcement Learning (NeurIPS 2026 · similarity 0.87)
- Iteratively Refined Behavior Regularization for Offline Reinforcement Learning (NeurIPS 2024 · similarity 0.87)
- Adaptive Advantage-Guided Policy Regularization for Offline Reinforcement Learning (ICML 2024 · similarity 0.87)
- Simple Ingredients for Offline Reinforcement Learning (ICML 2024 · similarity 0.87)
- Behavioral Entropy-Guided Dataset Generation for Offline Reinforcement Learning (ICLR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.