Research map / Reinforcement learning / Visual and representation RL
ARS: Adaptive Reward Scaling for Multi-Task Reinforcement Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Adaptive Reward Scaling (ARS), which rescales per-task rewards from replay-buffer mean statistics and periodically resets networks, applied on top of SAC-MT for multi-task RL on Meta-World MT10/MT50.
Problem
Multi-task RL suffers when tasks have very different reward magnitudes, causing critic bias toward high-reward tasks and failure on tasks with low reward scale (e.g., push, pick-place, peg-insert-side).
Approach
Scale each task's reward by a factor equal to the maximum task mean reward divided by that task's mean reward in its replay buffer (an equalizer rule), recomputed only at periodic network resets that reinitialize the policy and Q-networks while keeping replay buffers.
Key results
- ARS-LoRA reaches 99.5% average success on MT10, the first to solve MT10 from scratch multi-task training per the authors.
- ARS-LoRA reaches 83.2% average success on MT50 bottom-50 measure, versus 47.6% for SAC-MT and 64.2% for MOORE; ARS-LN reaches 78.3%.
- Integrating ARS into MOORE raises MT50 average success from 64.2% to 84.6%, and boosts PCGrad and Soft Modular on both benchmarks.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| apply ARS adaptive reward scaling and periodic resets | SAC-MT · Meta-World MT10 | average success rate (%): 69.4 ± 0.8 -> 97.3 ± 2.1 | Helped |
| apply ARS adaptive reward scaling and periodic resets | SAC-MT · Meta-World MT50, bottom-50 | average bottom-k success ratio (%): 47.6 ± 2.0 -> 65.9 ± 2.9 | Helped |
| add layer normalization to the critic | ARS · Meta-World MT10 | average success rate (%): 97.3 ± 2.1 -> 98.6 ± 1.5 | Helped |
| add LoRA to ARS-LN | ARS-LN · Meta-World MT10 | average success rate (%): 98.6 ± 1.5 -> 99.5 ± 0.8 | Helped |
| remove the reset mechanism (ablation) | ARS · Meta-World MT10 | average success ratio: 97.3±2.1 -> 69.8 ±8.1 | Hurt |
| remove adaptive reward scaling (ablation) | ARS · Meta-World MT10, difficult tasks | average success ratio: 91.7±7.3 -> 0.8±1.5 | Hurt |
| replace mean-reward scaling with reward normalization (ablation) | ARS · Meta-World MT10 | average success ratio: 97.3±2.1 -> 82.4 ± 27.9 | Hurt |
| replace mean-reward scaling with standard-deviation scaling (ablation) | ARS · Meta-World MT10 | average success ratio: 97.3±2.1 -> 94.3±7.1 | Hurt |
| increase network hidden dimensions (ablation) | ARS-LN · Meta-World MT50 | average success rate (%): 78.3±3.7 -> 88.9±2.6 | Helped |
| increase network hidden dimensions (ablation) | ARS · Meta-World MT50 | average success rate (%): 65.9±2.9 -> 78.7±2.9 | Helped |
Surprising findings
- Uniformly applying a 100x reward scale to all tasks fails on several tasks, while scaling only the hard tasks succeeds, showing fixed scaling can hurt overall performance.
Limitations
- Evaluated only on Meta-World manipulation tasks with SAC-based off-policy methods.
- Comparison with MOORE is not parameter-matched in the main table; MOORE uses about 4x more parameters.
Experimental setup
Datasets: Meta-World MT10 Meta-World MT50
Benchmarks: Meta-World MT10 Meta-World MT50
Metrics: average success rate bottom-k success rate effective solvable task ratio (ESTR)
Baselines: SAC-MT SAC-MT-MH Soft Modularization PCGrad PaCo MOORE SMT PopArt-style normalization
Builds on: SAC SAC-MT Nikishin et al. 2022 parameter reset Automatic Reward Scaling PopArt
Closest papers by meaning
- Hard Tasks First: Multi-Task Reinforcement Learning Through Task Scheduling (ICML 2024 · similarity 0.89)
- Exploring Data Scaling Trends and Effects in Reinforcement Learning from Human Feedback (NeurIPS 2025 · similarity 0.87)
- MetaCARD: Meta-Reinforcement Learning with Task Uncertainty Feedback via Decoupled Context-Aware Reward and Dynamics Components (AAAI 2024 · similarity 0.87)
- Mastering Massive Multi-Task Reinforcement Learning via Mixture-of-Expert Decision Transformer (ICML 2025 · similarity 0.87)
- Learning Task Belief Similarity with Latent Dynamics for Meta-Reinforcement Learning (ICLR 2025 · similarity 0.87)
- AMAGO-2: Breaking the Multi-Task Barrier in Meta-Reinforcement Learning with Transformers (NeurIPS 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.