Research map / Reinforcement learning / Visual and representation RL

Meta-learning how to Share Credit among Macro-Actions

NeurIPS 2025 · Reinforcement learning › Visual and representation RL · approach: In-context and adaptive RL · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes MASP, a regularizer that pushes Q-values of similar macro-actions toward each other via a similarity matrix meta-learned jointly with the policy, improving Rainbow-DQN with macro-actions on Atari and Street Fighter II.

Problem

Adding macro-actions to the action space can hurt exploration because the branching factor grows, and each macro-action is treated as independent and atomic in standard value-based RL.

Approach

A soft penalty ||Q - ΣQ||² that ties Q-values of similar (macro-)actions together, with the similarity matrix Σ meta-learned via meta-gradients jointly with the Q-network, so credit is shared among related macro-actions and the effective action dimension is reduced.

Key results

• On Atari, +MASP reaches Breakout 884.4 vs 379.5 for Rainbow-DQN and Seaquest 23,768.8 vs 19,176.0, with HN ≥100 in many games.
• Rainbow with naive macro-actions degrades as the macro set grows (e.g., Breakout 13.2 at k=1024), while MASP stays at 379.5 at k=1024.
• On StreetFighter II Macro-Actions II, MASP reaches 593,784 vs 537,388.8 for Rainbow-DQN and 565,319.2 for +Macro-Actions.

Measured effects reported in the paper

Surprising findings

• Naively adding macro-actions to Rainbow-DQN lowers performance on most Atari games relative to the baseline.
• Transferring a frozen Σ from another game (e.g., Breakout Σ to Space Invaders) can match or exceed in-domain performance in some cases.

Limitations

• Computational cost of the similarity matrix grows quadratically with the action space; overhead about 20% wallclock.
• Benefit depends on the quality of macro-action extraction, which uses a frequency heuristic on human trajectories.
• Σ is state-independent and its interpretability and transfer properties are not fully understood.

Experimental setup

Datasets: Atari Grand Challenge Dataset Arcade Learning Environment Gym Retro StreetFighter II MiniGrid

Benchmarks: Atari 2600 (57 games) StreetFighter II MiniGrid DoorKey, FourRooms, LockedRoom, ObstructedMaze-Full

Metrics: game score human-normalized score success rate

Baselines: Rainbow-DQN Rainbow-DQN + Macro-Actions

Builds on: Rainbow DQN DQN meta-gradient reinforcement learning universal value function approximators

Closest papers by meaning

• First-Explore, then Exploit: Meta-Learning to Solve Hard Exploration-Exploitation Trade-Offs (NeurIPS 2024 · similarity 0.87)
• Action abstractions for amortized sampling (ICLR 2025 · similarity 0.86)
• Discovering Temporally-Aware Reinforcement Learning Algorithms (ICLR 2024 · similarity 0.85)
• Can Learned Optimization Make Reinforcement Learning Less Difficult? (NeurIPS 2024 · similarity 0.85)
• Achieving Sample and Computational Efficient Reinforcement Learning by Action Space Reduction via Grouping (ICLR 2024 · similarity 0.85)
• DISCOVER: Automated Curricula for Sparse-Reward Reinforcement Learning (NeurIPS 2025 · similarity 0.85)