Research map / Reinforcement learning / Visual and representation RL
Learning Uncertainty-Aware Temporally-Extended Actions
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes UTE, an uncertainty-aware action repetition method that uses an ensemble of option-value heads (TempoRL-style skip policy) with a UCB-style mean+λ·std extension choice, evaluated on Chain MDP, Gridworlds, Atari, and Pendulum.
Problem
Action repetition (temporal extension of actions) can degrade performance when sub-optimal actions are repeated, since existing methods ignore uncertainty of option values over the repeated horizon.
Approach
Select the extension length by maximizing ensemble mean plus λ times ensemble standard deviation of option-value estimates, with heads trained toward a shared target, n-step Q-learning for the action values, and an optional adaptive λ chosen via sliding-window UCB bandit; λ<0 gives pessimism and λ>0 gives optimism.
Key results
- On Chain MDP (chain length 70), UTE with λ=+2.0 reached normalized AUC 0.191 vs 0.052 for TempoRL and 0.131 for epsilon z-Greedy.
- On Gridworlds Zigzag with linear epsilon decay, UTE achieved normalized AUC 0.84 vs 0.62 for epsilon z-Greedy and 0.14 for TempoRL.
- On Atari Road Runner, UTE with n-step learning averaged 12323.2 vs 8131.5 for TempoRL and 3733.8 for epsilon z-Greedy; n-step learning improved mean normalized score from 1.39 to 1.81 (30.2%).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use uncertainty-aware action extension | action-repetition policy · Chain MDP | normalized AUC: 0.904 -> 0.919; 0.740 -> 0.758; 0.246 -> 0.560; 0.052 -> 0.191 | Helped |
| use uncertainty-aware action extension | action-repetition policy · Bridge and Zigzag Gridworlds | normalized AUC: Bridge: 0.76 -> 0.86, 0.92 -> 0.92, 0.59 -> 0.83; Zigzag: 0.62 -> 0.84, 0.76 -> 0.89, 0.36 -> 0.76 | Helped |
| use n-step rather than 1-step Q-learning for the action-value function (ablation) | UTE · Atari 2600 games | average reward: The paper reports a 30.2% improvement (from 1.39 to 1.81). | Helped |
| select lambda adaptively (ablation) | UTE · Atari 2600 games | average reward | Mixed |
| set uncertainty parameter lambda to +2.0 instead of -2.0 (ablation) | UTE · Chain MDP, length 70 | normalized AUC: 0.01 -> 0.19 | Helped |
| use a more negative uncertainty parameter (ablation) | UTE · Gridworlds with risky lava regions | normalized AUC | Helped |
| use a fixed action repetition length | DDQN · Atari 2600 games | average reward | Hurt |
Surprising findings
- TempoRL performed substantially worse than vanilla DDQN in function-approximation Gridworlds, contrary to results reported in the original tabular setting.
- Fixed action repeat (j=4) fails at learning in most Atari games, highlighting the necessity of learning the extension policy.
Limitations
- Results on Atari are reported over few seeds (7) and only four games in the main comparison, with a short 10M-frame budget.
- Optimal uncertainty parameter λ is environment-dependent; adaptive λ is somewhat slower to learn.
- Ensemble of extension-policy heads adds computational and hyperparameter overhead.
Experimental setup
Datasets: Chain MDP Lava Gridworlds (Bridge, ZigZag, Cliff) Atari 2600 (Crazy Climber, Road Runner, Sea Quest, Up n Down) Pendulum-v0
Benchmarks: Chain MDP Gridworlds Bridge/ZigZag Atari 2600 (4 games) Pendulum-v0
Metrics: normalized AUC of reward average reward over last 100k steps coverage average reward over last 10k steps
Baselines: DDQN Fixed Repeat epsilon z-Greedy DAR TempoRL B-DQN DDPG FiGAR t-DDPG
Builds on: TempoRL Bootstrapped DQN Double DQN DAR epsilon z-greedy
Closest papers by meaning
- From Static Policies to Adaptive Priors in Offline Reinforcement Learning (NeurIPS 2026 · similarity 0.86)
- Improving planning and MBRL with temporally-extended actions (NeurIPS 2025 · similarity 0.86)
- Uncertainty-Based Smooth Policy Regularisation for Reinforcement Learning with Few Demonstrations (NeurIPS 2025 · similarity 0.85)
- Option-aware Temporally Abstracted Value for Offline Goal-Conditioned Reinforcement Learning (NeurIPS 2025 · similarity 0.85)
- Learning to Reuse Policies in State Evolvable Environments (ICML 2025 · similarity 0.85)
- Beyond-Expert Performance with Limited Demonstrations: Efficient Imitation Learning with Double Exploration (ICLR 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.