Research map / Reinforcement learning / Visual and representation RL

NASDAQ: Normalized Observation Space Dynamics-Augmented Q-Learning

NeurIPS 2026 · Reinforcement learning › Visual and representation RL · approach: Latent-space world models · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Identifies unbalanced per-dimension reconstruction losses as the bottleneck of observation-predictive RL on low-dimensional tasks, proposes SARON (robust, shift-adaptive observation normalization) and NASDAQ, a TD3-based framework that adds normalized next-observation prediction and short-term value prediction auxiliary tasks.

Problem

Observation-predictive RL (e.g., OFENet) performs poorly on challenging low-dimensional control tasks because reconstruction losses are dominated by high-range observation dimensions, neglecting small-range ones.

Approach

Normalize low-dimensional observations with an online, episode-based EMA normalizer (SARON) that excludes outlier low-return episodes, so per-dimension reconstruction losses and gradients are balanced; then perform coupled dynamics prediction in the normalized observation space, unified with pixel min-max normalization across low- and high-dimensional inputs.

Key results

• NASDAQ with SARON achieves Gym IQM 1.50 and DMC (proprioceptive) mean 805 / IQM 898, the best among compared methods including MR.Q (791 mean) and OFENet+TD3 (567 mean).
• On DMC (visual), NASDAQ reaches mean 606 and IQM 701, above MR.Q (598 / 686) and DreamerV3 (463 / 452).
• SARON improves OFENet+TD3 on Gym (mean 1.48 to 1.64) and DMC proprioceptive (mean 567 to 587), and NASDAQ w/o auxiliary tasks with SARON reaches DMC mean 744 vs 593 without ON.

Surprising findings

• Short-term value prediction slightly hurts performance on low-dimensional (non-visual) tasks while helping visual tasks, attributed to a bias-variance trade-off.
• NASDAQ trains faster than the model-free DrQ-v2 baseline in visual settings because its encoder parameters are reused across objectives.

Limitations

• Benchmark-level auxiliary loss weights are fixed and not universal across benchmarks; no adaptive weighting.
• SARON only applies to low-dimensional observations; no normalization for high-dimensional inputs beyond min-max for pixels.
• Short-term value prediction gives a slight negative effect on non-visual tasks.

Experimental setup

Datasets: Gym (MuJoCo -v5) DMC (proprioceptive) DMC (visual) Atari100k

Benchmarks: Gym DMC (proprioceptive) DMC (visual) Atari100k

Metrics: mean median interquartile mean (IQM) deep-TD3-normalized score human-normalized score training wall-time

Baselines: OFENet+TD3 TD7 SPR TD-MPC2 DreamerV3 MR.Q DrQ-v2 Rainbow (enhanced) simple ON (SB3)

Builds on: OFENet TD3 MR.Q SPR Adam

Closest papers by meaning

• Provably Filtering Exogenous Distractors using Multistep Inverse Dynamics (ICLR 2022 · similarity 0.86)
• Rich-Observation Reinforcement Learning with Continuous Latent Dynamics (ICML 2024 · similarity 0.85)
• Towards Principled Representation Learning from Videos for Reinforcement Learning (ICLR 2024 · similarity 0.85)
• Learning Low-dimensional Latent Dynamics from High-dimensional Observations: Non-asymptotics and Lower Bounds (ICML 2024 · similarity 0.84)
• MAD-TD: Model-Augmented Data stabilizes High Update Ratio RL (ICLR 2025 · similarity 0.83)
• AlphaQCM: Alpha Discovery in Finance with Distributional Reinforcement Learning (ICML 2025 · similarity 0.83)