Research map / Reinforcement learning / Offline RL
Adaptive Neighborhood-Constrained Q Learning for Offline Reinforcement Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes adaptive neighborhood constraint for offline RL's Bellman target, with radius adapted per data point by advantage, implemented as ANQ via bilevel optimization and expectile regression.
Problem
Offline RL suffers from extrapolation errors from OOD actions; existing constraints (density, support, sample) are either overly conservative or require accurate behavior policy modeling.
Approach
Restrict Bellman target actions to union of per-sample neighborhoods of dataset actions, with radius shrinking for high-advantage data and growing for low-advantage data; enforce via bilevel optimization (inner: auxiliary perturbation policy under a Lagrangian; outer: expectile regression for V) and extract policy by weighted regression toward optimized actions.
Key results
- ANQ achieves 1194.5 total on Gym locomotion versus 1139.9 for the next best (SPOT), and 441.6 total on AntMaze versus 392.4 for IQL.
- On AntMaze large-diverse, ANQ scores 55.8 vs. 46.0 for IQL and 36.3 for SPOT.
- Robustness experiments show ANQ outperforms CQL, IQL, and SPOT on noisy mixtures of random and expert data and on reduced AntMaze datasets.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use adaptive neighborhood-constrained Q learning instead of the strongest aggregate baseline | Offline RL on Gym locomotion tasks · Gym locomotion total | normalized score: 1139.9 -> 1194.5 | Helped |
| Use adaptive neighborhood-constrained Q learning instead of the strongest aggregate baseline | Offline RL on AntMaze tasks · AntMaze total | normalized score: 392.4 -> 441.6 | Helped |
| Use adaptive neighborhoods instead of uniform neighborhoods (α=0) (ablation) | ANQ on noisy datasets · Noisy datasets | normalized score | Helped |
| Increase α to an appropriately large value to adapt neighborhood radius by advantage (ablation) | ANQ · Offline RL tasks | normalized score | Helped |
| Set α excessively high (α=20) (ablation) | ANQ · Offline RL tasks | normalized score | Hurt |
| Vary λ to increase neighborhood radius from a very constrained setting (ablation) | ANQ · Offline RL tasks | normalized score | Mixed |
| Use a sufficiently large neighborhood in limited-data settings by reducing λ (ablation) | ANQ on reduced AntMaze datasets · Limited-data AntMaze | normalized score | Helped |
Surprising findings
- Sample-constraint style and density-constrained CQL perform poorly on AntMaze (CQL total 218.0 vs IQL 392.4 in the reported table), while the neighborhood approach generalizes beyond the dataset.
Limitations
- Adaptive radius uses advantage as the quality criterion, which may not be optimal; uncertainty-based adaptation is suggested as an alternative.
- Theoretical guarantees rely on assumptions such as the NTK regime and Lipschitz transition dynamics that do not strictly hold for practical deep networks.
- Performance is sensitive to the Lagrange multiplier lambda, and excessively large alpha (alpha=20) degrades performance.
Experimental setup
Datasets: D4RL halfcheetah D4RL hopper D4RL walker2d D4RL antmaze
Benchmarks: D4RL Gym-MuJoCo locomotion D4RL AntMaze
Metrics: D4RL normalized score
Baselines: BCQ BEAR DT AWAC OneStep RL TD3BC CQL IQL SPOT
Builds on: TD3+BC IQL BCQ TD3
Closest papers by meaning
- Constrained Latent Action Policies for Model-Based Offline Reinforcement Learning (NeurIPS 2024 · similarity 0.91)
- Adaptive Advantage-Guided Policy Regularization for Offline Reinforcement Learning (ICML 2024 · similarity 0.91)
- Exclusively Penalized Q-learning for Offline Reinforcement Learning (NeurIPS 2024 · similarity 0.91)
- A Perspective of Q-value Estimation on Offline-to-Online Reinforcement Learning (AAAI 2024 · similarity 0.90)
- On the Complexity of Offline Reinforcement Learning with Q*-Approximation and Partial Coverage (NeurIPS 2026 · similarity 0.90)
- A2PO: Towards Effective Offline Reinforcement Learning from an Advantage-aware Perspective (NeurIPS 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.