Research map / Multi-agent systems and games / Human-AI and goal-conditioned
Agent-Aware Training for Agent-Agnostic Action Advising in Deep Reinforcement Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes A7, an action advising framework where a BYOL-style action-conditioned proxy feature extractor (action-BYOL) selects states for teacher queries via feature similarity, with a behavior-cloned reuse model and an intrinsic reward for advised samples.
Problem
Sampling inefficiency in deep RL; existing action advising methods decide when to request expert actions using agent-specific uncertainty (easily misled by agent imperfections) or agent-agnostic novelty (wasting advice on already-learned states).
Approach
Decide when to ask for advice by the similarity of state features from a proxy encoder trained with action-conditioned BYOL on the agent's own transitions (state, action, next state), combined with a dropout-based behavior-cloning reuse model and a decaying distance-based intrinsic reward for advised samples.
Key results
- A7 achieves the highest AUC in all eight environments, e.g., 0.85 vs 0.85 for best baseline SUA-AIR on Freeway... specifically 0.85 on Freeway vs 0.69 for SUA-AIR and 0.54 on Qbert vs 0.26 for RCMP.
- On LunarLander best test score 266.35 ± 34.12 for A7 vs 224.95 for SUA-AIR; Qbert 4096.45 vs 4024.95 for SUA-AIR.
- A7 inference time is 2.4ms, about 1.5x the no-advising baseline (self-reported limitation).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use A7 instead of the strongest listed baseline | A7 · GridWorld | AUC: 0.85 -> 0.87 | Helped |
| Use A7 instead of the strongest listed baseline | A7 · LunarLander | AUC: 0.83 -> 0.85 | Helped |
| Use A7 instead of the strongest listed baseline | A7 · Freeway | AUC: 0.69 -> 0.85 | Helped |
| Use A7 instead of the strongest listed baseline | A7 · Qbert | AUC: 0.28 -> 0.54 | Helped |
| Use A7 instead of the strongest listed baseline | A7 · Seaquest | AUC: 0.31 -> 0.37 | Helped |
| Use A7 instead of the strongest listed baseline | A7 · Pong | AUC: 0.61 -> 0.62 | Helped |
| Use A7 instead of the strongest listed baseline | A7 · Enduro | AUC: 0.56 -> 0.64 | Helped |
| Use A7 instead of the strongest listed baseline | A7 · SpaceInvaders | AUC: 0.57 -> 0.60 | Helped |
| Replace A7's contrastive advice selector with EA and set intrinsic reward to a fixed value (ablation) | A7 · six Atari scenarios | AUC | Hurt |
| Remove the intrinsic reward generator (ablation) | A7 · six Atari scenarios | AUC | Hurt |
| Increase advice budget from 5k to 25k (ablation) | A7 · Freeway and Qbert | performance: 5k to 25k | Helped |
| Increase advice budget from 15k to 35k (ablation) | A7 · Freeway | performance: 15k to 35k | No clear effect |
Surprising findings
- Agent-agnostic ANA performs poorly on Seaquest and Qbert, attributed to susceptibility to noise.
- Increasing the advice budget from 25k to 35k in Freeway and Qbert yields minimal further gain, suggesting a tipping point.
Limitations
- Requires training three models, adding time overhead.
- Limited to discrete action spaces.
- Relies on a pre-trained teacher model and assumes high-quality advice.
Experimental setup
Datasets: GridWorld LunarLander Atari Enduro Atari Freeway Atari Pong Atari Qbert Atari Seaquest Atari SpaceInvaders
Benchmarks: GridWorld LunarLander Atari Enduro Atari Freeway Atari Pong Atari Qbert Atari Seaquest Atari SpaceInvaders
Metrics: area under the learning curve (AUC) best test evaluation score
Baselines: No Advising Early Advising Random Advising IAA RCMP SUA-AIR ANA
Builds on: BYOL Double DQN Dueling DQN Random Network Distillation Bootstrapped DQN
Closest papers by meaning
- Learning Agentic Policy from Action Guidance (NeurIPS 2026 · similarity 0.88)
- ML-Agent: Reinforcing LLM Agents for Autonomous Machine Learning Engineering (ICML 2026 · similarity 0.86)
- Boosting Virtual Agent Learning and Reasoning: A Step-Wise, Multi-Dimensional, and Generalist Reward Model with Benchmark (ICML 2025 · similarity 0.85)
- PLATO: Pointer Learner for Agent and Task Openness (NeurIPS 2026 · similarity 0.85)
- Generalized Intention Modeling in Multi-Agent Reinforcement Learning (NeurIPS 2026 · similarity 0.85)
- Robust Instruction Compliance in Cooperative Multi-Agent Reinforcement Learning (NeurIPS 2026 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.