Research map / Multi-agent systems and games / Human-AI and goal-conditioned
Integrating Suboptimal Human Knowledge with Hierarchical Reinforcement Learning for Large-Scale Multiagent Systems
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes hhk-MARL, a hierarchical MARL framework that injects suboptimal human knowledge encoded as trainable fuzzy-logic rules, integrated via hypernetworks plus a graph-based group controller, and tests it on SMAC with IQL, QMIX, and Qatten.
Problem
Learning efficient coordination from scratch is hard in large-scale multi-agent systems due to exponential joint state-action spaces, sparse rewards, and start-up problems; existing human-knowledge transfer often requires high-quality step-by-step demonstrations.
Approach
Represent abstract, suboptimal human knowledge as trainable fuzzy logic rules at the top level, let agents decide how much to use it through a hypernetwork-based knowledge integration (a "CPG"-like mechanism), and add a graph-based group controller that weights agents by cooperation tendencies during training.
Key results
- Knowledge-guided hhkIQL matches the best baseline QMIX in performance and baseline IQL fails in 27m vs 30m and IQL and QMIX fail in 35m vs 40m, where the proposed approaches overcome start-up issues.
- Using only human-knowledge fuzzy rules yields a 0% win rate, but integrated with MARL it accelerates training and improves final performance across IQL, QMIX, and Qatten.
- Random (inappropriate) knowledge still outperforms the IQL baseline, indicating the knowledge integration module filters negative knowledge.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add hhk-MARL to IQL, QMIX, or Qatten | IQL, QMIX, and Qatten · SMAC scenarios: 5m vs 6m, 10m vs 11m, 18m vs 20m, 27m vs 30m, and 35m vs 40m | win rate | Helped |
| Add hhk-MARL to IQL | IQL · SMAC 27m vs 30m | win rate | Helped |
| Add hhk-MARL to IQL and QMIX | IQL and QMIX · SMAC 35m vs 40m | win rate | Helped |
| Add the group controller (ablation) | IQL · SMAC 10m vs 11m | convergence rate | Helped |
| Add human knowledge integration (ablation) | IQL · SMAC 10m vs 11m | performance | Helped |
| Add trainable weights to fuzzy logic rules (ablation) | IQL with human knowledge · SMAC 10m vs 11m | convergence rate | Helped |
| Increase the human knowledge rule set from 1 rule to 3 or 5 rules (ablation) | hhkIQL · SMAC 10m vs 11m | final performance | Helped |
| Use 1 human knowledge rule rather than 3 or 5 rules (ablation) | hhkIQL · SMAC 10m vs 11m | learning speed | Helped |
| Replace human preference values with random values (ablation) | hhkIQL · SMAC 10m vs 11m | performance | Helped |
Surprising findings
- Fuzzy rules alone achieve a 0% win rate, yet combined with MARL they still substantially improve learning.
- Randomized (inappropriate) human knowledge still outperforms the baseline IQL, suggesting the benefit is partly robust to knowledge quality.
Limitations
- Agents are assumed homogeneous because modules are shared among agents; heterogeneous agents are not covered.
- Only verified on SMAC; other domains with more agents are untested due to compute limits.
- Fuzzy logic is a relatively primitive knowledge representation, and the paper does not establish what knowledge is most useful.
Experimental setup
Datasets: StarCraft Multi-agent Challenge (SMAC) scenarios 5m vs 6m, 10m vs 11m, 18m vs 20m, 27m vs 30m, 35m vs 40m
Benchmarks: SMAC 5m vs 6m SMAC 10m vs 11m SMAC 18m vs 20m SMAC 27m vs 30m SMAC 35m vs 40m
Metrics: win rate learning curve convergence speed
Baselines: IQL QMIX Qatten
Builds on: QMIX Qatten IQL fuzzy logic knowledge representation
Closest papers by meaning
- Cautiously-Optimistic Knowledge Sharing for Cooperative Multi-Agent Reinforcement Learning (AAAI 2024 · similarity 0.87)
- M³HF: Multi-agent Reinforcement Learning from Multi-phase Human Feedback of Mixed Quality (ICML 2025 · similarity 0.87)
- Adaptive Collaboration with Humans: Metacognitive Policy Optimization for Multi-Agent LLMs with Continual Learning (ICLR 2026 · similarity 0.86)
- Learning to Deliberate: Meta-policy Collaboration for Agentic LLMs with Multi-agent Reinforcement Learning (AAAI 2026 · similarity 0.86)
- Efficient Adaptation in Mixed-Motive Environments via Hierarchical Opponent Modeling and Planning (ICML 2024 · similarity 0.86)
- HMARL-CBF – Hierarchical Multi-Agent Reinforcement Learning with Control Barrier Functions for Safety-Critical Autonomous Systems (NeurIPS 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.