Research map / Multi-agent systems and games / Cooperative multi-agent RL
M³HF: Multi-agent Reinforcement Learning from Multi-phase Human Feedback of Mixed Quality
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes M³HF, a MARL framework that pauses training for multi-phase human feedback of mixed quality, parses it with an LLM into predefined reward templates, and weights reward components via decay and performance-based adjustment, evaluated on macro-action Overcooked.
Problem
Reward design in multi-agent RL is hard: sparse or complex rewards lead to slow convergence or misaligned coordinated behaviors, and existing human-feedback methods for MARL are offline, single-phase, or require expert demonstrations.
Approach
Agents periodically pause training to generate rollouts; humans (experts or non-experts) give feedback on them; an LLM parses the feedback per agent and instantiates predefined reward templates; the reward pool is combined with weights that decay over generations and are boosted or reduced based on the change in original-task performance.
Key results
- M³HF outperforms MAPPO, IPPO and the macro-action baseline across Overcooked layouts A, B, C for both salad recipes, per the learning curves.
- On Overcooked-B Lettuce-Onion-Tomato, full M³HF reaches 102.7 ± 10.8 average return versus 68.7 ± 4.1 for LLM parsing without weight adjustment and 45.3 ± 5.2 for raw feedback.
- At iteration 400, M³HF reaches 164.8 ± 1.2 versus 68.9 ± 10.1 for the best IRAT variant and 19.2 ± 4.5 for IPPO.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| parse human feedback into structured rewards instead of using raw feedback (ablation) | M³HF on IPPO · Overcooked-B, Lettuce-Onion-Tomato Salad | Average Return: 45.3 ± 5.2 -> 68.7 ± 4.1 | Helped |
| add performance-based weight adjustment to parsed feedback rewards (ablation) | M³HF on IPPO · Overcooked-B, Lettuce-Onion-Tomato Salad | Average Return: 68.7 ± 4.1 -> 102.7 ± 10.8 | Helped |
| use multi-phase feedback instead of feedback only once at the start (ablation) | M³HF on IPPO · Overcooked-B, Lettuce-Onion-Tomato Salad | Average Return: 43.1 ± 10.3 -> 102.7 ± 10.8 | Helped |
| add an intrinsic reward for picking up or chopping any ingredient (ablation) | IPPO with IRAT-rw_1 · Overcooked-B, Lettuce-Onion-Tomato Salad, iteration 1000 | Average Return: 27.4 ± 4.9 -> 94.9 ± 10.7 | Helped |
| add an intrinsic reward for reaching the knife after picking up an ingredient (ablation) | IPPO with IRAT-rw_2 · Overcooked-B, Lettuce-Onion-Tomato Salad, iteration 1000 | Average Return: 27.4 ± 4.9 -> 34.5 ± 14.0 | Helped |
| add an intrinsic reward for chopping an ingredient after reaching the knife (ablation) | IPPO with IRAT-rw_3 · Overcooked-B, Lettuce-Onion-Tomato Salad, iteration 1000 | Average Return: 27.4 ± 4.9 -> 33.8 ± 9.9 | Helped |
| use M³HF instead of the intrinsic-reward variants (ablation) | M³HF and IRAT on IPPO · Overcooked-B, Lettuce-Onion-Tomato Salad, iteration 400 | Average Return: 68.9 ± 10.1, 1.1 ± 2.1, 10.8 ± 9.1 -> 164.8 ± 1.2 | Helped |
| provide erroneous or irrelevant feedback at each generation | M³HF on IPPO · Overcooked-B, Lettuce-Onion-Tomato Salad | Average Return | Hurt |
| replace human feedback with VLM-generated feedback | M³HF · Overcooked | Average Return | No clear effect |
Surprising findings
- VLM (Gemini-1.5-Pro) feedback yielded little benefit over the baseline because it lacked specific, translatable guidance.
- Low-quality feedback caused performance only slightly below IPPO rather than degrading it significantly.
Limitations
- Evaluated only in a single environment family (macro-action Overcooked) with three agents.
- Human feedback is simulated for the mixed-quality experiments rather than collected from real users; the robustness proposition relies on assumptions such as convergence and accurate performance estimation.
- Reliance on predefined reward templates limits expressiveness of the reward functions.
Experimental setup
Datasets: Overcooked (macro-action variant, layouts A/B/C)
Benchmarks: Overcooked-A Overcooked-B Overcooked-C
Metrics: mean episode return
Baselines: MAPPO IPPO Mac-IAICC/Mac-CAC macro-action baseline IRAT Raw Feedback Single-phase feedback VLM feedback
Builds on: IPPO MAPPO EUREKA PbMARL Markov Game
Closest papers by meaning
- Adaptive Collaboration with Humans: Metacognitive Policy Optimization for Multi-Agent LLMs with Continual Learning (ICLR 2026 · similarity 0.88)
- GUIDE: Real-Time Human-Shaped Agents (NeurIPS 2024 · similarity 0.88)
- O-MAPL: Offline Multi-agent Preference Learning (ICML 2025 · similarity 0.88)
- Personalizing Reinforcement Learning from Human Feedback with Variational Preference Learning (NeurIPS 2024 · similarity 0.88)
- Uni-RLHF: Universal Platform and Benchmark Suite for Reinforcement Learning with Diverse Human Feedback (ICLR 2024 · similarity 0.88)
- DUO: Diverse, Uncertain, On-Policy Query Generation and Selection for Reinforcement Learning from Human Feedback (AAAI 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.