Research map / Multi-agent systems and games / Human-AI and goal-conditioned
Vision-Based Generic Potential Function for Policy Alignment in Multi-Agent Reinforcement Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes V-GEPF, a hierarchical vision-based reward shaping method where a CLIP-based generic potential function, selected every 50 epochs by a vLLM (MiniCPM-Llama3-V) from a pool of instructions, shapes MAPPO's reward in Google Research Football.
Problem
MARL policies in complex, long-horizon sparse-reward multi-agent tasks often diverge from human common sense, and hand-designed potential or rule-based dense rewards are labor-intensive and lack high-level semantic understanding.
Approach
Use a frozen VLM (CLIP) image-text cosine similarity as a generic potential function for potential-based reward shaping, and a vLLM that reads replay videos, training records, and a pool of instructions to adaptively switch which potential function is active during training.
Key results
- In GRF 11 vs 11 hard, V-GEPF significantly outperforms MAPPO, MAPPO-xT, HAPPO, and IPPO in win rate, with baselines plateauing around 15 million steps.
- In GRF 11 vs 11 easy, V-GEPF reaches a higher final win rate than all baselines, though HAPPO, MAPPO, and MAPPO-xT converge faster.
- Coefficient rho=0.5 was chosen from {1, 0.5, 0.1} in the 11 vs 11 hard scenario; rho=1 yielded a low win rate.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add V-GEPF hierarchical potential-based reward shaping | MAPPO · GRF 11 vs 11 easy | win rate | Helped |
| Add V-GEPF hierarchical potential-based reward shaping | MAPPO · GRF 11 vs 11 hard | win rate | Helped |
| Add V-GEPF hierarchical potential-based reward shaping | MAPPO · GRF academy counterattack easy | win rate | Helped |
| Add V-GEPF hierarchical potential-based reward shaping | MAPPO · GRF academy counterattack hard | win rate | Helped |
| Add adaptive skill selection across training stages (ablation) | V-GEPF · GRF 11 vs 11 | policy style balance | Helped |
| Set reward coefficient ρ to 1 (ablation) | V-GEPF · GRF 11 vs 11 hard | win rate | Hurt |
| Set reward coefficient ρ to 0.1 instead of 0.5 (ablation) | V-GEPF · GRF 11 vs 11 hard | win rate | Hurt |
| Use a fixed VLM-based potential function instead of adaptive skill selection (ablation) | V-GEPF · GRF 11 vs 11 | policy style balance | Hurt |
Surprising findings
- A fixed VLM-based potential function produces policies whose style strongly matches its instruction, which the authors argue motivates adaptive selection.
- Baselines converge faster in the easy task, with V-GEPF's advantage appearing only in final win rate.
Limitations
- The vLLM selection and potential function pool are hand-designed, and the approach is evaluated only in one domain (football).
- Potential-based reward with a dense signal requires careful tuning of the coefficient rho, which the authors show is sensitive.
- Slower convergence than some baselines in the easy task.
Experimental setup
Datasets: Google Research Football scenarios FIFA World Cup 2018 event data (for xT baseline)
Benchmarks: Google Research Football 11 vs 11 easy Google Research Football 11 vs 11 hard Google Research Football academy counterattack easy/hard
Metrics: average win rate radar-chart style metrics (shots, shot success, passes, pass
Baselines: MAPPO IPPO HAPPO MAPPO-xT
Builds on: MAPPO potential-based reward shaping CLIP HAPPO generalized advantage estimation
Closest papers by meaning
- Code as Reward: Empowering Reinforcement Learning with VLMs (ICML 2024 · similarity 0.87)
- Centralized Reward Agent for Knowledge Sharing and Transfer in Multi-Task Reinforcement Learning (NeurIPS 2025 · similarity 0.87)
- VLMs-Guided Representation Distillation for Efficient Vision-Based Reinforcement Learning (CVPR 2025 · similarity 0.87)
- GoalLadder: Incremental Goal Discovery with Vision-Language Models (NeurIPS 2025 · similarity 0.86)
- All Roads Lead to Rome: Incentivizing Divergent Thinking in Vision-Language Models (CVPR 2026 · similarity 0.86)
- COVR: Collaborative Optimization of VLMs and RL Agent for Visual-Based Control (AAAI 2026 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.