Research map / Multi-agent systems and games / LLM agents
SEEA-R1: Tree-Structured Reinforcement Fine-Tuning for Self-Evolving Embodied Agents
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SEEA-R1, an RFT framework combining MCTS-based Tree-GRPO for policy learning with a multimodal generative reward model (MGRM) that enables self-evolving embodied agents, evaluated on ALFWorld and EmbodiedEval.
Problem
Reinforcement fine-tuning for embodied agents suffers from sparse delayed rewards in long-horizon multi-step tasks and reliance on handcrafted, environment-specific reward functions that do not generalize.
Approach
Integrates Monte Carlo Tree Search into GRPO (Tree-GRPO) to convert sparse outcome rewards into step-wise Q-value process signals, and replaces handcrafted rewards with a multimodal generative reward model (MGRM) that predicts success/continue/failure, trained with SFT (with GT) or GRPO with TTRL-style majority-vote pseudo-labels (without GT) in an iterative self-evolution loop.
Key results
- On ALFWorld unseen (MLLM setting), SEEA-R1 w/ GT reward reaches 46% average success vs 24% for GPT-4o and 11% for Qwen2.5-VL.
- Without GT reward (MGRM self-supervised), SEEA-R1 reaches 44% average success on ALFWorld unseen MLLM setting.
- On EmbodiedEval, SEEA-R1 (w/ GT reward) achieves 19.88% overall vs 18.29% for Qwen2.5-VL-7B and 16.77% for MCTS+SFT/DPO.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| train with Tree-GRPO and MGRM without ground-truth rewards | SEEA-R1 · ALFWorld test, text-only | success rate: 80.30\% | Helped |
| train with Tree-GRPO | Qwen2.5-VL-7B · EmbodiedEval | overall success rate: 18.29\% -> 19.88\% | Helped |
| use Tree-GRPO instead of MCTS + DPO (ablation) | Qwen2.5-VL-7B · ALFWorld unseen | peak success rate: 25.37\% -> 32.46\% | Helped |
| use Tree-GRPO instead of MCTS + SFT (ablation) | Qwen2.5-VL-7B · ALFWorld unseen | peak success rate: 16.79\% -> 32.46\% | Helped |
| increase training to 30 iterations | Tree-GRPO · ALFWorld unseen | success rate: 11.57\% to 39.48\% | Helped |
| increase sample and batch sizes to Sample=1024, Batch=512 (ablation) | Tree-GRPO · ALFWorld unseen | final success rate: 41.79\% | Helped |
| use Sample=128 and Batch=32 (ablation) | Tree-GRPO · ALFWorld unseen | peak success rate: 25.75\% | Hurt |
| use a self-supervised MGRM instead of persistent ground-truth rewards (ablation) | SEEA-R1 · ALFWorld | success rate | Helped |
| use supervised MGRM (ablation) | SEEA-R1 · ALFWorld | success rate | Helped |
| replace the trained MGRM with a frozen MGRM (ablation) | SEEA-R1 · ALFWorld | success rate | Hurt |
| use ReAct-style single-path inference without MCTS search | SEEA-R1 · ALFWorld | success rate: 11.57 -> 36.19 | Helped |
| use ReAct-style single-path inference without MCTS search | SEEA-R1 · ALFWorld | total tokens (test): 10{,}701{,}255 -> 6{,}554{,}090 | Helped |
Surprising findings
- Inference with single-path ReAct sampling achieves 3.1x higher success than the base model while using 38.8% fewer tokens.
- Tree-GRPO improvements on EmbodiedEval over the base model are small (19.88% vs 18.29%).
Limitations
- Fails to fully address highly dynamic and unpredictable real-world environments.
- Experiments limited to a single benchmark domain (ALFWorld) plus small real-world trial; scaling to larger models and more environments left to future work.
- World-model-based feedback (Kling) was found unreliable due to hallucinations and slow generation.
Experimental setup
Datasets: ALFWorld EmbodiedEval
Benchmarks: ALFWorld EmbodiedEval MMBench MMStar MMMU HallusionBench AI2D OCRBench
Metrics: success rate average steps total tokens average success rate
Baselines: GPT-4o Qwen2.5-VL-7B Florence-2 Idefics-2 MiniGPT-4 InstructBLIP RL4VLM Vision-R1-7B Ocean-R1 Qwen-VL-Max LLaVA-NEXT-72B LLaVA-OneVision-72B
Builds on: GRPO DeepSeek-R1 Monte Carlo Tree Search DAPO TTRL ReAct
Closest papers by meaning
- Self-Improving Embodied Foundation Models (NeurIPS 2025 · similarity 0.89)
- Embodied-R1.5: Evolving Physical Intelligence via Embodied Foundation Models (NeurIPS 2026 · similarity 0.89)
- Efficient Language-instructed Skill Acquisition via Reward-Policy Co-Evolution (AAAI 2025 · similarity 0.88)
- Diving into Self-Evolving Training for Multimodal Reasoning (ICML 2025 · similarity 0.88)
- MAGE: Multi-Agent Self-Evolution with Co-Evolutionary Knowledge Graphs (NeurIPS 2026 · similarity 0.88)
- CoMAS: Co-Evolving Multi-Agent Systems via Interaction Rewards (ICLR 2026 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.