Research map / Multi-agent systems and games / LLM agents

CaP-X: A Framework for Benchmarking and Improving Coding Agents for Robot Manipulation

ICML 2026 · Multi-agent systems and games › LLM agents · approach: GUI agent memory and trajectories · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces CaP-X, a framework (CaP-Gym, CaP-Bench, CaP-Agent0, CaP-RL) for benchmarking and improving code-as-policy coding agents on robot manipulation, showing success depends on primitive abstraction and that multi-turn, visual differencing, skill synthesis and ensembling help.

Problem

Code-as-Policy robot agents are often evaluated with high-level human-designed primitives, so it is unclear how much performance comes from the agent versus designer scaffolding; their capability over low-level primitives is underexplored.

Approach

Systematically vary primitive abstraction (high-level vs low-level APIs), interaction (single vs multi-turn), and visual grounding (raw images vs text via a Visual Differencing Module) in a Gymnasium-style code environment; then build a training-free agent (CaP-Agent0) with VDM, auto-synthesized skill library and parallel multi-model ensembling, and RL-post-train a 7B coder with GRPO (CaP-RL).

Key results

• Across 12 models on CaP-Bench, success rates rise with primitive abstraction; no frontier model matched human expert single-turn success (88.5%).
• CaP-RL on Qwen2.5-Coder-7B raised Cube Lift sim success from 25% to 80% and real-world Cube Lift to 84% on Franka (N=25).
• CaP-Agent0 achieves higher average success than OpenVLA, pi0 and pi0.5 on LIBERO-PRO perturbation tasks (e.g., 0.22/0.18 Pos/Task on libero-object vs pi0.5 0.17/0.01).

Surprising findings

• Adding raw RGB images to the agent's context (M2) degraded performance relative to text-only feedback (M1).
• Low-level primitives with multi-turn feedback and visual differencing can reach parity with high-level multi-turn performance.

Limitations

• Primary CaP-Bench analysis uses only 7 simulation tasks with 100 trials each.
• Human reference is a small group of 7 paper authors; comparisons rely on this reference.
• CaP-RL trained only on three privileged-state tasks; programmatic control is brittle for contact-rich behaviors.

Experimental setup

Datasets: Robosuite LIBERO-PRO BEHAVIOR

Benchmarks: CaP-Bench (7 tasks) LIBERO-PRO BEHAVIOR

Metrics: task success rate code compilation success rate average dense reward navigation success rate

Baselines: OpenVLA pi0 pi0.5 Human expert code S3 single-turn baseline

Builds on: Code-as-Policy GRPO Gymnasium LIBERO-PRO Robosuite

Closest papers by meaning

• Towards Reliable Code-as-Policies: A Neuro-Symbolic Framework for Embodied Task Planning (NeurIPS 2025 · similarity 0.90)
• ManiTaskGen: A Comprehensive Task Generator for Benchmarking and Improving Vision-Language Agents on Embodied Decision-Making (NeurIPS 2026 · similarity 0.89)
• Efficient Skill Grounding via Code Refactoring with Small Language Models (ICML 2026 · similarity 0.89)
• RoboCodeX: Multimodal Code Generation for Robotic Behavior Synthesis (ICML 2024 · similarity 0.88)
• RoboCerebra: A Large-scale Benchmark for Long-horizon Robotic Manipulation Evaluation (NeurIPS 2025 · similarity 0.88)
• VLA-REPLICA: A Low-Cost, Reproducible Benchmark for Real-World Evaluation of Vision-Language-Action Models (NeurIPS 2026 · similarity 0.88)