Research map / Vision-language and multimodal / Multimodal LLMs
ClevrSkills: Compositional Language And Visual Reasoning in Robotics
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces ClevrSkills, a ManiSkill2-based robotics benchmark of 33 multi-modal-prompted tasks in three compositional levels plus a 330k-trajectory dataset, and shows that current VLM-based policies (JAT, Octo, RoboFlamingo, StreamRoboLM) fail at zero-shot composition.
Problem
Existing compositional reasoning benchmarks (CLEVR, bAbI, GSM8K) do not involve actions, and robotics benchmarks either test only low-level manipulation or abstract away skills with oracles, so compositional generalization of VLM-based robot policies is not measured in a closed loop.
Approach
A curriculum of 33 simulated manipulation tasks (L0 skills, L1 compositions, L2 complex compositions) with multimodal prompts, scripted solver-based oracle policies generating 330k trajectories with dense language, bounding box, and reward annotations, used to test zero-shot and fine-tuned compositional generalization of VLM-based robot policies.
Key results
- On L0 tasks with unseen seeds, StreamRoboLM (OPT) reaches 62.91% success versus 23.75% for JAT, 35.41% for RoboFlamingo and 34.16% for Octo.
- Zero-shot on L1 tasks, all models except Octo score 0-0.33% success; Octo reaches 0.33%.
- On L2 after fine-tuning, Octo reaches 5.83% success, the best among models, while the oracle achieves 100%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Fine-tune JAT on L1 tasks after L0 training | JAT · L1 | success rate: 0.0 -> 0.60 | Helped |
| Fine-tune RoboFlamingo on L1 tasks after L0 training | RoboFlamingo · L1 | success rate: 0.0 -> 1.00 | Helped |
| Fine-tune Octo on L1 tasks after L0 training | Octo · L1 | success rate: 0.33 -> 5.00 | Helped |
| Fine-tune StreamRoboLM (Opt) on L1 tasks after L0 training | StreamRoboLM (Opt) · L1 | success rate: 0.0 -> 3.33 | Helped |
| Fine-tune StreamRoboLM (Llama3) on L1 tasks after L0 training | StreamRoboLM (Llama3) · L1 | success rate: 0.0 -> 3.33 | Helped |
| Fine-tune JAT on L2 tasks | JAT · L2 | success rate: 0.83 -> 0.0 | Hurt |
| Fine-tune RoboFlamingo on L2 tasks | RoboFlamingo · L2 | success rate: 0.0 -> 1.66 | Helped |
| Fine-tune Octo on L2 tasks | Octo · L2 | success rate: 0.83 -> 5.83 | Helped |
| Fine-tune StreamRoboLM (Opt) on L2 tasks | StreamRoboLM (Opt) · L2 | success rate: 0.0 -> 4.16 | Helped |
| Fine-tune StreamRoboLM (Llama3) on L2 tasks | StreamRoboLM (Llama3) · L2 | success rate: 0.0 -> 3.33 | Helped |
| Use Llama3.2 3B instead of OPT1B as StreamRoboLM base model | StreamRoboLM · L0 unseen objects/textures | success rate: 41.66 -> 55.41 | Helped |
| Use Llama3.2 3B instead of OPT1B as StreamRoboLM base model | StreamRoboLM · L0 unseen seeds | success rate: 62.91 -> 62.50 | No clear effect |
Surprising findings
- JAT performs similarly on seen and unseen objects/textures, which the authors attribute to its smaller vision backbone avoiding overfitting.
- Models obtain decent average reward even when binary success is low, suggesting strict success criteria contribute to poor results.
Limitations
- Benchmark is fully simulated with no real-world counterpart.
- Only one robot embodiment (xArm 6 with vacuum gripper) is used by default.
- Tasks are structured and constrained rather than abstract or free-form.
Experimental setup
Datasets: ClevrSkills
Benchmarks: ClevrSkills L0 ClevrSkills L1 ClevrSkills L2
Metrics: success rate average reward reward per step
Baselines: JAT Octo RoboFlamingo StreamRoboLM
Builds on: ManiSkill2 VIMA RoboFlamingo JAT Octo
Closest papers by meaning
- JRDB-Reasoning: A Difficulty-Graded Benchmark for Visual Reasoning in Robotics (AAAI 2026 · similarity 0.89)
- RoboDreamer: Learning Compositional World Models for Robot Imagination (ICML 2024 · similarity 0.88)
- RoboProcessBench: Benchmarking Process-Aware Understanding in Visual-Language Robotic Manipulation (NeurIPS 2026 · similarity 0.88)
- Seeing Across Views: Benchmarking Spatial Reasoning of Vision-Language Models in Robotic Scenes (ICLR 2026 · similarity 0.88)
- Unveiling the Compositional Ability Gap in Vision-Language Reasoning Model (NeurIPS 2025 · similarity 0.88)
- RoboCerebra: A Large-scale Benchmark for Long-horizon Robotic Manipulation Evaluation (NeurIPS 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.