Research map / Vision-language and multimodal / Robot action and navigation
PaLM-E: An Embodied Multimodal Language Model
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add pretrained LLM (vs. non-pretrained) for state inputs (ablation) | PaLM-E state-input model · TAMP 6 objects | success rate p1: 56.5 to 95.3 | Helped |
| co-train on full mixture (vs. TAMP-only data) with ViT-4B (ablation) | PaLM-E ViT-4B · TAMP 1% data | success rate p1: 30.6 to 74.1 | Helped |
| use object-centric OSRT encoder instead of ViT variants (ablation) | PaLM-E TAMP input representation · TAMP 1% data | success rate p1: OSRT 82.5 vs ViT-4B full mixture 74.1 | Helped |
| remove entity referrals from state inputs (ablation) | PaLM-E state model · TAMP 8 objects | success rate p2: 89.3 to 49.3 | Hurt |
| freeze LLM and train only input encoders (vs unfrozen) (ablation) | PaLM-E-12B on VQAv2 · VQAv2 test-dev | accuracy: 77.7 unfrozen vs 70.3 frozen | Hurt |
| co-train on full mixture (vs single robot data) with pretrained LLM+ViT (ablation) | PaLM-E-12B Language-Table Task 1 · Language-Table sim, 10 demos | success rate: 20.0 to 70.0 | Helped |
| scale model from 12B to 84B | PaLM-E full mixture · Language-Table sim, Tasks 1-3 | success rate: Task 1 80.0 to 90.0; Task 2 40 demo 58.8 to 53.8; Task 3 80 demo 56.3 to 64.4 | Mixed |
| add pretrained LLM+ViT and full mixture (vs single robot) for failure detection (ablation) | PaLM-E-12B mobile manipulation · Failure detection F1 | F1: 0.54 to 0.91 | Helped |
| unfreeze LLM (end-to-end training) vs frozen LLM for affordance prediction (ablation) | PaLM-E-12B mobile manipulation · Affordance prediction F1 | F1: 0.91 frozen vs 0.87 unfrozen full mixture | Hurt |
| scale LLM with end-to-end multimodal training (ablation) | PaLM-E unfrozen models · General NLG tasks | relative NLG degradation: -87.3% (12B) to -3.8% (562B) | Helped |
| scale generalist model to 562B (OK-VQA SOTA claim) | PaLM-E · OK-VQA val | accuracy: 66.1 vs PaLI 64.5 | Helped |
| PaLM-E outperforms zero-shot PaLI on TAMP VQA | PaLM-E vs PaLI · TAMP embodied VQA | success rate q2: 98.2 vs 0.0 | Helped |
Closest papers by meaning
- EmbodiedBench: Comprehensive Benchmarking Multi-modal Large Language Models for Vision-Driven Embodied Agents (ICML 2025 · similarity 0.91)
- BEAR: Dissecting Embodied Abilities in Multimodal Language Models through Skill-level Evaluation and Diagnosis (ICML 2026 · similarity 0.90)
- NavBench: Probing Multimodal Large Language Models for Embodied Navigation (NeurIPS 2025 · similarity 0.90)
- RoboMP$^2$: A Robotic Multimodal Perception-Planning Framework with Multimodal Large Language Models (ICML 2024 · similarity 0.89)
- MultiPLY: A Multisensory Object-Centric Embodied Large Language Model in 3D World (CVPR 2024 · similarity 0.89)
- ManipLLM: Embodied Multimodal Large Language Model for Object-Centric Robotic Manipulation (CVPR 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.