Research map / Vision-language and multimodal / Multimodal LLMs
ESCA: Contextualizing Embodied Agents via Scene-Graph Generation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes ESCA, an embodied-agent framework that augments MLLM perception with open-domain spatial-temporal scene graphs from SGClip, a CLIP-based model trained on 87K self-labeled videos via neurosymbolic weak supervision.
Problem
MLLM-based embodied agents make frequent perception errors because they do not reliably link low-level visual features to high-level textual semantics, causing weak grounding in environments like EmbodiedBench.
Approach
Selective scene-graph contextualization: the MLLM extracts task-relevant concepts, a CLIP-based promptable scene-graph model (SGClip) scores entities, attributes and relations on grounded segments, and the probabilistic scene graph is summarized back into the agent prompt. SGClip is trained without human labels by aligning model-generated captions/specs with its own scene graphs via Scallop-based
Key results
- On EB-Navigation, InternVL-2.5-38B with ESCA averages 51.66% vs 47.33% base, exceeding GPT-4o base (51.33%).
- ESCA reduces the perception error rate from 69% to 30% in agent failures.
- With 5% of ActivityNet (~800 videos), SGClip reaches 92.10% accuracy, approaching InternVideo2-6B end-to-end finetuning.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add ESCA to InternVL-2.5-38B-MPO | InternVL-2.5-38B-MPO · EB-Navigation | Avg: 47.33 -> 51.66 | Helped |
| Add ESCA to Gemini-2.0-flash | Gemini-2.0-flash · EB-Navigation | Avg: 40.68 -> 42.00 | Helped |
| Add ESCA to Qwen2.5-VL-72B-Ins | Qwen2.5-VL-72B-Ins · EB-Navigation | Avg: 44.99 -> 49.33 | Helped |
| Add ESCA to GPT-4o | GPT-4o · EB-Navigation | Avg: 51.33 -> 54.67 | Helped |
| Add ESCA to InternVL-2.5-38B-MPO | InternVL-2.5-38B-MPO · EB-Manipulation | Avg: 19.31 -> 24.30 | Helped |
| Add ESCA to Gemini-2.0-flash | Gemini-2.0-flash · EB-Manipulation | Avg: 11.81 -> 21.94 | Helped |
| Add ESCA to Qwen2.5-VL-72B-Ins | Qwen2.5-VL-72B-Ins · EB-Manipulation | Avg: 4.72 -> 14.59 | Helped |
| Add ESCA to GPT-4o | GPT-4o · EB-Manipulation | Avg: 23.47 -> 34.44 | Helped |
| Reduce agent perception errors using ESCA | MLLM-based embodied agents · Embodied environments | perception error rate: 69% to 30% | Helped |
| Fine-tune SGClip with 5% of ActivityNet training data | SGClip · ActivityNet | accuracy: 92.10% | Helped |
Surprising findings
- Injecting full scene graphs can degrade performance, motivating selective grounding.
- Grounding DINO increases false positives, hurting the long-horizon navigation subset (drop to 11.66 and 6.00).
Limitations
- LLM-based high-level planning adds latency, unsuitable for real-time low-level control.
- Relies on 2D inputs and lacks 3D/point cloud representations.
- No formal mechanism to verify intermediate and final states during execution.
Experimental setup
Datasets: ESCA-Video-87K LLaVA-Video-178K ActivityNet OpenPVSG Action Genome VidVRD EmbodiedBench VLMBench
Benchmarks: EB-Navigation EB-Manipulation OpenPVSG Action Genome VidVRD ActivityNet
Metrics: success rate entity class R@1 binary relation R@10 accuracy error rate
Baselines: Grounding DINO YOLO11 CLIP InternVL-6B BIKE Text4Vis
Builds on: LASER CLIP Grounding DINO SAM2 Scallop
Closest papers by meaning
- EmbodiedBench: Comprehensive Benchmarking Multi-modal Large Language Models for Vision-Driven Embodied Agents (ICML 2025 · similarity 0.90)
- Learning to Evolve Scenes: Reasoning about Human Activities with Scene Graphs (NeurIPS 2026 · similarity 0.89)
- BEAR: Dissecting Embodied Abilities in Multimodal Language Models through Skill-level Evaluation and Diagnosis (ICML 2026 · similarity 0.88)
- VisualAgentBench: Towards Large Multimodal Models as Visual Foundation Agents (ICLR 2025 · similarity 0.88)
- UP-VLA: A Unified Understanding and Prediction Model for Embodied Agent (ICML 2025 · similarity 0.88)
- EmbodiedScan: A Holistic Multi-Modal 3D Perception Suite Towards Embodied AI (CVPR 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.