Research map / Large language models / Retrieval and knowledge
OpenEQA: Embodied Question Answering in the Era of Foundation Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces OpenEQA, an open-vocabulary embodied question answering benchmark covering episodic-memory and active exploration settings, with an LLM-based scoring metric (LLM-Match), and benchmarks foundation-model agents against human performance.
Problem
Existing embodied QA benchmarks are closed-vocabulary, template-based, or limited to single frames or simulated scenes, so they do not test open-ended question answering about real environments with foundation models.
Approach
Reformulate EQA into episodic-memory (EM-EQA) and active (A-EQA) variants with human-written open-vocabulary questions over real scanned/recorded indoor scenes, and score free-form answers with a GPT-4 judge (LLM-Match) that correlates strongly with humans.
Key results
- GPT-4V achieves the best agent score of 49.6% LLM-Match on EM-EQA, versus 86.8% for human agents.
- LLM-Match (GPT-4 judge) reaches Spearman rho = 0.909 with human evaluators, comparable to inter-human agreement of 0.91-0.93.
- Blind GPT-4 reaches 33.5% on EM-EQA, showing that many questions can be guessed without visual context; spatial questions remain near blind-LLM performance.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use multi-frame GPT-4V instead of GPT-4 with frame captions | EM-EQA agent · EM-EQA ALL | LLM-Match: 43.6±1.1 -> 49.6±2.0 | Helped |
| Use frame captions from LLaVA-1.5 rather than a blind GPT-4 | GPT-4 EQA agent · EM-EQA ALL | LLM-Match: 33.5±1.0 -> 43.6±1.1 | Helped |
| Use frame captions from LLaVA-1.5 rather than a blind LLaMA-2 | LLaMA-2 EQA agent · EM-EQA ALL | LLM-Match: 28.3±1.0 -> 36.8±1.1 | Helped |
| Represent episodic memory with ConceptGraph rather than LLaVA-1.5 frame captions | GPT-4 EQA agent · EM-EQA ALL | LLM-Match: 43.6±1.1 -> 36.5±1.0 | Hurt |
| Represent episodic memory with SVM rather than LLaVA-1.5 frame captions | GPT-4 EQA agent · EM-EQA ALL | LLM-Match: 43.6±1.1 -> 38.9±1.0 | Hurt |
| Use GPT-4 instead of LLaMA-2 as the language model | EQA agents using LLaVA-1.5 frame captions · EM-EQA ALL | LLM-Match: 36.8±1.1 -> 43.6±1.1 | Helped |
| Use active exploration rather than the provided human-like episode history | GPT-4V EQA agent · HM3D | LLM-Match: 46.6±3.1 -> 41.8±3.2 | Hurt |
| Improve the agent from GPT-4V performance toward human performance | OpenEQA agents · EM-EQA ALL | LLM-Match: 49.6±2.0; 86.8±0.6 | Mixed |
Surprising findings
- Scene-graph-based agents (ConceptGraph, sparse voxel map) are no better than frame-caption agents, even on spatial reasoning questions.
- Blind LLMs are strong baselines, scoring 33.5% on EM-EQA with GPT-4 and 28.3% with LLaMA-2.
Limitations
- Active EQA baseline uses simple frontier exploration and scores poorly on efficiency, leaving efficient exploration open.
- Benchmark covers only navigation actions and indoor scenes (ScanNet, HM3D).
- GPT-4V results computed on a 500-question subset due to API limits.
Experimental setup
Datasets: ScanNet HM3D Gibson MMBench
Benchmarks: OpenEQA
Metrics: LLM-Match efficiency score (SPL-style) Spearman correlation with human judgement
Baselines: GPT-4 blind LLM LLaMA-2-70B blind LLM GPT-4 w/ LLaVA-1.5 captions GPT-4 w/ ConceptGraph GPT-4 w/ Sparse Voxel Map GPT-4V Human agent
Builds on: MMBench evaluation protocol ConceptGraph Socratic models frontier exploration EQA-v1
Closest papers by meaning
- IndustryEQA: Pushing the Frontiers of Embodied Question Answering in Industrial Scenarios (NeurIPS 2025 · similarity 0.91)
- ECBench: Can Multi-modal Foundation Models Understand the Egocentric World? A Holistic Embodied Cognition Benchmark (CVPR 2025 · similarity 0.87)
- Multi-modal Situated Reasoning in 3D Scenes (NeurIPS 2024 · similarity 0.87)
- Embodied Scene Understanding for Vision Language Models via MetaVQA (CVPR 2025 · similarity 0.87)
- EgoThink: Evaluating First-Person Perspective Thinking Capability of Vision-Language Models (CVPR 2024 · similarity 0.87)
- Exemplar2VQA: A Scalable Exemplar-Driven Visual Question Answering Generation Framework via Multi-Agent Coding (NeurIPS 2026 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.