Research map / Multi-agent systems and games / Human-AI and goal-conditioned
IndustryEQA: Pushing the Frontiers of Embodied Question Answering in Industrial Scenarios
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces IndustryEQA, a safety-focused embodied question answering benchmark of 1,344 QA pairs over Isaac Sim warehouse episodic-memory videos, and evaluates blind LLMs, multi-frame VLLMs and video VLLMs with LLM-judged direct and reasoning scores.
Problem
Existing EQA benchmarks focus on household scenes and neglect safety-critical industrial settings and reasoning justifications, limiting assessment of agent readiness for industrial use.
Approach
Builds warehouse episodic-memory videos in NVIDIA Isaac Sim with humans and hazard configurations inspired by OSHA guidance, generates QA pairs with Gemini 2.5 Pro and refines them with human experts, adding direct and reasoning answers scored by an LLM judge.
Key results
- o4-mini achieves the highest scores, with Direct Score 69.67% on small warehouses and Reasoning Score 67.71% on small warehouses.
- Blind text-only LLMs score roughly 36-44% on direct answers, well below multi-frame and video VLLMs that often exceed 65%.
- Gemini-2.5-Flash (video VLLM) reaches 70.24% direct score on large warehouses, the top direct score in that column.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Provide sampled video frames instead of text-only questions | VLLMs versus blind LLMs · IndustryEQA | Direct Score (%) | Helped |
| Increase the number of sampled video frames (ablation) | Multi-Frame VLLMs · IndustryEQA, Small warehouse | Direct Score | Helped |
| Increase the number of sampled video frames (ablation) | Multi-Frame VLLMs · IndustryEQA, Small warehouse | Reasoning Score | Helped |
| Increase the number of sampled video frames (ablation) | Multi-Frame VLLMs · IndustryEQA, Large warehouse | Direct Score | Helped |
| Increase the number of sampled video frames (ablation) | Multi-Frame VLLMs · IndustryEQA, Large warehouse | Reasoning Score | Helped |
| Use GPT-4o-mini rather than Gemini-2.0-Flash as the LLM judge (ablation) | IndustryEQA evaluation · IndustryEQA | Direct Score | Mixed |
| Use Gemini-2.0-Flash rather than GPT-4o-mini as the LLM judge (ablation) | IndustryEQA evaluation · IndustryEQA | Reasoning Score | Helped |
Surprising findings
- Reasoning scores are consistently lower than direct scores across models, showing justification is harder than factual answering.
- Temporal understanding is the hardest category while attribute recognition is the easiest.
Limitations
- Limited art asset variety restricts warehouse layouts and some hazard types.
- Simulation events are conservative; no event-driven dynamics, time-of-day changes or extreme events.
- Only 76 episodic videos, one per scenario, limiting diversity.
Experimental setup
Datasets: IndustryEQA
Benchmarks: IndustryEQA
Metrics: Direct Score Reasoning Score (LLM-Match normalized 1-5)
Baselines: GPT-4o Gemini-2.0-Flash DeepSeek-R1 DeepSeek-V3-0324 LLaMA-4-Scout Qwen2.5-VL-72B InternVL2.5-78B Claude-3.5-Haiku GPT-4.1 o4-mini Gemini-2.5-Flash
Builds on: OpenEQA Isaac Sim Embodied Question Answering
Closest papers by meaning
- OpenEQA: Embodied Question Answering in the Era of Foundation Models (CVPR 2024 · similarity 0.91)
- Multi-modal Situated Reasoning in 3D Scenes (NeurIPS 2024 · similarity 0.86)
- AGENTSAFE: Benchmarking the Safety of Embodied Agents on Hazardous Instructions (CVPR 2026 · similarity 0.85)
- IS-Bench: Evaluating Interactive Safety of VLM-Driven Embodied Agents in Daily Household Tasks (AAAI 2026 · similarity 0.85)
- ECBench: Can Multi-modal Foundation Models Understand the Egocentric World? A Holistic Embodied Cognition Benchmark (CVPR 2025 · similarity 0.85)
- Embodied Scene Understanding for Vision Language Models via MetaVQA (CVPR 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.