Research map / Vision-language and multimodal / Visual question answering
When Robots Should Say ''I Don't Know'': Benchmarking Abstention in Embodied Question Answering
Tao Wu, Chuhao Zhou, Guangyu Zhao, Haozhi Cao, Yewen Pu, Jianfei Yang
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add coarse abstention prompt (ablation) | Qwen2.5-VL-7B · EM-EQA | abstention recall / precision: recall 28.61 -> 75.86; precision 76.85 -> 56.84 | Mixed |
| add fine abstention prompt (ablation) | HM-EQA agent · A-EQA | abstention recall (%): 27.17 -> 99.46 (precision 56.82 -> 49.86) | Mixed |
| enable thinking mode (ablation) | Qwen3-VL-32B · EM-EQA | abstention recall (%): 31.30 -> 21.70 | Hurt |
| enable thinking mode (ablation) | Qwen3-VL-8B · EM-EQA | abstention recall (%): 33.92 -> 27.02 | Hurt |
| enable thinking mode (ablation) | Qwen3-VL-2B · EM-EQA | abstention recall (%): 23.59 -> 25.55 | No clear effect |
| enable thinking mode (ablation) | Gemini-2.5-Pro · EM-EQA | abstention recall (%): 44.13 -> 43.09 | No clear effect |
| supervised fine-tuning on abstention (ablation) | Qwen2.5-VL-7B · EM-EQA | abstention recall (%): 26.94 -> 83.27 | Improved |
| replace visual inputs with random ones in SFT (ablation) | 7B-SFT · EM-EQA | abstention recall (%): 83.27 -> 83.27 | No clear effect |
| remove visual input entirely (text-only SFT) (ablation) | 7B-SFT · EM-EQA | abstention recall (%): 83.27 -> 86.12 | No clear effect |
| scale model within Qwen families (ablation) | Qwen2.5-VL / Qwen3-VL · AbstainEQA | abstention recall: e.g. 4B 27.32 vs 7B 28.61 across families | Mixed |
| ambiguous (abstention) queries instead of answerable ones | 3D-Mem · A-EQA | success rate (%): 77.17 -> 61.41 | Hurt |
Closest papers by meaning
- OpenEQA: Embodied Question Answering in the Era of Foundation Models (CVPR 2024 · similarity 0.94)
- IndustryEQA: Pushing the Frontiers of Embodied Question Answering in Industrial Scenarios (NeurIPS 2025 · similarity 0.88)
- Implicit Intelligence - Evaluating Agents on What Users Don’t Say (ICML 2026 · similarity 0.87)
- ECBench: Can Multi-modal Foundation Models Understand the Egocentric World? A Holistic Embodied Cognition Benchmark (CVPR 2025 · similarity 0.86)
- EgoThink: Evaluating First-Person Perspective Thinking Capability of Vision-Language Models (CVPR 2024 · similarity 0.86)
- AgentHop: A Diagnostic Benchmark for Agentic Multi-Hop Scientific Question Answering (NeurIPS 2026 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
