Research map / Vision-language and multimodal / Robot action and navigation
AHA: A Vision-Language-Model for Detecting and Reasoning Over Failures in Robotic Manipulation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces AHA, an open-source VLM fine-tuned on FailGen-generated simulated robot failure data to detect and explain manipulation failures in free-form language, and shows gains when plugged into reward reflection, TAMP, and zero-shot data generation pipelines.
Problem
Robots using LLMs/VLMs struggle to detect and reason about manipulation failures; prior work treats failure detection as binary success classification without explanations.
Approach
Frame failure detection as free-form reasoning: predict a yes/no sub-task success and, if failed, generate a natural-language explanation. Procedurally perturb successful RLBench demonstrations via the FailGen wrapper to create a large failure dataset for instruction-tuning a LLaVA-style VLM.
Key results
- AHA-13B reaches 0.702 binary success and 0.583 cosine similarity on the AHA test set, and 0.643 binary success on RoboFail, outperforming all compared VLMs on most metrics.
- In downstream Eureka-style reward reflection on ManiSkill, AHA-13B outperforms GPT-4o by 22.34% in task success rate.
- In PRoC3S TAMP integration, AHA-13B outperforms GPT-4o by 36.7% in task success rate.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Fine-tune LLaVA-v1.5-13B on the Aha dataset for failure reasoning | AHA-13B · Aha dataset (Test set) | ROUGE-L: 0.226 -> 0.446 | Helped |
| Fine-tune LLaVA-v1.5-13B on the Aha dataset for failure reasoning | AHA-13B · ManiSkill-Fail | Binary Success (%): 0.971 -> 1.000 | Helped |
| Fine-tune LLaVA-v1.5-13B on the Aha dataset for failure reasoning | AHA-13B · RoboFail | ROUGE-L: 0.236 -> 0.280 | Helped |
| Use AHA-13B rather than GPT-4o for reward reflection | Eureka reward generation in ManiSkill · Five ManiSkill reinforcement-learning tasks | task success rate: +22.34% | Helped |
| Use AHA-13B rather than GPT-4o for failure reasoning | PRoC3S task planning · Three TAMP tasks | success rate: +36.7% | Helped |
| Replace GPT-4V with AHA-13B in sub-task verification | Manipulate-Anything · Four RLBench tasks | task success: 5% | Helped |
| Increase Aha instruction-tuning data size (ablation) | AHA · ManiSkill-Fail | performance across four evaluation metrics: average quadratic fit gradient of 0.0022 | Helped |
Surprising findings
- GPT-4o-ICL (5-shot) achieves higher binary success on ManiSkill-Fail (0.971) than AHA-13B's comparable metrics in some cases, and AHA-7B outperforms AHA-13B on ManiSkill-Fail ROUGE-L (0.609 vs 0.600).
- AHA-13B scores 65.2 on MMBench versus 67.7 for LLaVA-13B, indicating some loss of general VQA ability after fine-tuning.
Limitations
- Outputs are closely aligned with the seven failure modes in the taxonomy and may not cover open-ended failures.
- Training data is generated only from simulation and failure taxonomy-based perturbations rather than diverse policy rollouts.
- Binary success on RoboFail and some metrics remain modest, e.g. ROUGE-L of 0.280 on RoboFail.
Experimental setup
Datasets: AHA dataset (FailGen-generated from RLBench) VQA (LLaVA instruction data) LVIS ManiSkill-Fail RoboFail MMBench ScienceQA TextVQA POPE VizWiz
Benchmarks: AHA dataset (Test) ManiSkill-Fail RoboFail MMBench ScienceQA TextVQA POPE VizWiz
Metrics: ROUGE-L cosine similarity binary success rate LLM fuzzy match
Baselines: LLaVA-v1.5-13B LLaVA-NeXT-34B Qwen-VL Gemini-1.5 Flash GPT-4o GPT-4o-ICL (5-shot)
Builds on: LLaVA Eureka PRoC3S Manipulate-Anything RoboFail
Closest papers by meaning
- Failing Forward: Adaptive Failure-Informed Learning for Vision-Language-Action Models (NeurIPS 2026 · similarity 0.93)
- SAFE: Multitask Failure Detection for Vision-Language-Action Models (NeurIPS 2025 · similarity 0.92)
- Self-Refining Vision Language Model for Robotic Failure Detection and Reasoning (ICLR 2026 · similarity 0.92)
- RoboProcessBench: Benchmarking Process-Aware Understanding in Visual-Language Robotic Manipulation (NeurIPS 2026 · similarity 0.91)
- SAM2Act: Integrating Visual Foundation Model with A Memory Architecture for Robotic Manipulation (ICML 2025 · similarity 0.88)
- PAC Bench: Do Foundation Models Understand Prerequisites for Executing Manipulation Policies? (NeurIPS 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.