Research map / Vision-language and multimodal / Multimodal LLMs

Self-Refining Vision Language Model for Robotic Failure Detection and Reasoning

ICLR 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Chain-of-thought multimodal reasoning · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes ARMOR, a Qwen2.5-VL-based multi-task model with detection and reasoning heads that iteratively self-refines over rounds, trained on heterogeneous sparse/dense failure labels with offline and online imitation, and selects predictions by entropy.

Problem

Robotic failure detection and open-ended failure reasoning from video, where binary failure labels are abundant but natural-language reasoning labels are scarce and failure modes are hard to enumerate.

Approach

Treat detection and reasoning as a multi-task round-based self-refinement process with separate classification and language heads, trained with offline expert-conditioned imitation and online rollouts on heterogeneous sparse/dense data, and select the lowest-entropy trajectory at inference.

Key results

• ARMOR reaches 0.917 detection accuracy and 0.718 LLM fuzzy score on RLBench, versus 0.726 and 0.550 for SFT-S+D.
• On the Sparrow test set, ARMOR achieves 0.733 detection and 0.503 fuzzy score, improving over the 7B Qwen2.5-VL baseline (0.453 / 0.240).
• Refinement rounds raise RLBench reasoning LLM fuzzy score from 0.475 (round 0) to 0.717 (round 3) with decreasing variance across 4 seeds.

Surprising findings

• Naively mixing sparse and dense data in SFT (SFT-S+D) collapses reasoning on the Maniskill transfer setting (0.644 to 0.177 fuzzy score).
• Open-source VLMs such as Qwen2.5-VL and Cosmos-Reasoning perform at roughly chance level on detection accuracy.

Limitations

• Refinement can fail to converge to correct reasoning, as shown in the ARMBench example where reasoning drifts while detection remains correct.
• Evaluation relies on an LLM judge (Claude-3.7-Sonnet) and on balanced test sets; baseline evaluation via regex parsing is brittle.
• Sparrow-Fail and ARMBench are from a single warehouse setup and the new dataset is not released.

Experimental setup

Datasets: RLBench-Fail Maniskill-Fail Sparrow-Fail ARMBench

Benchmarks: RLBench-Fail Maniskill-Fail Sparrow-Fail ARMBench

Metrics: binary detection accuracy LLM fuzzy match ROUGE-L

Baselines: Qwen2.5-VL Cosmos-Reasoning LLaVA-NeXT Claude-3.7 SFT-D SFT-S+D AHA

Builds on: RISE AHA Qwen2.5-VL online imitation learning

Closest papers by meaning

• AHA: A Vision-Language-Model for Detecting and Reasoning Over Failures in Robotic Manipulation (ICLR 2025 · similarity 0.92)
• SAFE: Multitask Failure Detection for Vision-Language-Action Models (NeurIPS 2025 · similarity 0.90)
• Failing Forward: Adaptive Failure-Informed Learning for Vision-Language-Action Models (NeurIPS 2026 · similarity 0.89)
• Code-as-Monitor: Constraint-aware Visual Programming for Reactive and Proactive Robotic Failure Detection (CVPR 2025 · similarity 0.86)
• DiffusionVLA: Scaling Robot Foundation Models via Unified Diffusion and Autoregression (ICML 2025 · similarity 0.85)
• SOLE-R1: Video-Language Reasoning as the Sole Reward for On-Robot Reinforcement Learning (NeurIPS 2026 · similarity 0.85)