Research map / Vision-language and multimodal / Visual question answering
Vision-Language Models for Robot Success Detection
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Fine-tunes MiniGPT-4 (Vicuna-13B) as a binary VQA success detector for robot trajectories from Berkeley Bridge and Berkeley UR5, studying few-shot accuracy, cross-dataset transfer, co-finetuning, and visual/language robustness.
Problem
Robot task success detection needs to generalize across visual, language, and task variation; the paper asks whether fine-tuned VLMs can serve as accurate success detectors, especially in low-data regimes and across datasets.
Approach
Formulate success detection as a binary VQA problem and fine-tune an open-source VLM (MiniGPT-4) on a handful of in-distribution robot trajectories, then measure how accuracy transfers across datasets, co-finetuning mixes, and visual versus language perturbations.
Key results
- Fine-tuning on 5 UR5 trajectories reaches 92% balanced accuracy on cloth sweeping and 94% on cup stacking, versus about 50% zero-shot.
- Fine-tuning on 400 Bridge trajectories gives 70.3% test balanced accuracy on 25 unseen Bridge tasks.
- Augmenting visual inputs drops cloth-sweep balanced accuracy from 95% to 75%, while paraphrasing the prompt drops it only to 94%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| fine-tune with 5 trajectories instead of no fine-tuning (ablation) | MiniGPT-4 · UR5 cloth sweep | balanced accuracy: 49% -> 92% | Helped |
| fine-tune with 5 trajectories instead of no fine-tuning (ablation) | MiniGPT-4 · UR5 cup stacking | balanced accuracy: 56% -> 94% | Helped |
| increase training trajectories from 5 to 10 (ablation) | MiniGPT-4 · UR5 cloth sweep | balanced accuracy: 92% -> 86% | Hurt |
| increase training trajectories from 5 to 10 (ablation) | MiniGPT-4 · UR5 cup stacking | balanced accuracy: 94% -> 91% | Hurt |
| fine-tune with 25 rather than 5 trajectories (ablation) | MiniGPT-4 · UR5 cloth sweep | balanced accuracy: 92% -> 95% | Helped |
| fine-tune with 25 rather than 5 trajectories (ablation) | MiniGPT-4 · UR5 cup stacking | balanced accuracy: 94% -> 97% | Helped |
| fine-tune on Bridge rather than UR5 trajectories (ablation) | MiniGPT-4 · UR5 cloth sweep | balanced accuracy: 95% -> 52% | Hurt |
| co-fine-tune with 400 Bridge trajectories (ablation) | MiniGPT-4 · UR5 cloth sweep | balanced accuracy: 95% -> 90% | Hurt |
| apply visual augmentations to test images (ablation) | MiniGPT-4 · UR5 cloth sweep | balanced accuracy: 95% -> 75% | Hurt |
| paraphrase the task prompt (ablation) | MiniGPT-4 · UR5 cloth sweep | balanced accuracy: 95% -> 94% | No clear effect |
| train on 400 Bridge trajectories and evaluate on unseen Bridge tasks | MiniGPT-4 · Bridge unseen tasks | balanced accuracy: 87.6% -> 70.3% | Mixed |
Surprising findings
- Fine-tuning solely on Bridge data makes the detector predict mostly failures on UR5 (52% balanced accuracy), and co-finetuning with Bridge hurts UR5 performance relative to UR5-only training.
Limitations
- Transfer across environments and tasks is weak under distribution shift, with UR5 models failing to generalize to unseen UR5 tasks.
- Accuracy drops substantially under visual augmentations, indicating vision is the bottleneck.
- Evaluation uses small test sets (100 UR5 observations, 25 Bridge trajectories).
Experimental setup
Datasets: Berkeley Bridge Berkeley UR5 Demonstration Dataset
Benchmarks: Berkeley UR5 cloth sweeping Berkeley UR5 cup stacking Berkeley Bridge unseen tasks
Metrics: balanced accuracy false positive rate false negative rate precision
Baselines: zero-shot MiniGPT-4 RoboFUME
Builds on: MiniGPT-4 RoboFUME
Closest papers by meaning
- SAFE: Multitask Failure Detection for Vision-Language-Action Models (NeurIPS 2025 · similarity 0.88)
- PIVOT: Iterative Visual Prompting Elicits Actionable Knowledge for VLMs (ICML 2024 · similarity 0.88)
- GoalLadder: Incremental Goal Discovery with Vision-Language Models (NeurIPS 2025 · similarity 0.86)
- TTRV: Test-Time Reinforcement Learning for Vision Language Models (CVPR 2026 · similarity 0.86)
- LLaRA: Supercharging Robot Learning Data for Vision-Language Policy (ICLR 2025 · similarity 0.86)
- Vision-Language Models are Zero-Shot Reward Models for Reinforcement Learning (ICLR 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.