Research map / Vision-language and multimodal / Robot action and navigation
Code-as-Monitor: Constraint-aware Visual Programming for Reactive and Proactive Robotic Failure Detection
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Code-as-Monitor, which uses GPT-4o-generated monitor code over tracked "constraint elements" (geometric points/lines/surfaces from a trained constraint-aware segmenter ConSeg) to perform both reactive and proactive robotic failure detection.
Problem
Open-set robotic failure detection is needed both reactively (after failures occur) and proactively (preventing foreseeable failures), but VLM VQA-based detectors are slow and coarse in 3D spatio-temporal perception.
Approach
Formulate both detection modes as spatio-temporal constraint satisfaction problems; a VLM writes monitor code once per subgoal that evaluates tracked geometric constraint elements (points, lines, surfaces abstracted from entities/parts), enabling real-time checks without repeated VLM calls.
Key results
- Under severe disturbances in CLIPort Stack in Order (p=0.3, q=3), CaM reaches 63.33% success vs 15.00% for CLIPort alone and 40.00% for Inner Monologue/DoReMi, and reduces execution time 14.4% vs DoReMi.
- In OmniGibson, CaM reduces execution time by 34.8% and token count by 52.2% versus DoReMi.
- ConSeg-13B achieves 60.2 gIoU on part-level ConstraintSeg vs 23.4 for LISA-13B, while being comparable on ReasonSeg (55.7 vs 56.6 gIoU).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use CaM for monitoring instead of DoReMi | CLIPort Stack in Order (p=0.3) · CLIPort Stack in Order (p=0.3) | success rate (%): 76.67 -> 88.33 | Helped |
| Use CaM for monitoring instead of DoReMi | CLIPort Stack in Order (q=3) · CLIPort Stack in Order (q=3) | execution time (s): 54.2 -> 36.8 | Helped |
| Use CaM to count blocks with constraint elements instead of DoReMi VQA | CLIPort Sweep Half the Blocks · CLIPort Sweep Half the Blocks | success rate (%): 16.67 -> 75.00 | Helped |
| Use CaM instead of DoReMi | Omnigibson Pour Tea · Omnigibson Pour Tea | success rate (Disturbance a): 0 -> 40 | Helped |
| Use CaM instead of DoReMi | Omnigibson tasks · Omnigibson | execution time: 34.8% reduction | Helped |
| Use CaM instead of DoReMi | Omnigibson tasks · Omnigibson | token usage: 52.2% decrease | Helped |
| Add constraint elements to CaM (ablation) | CaM on real-world reasoning pick-and-place · Reasoning Pick & Place: grasp animals by distance to fruits | success rate (%): 20% -> 60% | Helped |
| Add multi-view observations (ablation) | CaM on Omnigibson Stow Book · Omnigibson Stow Book | average success rate (%): 40% -> 65% | Helped |
| Use constraint-aware segmentation instead of DINOv2 semantic points (ablation) | CaM on Omnigibson Stow Book · Omnigibson Stow Book | average success rate (%): 42.5% -> 65% | Helped |
| Connect points to form constraint elements (ablation) | CaM on Omnigibson Stow Book · Omnigibson Stow Book | average success rate (%): 55% -> 65% | Helped |
| Add constraint instance-level training data (ablation) | ConSeg · ConstraintSeg instance-level | gIoU: 51.5 -> 62.1 | Helped |
Surprising findings
- Using DINOv2-derived semantic points instead of constraint-aware segmentation drops success from 65% to 42.5%, showing element quality matters more than generic keypoints.
- DoReMi's success rate in some OmniGibson tasks is lower than ReKep alone, attributed to VLM limited spatio-temporal understanding from single images.
Limitations
- Relies on multi-view RGB-D and accurate depth for 3D element construction; occlusion degrades element generation.
- Failure judgment success rate not assessed because monitoring is continuous code execution rather than per-query judgment.
Experimental setup
Datasets: BridgeData V2 ADE20K COCO-Stuff PACO-LVIS PASCAL-Part refCLEF refCOCO refCOCO+ refCOCOg LLaVA-Instruct-150k ReasonSeg
Benchmarks: CLIPort stack-in-order and sweep tasks OmniGibson Slot Pen, Stow Book, Pour Tea RLBench real-world UR5 + Leap Hand pick & place ReasonSeg ConstraintSeg
Metrics: success rate execution time token usage gIoU cIoU
Baselines: DoReMi Inner Monologue ReKep PixelLM LISA-13B Foundation Model Combination
Builds on: LISA ReKep DoReMi Inner Monologue CoTracker
Closest papers by meaning
- AHA: A Vision-Language-Model for Detecting and Reasoning Over Failures in Robotic Manipulation (ICLR 2025 · similarity 0.87)
- SAFE: Multitask Failure Detection for Vision-Language-Action Models (NeurIPS 2025 · similarity 0.87)
- RoboProcessBench: Benchmarking Process-Aware Understanding in Visual-Language Robotic Manipulation (NeurIPS 2026 · similarity 0.86)
- Self-Refining Vision Language Model for Robotic Failure Detection and Reasoning (ICLR 2026 · similarity 0.86)
- Failing Forward: Adaptive Failure-Informed Learning for Vision-Language-Action Models (NeurIPS 2026 · similarity 0.86)
- Robotic Visual Instruction (CVPR 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.