Research map / Vision-language and multimodal / Human and embodied modeling
Mass-Producing Failures of Multimodal Systems with Language Models
Shengbang Tong, Erik Jones, Jacob Steinhardt
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace CLIP-similarity scraping with random COCO pairs (ablation) | MultiMon · downstream text-to-image failure rate | failure rate: 80% -> 20% | Hurt |
| remove scraped individual failures from categorization prompt (ablation) | MultiMon (GPT-4) · CLIP systematic failures | systematic failures found: 14 -> 2 | Hurt |
| remove scraped individual failures from categorization prompt (ablation) | MultiMon (GPT-4) · CLIP systematic failures | average success rate: 29.3% | Hurt |
| use GPT-4 instead of GPT-3.5 as categorizer (ablation) | MultiMon · CLIP systematic failures | systematic failures found: 8 -> 14 | Improved |
| use Claude instead of GPT-4 as categorizer (ablation) | MultiMon · CLIP systematic failures | systematic failures found: 14 -> 11 | Hurt |
| use GPT-3.5 instead of GPT-4 as categorizer (ablation) | MultiMon · shared systematic failures | average success rate: 80.2% -> 69.5% | Hurt |
| use Claude instead of GPT-4 as categorizer (ablation) | MultiMon · shared systematic failures | average success rate: 80.2% -> 83.3% | No clear effect |
| use Claude or GPT-3.5 as generator instead of GPT-4 (ablation) | MultiMon · CLIP success rate | success rate | Mixed |
| use COCO vs SNLI as input corpus (ablation) | MultiMon · systematic failures found | failure types | Mixed |
| steer scraping toward self-driving | MultiMon · self-driving systematic failures | success rate: 100% | Improved |
| steer scraping toward self-driving: object-specific attributes failure | MultiMon · self-driving systematic failures | success rate: 41.0% | Hurt |
| steer generation toward self-driving | MultiMon · self-driving individual failures | success rate: 74.56% | Improved |
Closest papers by meaning
- MEGA-Bench: Scaling Multimodal Evaluation to over 500 Real-World Tasks (ICLR 2025 · similarity 0.85)
- SDEval: Safety Dynamic Evaluation for Multimodal Large Language Models (AAAI 2026 · similarity 0.84)
- SO-Bench: A Structural Output Evaluation of Multimodal LLM (CVPR 2026 · similarity 0.84)
- LLaVAShield: Safeguarding Multimodal Multi-Turn Dialogues in Vision-Language Models (CVPR 2026 · similarity 0.84)
- SAFE: Multitask Failure Detection for Vision-Language-Action Models (NeurIPS 2025 · similarity 0.84)
- On Path to Multimodal Generalist: General-Level and General-Bench (ICML 2025 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
