Research map / Vision-language and multimodal / Multimodal LLMs
m&m’s: A Benchmark to Evaluate Tool-Use for multi-step multi-modal Tasks
Zixian Ma, Weikai Huang, Jieyu Zhang, Tanmay Gupta, Ranjay Krishna
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use multi-step planning instead of step-by-step planning (ablation) | LLM tool planners · m&m's | tool-F1: gap smaller for Llama-3-70B and GPT-4; GPT-3.5 pass rate +21.8% | Improved |
| add verification feedback to parsing feedback (ablation) | Llama-2-7B · m&m's pass rate | pass rate: 28.23 -> +18.14 | Improved |
| add verification feedback to parsing feedback (ablation) | Llama-2-70B · m&m's argname-F1 | argname-F1: 52.49 -> +12.87 | Improved |
| add verification or execution feedback (ablation) | weaker LLMs such as Llama-2-13B · m&m's tool selection | tool-F1: 42.27 -> -3.45 | Hurt |
| add feedback to already strong models (ablation) | GPT-4 and Llama-3-70B · m&m's | pass rate: Llama-3-70B PV -0.45; GPT-4 PV +1.13 | No clear effect |
| use verification feedback instead of raw execution feedback (ablation) | argument name prediction · m&m's | argname-F1: Llama-2-70B 12.87 (PV) vs 8.97 (PE) | Improved |
| generate plans as Python code instead of JSON (ablation) | LLM tool planners · m&m's | pass rate | Hurt |
| generate plans as Python code instead of JSON (ablation) | LLM tool planners except Llama-2-7B · m&m's | tool-F1: <3% difference | No clear effect |
| step-by-step planning terminates after positive feedback | step-by-step planners · m&m's | tool-F1 | Hurt |
Closest papers by meaning
- MEGA-Bench: Scaling Multimodal Evaluation to over 500 Real-World Tasks (ICLR 2025 · similarity 0.87)
- MM-OptBench: A Solver-Grounded Benchmark for Multimodal Optimization Modeling (NeurIPS 2026 · similarity 0.86)
- MINT: Evaluating LLMs in Multi-turn Interaction with Tools and Language Feedback (ICLR 2024 · similarity 0.86)
- Tool-Planner: Task Planning with Clusters across Multiple Tools (ICLR 2025 · similarity 0.86)
- MMhops-R1: Multimodal Multi-hop Reasoning (AAAI 2026 · similarity 0.85)
- CRAFT: Customizing LLMs by Creating and Retrieving from Specialized Toolsets (ICLR 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
