AI Paper Atlas

Research map / Vision-language and multimodal / Multimodal LLMs

m&m’s: A Benchmark to Evaluate Tool-Use for multi-step multi-modal Tasks

Zixian Ma, Weikai Huang, Jieyu Zhang, Tanmay Gupta, Ranjay Krishna

ECCV 2024 · Vision-language and multimodal › Multimodal LLMs · approach: Multimodal LLM benchmarks · 15 citations (OpenAlex)

Read the paperCheck your idea against 137,217 papers

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
use multi-step planning instead of step-by-step planning (ablation)LLM tool planners · m&m'stool-F1: gap smaller for Llama-3-70B and GPT-4; GPT-3.5 pass rate +21.8%Improved
add verification feedback to parsing feedback (ablation)Llama-2-7B · m&m's pass ratepass rate: 28.23 -> +18.14Improved
add verification feedback to parsing feedback (ablation)Llama-2-70B · m&m's argname-F1argname-F1: 52.49 -> +12.87Improved
add verification or execution feedback (ablation)weaker LLMs such as Llama-2-13B · m&m's tool selectiontool-F1: 42.27 -> -3.45Hurt
add feedback to already strong models (ablation)GPT-4 and Llama-3-70B · m&m'spass rate: Llama-3-70B PV -0.45; GPT-4 PV +1.13No clear effect
use verification feedback instead of raw execution feedback (ablation)argument name prediction · m&m'sargname-F1: Llama-2-70B 12.87 (PV) vs 8.97 (PE)Improved
generate plans as Python code instead of JSON (ablation)LLM tool planners · m&m'spass rateHurt
generate plans as Python code instead of JSON (ablation)LLM tool planners except Llama-2-7B · m&m'stool-F1: <3% differenceNo clear effect
step-by-step planning terminates after positive feedbackstep-by-step planners · m&m'stool-F1Hurt

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy